It only won’t matter if the mods in the community your switching from aren’t actually bad. If it’s only you who doesn’t like them, then it’s you who’s the problem, and your leaving fixed it for that community.

But if it’s true that the mod in that community is a real problem, most of the people in that community will move with you, and the problematic mod will be left with an empty community and nothing to moderate.

In either case, that’s the system working correctly.