Your measure of "too many people" is that you see people in your city and never see them again? That's an amazingly self-centered metric for deciding how many people there should be.

And it's very easy to fix your problem. Just move to a small town.