I work at a place where a manager has around 20 people, and Google is told to have much bigger departments. But are you meaning people manager or technical manager? If your manager really directs your work, then yes 20 is too much.