In my opinion, a phone needs to be extremely portable and easy to use with a single hand. I'm a tall man and the thumbs on my large hands have no trouble reaching anywhere in a 3.5" screen. I'm also rather athletic in build, so a device the size and weight of an iphone is just within the bounds of what I consider acceptable to carry all day in a front trouser pocket. I've experimented with phones that came in other sizes. The smaller ones sacrificed too much screen real estate, the larger ones were a bit unwieldy to touch / carry.

This is only my opinion and YMMV, of course. But for me at least, it's so wonderful that there are products out there that take ergonomics into account instead of just dashing for the biggest possible spec.