When we started DotSpots in 2007, we leased physical servers from iWeb. It was far cheaper than EC2, but you lose of ton of agility when you can't pave a server and rebuild it from the command line. We switched to EC2 and starting paying a bit more, but we've been able to expand capacity much more nimbly than before.
The only thing I'd recommend running outside of EC2 would be memcache, but that's only if you're really pinching pennies. If you're an early stage startup and you're not running virtual machines, you might be doing something wrong (IMHO).
Another and somewhat more subtle consideration: if you're deploying web crawlers or related tools from EC2, an increasing number of web sites are blocking access from the Amazon IP blocks due to the volume of vermin.
Depends on your latency needs, how expensive your data is to compute from its source, how big it is, etc. In some cases it'll be cheaper to pay the cost to exit EC2, hit the big memcache box and re-enter EC2 than it would be to rent a few Quad-Mem boxes.
FWIW, our fleet is 100% EC2 - this tradeoff didn't work for us.
It's literally just blank files being requested 10s of millions of times a day, they serve no data which makes a caching layer unnecessary. I just need the log files generated by the requests.
The only thing I'd recommend running outside of EC2 would be memcache, but that's only if you're really pinching pennies. If you're an early stage startup and you're not running virtual machines, you might be doing something wrong (IMHO).