The problem was, it was a better GeoCities. Vanity blogs for people with internet friends.
Facebook used real names, so it helped people connect with their real friends; even the ones they weren't confortable being "internet friends" with.
I'm not saying this was genius. It was probably just luck. Or the result of Facebook being optimized for a single college, which was a microcosm of the real world - getting everyone to sign-up was only possible using real names.
Of course, it helped that Facebook has stayed small, and stayed pretty close to its pragmatic hacker roots.
I thought it would be more technical. Instead it was mostly:
The scale was a new challenge to most people back then, we did the best we could - which most of the time was more than good enough - learnt a lot of good lessons and had a great team. In the end, we lost because we focused on advertisers and increasing the companies headcount for the sake of it.
Later in the MetaFilter comments, Erik posted a link to video+slides of a session he co-presented at MIX 10. This session had technical content about the architecture of some of MySpace's stack:
It annoys me a bit that people talk about scaling problems as being brand new then. A lot of the first generation of dotcoms had them too (Amazon, eBay, Yahoo, etc).
In general they were solved similarly - build custom middleware and go as distributed as possible.
Now I realize that's a simplistic answer to a very complicated problem but really guys, this isn't unprecedented and the hardware is a lot more capable and cheaper than the 1997 equivalents.
I think the scaling problem MySpace faced (and social sites in general face) was somewhat different than the sites you mention. Every page is dynamic, and requires real time updates. You could probably cache an amazon product page for a week. Also, the information on a page in a social network in much less siloed than a product page on amazon, or an auction on ebay. You are pulling in user information from across your base. So it doesn't lend itself as easily to a basic shard and cache model.
That's like saying Facebook is PHP or Twitter is Rails. Sure it started that way but when the shit hit the fan they probably went heavy and deep into lower level languages working much closer to the metal to solve the problems they were having.
He stated they had a third of the number of servers as Facebook back when the two were roughly equal. They were running on the older pre-java based ColdFusion 5. I'd say they scaled pretty well regardless of your opinion of the service.
I have fond memories of Digg, at least. Loved the user activism around the HD-DVD key days. That's kind of the last thing I remember was really awesome about that site.
I'd be really curious to see a good analysis or even set of anecdotes about why it went downhill. Seems like basically they ignored their userbase.
The problem was, it was a better GeoCities. Vanity blogs for people with internet friends.
Facebook used real names, so it helped people connect with their real friends; even the ones they weren't confortable being "internet friends" with.
I'm not saying this was genius. It was probably just luck. Or the result of Facebook being optimized for a single college, which was a microcosm of the real world - getting everyone to sign-up was only possible using real names.
Of course, it helped that Facebook has stayed small, and stayed pretty close to its pragmatic hacker roots.