I'm always reluctant about going completely schemaless. Reminds me of the blog post by FriendFeed about how they use MySQL, highly recommended: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

I feel like like there needs to be a better middleground for having some schema but being able to augment it easily with metadata that you're not querying against (yet). Then later extracting the metadata into queryable columns.

I wrote a post outlining some ideas of how to do this: https://github.com/shazow/everything/blob/master/idea/arbitr...

I've only implemented bits and pieces of this in practice, huge convenience so far.

I think this is exactly right. Have a document store and a (separate?) index store.

James Golick implemented this idea in Ruby last year: https://github.com/jamesgolick/friendly

We used Friendly at my previous company wego.com and t worked quite well although I don't know shehar they at still using it. (Friendly doesnt look like it's in activ development)

I don't understand the fetish with caching every variation for every user.

Here's a thought: dynamic apps should render a page exactly one way in html, and rely on javascript and cookies to post-process the site so it APPEARS to be customized for that particular user. That includes admin widgets.

That particularly applies for displaying how many minutes ago something was generated. Serve it with a date-time format, use javascript to post-process into "x seconds ago" or "x minutes ago".

If someone isn't logged in, or isn't an admin, and they hack their javascript to display user or admin stuff, who cares? The user/admin requests sent to the webserver won't succeed anyway, because they rely on having an admin session cookie.

If there's meaningful rather than just UI stuff that users or admins get to see, then you have to cache that separately, but you can even do stuff like loading it dynamically with js so the publicly visible (cached) content can still be used, and you cut down the amount of stuff your server has to auto-generate. It can cache the separate pages (xml, json, whatever) that serve the logged-in user content, as well.

Note: the talk is from May 2010.

I wonder if Steve (or rather, jedberg or someone else at reddit) were to give the talk today, if 'memcache' and 'memcachedb' would both be replaced by Redis?

We replaced memcachedb with Cassandra a while ago, because memcachedb pretty much hit a wall at some point.

As for replacing memcached, I'm certainly open to it, but from what I've read, the performance of memcache for what we use it for is better than redis.

Maybe? There's something to be said for memcache's simplicity, however.

another lesson could be: "don't make a site that appeals to people who use ad-block. revenue won't keep up with demand."

many users (myself included) add an exception to adblock for reddit. I don't mind helping their ad revenue when the ads aren't that intrusive.

That's a good basic overview on what it takes to keep a site the size of Reddit (a year ago) afloat. I need to think more about caching, personally.

Reddit has done a great job of serving a massive amount of traffic, given the size of their staff especially.

Are you trolling? Site stability issues very rarely have anything to do with the underlying programming language, unless you're using some experimental language that no one else is using for web development (and even then, Arc seems to be working pretty well for HN).

It's half a dozen employees constantly trying to keep up with what is now a 1 billion page views a month site. From what I understand they're carefully trying to balance getting new hires (which Conde Nast now lets them do) and keep the damn ship afloat.

In the very first place, actually, they wrote the initial version in Lisp. The Python code was a later rewrite.

Reddit was stable for a long time. It's really only been the last few months (~6?) that they've been suffering serious service outages.

Actually, our uptime has been better for the last 6 months than the previous few years.

I don't go on reddit often enough to have any idea about the uptime, but out of curiosity, do you think there's anything that could give someone the impression it was worse recently?

There's enough users that each 3 second downtime is long enough to get someone who submits stories like 'WHY WAS REDDIT DOWN' and enough upvotes to stay on the front page long enough for everyone else to see it. And then recency effect kicks in as soon as reddit is down for another 4 seconds later on.

That's a bit of a stretch... That "reddit is under heavy load" page is an abomination.

Try turning admin mode off. :)

The site is unstable doesn't mean Python is to blame. Perhaps you can elaborate why do you think so.

I dont think reddit's stability is as bad as it looks. Just that Reddit's community is pretty vocal on it. IIRC, they have availability in high 90s (may be jedberg can confirm).

Yes, we are above 99%. We just have a community that is very vocal with things are broken, and then continue to be vocal about it after it is fixed because it is funny.

For example, we heard "search sucks" for weeks after we switched to the new service, until we announced the switch, and then all of a sudden the tune changed. :)