Hacker News new | past | comments | ask | show | jobs | submit login

Yea. I work as a software engineer at a company that sees huge swings in traffic on our product (by nature of the product). It's not uncommon for us to see 20x swings in traffic day-to-day. It's taken us a long time to be able to handle that and even then shit still goes wrong.

I'm much more inclined to believe the downtime is caused by scaling issues rather than some big conspiracy,

EDIT: Just to give some insight, one of the problems we run into is that usually only one issue is exposed during an outage. For example, lets say we have 10 functions (or services or whatever) in a row that all call each other. All the odd numbered ones have issues where they fail at scale. During the first downtime, function one will fail and start to throw errors, but this means the full traffic load isn't reaching the rest of the functions. By the time the problem is solved, traffic has usually died down, so function 1 is fixed, but 3,5,7,9 still have issues that haven't been exposed.

Additionally, at a quickly growing company like Coinbase, the code isn't going to stay the same for very long, so even once those functions are fixed, new issues are introduced.

It's easy to say they should be load testing at 10x their expected load to prevent these type of issues, but it's really hard to replicate production traffic. Users do weird and unexpected shit that can cause problems.




There's an article on the frontpage of HN from NYTimes that mentions Coinbase has struggled with the massive spike in traffic recently.

It's funny how quickly people jump to overly-complex or borderline conspiratorial explanations when the obvious is staring us in the face...

Coinbase might be a large well funded organization but it's still run by developers with giant backlogs of tickets and conflicting priorities to build new stuff vs maintain old stuff.


> but it's really hard to replicate production traffic

This times 1,000.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: