I've worked at smaller companies. Even still, it's not hard to hit relatively large amounts of data, depending on the field.
As an example, any kind of analytics could generate terabytes of data a day... per customer.
A side project I am building will have to handle billions of events per day. Per customer. There are 0 customers (this is for fun, not profit), but as soon as it would hit one customer I would need to consider an approach that scales.
How many companies have similar requirements?
But that's actually besides the point.
Microservice architecture, or any architecture that focuses on isolated, asynchronous components, adds complexity. Of course.
But it also reduces work in other areas. If you build async, isolated services, you no longer have to deal with catastrophic service failure. Cascading failures go away at the async bound.
For many of us, I imagine we've spent a lot of time fighting fires at organizations where one service going down was a serious problem, causing other services to fail, and setting your infrastructure ablaze. Hence a bias towards solving that problem upfront.
> As an example, any kind of analytics could generate terabytes of data a day... per customer.
Wait, what? I've never worked anywhere where one customer generated terabytes of data per day, and I've worked on very large commercial enterprise software.
The only thing I have experience with that produces anything close to that kind of data per customer is in genetic sequencing, and you only do a customer once. (Even that isn't a TB in its bulkiest, raw data form, and the formats used for cross-customer analysis are orders of magnitude smaller).
> For many of us, I imagine we've spent a lot of time fighting fires at organizations where one service going down was a serious problem, causing other services to fail, and setting your infrastructure ablaze.
The reason so many of us have worked in places like that is that those places 'got stuff done' and survived and grew.
I was also very confused by OP as well. I’m thinking maybe by customer he means a business using his analytics service. That would explain why it needs to scale for his first customer, and why one customer could have terabytes of data.
As an example, any kind of analytics could generate terabytes of data a day... per customer.
A side project I am building will have to handle billions of events per day. Per customer. There are 0 customers (this is for fun, not profit), but as soon as it would hit one customer I would need to consider an approach that scales.
How many companies have similar requirements?
But that's actually besides the point.
Microservice architecture, or any architecture that focuses on isolated, asynchronous components, adds complexity. Of course.
But it also reduces work in other areas. If you build async, isolated services, you no longer have to deal with catastrophic service failure. Cascading failures go away at the async bound.
For many of us, I imagine we've spent a lot of time fighting fires at organizations where one service going down was a serious problem, causing other services to fail, and setting your infrastructure ablaze. Hence a bias towards solving that problem upfront.