And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the disk 40 Gb. He could have easily just allocated 100 Gb and call it a day, this way we had to go to him 3 times to scale it up in 10 Gb steps each time, including a the stress of figuring out why things are failing (something that the admin should have seen on his monitoring system).

Admins are my heros, but please if you allocate disk space, just take the biggest expected backup and multiply it by π. And if you need to be stingy with storage for some reason, be stingy, but decide when and where to be stingy — and at least keep an eye on the monitoring and upsize the storage before it is too late.

They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour.

They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't really use their dev or stage environments, or have a process for documenting and reproducing bugs).

Seems pretty reasonable to me.

- set up some service that talks to a s3 bucket

- set up that bucket in the same region/datacenter

- send a decent but not insane amount of traffic through there (several hundred Gb per day)

- assume that you won’t get billed any data transfer fees since you’re talking to a bucket in the same data center

- receive massive bill under “EC2-Other” line item for NAT data transfer fees

- realize that AWS routes all traffic through NAT gateway by default even though it’s just turning around and going back into the data center it came from and billing exorbitant fees for that

- come to the conclusion that this is obviously a racket designed to extract money from unsuspecting people because there is almost no situation where you would want to do that by default and discover that hundreds to thousands of other people have been screwed in the exact same way for years (and there is a documented trail of it[1])

1: https://www.lastweekinaws.com/blog/the-aws-managed-nat-gatew...

Even has ha mode.

If the solutions are:

* rewrite that part to add retention, or use better compression, or spend next month deciding which data to keep and which can be removed early

* Wiggle a thing in panel/API giving it more space

The second will win every single time unless there is pushback or it hits the 5% of the developers that actually care to make good architecture not just deliver tickets.

On a per-month basis, the grossly exaggerated number is in the single thousands. The non-exaggerated number is down in the double digits.

$50/TB is a lowball if you want much of the data to be on SSDs, but taking an analysis server and stuffing in 20TB of SSD (plus RAID, plus room for growth) is a very small cost compared to repeated debugging sessions. Especially because the SSD has to deal with about 0.01 DWPD.

And that was a number hundreds of times higher than what they were replying to, just to make a point.

I wonder if there's anything as good in the open-source world. The closest thing I can think of is Clickhouse's "new" JSON type, which is backed by columnar storage with dynamic columns [2].

[1] https://www.uber.com/en-BR/blog/reducing-logging-cost-by-two... [2] https://clickhouse.com/docs/en/integrations/data-formats/jso...

The design described there is what Uber should be logging in the first place. Instead they are logging the fully resolved message and then compressing back into the templated form.

However, the compression back into the templated form is a good idea if you have third party logs that you want to store where you can not rewrite the logging to generate the correct form in the first place.

Current single disk solutions are around $25/TB for HDDs and ~$100/TB for NVMe.

At a minimum you're looking at $54k just for raw capacity-- assuming no backup, no chassis, no networking, and no redundancy.

More reasonable estimations would be in excess of $400/TB.

Unless you want to claim storage costs $5,000/TB for 3 MB/s of I/O “multiple gigabytes per hour” with 90 day retention for a team worth of logging is not stupid on its face. Not to say that is a efficient or smart solution, but certainly not a “look at this insane request by developers” the person I was originally responding to was making it out to be.

Personally, I would probably question the competence of the team if they had that sort of logging rate with manual logging statements, but I am merely pointing out that “multiple gigabytes per hour” for 90 days is not crazy on its face and a plausible business case could be made for it even with a relatively modest engineering team.

It's difficult to estimate the log throughput in this scenario. Cisco on debug all can overload the device's CPU; systems like sssd can generate MB of logs for a single login.

All of this is really missing the core issue though. A 2PB system is nontrivial to procure, nontrivial to run, and if you want it to be of any use at all you're going to end up purchasing or implementing some kind of log aggregation system like Splunk. That incurs lifecycle costs like training and implementation, and then you get asked about retention and GDPR.... and in the process, lose sight of whether this thing you've made actually provides any business value.

IT is not an ends in itself, and if these logs are unlikely to be used the question is less about dollars-per-developer-hour and more about preventing IT scope creep and the accumulation of cruft that can mature into technical debt.

For the 20TB case, you can fit that on 1 to 4 drives. It's super cheap. Plus probably a backup hard drive but maybe you don't even need to back it up.

For the 2PB case, you probably want multiple search servers that have all the storage built in. There's definitely cost increases here, but I wouldn't focus too much on it, because that was more of a throwaway. Focus more on the 20TB version.

> That incurs lifecycle costs like training and implementation

Those don't relate much to the amount of storage.

> and then you get asked about retention and GDPR....

It's 90 days. Maybe you throw in a filter. It's not too difficult.

> if these logs are unlikely to be used

The devs are complaining about the search features, it sounds like the logs are being used.

> preventing IT scope creep and the accumulation of cruft that can mature into technical debt

Sure, that's reasonable. But that has nothing to do with the amount of storage.

More like $15/TB. $100K for 2 PB of storage with redundancy and backups is quite reasonable.

$300 is moving towards refurb / shucked prices.

Where? For new prices I'm seeing $350 at amazon, $350 at B&H, $280 direct from newegg, $280 at serverpartsdeals.

When you buy a SAN to present a bunch of disks as one thing to the rest of the machines.

Don't get me wrong, I understand that a single-digit kUSD/month is peanuts against developer productivity gains, but I still wouldn't be able to take a developer making that suggestion seriously. I would also seriously question internal processes, GDPR (or equivalent) compliance, and whether the system actually brings benefit or if it is just lazy "but what if" thinking.

End number doesn't matter, what matters is developers thinking about how long data should be stored and what data should be stored.

Not doing that analysis and overprovisioning 4x will just cause disaster in 2 years instead of 6 months.

So what? Just buy more. Storage is cheap.

It's hard to have a discussion here without understanding the scales involved. Is the problem that they're wasting 100 GB or 100 TB? And if the issue is truly that they're wasting 100 TB, then clamp down on it as part of cost reduction efforts. The truth in most organizations is you get rewarded for eliminating mountains of waste, but trying to prevent the waste in the first place brands you as someone difficult to work with who is standing in the way. Why not lean into that?

But this way, I directly get an idea of the increase in storage we will need over the next year in order to plan the next hardware expansion.

As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.

It is also trivial to set up, and does not require me to figure out how to set up an OS alert, or trust that whatever alert process is running. So it is an essentially fail proof alert that works the same on any OS.

What’s a good argument against it?

If you just "I want 20 GB of storage", i'm not going to give it to you.

Storage is cheap in relation to other things. Just have a good reason to why you need it.

So, you are not expecting that your co-workers have good reasons for what they are doing? Maybe the hiring bar at your place is too low then.

I prefer to work at places where my default assumption is that everybody around me is smart and responsible. Lifts lots of worries off my shoulders (and tends to benefit the stock price over time too and thereby my income).

If I wanted unquestioned control, I'd run my own shop. If I want the best product, then I hope that people question my assumptions.

The way this was phrased was more from the angle "who knows what these guys were thinking; if they can't give me a good reason, no way they will get storage space as I don't trust that they make good decisions on their own".

Let's say I work in DevOps and want to optimize cloud costs. In that case, I would challenge the size of everything, the use of higher-costs services, the number of regions, all that - but the team might want more regions and bigger resources to improve latency and performance, and use more high-cost services for developer experience, and ship features without having to think about utilization.

It's a tug of war, and only works when you have forces on both sides to balance out. Being too conservative might stall innovation or make things too slow to save a buck, not being conservative enough might drain funds or make things impossible to scale.

Yeah, any workplace in which the word "war" was used in the context of colleague interaction saw me leave within a few months.

I like to plan those things ahead of time with all stakeholders involved, then we work together instead of against each other.

Unless you work with clones of yourself, there will always be differences in opinions and priorities, and not every feature and bug fix can be a company-wide stakeholder meeting, and you certainly will not get any social points for trying to micro-manage other teams.

But the attitude needs to be "let's put the requirements on the table and see what we can do" instead of "you don't get what you want unless you give me a good reason". The latter comes from an angle of distrust which I'm arguing against. The former comes from an angle of collaborative problem solving.

In a company in which I go to a team relevant to a project and like to engage in a discussion and am met with an attitude of "unless you give us a good reason we'll stop talking to you", the atmosphere is not one that will keep me personally for long. YMMV.

> I believe you are intentionally misunderstanding.

You are free to believe what you like. Opening a reply with such a sentence is pretty sad though. It does not foster a healthy atmosphere, nor does it match reality, I might add.

Your response hitched on a single word ("war") within a common phrase ("tug of war", a game). While it might have been accidental, such answers mislead from the actual discussion (and tends to be used as distractions when no good answer is present).

> Of course there will be differences. That's why you sit down and plan things together, pulling in and coordinating with all _relevant_ stakeholders.

When you discuss new architectures or large projects, this is a given, but this covers only a small portion of company operation - the rest is organic day-to-day work, which slowly but surely distorts initial assumptions. Slowly boiling the frog, so to speak. Think one team making changes that affect request patterns, another team making something that is accidentally quadratic, and a third team suddenly asking for a large number of cloud resources to carry this that should absolutely be challenged.

And at the same time, teams are under different organization units with different budgets, schedules, leaderships and priorities - and most certainly don't care about daily scrum work of other teams.

> In a company in which I go to a team relevant to a project and like to engage in a discussion and am met with an attitude of "unless you give us a good reason we'll stop talking to you", the atmosphere is not one that will keep me personally for long. YMMV.

No one said "we'll stop talking to you", but "you get what can be justified". If you take offense to be challenged and would rather work somewhere else, you do you, but if you can't justify your request I'd argue that you are not doing your job properly in the first place.

Are you going to rearchitect my system for me?

“There’s 10 people here whose time I bill out at $250/hr each, spending an hour discussing whether to buy a $1,000 software license? Why?”

"Some of us don't like jumping into things without looking."

I wager your boss would not be amused with me.

But, unlike Toyota, we do have disk space alerts.

Sometimes the problem is also entirely political, the management needs to tell client and charge them for more storage and won't accept the change till that happens. Meanwhile clock is ticking...

The problem in this case was twofold:

- admin had the job to implement database backups. He didn't factor in backup size when allocating disk space. So this was wntirely his own fault.

- the database does store certain transactions for a certain period, so this grew initially until it it setteled at a certain level. Because the margin of storage was slim, this caused the problem

Because most of our cases where that happens was either lack of planning or lack of communicating that plan. By far most common one was "neither dev nor client knows the data volume in longer period". Which is fine as long as that's also communicated, but that's also often a problem.

But I'm not denying of course that there are just shitty incompetent ops departments, just for the other customer we had dealing with ops department that had:

* backup storage (which was some remote FTPS server IIRC) provisioned so slow the backup wouldn't copy within 24 hours. And the backup size was below TB. * weeks long delays with any resize.

Good to know. FWIW i should also be avoided. It's tempting to use, since most programs use it as a counter, so it /should/ standardize the logfilesizes. But in practice it's very tricky to get a definitive disk space requirement with it.

The instances for the service itself had 20GB of EBS allocated to them. Luckily they don’t need much local storage. But that’s typical here. There’s a Jenkins instance that is even more of a pain. I’m not responsible for it but every week or two one of the worker nodes runs out of space because they’re given 8GB of storage space. I’m just watching the disaster unfold over the course of a year and a half as I’m constantly telling that team to just up the storage space on the worker nodes instead of constantly having to fiddle with cron jobs.

It’s not even an expense thing. They just… don’t want to increase the storage space. It drives me insane.

I'm not saying this is real, but the worry certainly is.

Or comparing the cost of one store not being able to open on time because the RDS database's space ran out. VPs and directors start yelling and there's suddenly like 20+ people involved in figuring out why this one store didn't open on time. What's the cost of that compared to just giving the DB 250GB of space so this never comes up again?

But you are also 100% correct and I've seen that happen here, too. There's some instances I'm responsible for that were using EFS for their local storage. Costing thousands of dollars every month for absolutely no reason. I switched those to reasonably-sized EBS volumes and that alone was half of my annual savings goal.

I was completely flabbergasted seeing these instances using EFS while others were stuck on 8GB EBS volumes. Backups on the EFS drives had ballooned to the many TBs. And the backups were worthless! Instances themselves are ephemeral. They use S3 for long-term storage & metadata is on a database. Those are the things that should be backed up & their cost compared to EFS is minuscule.

> compared to just giving the DB 250GB of space so this never comes up again?

As long as there is reasonable confidence in that this is actually the case, then just provision the space and be done with it. That requires a certain understanding of future space requirements/expectations, and anything even just so slightly running away / leaking space will hit any limit given enough time. So, due diligence requires looking at whether it's actually needed.

I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering practices, not by limiting resources.

There is absolutely no technical reason to provide all of your storage needs with a single quality of disks. You may do that to get some economies of scale, but if the impact is that high, you should rethink it. You can just as well have two different solutions, one with a huge amount of disk space, little redundancy and low performance, and one with a limited amount of space, plenty of redundancy and high performance.

And yes, I know that goes against every common ops procedure, because disks are so cheap. But the next thing you hear is always that disks are not cheap at all, and you really can't have it both ways.

If you have two tiers available, someone will use the less reliable one because that’s how they fit their budget but then because it’s “in production” they’ll expect the same level of service.

All of these are manageable but what you’re really hearing is that the technical issues are really the tip of the social iceberg most organizations have. One of the reasons people pick AWS isn’t just that it’s usually cheaper than the full cost of rolling your own but that lots of these things don’t affect you: you never fail to provision an EBS volume because the VPs of finance and IT are still arguing about procuring a new rack of disks, people can’t request endless customizations because the options are “take it or leave it”, etc.

Still cheap, but not that cheap.

SOmewhat recently I toyed with an idea of HDDs with an 'interposer' which would be just a dumb ATA2iSCSI interface, with Ethernet or WiFi connectivity.

That would allow you to place those drives literally anywhere and with a minimal footprint.

Sadly it would be too costly for the home usage (I assume about $30/unit at best) and for the enterprise usage... 1Gbit is too slow, 10Gbit is too hot and bulky and then you can't sell a $3 plastic case for $100 'vendor approved with light-path(r)(tm) diagnostic indication' for each drive.

Why in the world would you buy one "Enterprise" disk instead of three "consumer" disks?

None of this means that you should trust any particular disk enough not to need redundancy, backups, etc. Companies can and do make trade offs based on their needs and management competency and people have been shifting software for a generation to rely less on the hardware – back when Sun announced ZFS, one of the major appeals was that you could drop expensive hardware RAID controller dependencies in favor of cheap boxes of disks – but there isn’t a single global optimum point. A lot of enterprise purchases are driven by being able to satisfy your most demanding users with the same service as everyone else so you can avoid needing your admins to be trained and experienced with dozens of different storage systems. That last part especially extends to testing: for example, does your rack of consumer drives with software redundancy come back up cleanly after a kernel panic or power outage, especially a nasty one like a fluctuating brownout? Depending on your budget, needs, and technical bench depth you might reasonably conclude that the savings are worth the ops work, or that it’s safer to pay an enterprise storage vendor who’ll certify that they’ve done that testing and will have tech support on-sight within an hour, or that you’ll use AWS/Azure/GCP because they do even more of that tedious but important work. All of those can be right, but I’ve typically found that people in the first two categories think they’re doing better than they are and would be paying less for better service in the cloud.

So, internally the storage team might quote say $1000/TB (simple numbers for an example) for a given storage quantity. And behind the scenes they'll likely have at least redundant storage arrays, plus backups and 24/7 monitoring for all of the data, etc.

Mom and Pop businesses using single servers can do all they want in regards to drives, I don't care. I would argue they shouldn't have servers, but if you do have servers, you should at minimum use RAID etc, not a drive plugged into your USB port.

Absolutely none of that has to apply to the sort of situation we are talking about here.

The only real reason to spend more per disk is when you know all your disks are going to fail, and extending the lifespan per average disk will definitely save you more than the enterprise markup costs. So you better have dozens or hundreds of disks in the first place.

For any truly valuable data, you should at minimum have a backup in one different physical location. That backup should be include at least one redundant disk. None of that is worth spending a dime on more expensive hardware.

All disks fail eventually. Outliers may run longer than the MTBF for that drive model, but they all fail eventually.

And backups are fine for restoring data, but they don't help provide access to that data in a timely fashion. That's why people use SANs, AWS etc.

The cost of a SAN storage array is a nit for a business making cars (like Toyota) or selling insurance (like my company).

And again, a backup is fine for deleted data, or fire/ransomware. But for day to day operations, no one is really willing to wait for you to restore from a local backup, much less an offsite backup.

And they are willing to wait for you to rebuild your raid array?

Either you lost your data from a disk failure, or you are waiting to lose your data from a disk failure. How are we not on the same page here?

More disk space isn't one drive, but redundancy too And backups. And even network bandwidth.

This thread sort of highlights the problems. DEVs don't get it. Nor does management.

When would we get new filers?

Dunno, because the datacenter was also full.

Unfortunately, this strategy frustrates users who have a genuine need to expand storage and who are unable to efficiently obtain an exception to the process.

Everywhere I’ve seen that strategy tried had massive overspending and outages due to it. What’s worked best is cloud style usage billing because that aligns the incentives with the people making the decisions, but it’s really non-trivial to get the accurate full cost.