DevOps means different things to different people.

most people use the term to mean 'a coder who can configure apache'

this whole incident reeks of a lack of OpSec and general Ops knowledge.

Lack of offsite backups is one thing, but if you're using cloud infrastructure you never place all eggs in one basket.

your instances have to be disposable and reproducible, at the first hint of a break in my default action is to torch and rebuild.

> your instances have to be disposable and reproducible, at the first hint of a break in my default action is to torch and rebuild.

Or rather shutdown, preserve, start replacements. IMHO your automation should be able to deal with the old preserved instances being left alone.

DevOps here.

There are some things you aren't going to expect (compromise of your AWS console). This could have been solved by having MFA enabled, as well as having the app push backups in realtime, versioned with delete protection, to S3 buckets under the control of another account (write access, but no delete access).

Show of hands how many people here are doing it this way.

Seriously, if your root account and all full admin accounts aren't using MFA you're just asking for it. Also if you're not using purpose specific access keys, you're just asking for it. If the first thing you do isn't calling AWS support, wow...

Couldn't agree more. Everything under a single platform, no MFA, no (real) offsite backup, and on top of that they spent 12 hours corresponding with the attacker, instead of immediately calling Amazon to ask their help to shut down everything, while they still had time?

I'm sorry, but this is a succession of things not to do in terms of system operations. Probably the team never managed mission critical platforms before, and hopefully they now learned the lesson.

How many companies have not yet learned that lesson? There are probably a lot of codespaces on AWS. My reasoning is that if you make it so that a developer can set up a virtual datacenter but does not have the background of actually running such an installation then you're going to have to assume that it is probably quite fragile.

Software people tend to make all kinds of assumptions about hardware that do not work out in practice.

That the problem with the recent "DevOps" trend. Lots of people coming from a "Dev" background, but no real "Ops".

And now that spinning up a couple of servers on AWS and creating snapshots on-the-fly are so easy, it gives the false impression that you don't need much to act as a sysadmin.

Not to be too flippant, but the company's closing shop. So, yeah, the DevOps are fired, along with everybody else.

As for the rest of us: AWS is a great one-stop shop. Unfortunately, using just AWS puts you in the "all the eggs in one basket" scenario that we were warned against as children.

Two-factor authentication is a second basket.

Sending a copy to Glacier is a second basket.

Does Amazon not have 30-day undelete for bulk storage? Seems crazy.

> Two-factor authentication is a second basket.

Not in my books. Recommended to be sure, but this has far too many single points of failure. To name a few:

- Software corrupts data

- Hardware corrupts data

- Social engineering bypasses 2FA

> Two-factor authentication is a second basket.

Two-factor authentication is a second basket is a better carrying strap but it's still on a single basket.