"Say we want to deploy a web service as four containers running off the “httpd” image....This is simple to ask for but deceptively hard to actually make happen."
...and yet, would take almost no work to set up using a non-dockerized workflow. I don't understand why so many people are putting themselves through this. It's becoming common to hear people from dinky little startups going down dark rabbit holes trying to build their infrastructure like Google -- but it's a total distraction!
If you're running a service with a small number of machines, don't do any of this. Architect a sensible multi-AZ deployment (i.e. a cluster of 2-3 machines, an ELB, a VPN, and a firewall/bastion server), spin up instances by hand, and upgrade things as needed. Create AMIs for your machine classes, and get yourself used to working with a sensible upgrade schedule. Doing this for a small number of machines (e.g. N <= 25) won't take an appreciable amount of your time.
Once you start to have more machines than that, you'll probably also have the resources to get someone who knows what they're doing to set up more "magical" automated management schemes. Don't bury yourself in unnecessary complexity just because it's the hot new tech of the moment.
I mostly agree with your basic point, but I disagree that AWS is a necessary part of the situation. If you aren't dynamically spinning VM's up/down depending on usage - that is, if your EC2 instances are a pretty fixed list of always-on machines - you're throwing money away, and you're spending more time and effort to make use of proprietary infrastructure you aren't using as intended.
If you have a 'static' number of servers for your app - just rend a bunch of VPS. Use whatever automated setup tool works for you. I'm a big fan of Debian config & meta packages, but ansible, chef, hell even fucking shell scripts will work.
I agree in general. You want to keep everything as simple as possible. If you have a simple, small service, or don't deploy too frequently, traditional server management will be better
...and yet, would take almost no work to set up using a non-dockerized workflow. I don't understand why so many people are putting themselves through this. It's becoming common to hear people from dinky little startups going down dark rabbit holes trying to build their infrastructure like Google -- but it's a total distraction!
If you're running a service with a small number of machines, don't do any of this. Architect a sensible multi-AZ deployment (i.e. a cluster of 2-3 machines, an ELB, a VPN, and a firewall/bastion server), spin up instances by hand, and upgrade things as needed. Create AMIs for your machine classes, and get yourself used to working with a sensible upgrade schedule. Doing this for a small number of machines (e.g. N <= 25) won't take an appreciable amount of your time.
Once you start to have more machines than that, you'll probably also have the resources to get someone who knows what they're doing to set up more "magical" automated management schemes. Don't bury yourself in unnecessary complexity just because it's the hot new tech of the moment.