How do you change network configuration, load balancers or other external configuration in lock step with your application? Scripting by hand?

And deployments are a just small part of configuration management. How do you find out which of your applications use libz 1.2.3 (to measure CVE applicability for example)? By looking in each and every one of them?

How do you find out which application run a cron job? And which application connect to backend x without going through the load balancer?

Which application has a client certificate about to expire? How do you guarantee two applications have the same shared secret, and change it in lock step?

With configuration management all this is just a look up away. And best of all, most of this information is authoritative.

When I come somewhere new, the use of these tools instead of a directory of miscellaneous scripts is like night and day. It literally turns a two day job into a ten minute one when the environments are complex.

Docker is great for a lot of things. It enables to do daring things to your environment even if it's so complex you don't fully understand it all because you can just shuffle containers around. But I would never use it instead of configuration management.

I would argue I accomplish most configuration management via the Dockerfile. In Docker 1.7, even if I want to use a different filesystem like ZFS, I can.

I agree with you that I probably have missed many important features in my architecture. However, it's simple and works for what I got. It sounds like in the future, I'll need a combination of Puppet / Chef & Docker rather than relying purely on ECS. For now though, it's quick & easy. Thanks for the thoughts, they all make me think.

It sounds like the poster is doing something similar to a 'Gold System Image' though with Container technologies.

Configuration Management is great, but I'm with the view that you should start with a combination of gold images/system builds and then stack on Cfg Mgmt on top of that.

Theory is you will have a known base and identical builds etc, otherwise you would get subtle drift in your configurations, especially for things which are not explicitly under control of Config Mgmt (shared library versions, packages).

Of course, this is not always feasible but if I were to start from scratch, I would probably try to do it this way.

You want to have your "Gold system image" available and this is most certainly what you should be deploying from; however your configuration management - whether it's Chef, Puppet, whatever - should be able to take a base operating system and get it setup completely from scratch.

This then solves the problem of ensure that you can completely reproduce your application from scratch; but also removes the possibly horrendous slow scale up time by using a "Gold image" that has already done this.

My current process is: Jenkins runs puppet/chef, verifies that the application is healthy after the run and everything is happy, calls AWS and images the machine, then it iterates over all instances in my load balancer 33% at a time and replaces them with the new image resulting in zero-downtime. Of course another solution is to pull out those instances and apply the update, and then put them back in.

And I'm sure someone else will have their $0.02 on their own process, which actually I'd love to hear :-)

I have an Ansible repo with an init task, which configures new boxen (makes sure proper files, dependencies etc. exist). Then to deploy, I have another task that ships a Dockerfile to the target boxen group (dev, staging, or prod) and has them build the new image, then restart. This happens more-or-less in lockstep across the whole group, and scaling up is relatively easy - just provision more boxen from AWS and add the IPs to the Ansible inventory file. Config is loaded from a secrets server, each deploy uses a unique lease token that's good for 5 minutes and exactly one use.

I'd love to hear how to improve this process, since I'm dev before ops. My next TODO is to move Docker image building locally and deploy the resulting tarball instead (though that complicates the interaction with the secrets server).

Do you end up having to update multiple containers with the same patch?

Context: I've looked at Docker, but not used any containerization yet.

One potential solution that came to mind was that if there was a standard way of deploying an application into containers, and Google/Amazon/Microsoft provided auto-updating containers, the maintenance of a secure container would be in the hands of companies who (hopefully) have the resources necessary to keep the entire stack up-to-date.

All containers running code are based on these images, so the updates are picked up on the next build/deploy.

This in the sense, i guess, that if they have a security flaw in their php that gives disk access, all the attacker will see is the content of the php container as the database will be on the next container over.

Then again the containerization seems to have come alongside devops, where the mantra seems to be "update early, update often, to hell with stable branches".

If a security flaw exists in one container due to the stack not being updated, isn't there a pretty good chance that it also exists in the other containers?

Also, for any given container, there probably still is a way for an attacker to do immense amounts of damage. With the database container you can steal customer data. With the PHP container you can remotely instruct the database to do whatever you want, or just point the code at your own database.

Pull a new base image, have your build system rebuild and deploy the relevant containers.

The same way you do in any application, container or not - you look up your documentation ;)

Love it for quick scripting containers (like netcat) too since it downloads in line 2 seconds.

[1]: https://github.com/gliderlabs/docker-alpine

Certainly can be worked around, but it's a little annoying.

1.) Create a new task and update the service with the task. From the documentation (http://docs.aws.amazon.com/AmazonECS/latest/developerguide/u...):

> If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service, one task at a time. The service scheduler creates a task with the new task definition (provided there is an available container instance to place it on), and after it reaches the RUNNING state, a task that is using the old task definition is drained and stopped. This process continues until all of the desired tasks in your service are using the new task definition.

2.) (way I'm doing it) Update the desired count to 0 and utilize wait_until (http://docs.aws.amazon.com/sdkforruby/api/Aws/ECS/Client.htm...) services are stable. After this, update desired count to 1.

You're going to get some downtime with #2. I'm not sure about #1 because I don't understand how something like an API service could be run without error'ing out that the port it's trying to bind to and forward is not available. So I'm not sure how it would ever reach the running state.. but I haven't experimented going about it that way. I can handle the small amount of downtime.

I think the key thing is that users shouldn't have to have a knowledge of Sysops or Devops to be able to take advantage of the benefits of containers. Most developer we've spoken to don't have time to learn a completely new paradigm, they just want the benefits.

ECS is pretty exciting, but isn't the same level of abstraction accomplished with the GCE/Fleet/k8s? Even the Kubernetes or Fleet install process on EC2 lets you quickly forget about the actual boxes.

The evolution of CaaS would be sth. like hyper.sh. Due to its hypervisor nature, the whole cluster+scheduling thing can be made transparent to developers, who simply "compose" the app spec and submit to run.

Either way, thanks.