Also, I really don't see the problem with the volume mounting approach. It may look ugly on the command-line, but when you're using an orchestration tool, it is actually quite painless and solves a lot of the issues mentioned in the article.

I do completely agree that one should completely avoid running a custom entrypoint. They are often written and then forgotten and can lead to really nasty bugs.

Lastly (this is more a Docker criticism, not the article) writing to stdout is all well and good, but Docker does a terrible job handling it. There is no way to truncate logs to stdout coming from a Docker container so Docker just holds onto the entirety of the log contents until the container is removed. For long running applications, this makes logging to stdout a deal-breaker. In a move absolutely contrary to my last point, I commonly use a custom entrypoint purely to handle logging. It passes all arguments to the application and then redirects stdout and stderr to cronolog which writes to a volume in my log pulling container.

  /var/lib/docker/containers/*/*-json.log {
      copytruncate
      rotate 2
      size 40M
      missingok
      nocreate
      nocompress
  }

More recent docker releases have added other logging drivers, and added options to the "json-file" logging driver for "max-size" and "max-file", which sounds appropriate, but that came after I worked around the problem and I haven't upgraded yet.

I think it goes without saying that practicality/pragmatism trumps convention or philosophy every time. I suppose the question we have to ask ourselves every day is "are we being resistant to change or are we being pragmatic?" Anyway, he does cover this when he mentions applications you don't have control over. So I'm not sure it's a fair criticism. This post seems to be aimed directly at developers or people with the latitude to effect change with them.

In this example, it's only six simple values but what happens when you have 10 or 20? Or you have 10 applications with six values each that need to be deployed to four different environments?

Or what about if there are multiple teams making concurrent changes at the same time? What if some application starts failing due to to a recent variable change and you want to revert back or track down who made the change?

I feel like once your application grows to more than just a few simple values, you end up creating a big file to populate these values and you end up back to using configuration files.

If you need the flexibility of environment variables for a semi-configurable non-secure variable, use them to overwrite a sensible default.

In general 12-factor gets my hackles up as it comes across as dictatorial with explaining why. Even when I'm wrong I like to be gently convinced rather than hit over the head with rule book. Can someone point me to an extensive source that clearly justifies each factor? Ideally with an actual debate about each point (as this often surfaces the strongest parts of the case for something)

One reason the factors are presented as prescriptive because apps that don't do this won't work on Heroku.

Is there a specific factor you'd like to deep dive into?

I'll pick one to start: Environment.

There are many ways you can set and read configuration for an app: env, config files, config tools like chef or puppet, config database like zookeeper or etc. if we are talking about config like a database URL you could also use a service discovery system.

Env represents the simplest contract between your app and whatever platform is running it (the OS, Docker, Heroku, ECS).

If the platform can update env and restart the processes to get the new settings, no other config management is necessary.

It's UNIX, it's simple, and it helps you bootstrap any more specialized config management if you need it (set ZOOKEEPER_URL or CHEF_SERVER_URL).

So ENV feels like a factor to become very prescriptive about.

The biggest debate I can see is if ENV is sufficient to build our micro services on, or if service discovery "magic" is necessary too. I.e Zookeeper, Airbnb SmartStack or Docker Ambassador containers.

For the vast majority of apps, ENV is sufficient.

I personally still build my more complex apps around ENV and at all costs avoid needing to use a service discovery system. The added complexity and operations isn't worth it to me.

I have a strong hunch that service discovery won't become an app development pattern that everyone uses until a managed platform (like Heroku) offers it. Perhaps this is where Docker, Swarm and Tutum is headed.

Files on disk have the problem of being persistent, though, and being subject to Unix permissions, instead of the process you're explicitly giving the env variables to.

The solution I work on is to keep files in a non-persistent filesystem that audits access and ensures tight permissions (Keywhiz), though in many cases a tmpfs and auditd will do the same.

But it's pretty common to put service credentials into a config file, so it's an anti-pattern to version-control them. It's _way_ safer not to, which means you shouldn't be version-controlling the thing that runs your container? This is sort of tricky. We're doing it by volume-mapping a non-version-controlled file for database credentials, and storing the rest of the configuration in the database.

We're doing this: the env vars are stored as a stage/container/key hierarchy in version-controlled eyaml files (yaml with encryption at the value level, nice for git diffs). At deployment the eyaml gets decrypted by ops or jenkins converted into a container env map (in our case a kubernetes resource controller).

Additionally we tag deployed containers with the config's git hash to have reproducible deployments, which is actually pretty useful. (again we leverage kubernetes labels, but this principle should could be applied to other orchestration tech i guess).

If you're using Cloud Foundry, you put them in your manifest.yml and check that into source control. When you do `cf push`, they'll be updated.

Disclaimer: I work for Pivotal, who donate the most engineering to CF.

I worked with countless apps and developers at Heroku on getting their apps running well on the platform. There was always one great mystery: why not build our apps a bit differently (dare I say better) to work in the cloud?

The database connection pattern is spot on. For any network resource, try to connect and if there is a problem retry with back off.

Also log the connection error events that a monitoring tool can notify off of.

I've seen apps that have the absolute worst behavior around this error that will happen. The worst is crashing the app in a way that triggers thrashing around restarts.

We had to build tons of complex restart back off logic into the Heroku platform to handle this.

I often wish app frameworks made this easier. I think most devs don't do these things because it is a chore for s problem that only happens occasionally.

But what if Rails baked this into ActiveRecord?

At one point Rails only logged to files on disk. We came together to add stdout logging to the framework.

You have managed to capture the spirit of the post. The goal was to highlight areas where developers can take action, and how improving even the little things can go a long way to improving the entire system -- even the one you can't see.

In the early parts of my career I would often take pride in building complex systems to accommodate for misbehaving applications. Throw in some fancy Nagios alerts and a sleep depriving on-call rotation; I looked like a hero.

Then I learned how to write code.

This was the turning point in my career (2006). I was now brave enough to modify "legacy" production applications to take advantage of "new" infrastructure features like service discovery (use DNS records instead of IPs), and logging directly to syslog (asynchronously with proper ring buffers).

I was willing to learn any language too: Python, PHP, or Java, it did not matter because it allowed me to take action and contribute at the heart of the application.

I'm not saying platforms that also handle "misbehaving" applications or complex failure scenarios are unnecessary. I just consider those platforms as extra layers of protection, not a free pass to ignore building applications that take responsibility for reliability from startup to shutdown.

Where are you suppose to store secretive environment variables like database password or apikey/secret pair etc (say in Ubuntu server)? Is storing in something like ~/.profile or ~/.bashrc and doing `export SECRET_KEY='plaintext_secret'` on the server enough, or should they be treated in an even more secretive manner?

In case the database is pristine, ok I see, I do a "create table if not exist"

But if the database is at version N and I want to go version N+1 , what do I do? I mean I do know about database migration tools, but how does it integrates in your "pure 12factor" deployement, as it means when you deploy you need to have at least this order:

1. bring up the database 2. run the migration script 3. bring up the application

and the article was advocating to make things in a way that you don't need to have a "you MUST first run this, then that"

So maybe that's a schema A and a schema B, or maybe you have applied schema B which only app version 1.1 is optimized for , but version 1.0 is what is in production immediately following your database migration. So you can't make changes in schema B that would immediately render app 1.0 broken, which means you need to not box yourself into a corner with future assumptions as much as possible.

Ultimately if downtime is not an option you end up writing these capabilities in at every layer. Whether it's an api endpoint or code talking to a database you often have to make carefully thought out changes incrementally to ensure that things can all operate simultaneously, and often this ends up having metadata about the versions of everything as an option for taking different code paths.

This article touches on this in the way that if suggests making your app deal with an available database and one that is not available. Same with a field in a schema or a payload. To make your code less brittle instruct it what to do in both cases.

It's then up to the init daemon to attempt to restart that process, and up to the monitoring and orchestration tools to ensure that the environment returns to sanity over time.

If something is a precondition then the app shouldn't act like it wasn't just to make Docker configuration easier. It needs to fail fast.

Retrying database and other connections is sometimes the right thing to do in long running applications. But I dont' think application launch is the right time for it. Application launch is an opportunity to make sure that all dependencies were in place at some point. If things break later on, the odds of it being a temporary issue are much better.

"But I dont' think application launch is the right time for it. Application launch is an opportunity to make sure that all dependencies were in place at some point. If things break later on, the odds of it being a temporary issue are much better."

This is presuming you or some human has control over the lifecycle of an individual process.

The trend in both mobile and cloud native is the process model, which says the opposite: your app process can and will be killed or relaunched any time by the underlying OS. It may do so out of sequence with backing service availability. This, retries (with a time or count bound, perhaps) are a sensible default.

No, that doesn't matter at all. It's like with DBMS transactions. I want a defined point at which the system is in a known good state or fail in some detectable way.

For long running processes that get bounced automatically, there needs to be some sort of monitoring anyway. Monitoring is easier if the application does not linger endlessly in an inconsistent state.

I think we're (as an industry) getting to scale and complexity of systems that warrants systems to heal themselves for a range of predictable and well-understood failure modes, in a way that doesn't require my manual interference.

That way all your app has to worry about is "Every time I get started, I should try and connect to my dependent services. If it doesn't work, bail."

And the monitoring process gets to worry about things like "I should retry X times before giving up. There need to be at least Y instances of this process running."

What is interesting is that , for the stateful/persistent data processing, most large scale systems are rejecting transaction boundaries as we know them (fully serializable isolation and consistently) for relaxed consistency. There are some good articles and papers on how programming needs to change to enable better self healing / "recoverable to a known state" behaviour, such as CRDTs.

For cars if something is wrong then in some cases you can start and even drive but users get warning. If there is something really wrong car will not start.

So I think what author suggests is at least asking for trouble.

Almost everywhere as quoted:

"Everything in this post is about improving the deployment process for your applications, specifically those running in a Docker container, but these ideas should apply almost anywhere."

Kelsey's recommendations aren't that different from general resilient systems guidelines in the Erlang community, or any of the many notes on how distributed system development is Different.

"But what is sane default if you have to communicate with 3rd party server and your system is worthless when connection is not there? "

The sane default is to wait and keep trying to connect for at least a bounded period of time (and a sensible approach to backoff).

The point is that, if you're building distributed systems, you need to account for partial failure. One of the 12 factors is the process model: your app should be considered a process that will be killed and/or restarted at will. It might do so when your backing services are currently unavailable.

The sensible thing is to retry a bounded number times (with backoff) before giving up. Sometimes the underlying application platform also does this for you (by killing your app process and rescheduling it, if it doesn't come up after a certain time period).

Baking in timing dependencies as a failure mode makes software less resilient.

It suggests that you provide the locations of backing services in the config. This seems insane, since it means that you cannot move any backing services. Do they expect you to restart when you switch backing services? Do they expect you to run all your internal traffic through load balancers?

We provide each container with the address of the local service discovery server (Consul) and it finds what it needs itself, when it needs it. I assume everyone using this kind of setup in production is doing something similar?

12 Factor was pioneered by Heroku, so yes.

Edit: I don't understand the downvote. This is the factual answer. 12 Factor apps are supposed to delegate a lot of cross-cutting concerns to the platform, including the problem of "where does that service live?" and "can you give me another chunk of service?"

Why not let a PaaS do this for you? Heroku, Cloud Foundry, OpenShift or the others I've yet to learn about?

Disclaimer: I work for Pivotal who donate the majority of engineering effort to Cloud Foundry.

And the reality is that every app becomes distributed as soon as it has a database or client. A ton of legacy applications make the implicit assumption that the network is reliable, and fall over hard when it isn't true anymore.

I've worked on some legacy migrations and it's usually a process of incrementally chiselling out services and cleaning up hard-coded assumptions. Tedious but usually doable.