I have to say: before I began to read through, I was already convinced the author's idea was not a good one. But by the time I finished, she convinced me that it was a good plan in her situation.
There are lots of great tools in the Docker/container ecosystem these days, but I can see the argument that a major infrastructure migration should be done in incremental steps. Using containers as simply a Puppet replacement strikes me as a really good idea! You want your application to be _ready_ for containerization, before you try to introduce an orchestration system.
Although, when she mentioned that her team is meant to be "Heroku for the rest of the company", I couldn't help but think of Convox [0]. It's a great tool, it's open-source, and using it is really just like having your own Heroku. We used Convox at my last company, had amazing success with it, and I've been recommending it ever since.
Convox founder here. Thanks for the shout out! We are indeed building a private Heroku like platform so you don't have to.
It's a simple of layer around AWS so it doesn't add extra craziness some other approaches do.
As for the OP, her approach is spot on!
Containerizing your app is the first step in future proofing it. Doing so correctly will remove layers and therefore simplify things.
You now are looking at a package that can run on a laptop and run on a server with traditional techniques (upstart, systemd, etc).
You may take this to the next step and enter the orchestration world. Or not! Patience is a virtue, wait 6 more months and all the orchestration tools will be that much better.
It's all about simplifying. Adopt Dockerfile and shed Homebrew and Chef. Run containers on a "dumb" box with systemd and shed Packer.
Run containers on ECS and shed custom AMIs and userdata and nagios and...
In my experience, AWS alerting only sends one alert once per issue. If you're asleep when the single SMS comes through, tough. Nagios will nag you until you do something about it...
Maybe ECS is different. I haven't played with it, but I imagine it's "make an SNS topic" like everything else.
It's not containers, it's AWS. You get CloudWatch events that can turn into alerts for many parts of the platform. With the ECS and ELB/ALB integration, we get paged about downtime and 500s without Nagios.
I think when it comes to containers HN has been guilty of a shocking lack of scrutiny and unilaterally hyping Docker.
For example Docker was based on LXC till 0.9 but was so successful in hype and misdirection of the project it's based on that till today most commentators here do not seem to have a proper idea of the LXC project and whatever ideas they have are negative. How did this come to be?
I don't know how healthy it is for open source if the HN community lets VC funded projects essentially hijack open source projects in this way and allow misconceptions to grow about them. For the record LXC has always been a full scale container manager and was and remains a much simpler way to use containers than Docker.
Most of the big problems holding back containers like security, isolation, multi-tenancy are kernel side and not container manager side or ecosystem side and will be solved kernel side. Similarly most of the features of containers that are usable today are also thanks to kernel side.
Yet the folks who work on kernel namespaces, aufs, overlayfs and other container used technologies receive no rewards or even recognition while companies like Docker completely reliant on their work suck up the attention and hype. How is this sustainable?
How many know the author of Aufs or Overlayfs? How much support have these projects received and how do these projects sustain themselves.
How many know that cgroups were not namespaced, the issues this created for containers and that they recently got namespaced? How many know about the fantastic advantages but also pitfalls of user namespaces? Isn't this the kind of 'container discussion' we should we having on HN instead of Docker or vendor focussed discussions?
And thanks to the lack of scrutiny, Dockers complex use case of containers has passed unexamined and worse been conflated to containers. This has only increased technical debt for users just stepping in and runs the risk of putting users off containers due to the inherent complexity.
> HN has been guilty of shocking lack of scrutiny and unilaterally hyping Docker
Spot on. It's sad how often people behind code that does the real work (LXC, Overlayfs and many more) get overlooked, while marketing-fueled operations get all the glory.
It's also worrying to see that people driving technical decisions are so easily swayed by hype and fail to investigate what's under the hood.
Docker is a case where "the whole is greater than the sum of its parts". Yes, LXC and cgroups and aufs and overlayfs and whatever else do a lot of the heavy lifting, but Docker glued it all together in a way that was approachable by any developer, not just Linux geeks. I remember briefly looking into LXC ~6 years ago and being totally lost. Then Docker came along with a simple command line interface and growing repository of images.
If you want to be popular among developers, the "hello world" developer experience should be extremely simple. See: Stripe, Twilio, Docker, GitHub, etc.
That said, I do think it's too bad that the Docker client/daemon are the lowest-level building block most people are interacting with. It doesn't seem to follow the Unix philosophy of doing one thing well.
> but Docker glued it all together in a way that was approachable by any developer, not just Linux geeks.
> I remember briefly looking into LXC ~6 years ago and being totally lost. Then Docker came along with a simple command line interface and growing repository of images.
I'm not trying to discount your experience in any way, but, remarkably, my impression of Docker was exactly the opposite.
I personally find LXC to be very plain, simple and easy to understand, whereas docker seems intimidatingly opaque and confusing.
I was introduced to linux cgroups and namespaces via my use of docker. It is a fantastic tool for using these concepts quickly and I don't currently see rocket etc overthrowing dockers dominance in the market place.
However I do sense (and feel my self at times) a growing upset with the usability/ stability of the docker engine.
The OCI (Open Container Initiative) should allow new solutions to come to market without fragmenting the space. i.e. it should be possible to run any image with any container engine/Linux.
Docker is valued at $1B. It appears that justifying that valuation has taken them into a number of adjacent markets. I wish they were more focused on the Docker Engine itself.
Yep, the OCI is doing great stuff. At the bottom of the Cloud Foundry stack we have Garden-Runc (https://github.com/cloudfoundry/garden-runc-release). In past we had our own homegrown containerization technology and we're super happy to have been able to swap that out entirely in place of runc - it allows us to make use of the significant engineering talent contributed and has enabled us to move faster and reduce risk.
We're also buying into the image-spec for this reason.
Right now the focus is in underpinning CF, so although it is production ready functionally, it lacks the pretty fantastic user experience that Docker brings. Perhaps allowing us to reduce our focus on the containerizer, we'll be able to spend more time improving UX - which would be neat.
I believe the Kubernetes team is working on a simple implementation of the Kube API which is a thin wrapper around runc as well - so you're right, there are some cool things coming around as a result of the OCI.
> Yep, the OCI is doing great stuff. At the bottom of the Cloud Foundry stack we have Garden-Runc (https://github.com/cloudfoundry/garden-runc-release). In past we had our own homegrown containerization technology and we're super happy to have been able to swap that out entirely in place of runc - it allows us to make use of the significant engineering talent contributed and has enabled us to move faster and reduce risk.
Hi! I'm one of the maintainers of runC and I wanted to say that the CF stuff around runC is awesome. Keep it up. :D
> I believe the Kubernetes team is working on a simple implementation of the Kube API which is a thin wrapper around runc as well - so you're right, there are some cool things coming around as a result of the OCI.
Actually, ocid is mostly being worked on by people from the OCI community (runcom, mrunalp, myself and others). Though we are getting support from the Kubernetes folks with it, which is pretty cool. :D
Sweet. runC is awesome and using it is awesome. We are also super excited about the rootless work you personally have been doing, so thanks for your hard work in that area.
I can't take all the credit, Jess Frazelle was the reason that I started to work on an upstream version of binctr. But one of the cool things we can do with the rootless containers work (aside from making the lives of researchers easier -- something I've experienced personally) is to add OCI image building to KIWI (which is (open)SUSE's image building tool) and thus add it to OBS (the Open Build Service which is (open)SUSE's package building tool). I think there's a lot of cool things that will come out of rootless containers. :D
I'm going to be giving a talk about rootless containers at Linux.conf.au 2017 in case anyone is going to be attending.
That's the main issue with Docker right now. They've built an amazing static compiler that works with any language, but there's a question as to whether that compiler is worth 1B.
The nomad[0] team was just discussing today how much effort to put into our rkt support[1] (our support for rkt lags behind our docker support).
So far we don't see a lot of evidence of our users taking rkt into production over Docker, but I'm very curious if there's a significant number of users like Julia looking to avoid running docker in production.
From a nomad developer standpoint, while building long command strings is a bit awkward compared to calling docker's API, it's quite easy to support new features and manually run generated commands to test our driver's behavior.
As a user, I don't like have two schedulers (nomad/docker) on my nodes. I don't like having two daemons taking care of the overlay network either. And I don't like magic and things that break from time to time. We've had our fair share of broken Docker upgrades.
Hey Michael! Could you email rkt-dev about your questions as well? Particularly around the long command strings. There are other ways to launch rkt containers besides long args.
Our only question is more of a usage survey. Not sure it's our place to perform those on your mailing list. :)
So far the command line building is more pro than con. It's so nice to be able to copy & paste commands out of logs to see exactly how rkt is being run.
I'm just waiting for rkt to mature a bit. I personally don't see much value in running the Docker engine in a production context. It adds needless complexity to what should be a relatively simple solution.
I for one am definitely looking to replace Docker in production with Rkt. Just waiting for Rkt support in higher-level components (Kubernetes in my case) to mature a bit more.
Have you considered adding OCI support to Nomad? Is this something you're interested in? Currently we're working on getting an OCI-powered container runtime that works with k8s called ocid[1]. It'd be really awesome if nomad was interested in getting OCI (or even ocid) to work as a runner as well. :D
Note that you can use runC (the underlying runtime for Docker) in a similar way to rkt (except you also don't have to deal with images if you don't want to -- runC just runs a rootfs). And runC doesn't have a daemon as well, and is the canonical implementation of the Open Container Initiative runtime specification. We're also working on ocid, which is a way to run kubernetes purely on an OCI stack (no Docker required).
Though, shoutouts to the rkt guys. They really are awesome and have been helping out a lot in the OCI effort.
I don't get the author. So to sum it up, they like what Docker did with containers, but they don't want to use Docker, at least not immediately, because it requires to learn a few Docker commands and a bit other stuff and therefore they decided to build their own hybrid shit, which will take them probably more time to build than to learn Docker and then on top of that still need to train their employees on that as well, instead of using a well working, vibrant, healthy eco system, build by someone else, for free?
I cannot help myself but that sounds like the worst idea ever and totally backwards. Particularly when they she said that they want to run containers reliably in production, how do they think they are going to do this? The reason why Docker is a "little" bit more than just `Docker run` is so you can run containers reliably and in case of a fail over recover quickly. So how are they planning to run containers reliably in production without adopting an existing, well tested, well working framework?
>Particularly when they she said that they want to run containers reliably in production, how do they think they are going to do this?
As I understand it, they keep doing what they have been doing all along, including everything that currently makes their system reliable, and gradually add some of the desirable features of containers in a way they can fully understand.
That doesn't sound like the worst idea ever to me. The danger of ending up with too much of a homegrown snowflake is of course real. You're not wrong about that. But she described the risks of the alternative approaches pretty well I think.
I don't get what the issue is with the migration plan to Docker either. You have a big system with many components. The best way to start is build a Hello world app on Docker and orchestrate it with whatever framework you think suits best your requirements, let's say Kubernetes. Then run that hello world stuff for a while to get a POC and make yourself familiar with the sytem. After that start converting 1 of the many services to a dockerized service and run it side by side in your new infrastructure. Maybe even load balance some of the traffic progressively over to the new stuff until you are confident enough to run 100% load on that migrated servcie. By then you should be pretty experienced with the most part and have a great recipe to migrate the rest slowly over. I don't see a reason why you would want to start with a homegrown solution at any point in this process.
Yeah and I don't see how that is any good. Making small changes across your entire infrastructure means that everything will move extremely slowly. I cannot imagine that the entire infrastructure even deserves to be moved. What's wrong by just leaving some really old, solid working, but less critical stuff just on the old infrastructure until it gets phased out one day? It feels like a waste of time and resources to force a horizontal move across everything. Also technologies like containers and Docker are extremely fast moving at the moment. It might very well be that some parts cannot be moved immediately today, but in 6-12 months time they can and if you follow their migration plan then meanwhile a whole lot of other stuff which could really benefit from the new infrastructure will have to miss out. Also if you discover an issue very late then you ended up with moving a lot of things back and fourth, even if it is just a small step.
> but they don't want to use Docker, at least not immediately, because it requires to learn a few Docker commands
That's not what the blog post says. They don't want to use docker because it does a lot of things. They use rkt because it does fewer things. It's not about learning commands, it's about complexity of software. They selected rkt because it does less stuff, not because it's easier to learn.
article is really low on details. they use naked lxc? what does it mean just launch containers in docker and rkt? wishy washy it sounds to me. I want details.
We followed this idea at my workplace! It worked wonderfully; we slowly containerised every project we have, then deployed the containers as "dumb" "programs". Yesterday I began the orchestration step, and everything has gone extremely smoothly. Altogether this has taken about 12 months of on-and-off work, maybe a man-month all up.
> I think that just using containers by itself will force us to be disciplined about how we package and run services (you have to install all the stuff the service needs inside the container, otherwise the service will not work!).
>
> There’s no way for it to silently depend on the host configuration, because its filesystem is totally separate from the host’s filesystem.
I don't buy this, at least as presented. /etc/awesome/blah.xml does not spontaneously come into existence on a host on its own. No matter if you're building a docker image, a VM image, or managing long-lived hosts via a configuration management tool, you have to specify that /etc/awesome/blah.xml is created and has certain contents.
It sounds like the author is working on a poorly structured and documented puppet codebase, where all the dependencies of services aren't clear. Presumably if they can figure them out in order to build working docker images, they could also figure them out and then refactor the Puppet code to make it more maintainable.
At that point they could use their puppet code to generate their docker images, which fits nicely into the "horizontal change" philosophy.
> What is the motivation and benefit for running containers without docker?
I believe that Docker bet the company on the wrong business model. Therefore knowing that you can run containers without Docker can be desirable, as it gives you options in case Docker turns out to be not viable.
Docker made it easy to build and run containers from the command line. This technology was released at the right time and in the right place. And it got huge traction. Docker used this traction to get a very large amount of VC capital. Therefore they now need to show a very large return.
How do you go about achieving that large return? A common way to do this is to use one piece of technology that is popular as an anchor to lock customers into an entire stack. And then you monetize the stack, which is easier as it has higher value. The technology in this case is the container runtime, and the stack would be an full enterprise grade container orchestration system.
This strategy worked for others. Microsoft was a master in this. VMware as well, and so is Oracle. The reason I believe it will not work for Docker is:
1. The anchor technology is open source. Microsoft managed their lock-in via Windows, VMware via ESX, Oracle via their DB. It worked because those are proprietary. In the open source world you can just take Docker.
2. Even though the technology is open source, you can still try the ecosystem approach where you exploit certain network effects attached to your distribution of the code that protect you (e.g. vendor certifications). This is what worked for Red Hat. In my view it is unlikely to work for Docker. I don't think certifications are required as most container apps today are not vendor apps. And I would not know what else they could use.
3. For orchestration solution they are engineering against Google's Kubernetes. With all respect for Docker, this is going to be very hard as Google has a huge amount of experience in this area.
4. Due to Docker's need to protect their business model, there is no true community in the sense that there is an open exchange of ideas. The docker agenda prevails over technical decision making (like: should you have a docker daemon? Should Swarm be part of the Docker Engine?). Contrast that with Kubernetes which run in a very open manner by Google. This is why OpenStack got the traction, and why Eucalyptus and CloudStack died off. OpenStack had the most open community.
All of this can lead, IMHO, to a situation where Docker will not succeed in delivering Swarm as they see it. Kubernetes will become the default container orchestration system, and Docker is relegated to a small place in the stack. This situation would make it hard for Docker to justify their valuation.
Docker has realised the money is in the platform. So they're competing against Red Hat, who have OpenShift.
They're also my employers, Pivotal, and IBM, and HP, and SAP and every other company with a Cloud Foundry distribution.
The problem for Docker is that Red Hat, Pivotal et al started with selling an entire platform to F1000s. Docker and Kubernetes have great groundswell in the community, but the difference is that Docker need to make money from a sold platform and Google don't -- their play is GCP.
> For orchestration solution they are engineering against Google's Kubernetes.
Let's not forget Mesos, Nomad, Diego (the Cloud Foundry bits) and so on. And yes, it turns out this problem is actually very hard. Much harder than it looks.
> Due to Docker's need to protect their business model, there is no true community in the sense that there is an open exchange of ideas.
This is one of the reasons Pivotal donated Cloud Foundry IP to the Cloud Foundry Foundation, and it's why the Foundation rules are written to grant voting rights to members which contribute engineering effort. Red Hat are relevant as a large and critical part of an ecosystem. So it is for other successful OSS companies. But being the "owner" tends not to go as well. People who buy from us want to know that they can switch to IBM without much disruption and vice versa.
> This situation would make it hard for Docker to justify their valuation.
I can't blame them for trying. And I suppose they didn't see Kubernetes coming to suck the oxygen out of a big part of their platform.
The weird story of Docker will be of a company that created a PaaS, abandoned it for a component, then tried to build a PaaS.
Disclosure: as I noted above, I work for Pivotal. As Docker moves into platforms, we are competitors. Though we cooperate on runC.
Recently tried to run docker on Ubuntu Desktop 14.04. After googling a bunch of errors, trying different versions, but still failing decided to try rkt. That one worked out of the box, but apart from a bunch of simple images failed to run the caffe docker image.
So, my motivation of running without docker would be to understand every little thing they do to actually be able to use them and rely on them.
It installs fine, it just doesn't work. There is an issue on github with the same errors as I saw, where they discuss not fixing it, because of how uncommon the system is.
> What is the motivation and benefit for running containers without docker?
I regularly hear from people who want to run containers without Docker. There are several motivations, all of which are perfectly valid:
1. Learning. It's fun to build things from scratch to understand how they work under the hood.
2. Bad experience. Early versions of Docker were quite buggy, and we initially struggled to keep up with the colossal growth in usage and feature requests. As a result, many of the people who tried Docker in production too early were badly disappointed. Some of them decided Docker wasn't for them, and started looking for alternatives.
3. Extremely custom use case. If your deployment is larger, or more complex, or more specialized than 99.99% of deployments out there, then "mainstream" platforms like Docker might not be the right fit for you. Of course we try to make Docker as customizable as possible, to support more a wider spectrum of use cases with plugins. But realistically, no single platform can cover all use cases, and I don't think any platform ever will. Docker is no exception.
4. Philosophy disagreement. Different people have different opinions on how applications should be developed and deployed. Docker tries very hard to be agnostic - to accommodate as many opinions as possible. But we can't please everybody. If Docker does not fit your philosophy of development and deployment, then the natural response is to look for an alternative.
5. Competition. Many Docker competitors started out as extensions or modifications of Docker, and over time are looking to reduce their dependency on Docker.
I'm probably missing other reasons, but these are the ones I've been most exposed to. These are all reasonable reasons to not want to use Docker.
Our approach is that, if you want to run containers without Docker, we should make it as easy as possible. In practice that means spinning out as many of the underlying components as possible (what we call the "plumbing") so that you can assemble it yourself without being stuck using the entire Docker platform. For example:
- runc [https://runc.io] is a standardized "container executor", which we donated to the Linux Foundation as the reference foundation for the OCI spec.
- libnetwork [https://github.com/docker/libnetwork] is the low-level networking implementation (including overlay networking which is a very useful primitive for container clustering)
Our opinion is that, even if you don't use Docker, by using these components for your own purposes, you are indirectly contributing to making Docker better. Splitting out these components has also forced us to refactor Docker into a more modular, more robust design.
You missed another, big reason: "Fewer, simpler moving parts" is the main reason I would prefer if Kubernetes could run without Docker as a container runtime.
At the moment, Kubelet needs to control the Docker daemon (a somewhat brittle relationship), and a bunch of concepts — from networking to volume management — could be simplified by making the container runtime more K8s-specific and less general-purpose. A tighter coupling would also mean you'd always be running K8s against the exact container logic it was made for, as opposed to now, where the loose coupling means you'll be subject to bugs (and there are so many) in new Docker versions.
That and the fact that Docker is increasingly building in competing concepts (services, etc.) via swarm mode, which I don't care for.
Docker still has a place for things like image building, of course.
I would categorize that under 5: competition. Kubernetes started out as an extension/modification of Docker, and over time Docker has evolved some of the same features as its kub-based competitors. The result is an uncomfortable integration between partially competing platforms.
Personally I think the long-term solution is for kubernetes to adopt containerd - it's the small, stable, rock-solid container runtime that it needs.
Whether the Kubernetes community continues to build primarily on Docker, or switches to containerd - either way we will support them and help in every way we can. Behind the superficial internet drama, there is a solid engineering collaboration between both communities, particularly between Docker and Google. We may disagree on a lot, but at the end of the day we have a lot of users who rely on us, and we have a responsibility to work together to solve their problems.
> Kubernetes started out as an extension/modification of Docker, and over time Docker has evolved some of the same features as its kub-based competitors. The result is an uncomfortable integration between partially competing platforms.
That's quite inaccurate. It's is like saying that OpenStack started out as an extension to KVM. Both had very different (original) scopes.
> Whether the Kubernetes community continues to build primarily on Docker, or switches to containerd
Kubernetes has 3 container runtimes at the moment: Docker, Rocket, and the new remote runtime which allows you to plug in a separate server using a protobuf interface. There is a lot of focus on the remote runtime now. The CRI-O project for example is implementing a remote runtime that allows any OCI conformant runtime to be plugged into Kubernetes.
No. Kubernetes was launched at the first Dockercon, with a very specific focus of "we want to help expand what can be done with Docker by leveraging the significant operational experience we accumulated at Google". They delivered very successfully on that promise.
The focus on diversifying away from Docker is very recent, and is the direct result of the competitive tension I talk about earlier in the thread. It's completely understandable.
EDIT: A perfect example of that competitive dynamic is actually this very thread! the GP (`sysexit`) works at a competing vendor. You can tell in his post that de-emphasizing the dependency on Docker is important to him. In fact that talking point is so recognizable that I knew where he worked before even looking him up. There is nothing wrong with that, every vendor has an "official party line", and Docker is no exception. The point is, this is a delicate issue which has more to do with inter-personal and business dynamics than with code. And you should definitely verify the credentials and agenda of everyone speaking of authority on the topic of containers platforms. We vendors are everywhere, and we are biased as hell.
> The focus on diversifying away from Docker is very recent,
> and is the direct result of the competitive tension
I think you miss a reason at least equally as important, if not more: There are a lot of people who want to do non-container runtimes but still leverage the rest of Kubernetes. I don't want to say no to them (who am I to decide what tech is "good"?) but I don't want their code in my tree (too big already, too hard to maintain, too much email, etc).
You're absolutely right. I was focused on the subset of kub-based platforms which are container-focused and are overlapping more and more with Docker.
But you're right, kub itself is a neatly generic orchestration component which can be used with non-container things. That is an excellent reason for decoupling the underlying container runtime.
Docker have broken out some Docker logic into libraries, which is why the Garden team swapped out their driver to use runC. Kubernetes could presumably follow suit and target the libraries instead of the Docker daemon.
The experience of the Garden API (a containerising system that ultimately predates Docker) is that tighter coupling is actually a mistake.
If Garden had been tightly bound to its implementation or orchestrator, it'd still be using raw Linux system calls and running under a 2nd-generation orchestrator.
Instead it's now using runC, OS X or Windows APIs to create containers and runs under multiple consumers -- most heavily under Diego, a 3rd-generation orchestrator and Concourse, a project automation tool.
Disclosure: I work for Pivotal. Many, but not all, of the engineers who've worked on Garden work for Pivotal also.
My main concern is coming across a an issue only to research it and find a "won't fix"[closed] and 100 +1s asking for it. When I have something super important in actual production this scares me way too much to trust.
That's where the competition will come in handy. Docker Inc has its own corporate goals. Luckily, they are decoupling the underlying components more and more.
I can't comment much on docker as I have little experience but can I say this is a really honest and courageous post - I would have expected some corporate-speak not "we got things wrong early on", and a self-awareness and openness about the challenges being faced at a corporate level that has made me change my mind about the amount of trust to put in Docker the company.
Great! There is one question I would like to have answered, by you and you alone:
if I'm using SmartOS, and therefore zones, and all my applications and configurations are packaged into OS packages, and I use imgadm(1M), vmadm(1M), and Smart DataCenter (SDC), what do I need Docker for?
As I wrote previously, the question is for the founder of Docker and him alone, but since you volunteered your opinion, I am extremely pleased to inform you that I love making OS packages and performing configuration management at scale that way. I especially enjoy being able to ask the operating system's software management subsystem exactly which configuration changes have been done to it, and get a precise response back. I enjoy it even more when I am able to perform mass in-place upgrades of configuration, on tens of thousands of systems, in parallel, because I'm performing configuration management using operating system packages, or simply destroy the system's boot block, reboot it, and have the network instantiate it, from bare metal to fully configured virtual servers ready to serve data, by just using Kickstart, JumpStart(TM), or SDC, without any additional code. No Chef. No Puppet. No cfxengine. No Salt. No Ansible. No Docker. No KVM. No XEN. No Kubernetes. No Mesos. No extra software or daemons required or needed, other than what ships with the OS.
Fully isolated, virtualized UNIX servers. Servers are cattle, not pets. Deployments in up to 25 seconds.
That was the challenge. And it was fun to solve.
Have fun maintaining OS packages.
Oh I am! It's incredibly satisfying to be able to run systems where no human has ever logged into, and no file has ever been manually modified, and all the files and all the changes are accounted for by the operating system.
See also, not quite the same, but very much related and relevant to your comment:
Configuration management at scale, using an operating system's software management subsystem is the old, new-new thing. It is the foundation and the future of even more massive cloud computing. Now you'll be able to say someday in the future: "ha! I read that on Hacker News years ago!", and point to this post.
> by just using Kickstart, JumpStart(TM), or SDC, without any additional code. No Chef. No Puppet. No cfxengine. No Salt. No Ansible. No Docker. No KVM. No XEN. No Kubernetes. No Mesos.
Thank you for saying this!
Building OS packages and OS images and provisioning servers was done automatically 10 to 12 years ago with much less work.
It's still the preferred choice for many large tech companies and nobody talks about it because it just works and it's nothing new. Fast deployments, immutable infrastructure, modularity.
Much less work than messing with Docker, Mesos, Kubernetes, Chef, Puppet, Rkt, Salt, Ansible, cfengine, etcetera... it's much, much simpler to build OS packages and keep the metadata needed to build them under revision control (git, Mercurial, Bitkeeper, ...) And as a bonus, it is designed to plug into the change management process from the ground up, and not as an afterthought. (Deployment automation and configuration management are not enough for complete lifecycle management.)
So what are you bugging shykes for? You have achieved everything Docker offers, and you had fun in the process. Seems like you're smart enough to realize that.
Anyway, someday in the future I might look back on this and think "ha! someone put all that effort in even though NixOS already existed".
Is critical thinking, even just asking questions, undesirable now, unless someone is being praised to high heavens? Is that what we're down to now?
"ha! someone put all that effort in even though NixOS already existed"
SmartOS virtualization is based on Solaris zones, which existed long before people on GNU/Linux even realized that containers were the way to go. The groundwork, and my own work was laid down long before any of this became hype. Project Kevlar, nee zones was started around the same time as NixOS, and another participant already mentioned that large companies did large scale deployments and configuration management via OS packages, which is both true and correct.
You did not just ask a question as a form of critical thinking. You asked a question leading into a lecture on the technology you prefer.
Docker is just a simple wrapper that reduces the amount of knowledge needed to run a containerized process. It did not become hype because of any technological advantage, it became hype because this old technology became accessible not just for the kind of people/companies that don't (can't) run enterprise class Solaris deployments, but also the kind that don't have 20 years of experience in making robust well managed deployments.
Your process might seem easy and straightforward to you. But for the average developer and Linux enthousiast, there's not much that can the simplicity of the Dockerfile. Systemd or a more specialized system like Kubernetes to run the containers, and a simple script to provision your machine is all you need. You can pick all of this tech up in a few days and steadily grow into more advanced subjects like monitoring, logging, security.
Anyway, you're not helping anyone by just saying "SmartOS is better than Docker+(list of redundant and unrelated other tech)", maybe instead show us a one-page tutorial on how to set up a SmartOS system with a webapp stack (i.e. LAMP, MEAN or RoR) that can be executed by a newbie without a creditcard, registration, trial or phone call.
You did not just ask a question as a form of critical thinking.
Yes I did.
You asked a question leading into a lecture on the technology you prefer.
Only after you left your Good luck with packaging everything, which I have good reason to believe was cynical sarcasm.
It did not become hype because of any technological advantage, it became hype because this old technology became accessible not just for the kind of people/companies that don't (can't) run enterprise class Solaris deployments,
Performing configuration management at scale with OS packaging has nothing to do with Solaris - it works on any operating system which has a Bourne-family shell, AWK, and sed.
but also the kind that don't have 20 years of experience in making robust well managed deployments.
Then it would be a good idea for such people to actively seek mentors.
But for the average developer and Linux enthousiast, there's not much that can the simplicity of the Dockerfile.
That's what distinguishes the average developer. A true enthusiast will go out of their way to become a master of the art, so the assertion doesn't apply to them.
It's easy to hack several hundred thousand files together and deliver them via Docker. But, what happens when it's time for upgrades? Go through the entire hackfest again? Every time?
Anyway, you're not helping anyone
that depends on what your mentality is: if you're a true enthusiast, you're probably reading a lot, and when you stumble upon crumb trails like mine, the light bulb might go off reading what I wrote, and it might just be the clue you need (that's the idea). And you wouldn't be the first or the last person that happened to.
As for me, I'd gladly show how to do what interests you, only I have no blog, and no time to set one up, fight spam, hack attempts, and so on. I can afford one short answer like this, but I cannot afford months of engineering implementing the infrastructure for a blog and hardening it.
And by the way, nice stab at Joyent on your part, but SmartOS is free, open source software, and the entire ecosystem + the 14,000 packages never cost a penny. Even their cloud on a laptop ("COAL") preconfigured image is gratis.
As for packaging being hard... there is such an abundance of packaging documentation for any popular operating system... all you have to do is sit down and "warm up the chair" by reading it. Do you think I learned it some other way?
which I have good reason to believe was cynical sarcasm.
Well true, and I apologize. I have done some packaging for Debian, CentOS and recently Alpine and whilst the systems are smartly designed and effective, using them is neither easy nor enjoyable. Their complexity stems from the fact that they're smart and strict about dependencies. Docker sidesteps this by making images entirely self contained. This might not be your cup of tea, obviously it wastes system resources, but it does simplify the ordeal.
Then it would be a good idea for such people to actively seek mentors.
Or they could use Docker and focus their energy on building value for their company so at some point it can afford to employ proper system administrators. Who then may or may not decide to migrate the systems to SmartOS.
Do you think I learned it some other way?
I don't and neither did I. I just think that if knowledge-intensive engineering can be avoided it is worthwhile. Especially when an operation is young.
I have done some packaging for Debian, CentOS and recently Alpine and whilst the systems are smartly designed and effective, using them is neither easy nor enjoyable.
I cannot write anything of Alpine Linux, as I have never used it, but I have done Debian / Ubuntu packaging and configuration management with packages, and that, compared to RHEL / SLES / SmartOS / Solaris / HP-UX / IRIX, was the hardest to get right.
One of the reasons why Debian and Ubuntu are so hard to get right is that the Debian packaging guidelines are insane: for example, it is not allowed to have packages named in any way other than lowercase; there is no /opt, as the Debian people do not understand the filesystem hierarchy standard and consider it a hindrance, for reasons completely unknown to me.
So now imagine you had to build, say, your own PHP with OCI8 support: you cannot name the package php, nor can you deliver it into /usr, because the next upgrade from Debian could break your production. And there is no /opt, so neither vendors nor other 3rd parties have a place to safely deliver their software (/usr/local is not it, and is against standards).
What I'm trying to say is, you picked one of the worst operating systems, ever. Even RHEL, as bad as it is, is nowhere nearly as poorly designed as Debian GNU/Linux.
Pick a better OS. pkgsrc format is pretty straightforward, and AT&T SVR4 packaging[1] is the absolute best when it comes to doing configuration management. Unbeatable.
This might not be your cup of tea, obviously it wastes system resources, but it does simplify the ordeal.
If delivering software which JustWorks(SM) is an ordeal, then you need to step back and seriously re-think what needs to be corrected. Delivering software should be a breeze, and it should happen automatically, meaning that the software management subsystem of the OS should worry about it for you. But that also requires you to seriously revise how you put software together, because if it's an ordeal for you, something is majorly wrong. (Since I don't know your situation, I cannot comment what.)
Or they could use Docker and focus their energy on building value for their company so at some point it can afford to employ proper system administrators. Who then may or may not decide to migrate the systems to SmartOS.
This line of thinking, is, in my experience, completely, utterly incorrect on several levels:
developers should not act as if system administration is beneath them. One can never, ever become a master developer without becoming a top-notch system administrator: how can anyone develop software which is correctly integrated and runs on an operating system, if they do not understand how to administer that system? I have seen that over and over and over again in my decades of experience, and I have seen no developer who was able to code quality software which the system administrators didn't hate, because it was all hacked up and required lots of hacking to get working.
I just think that if knowledge-intensive engineering can be avoided it is worthwhile. Especially when an operation is young.
If you do that, it will come back to haunt you. With a vengeance. And the reason it will come back to haunt the company is because it's taking on a huge technical debt.
Are you familiar with the concept of taking on technical debt?
Containers existed before Docker and will exist after Docker, and articles like these are desperate attempts to remind everyone of that in the face of Docker's unilateral destruction of the concept. I know I'm right because of comments like these that presuppose "well, if you're doing containers, use Docker, the alternatives just aren't there." Except they were, before Docker, and after Docker.
Docker is a developer tool, not a container runtime. The 'containers' it presents require you to commit a daemon on every machine, commit to a weird storage and distribution story when, you know, files of containers served in a flat directory are totally adequate, and so on. Docker made extremely complicated choices for a lot of things and now everybody wishing to advance containers has to deal with presuppositions like these, where Docker exists and the motivation for not using it is unclear to a lot of people, you included.
Docker is a container runtime, not merely a developer tool. It was intended as such from the beginning. It is successfully used in production at large scale by a number of companies, including Uber.
The fact that you don't like Docker doesn't change any of that. If you don't like it, don't use it. But there is no justification for you to claim it is only a developer tool when it is obviously more than that.
Docker was originally built on top of LXC. It is now becoming more flexible with what is underneath, but there's a reason you need a VM running in the background on macOS to use Docker.
Docker is tool to interact with LXC (and hopefully other container systems).
Docker doesn't use LXC anymore[1]. The reason you need a VM is to get the Linux kernel, not LXC, and the reason you need that is because the macOS kernel (Darwin) doesn't have the necessary containerization features. On FreeBSD, for example, Docker runs natively, without a VM.
If MacOS doesn't have certain linux kernel features, neither does FreeBSD. How can Docker run on FreeBSD when FreeBSD has no cgroups? I think you misread/misinterpreted your reference.
> How can Docker run on FreeBSD when FreeBSD has no cgroups? I think you misread/misinterpreted your reference.
FreeBSD can run 64 bit and 32 bit Linux binaries via it's Linux emulator[1]. Plus FreeBSD has had containers since FreeBSD 4 which was released in March of 2000, except containers on FreeBSD are called jails.
> If MacOS doesn't have certain linux kernel features, neither does FreeBSD.
Just because one operating system doesn't have a feature that does not mean a different operating system does not have that feature.
> Jails are considerably different than containers. Jails are more like chroot jails on linux instead.
Thats incorrect both jails and "containers" are examples of OS level virtualization[1]. Instead of virtualizing the hardware you virtualize the operating system, ie system call table.
So what is your alternative? Run containers your own unique way and roll your own distribution mechanism and networking to fragment yourself from the tools in the "container" space?
When "your own unique way" requires less development time and efforts and it's easier to learn for new hires, yes.
That's why you hire system engineers - to write good automation.
The motivation is to make a smaller change to their operations and infrastructure. Right now they are running application services on linux machines using puppet to make the machine environment work. Switching to a container gives them reproducible environments that work on a developer workstation and a server. It also gives some operational benefits for how you run a process I think. You can do this without buying into the docker daemon, network overlay, orchestration. If you are currently able to deploy services on linux to production then you can do that, but with containers. This is an incremental win.
So, in general, tools like Docker are crap because they're like the Apple iPod equivalent of old standard Unix-style tools: bloated monolithic slapped together crap that's crammed with non-extensible non-intuitive extras, designed for people who don't want to know anything about how the sausage is made. But if you do want to do fancy production things, it's nice to have a tool that thousands of people have used for thousands of different things, and bake in all the weird fixes for weird problems over thousands of iterations of testing and bug-fixing.
However. If what you want is to simply understand what's going on under the hood, this is a great way to do it. The only thing better that I would recommend is to actually make your own Docker. It's not that difficult, you're basically just slapping together some system calls and execvp()ing some other standard tools, which is what Docker does, and you don't need to support any of the extra fancy features. But you gain an intimate understanding of what could be going on when stuff in production is breaking.
> The only thing better that I would recommend is to actually make your own Docker. It's not that difficult, you're basically just slapping together some system calls and execvp()ing some other standard tools, which is what Docker does
I don't get the differentiation between "rkt" and "docker". What complexity does Docker add? Separate networking? Difficulty communicating between processes?
Just trying to understand why rkt is less overhead...
A lot of it is historical. What rkt added conceptually when it was introduced was 1) a spec and test suite, 2) signed/content-addressable images and first class third party repos, 3) not running everything under a single daemon.
Since then, Docker has improved quite a bit on these so the most in your face practical differences are smaller, but there are still philosophical differences that affects it.
E.g. rkt comes out of CoreOS. CoreOS does a lot around embracing systemd to its full extent. Systemd can provide a lot of the capabilities that Docker did itself, and parts of rkt's design flows from that. E.g. restarting, querying status, capturing the logs, so in a systemd based system, Docker integrated fairly poorly in that systemd would be starting and keeping track of a Docker client rather than the process actually controlling the container, while rkt fits right in.
Again, the difference is getting smaller, and various tools like runc etc. from Docker now allows you narrow the gap even more (if you put in extra effort).
Try both, basically - they're similar enough that it's worth figuring out which "flavor" you like best.
One large difference is that docker has a daemon that exposes an HTTP API and acts more or less like an init for containers.
The docker daemon has historically had some stability issues as well as some security implications. Running a command line tool like rkt is a vastly smaller attack surface and less complex stack overall.
Also, not sure if this is still the case, but Docker's daemon ran as root on the host. Any vulnerability has the potential to give root on the host machine to an attacker. I don't think this has ever happened, but rkt's approach of using a process per container just makes much more sense in a security context (and for containerization in general).
The daemon needs superuser privileges to do its business, but your containers are not running as root. That lives behind the --privileged option and has ample guidance in the documentation against using it.
Wrong. Without usernamespacing your containers do run as root. If you type `docker run busybox id` it will print uid=0, and that uid is 0 in the container and out of it.
You are namespaced, so the linux kernel promises that even though you're root, you're not dangerous, and there is syscall filtering and shit going on.... but that historically has not really fared that well!
But your statement is false. You're root with and without privileged. Privileged gives you back CAPABILITIES which are different than USER, so your claim is bullshit.
He said "but your containers are not running as root".
That is objectively false.
uid = 0 is "privileged" basically everywhere in the kernel, from filesystem management (reading a file bindmounted in that's owned by root e.g.) to binding to low ports (like 80).
As you can see from the docs, it says "the most important security improvement is that, by default, container processes running as the root user will have expected administrative privilege (with some restrictions) inside the container but will effectively be mapped to an unprivileged uid on the host."
This implies the reverse, that if you don't use userns then your process as root in the container will be mapped to a privilege uid on the host.
This is all I'm saying is true. You clearly don't understand what I'm saying.
Could you please give a hint on what kind of stability issues you experienced? Do they still exist? My team is thinking about using Docker for high loaded system in production, and stability is one of our priorities here.
I don't personally operate docker in production, but googling "docker stability" or perusing their github issues should lead to the same anecdotes I'm referring to.
If you'll indulge me putting on my corporate hat for a minute: we've had some customers very happy scheduling a high volume of containers at a high rate on nomad:
Just started learning salt stack for infrastructure provisioning. But it makes so much sense just to build all requirements in the container itself. Did anybody already moved from tools like salt/puppet to container only approach?
rkt actually uses systemd-nspawn to run and namespace the container. What rkt "adds" before that is downloading, verifying managing the container image and setting up the cgroup for resource limitation.
systemd-nspawn is great! The workflow feels much more native than Docker on GNU/Linux host. The easy integration with networking, BTRFS snapshots, etc is very powerful.
I've successfully used Docker in production since it was a hobby project of dotCloud. I was answering the OP's question about why anybody would ever consider not using Docker for containers, so it's ironic that I'm now responding to your comment as the followup about my personal preferences.
Clever commentary trap: why not? Here's why. Oh, well, nobody's forcing you to use it.
The bloody comment I replied to called it that, and it's not controversial. Docker did not suggest running in production for many years and the production stuff is largely an afterthought. Again, I developed this opinion from running it for years.
It was not intended for production usage from the beginning, as you claimed, but I'm already tired of responding to this thread because I'm turning instantly gray for not buying into Docker so why would I bother proving that you're not correct?
You called it that. Read what you wrote. Here, I'll paste your entire comment as it is now, since you keep editing it:
> Containers existed before Docker and will exist after Docker, and articles like these are desperate attempts to remind everyone of that in the face of Docker's unilateral destruction of the concept. I know I'm right because of comments like these that presuppose "well, if you're doing containers, use Docker, the alternatives just aren't there." Except they were, before Docker, and after Docker.
> Docker is a developer tool, not a container runtime. The 'containers' it presents require you to commit a daemon on every machine, commit to a weird storage and distribution story when, you know, files of containers served in a flat directory are totally adequate, and so on. Docker made extremely complicated choices for a lot of things and now everybody wishing to advance containers has to deal with presuppositions like these, where Docker exists and the motivation for not using it is unclear to a lot of people, you included.
See the first sentence in the second paragraph? That's you. You wrote that. You did not present it as an opinion, but as a fact.
I edit to add because I'm on a phone. Nothing you've (annoyingly) pasted has been touched since it was submitted the first time. Please take your overly aggressive abuse of me to someone who cares, and reread what moondev wrote:
> Docker is by far the more mature and adopted development tool.
That's him. He wrote that. I added to it with something that you disagree with, and I'm rapidly tiring of interacting with you because you're making it extremely hard to remain civil.
But since you're the expert on Docker's production intentions, could you perhaps discuss Swarm and how it compares against competitive technology in the field? We can start small: what kind of scheduler does Swarm employ? Two-level, optimistic? What is your understanding of the runtime and performance bounds of the selected scheduling strategy? What is the expected latency for scheduling decisions as the number of executing containers grows? How does Swarm handle failure to schedule?
Could we then compare that scheduler against Aurora, Marathon, Kubernetes, Omega, and Borg? Why do you feel that Docker is production ready in light of the competitive work being done in this space? What do you feel is the difference between Mesos and Aurora? Between Docker and Kubernetes? Since Docker is intended for production usage, can you elaborate on some of the challenges you've experienced running it in production?
The title is "Docker: the Linux container runtime." So, your assertion that "Docker is a developer tool, not a container runtime" is simply false.
The text clearly describes deployment of docker containers as much more than simple development tools: "docker can run on any x64 machine with a modern linux kernel - whether it's a laptop, a bare metal server or a VM. This makes it perfect for multi-cloud deployments." That sounds like production deployment to me.
Whether or not you like Docker, or think it works well in production, is immaterial to this. Docker is a container runtime, it is intended for production deployment, and that means it is not merely a development tool as you claimed. The first website Docker ever published is proof of Docker's intentions. It was intended as a production tool from the very beginning.
Docker was always intended to run in production. That was the goal from the beginning. The developers recommended against running it in production before the 1.0 release because it was in beta. Developers of many products that are intended to run in production do the same thing before the first stable version is released.
That is not the same thing as saying that Docker was never intended to be used in production, and was only ever intended to be a development tool. That's the view I'm arguing against, and it's not trivial -- it denies the fundamental purpose of Docker.
If someone sent me an email equivalent to this blog post, my sole recommendation would be ProxMox running OpenVZ. OpenVZ gets you 90% of the way to Docker-style containers without breaking anything for her developers.
The current version of Proxmox uses LXC instead of OpenVZ, which is interesting since it's using the same kernel containerization features that Docker uses, unlike OpenVZ which required a custom kernel.
There are lots of great tools in the Docker/container ecosystem these days, but I can see the argument that a major infrastructure migration should be done in incremental steps. Using containers as simply a Puppet replacement strikes me as a really good idea! You want your application to be _ready_ for containerization, before you try to introduce an orchestration system.
Although, when she mentioned that her team is meant to be "Heroku for the rest of the company", I couldn't help but think of Convox [0]. It's a great tool, it's open-source, and using it is really just like having your own Heroku. We used Convox at my last company, had amazing success with it, and I've been recommending it ever since.
[0] https://jvns.ca/blog/2016/10/26/running-container-without-do...