is anyone running this stuff in production yet ? not just secrets.. but the whole Docker Swarm with the v3 compose file format thing ?
We have been testing out production clusters with Swarm 1.13 and I think its neat. But what do I know - I'm not a cloud hosting specialist. And the lack of success stories out there is making us nervous.
I can honestly say the only thing we found the full Docker line up (Containers, Swarm, Machine, etc.) good for large scale dev or staging/testing stuff. There are definitely uses for each of these tools, or other combinations, that have been successful for other people. It seems like Docker has a lot to focus on keeping up with the superior orchestration platforms that are out now. This in turn means quickly outdated (or missing) docs, inconsistent behavior of certain features, and other little "gotchas".
Depending on the size of your team, I would recommend Kontena (www.kontena.io) or Kubernetes (https://kubernetes.io/). While I haven't used the latter, a group of two devs is re-architecting our entire system to use Docker and Kontena. Nearing the end, the whole process seemed incredibly fast to iterate. While K8s seems far more proven in real production environments, it seemed much more daunting to us with more features than we needed for our use case.
Both of these are easy to fix. Configure etcd with peer certificates for clustering and a cert for client-server connections (https://coreos.com/etcd/docs/latest/v2/security.html). If you need encryption at rest, encrypt the filesystem.
If setting up a secure cluster is daunting, then use a distribution that handles it for you. OpenShift (https://www.openshift.org/) is built on kubernetes, and it's install is secure by default.
Disclaimer: I work for Red Hat, and spend lots of time on OpenShift consulting.
there are tons of these niggling issues that are cropping up.
the complexity of using k8s goes up exponentially every day. I have bo doubt it is a great piece of tech.. but at this point, it seems tailor made for consulting.
A rooted node has access to everything that lands on that node, and anyone who can reproducibly escape to root on a node from a container can do so on any node they can schedule on.
It's definitely something we'll fix in Kubernetes, but rooting workloads is the primary problem, and secondary acl defense in depth is good but won't block most attacks for long.
There's no way to schedule anything from a worker node -- Swarm follows a push model for all scheduling decisions; worker nodes never pull anything. This is the best ACL model possible: the one that doesn't exist because worker nodes have zero ability to perform actions.
Default ACLs are clearly the most important line of defense in an orchestrator's security model, because whether a container escape can happen is not something the orchestration system has control over.
I'm not sure I disagree, but pull vs push with the same ACL rules in place is the same outcome. A secure Kubernetes configuration would also not be able to schedule from a worker. Partition of secrets is important, but anyone able to trigger node compromise still sees secrets and workloads anywhere they can schedule.
At a design level, push removes an entire class of vulnerabilities, full stop. Pull requires good ACL'ing and properly implemented controls for the lifetime of the orchestration system's implementation. Pull makes the system vulnerable to both misconfiguration and incorrect ACL code implementation. Pull is clearly inferior.
Being able to trigger node compromise should have nothing to do with being able to schedule.
The thing is with security, default matter. Many people will not change default settings (especially in complex products like Kubernetes), that's why I suggested that people thinking about using Kubernetes should read and be aware of those risks.
On OpenShift I'd be very interested to see a list of the secure defaults that have been chosen, is there a list of the changes from base Docker/Kubernetes policies available anywhere?
We have been testing out production clusters with Swarm 1.13 and I think its neat. But what do I know - I'm not a cloud hosting specialist. And the lack of success stories out there is making us nervous.