I'm confused about how the documentation recommends using a Kubernetes operator to manage OS updates. That seems weird and backwards to me. I would rather see an immutable OS AMI in an auto-scaled group, and just replace the node instance whenever there is an update.
I can see a place for managing OS updates on an instance, but that seems more like "pets" than "cattle"... and I've always treated Kubernetes nodes like cattle, not pets. Isn't that the most common approach anyway?
We use 5000+ CoreOS nodes in production and never want to go back to replacing VMs with new images for each update again. In-place immutable updates are more efficient and faster. Unlike RPM based OSes that are hard to patch, transactional updates provide a safe way to perform safe in-place updates instead of wasteful operations such as replacing full VMs for small OS updates.
As a former CoreOS (now Red Hat) employee, I'm glad to hear of your success with the product and it's awesome to see that scale giving you time back in your day to concentrate on the big picture.
Not at all happy with things right now. You guys abruptly killed CoreOS which was great and left us to fend for ourselves. Fedora CoreOS doesn’t cut it and we don’t want open shift. We will need to move to either bottle-rocket or flatcar. A team member evaluated both and we leaning towards b-rocket since flatcar has a higher risk of dying by running out of funding during COVID (seen this movie before with CoreOS). Amazon’s customer support team is also a lot better.
1. update wasn't possible - that basically means that customers can't count on you
2. some decisions favored OpenShift instead of being general purpose
3. you rebuild the whole stack?! from scratch?!
4. the documentation is twice as bad as from CoreOS (which was already pretty bad)
5. the market now has more to offer, way better solutions. Fedora CoreOS is basically just a rebuild CoreOS that brings nothing new to the table and shaken up all customers. if I need to reinstall my nodes anyway I can evaluate k3os, bottlerocket, opensuse microos, rancher, whatever which didn't sleeped while you guys did rebuild your stuff.
I agree with you. This seems more complex than just having a auto scale group that auto rotates nodes after a certain amount of time and just picking a new update when the node launches.
I can provide a little background on this. In general yes I would recommend that you just use an ASG and roll out a new AMI. However that approach can be very expensive and time-consuming at truly massive scale (1000's or even 10's of thousands of machines).
Bottlerocket is built in part based on our experiences operating AWS Fargate, which obviously has as one of its needs the ability to patch a colossal number of hosts which are running people's containers, without downtime or disrupting their containers. Bottlerocket is designed to ensure that this is both efficient and safe. We aren't the only ones with this need. Many large orgs also have tremendous fleets, and its unacceptable to cause significant disruption by rotating at the host level.
Another aspect to consider is stateful workloads that are using the local disks. Bottlerocket lets you safely update your host if you are running something like a database or other stateful system where you don't really want to move your data around.
Not everyone will need to use this updating mechanism, but I think it will be very attractive to many of the larger organizations with a lot of infrastructure.
I agree with the confusion. There is the ability to rollout updates in a "wave"[1], but I'm not sure how this is better than a simple rollout strategy in kubernetes since a reboot of the node seems inevitable.
It seems to me, not to be combative, that if Fargate can't afford the "noschedule: node is old" overhead and customers of Fargate can't handle their containers restarting on a regular basis, there's something wrong with your management engine or with their design and implementation. Much of the point of containerization is that you can roll containers often and run enough of them that you never have a single point of failure. What part of that assumption is broken that destroying machines regularly doesn't work?
There are any number of reasons to avoid restarting things. Some customers are running code that has a cold start and needs some time to warm up its cache if it restarts. Some customers are running jobs (video rendering, machine learning training, etc) that might take literally days to complete. Interrupting these jobs and causing them to restart wastes the customer time and causes them to lose progress. Other containers may be hosting multiplayer game servers, and forcing them to restart would cause all people logged into the game instance to get disconnected or otherwise dropped from their game.
All of the above are use-cases that AWS Fargate is used for. Beyond this many folks simply don't like it when things happen unexpectedly outside of their control. We have Fargate Spot for workloads that can tolerate interruption, and we discount the price if you choose this launch strategy. However Fargate on-demand seeks to avoid interrupting your containers. You are in control of when your containers start and stop or autoscale.
This makes a ton of sense and I appreciate the response. I think what people aren't recognizing is that cloud services make you pay for performance, so doing things like relaunching containers which have slow warmup time literally costs extra money. While it's certainly important to design systems such that the containers can be tossed aside easily, that doesn't mean there isn't value in reducing how often that tossing aside occurs.
Any plans to reduce the minimum bill time for Fargate to accommodate short tasks?
With 1 minute minimum billing you have to turn to lambda for very short tasks or have a long running Fargate consuming tasks from some message bus.
If you choose lambda, your containers don’t work so you need to rebuild your runtime with lambda layers or ebs or squeeze into the lambda env.
If you choose messaging, say SQS from a lambda called by API gateway you’ve complicated your architecture and your Fargate instance is potentially hanging out billing, idle, and waiting for messages.
Fargate spot removed the last reason to consider AWS Batch. Short tasks could largely replace lambda.
This stuff is probably waaaay over my head, but isn't that why SIGTERM was made for ? To notify a running process that the host needs to be shutdown/restarted and to let the running process finish it's current task (current frame encoding / current multiplayer game / current request / ...) and that the state / cache / progress / ... needs to be saved.
The process on aws side would then be : send SIGTERM to all workloads. wait for [configurable] amount of time (maxed at xx hours) or until all workloads have exited (whichever comes first). Shutdown the node. Update the node. Start the node. Restart the workloads.
Yep you are right about SIGTERM, but let's think back to the original reason why we wanted to update the node: because of a patch, probably a security patch for a CVE?
What is the better option here? Implement a SIGTERM based process that allows the user to block the patch for a critical, possibly zero-day CVE for xx hours, remaining in a vulnerable state the entire time? Or implement a system that just patches the underlying host without interrupting the workloads on the box?
You aren't wrong, what you described is a possibility, but it is not the best possibility.
If there's a CVE vulnerability that is being actively exploited on your network, you should preempt running processes to deal with it, and absolutely must take the boot+nuke approach, because it already could be affecting any host that has not already been boot+nuked?
If there's not a CVE, AWS can significantly manage the lifecycle of their machines, and have ~5% of all of their machines "unschedulable" at any one time, waiting for existing processes to complete so that they may use an orderly restart before doing a boot+nuke. A SLA of "Tasks may never run longer than X days"(x=10-30) allows them to perform orderly restarts.
I don't know your background but the way you respond makes me think you have not been responsible for systems that multiple tenants rely on for varying workloads.
These assumptions you're making are dangerous because the variety of workloads across tenants is extreme. If you're going to do something like "kill compute no matter what" then you better have a good reason for it.
You may want to look at my resume. I've seen what happens when you don't "kill compute no matter what" - When compute does get killed no matter what (hardware problems happen quite often at scale), you have problems. I've also seen it done right. Clearly, Fargate has not - I could also tell you that from having used the service.
Nothing is "broken" about it. It's just that when you have tens of thousands of machines that might need an urgent security update, it's very inefficient and costly to destroy all of them at once instead of patching. Destroying machines regularly is not the same thing as frequently destroying all of them at once.
Constantly rotating nodes in and out of the cluster and restarting/relocating pods, even if mostly automated, causes a lot of needless infrastructure strain. It is IMO one of the most overlooked parts of Kubernetes, and I wish there was a better solution to maintain stable, long-running processes when needed.
Aside from being faster than replacing all of the hosts, the reason OpenShift does it this way is that you can't just burn down and replace a fleet of bare metal machines. While re-PXEing is possible, this takes a ton of time and stresses that infrastructure.
Doing the same on cloud, metal, OpenStack, VMware, etc means that your cluster's operational experience remains the same and in most cases is less disruptive.
edit: having your nodes controlled by your cluster has a number of other benefits aside from patching, like the Node Tuning Operator that can tweak settings based on the types of workloads running on that set of machines.
I can assure you that OpenShift doesn't take this path because it is "better". It does so because bare-metal is a significant part of their market and there isn't a better option to automate the process currently.
I once worked on a competing product (before the OS update operator was available) and the update-in-place model was always a disaster. Various problems like dns, service discovery, timeouts, breaking changes to dependency pkgs, etc make for a problematic process. Combine that with the frantic pace of k8s develeopment, short node compatibility window (2-3 minor k8s releases) and various CVEs - you end up debugging a lot of machines in unknown states that fail to rejoin clusters after reboots.
This has definitely not been my experience running many hundreds of Red Hat CoreOS nodes in production.
So far, aside from a few small flakey issues, having the cluster nodes _and_ the OpenShift cluster update in lock step has been dramatically simpler to manage.
Why? I view being able to do this as a huge advantage. Don't want to lose instance store state just for an OS update and love the kubernetes operator interface to be able to do this. The kubernetes operator also operates at a cluster level which means we don't need to write scripts to churn ASGs. It is eyebrow raising that they have only enabled this for Kubernetes and not ECS. I suspect that this is one of many signs that that the inferior and lock-in prone ECS service will be deprecated soon.
As strong as the engineering behind Bottlerocket seems to be, I'm not entirely sure who they built it for, except as a foundational component for AWS's managed offerings.
If you, as an AWS customer, decide to fully embrace AWS lock-in, then why would you run this yourself on an EC2 instance instead of running ECS or EKS? If you're trying to avoid AWS lock-in, why would you choose an OS that's locking you into AWS Systems Manager and Amazon Linux 2 for debugging needs?
Hi I'm a developer advocate in the container engineering org at AWS. I think there are a few misunderstandings here that I may be able to explain better.
First Bottlerocket is not Amazon Linux 2, it is its own minimal operating system, with most components built from the ground up in Rust. This is totally different than the Amazon Linux 2 you may be familiar with (and most other operating systems for that matter). Bottlerocket is optimized for running containers with high security isolation. The host OS is extremely minimal, it does not come with bash, an interpreter, ssh, or anything beyond the system basics needed to run containers. In fact it uses an immutable root filesystem. You aren't intended to run or install things directly on the host at all.
Everything that is installed and runs on Bottlerocket runs as containers, which are kept isolated from each other and the host with best practice security features such as SELinux. For example you can connect to a container on the host via AWS Systems Manager, or you can optionally enable a container that lets you connect to it via SSH. Once again the thing you are connecting to is the container on the host though, not directly to the host.
For this initial release of Bottlerocket we are focusing on providing image variants that are prepackaged with the requirements to serve as underlying container hosts for an ECS or EKS cluster. However we also intend Bottlerocket to eventually be something that anyone can use, anywhere, even outside of AWS, if they wanted to benefit from the secure by default, container first design of Bottlerocket.
But isn't a "developer advocate" trawling social media for opportunities to dump clarifications also a form of PR? It seems as if data tech has co-evolved the "good cop/bad cop" pattern. I'm not complaining though, so much better than "bad cop" alone.
I tried to see any performance characteristics of running bottlerocket, just to understand what the expectations are there. I assume since it was not mentioned it is either similar to Amazon Linux 2 or worse (but with security advantages). Can you request a follow on post on the aws blog that describes the performance impact of Bottlerocket? If performance is better, would be nice to know that as well of course.
AWS-Linux is so slow because of old packages that the new OS is guaranteed to be faster. It has a recent kernel & containerd. Podman would have made it even faster.
Would it be safe to say that bottlerocket is a container host for running containers under a hypervisor? Or is it intended to do full hw interfacing and run bare metal?
Bottlerocket runs containers using containerd, so containers are visible as processes on the host, from its perspective, not currently isolated from each other via a hypervisor. Bottlerocket limits containers ability to interact with the host or each other via SELinux, among other things.
We do have firecracker-containerd (https://github.com/firecracker-microvm/firecracker-container...) which is designed to allow the containerd runtime to launch containers as microVMs via Firecracker, with that additional layer of isolation via the KVM hypervisor. This stack is not currently fully compatible with K8s or ECS though, so it is not implemented using that approach yet. Rather Bottlerocket is built as a progressive improvement on the current state of container hosts, which is that many people are running all their containers on their hosts without any strong security hardening at all.
I think from the similar naming scheme of Firecracker and Bottlerocket you can already see the pieces of the puzzle that are in progress and the future potential though.
Thanks, I was more wondering what the relationship was between something like Debian GNU/linux and bottlerocket. From gp description it sounds like there's no "GNU userland", just the Linux kernel and some utility functions in rust - enough to launch containerd.
So if I have a heterogeneous collection of servers - I could install Debian, and run docker on Debian. It sounds like bottlerocket would more comfortably run on top of a hypervisor abstracting away the actual hw a bit? Eg on top of xen, kvm or VMware?
Obviously the Linux kernel can be made to run on a toaster, but maybe bottlerocket isn't ideal for that purpose?
Answering your initial question and this one: Bottlerocket today only runs in EC2, but we've tried to make it flexible enough to run outside of a hypervisor on bare metal in the future (in fact, a few engineers on the team are really excited to get it running on their RaspberryPi's at home; toasters haven't been added to our roadmap yet ;) ).
Bottlerocket has a GNU userland like many other distros. It is just one that is stripped down of many things including removal of interpreters, shells, and package managers.
If you want to explore more deeply, you can enable the admin container and jump into a shell on the host[1] to look at the filesystem and see what Bottlerocket's userspace looks like up close and personal. You can also see a bit more of this debugging/exploration tooling explained in an AWS Partner Blog[2].
I just installed Proxmox on a home server, and I’m using its CT containers (LXC) to run various services. Could I use this as a replacement for Proxmox?
Nope. Proxmox is mostly just an administrative UI, containers are a Linux feature. You don't need Proxmox for anything, you can just run containers "natively" in Linux or VMs with KVM + QEMU. The linked above is mostly just a Linux distro geared towards a pretty specific set of use-cases.
> Bottlerocket is optimized for running containers with high security isolation. The host OS is extremely minimal, it does not come with bash, an interpreter, ssh, or anything beyond the system basics needed to run containers. In fact it uses an immutable root filesystem. You aren't intended to run or install things directly on the host at all.Bottlerocket is optimized for running containers with high security isolation. The host OS is extremely minimal, it does not come with bash, an interpreter, ssh, or anything beyond the system basics needed to run containers. In fact it uses an immutable root filesystem. You aren't intended to run or install things directly on the host at all.
Sounds exactly like Qubes OS [0], but the latter is probably much more secure (based on Xen and VT-d). So in comparison Bottlerocket seems like trading some of security for performance.
Could bottlerocket be the basis for desktop system where it would be easy to switch between isolated environments? It would be very handy to have a maximally secure environment, at least from software perspective, for certain operations.
You may want to look into Qubes. It's not specifically designed for running containers like this is, but it does provide a single OS with multiple isolated environments for different programs.
As _underfl0w_ said, you're pretty much describing Qubes. Using it can be eye opening--nothing works together unless you let it, which is a good way to get a feel for your attack surface / unknown dependencies.
Qubes uses VM's. Subgraph OS is another option. It tries to achieve the same thing with containers, but their latest release is pretty old--not sure how alive that project is.
So did every other vendor. We have two CoreOSes, Flatcar, Container-Optimized OS from Google, Bottlerocket, k3OS, etc. Fortunately these aren't just different in name; there's a lot of experimentation going on around different ways to do updates, security, etc. I hope we'll eventually see some convergence after a few years.
Yep, very much agree. Container Linux was still very young and growing, but had some great promise. Incorporating the tech into RHEL and Fedora was a great idea IMHO. I think CoreOS was acquired for that specific purpose personally.
I don't think. Fedora CoreOS is more coupled to be an OpenShift node than a plain Container/K8s node.
it's sad what they did. And I was a Fedora user for my whole life, but that made me reconsider.
If you're using an ECS or EKS Cluster, you still have to run some operating system on the ECS Container Instances. The containers have to run somewhere.
There are a variety of options currently, including one from Amazon, but Bottlerocket seems designed to be a next-generation OS for container instances. It's extremely minimal, and designed to be as secure as possible, with transactional automatic updates.
Ideally, I would just deploy Bottlerocket as the underlying OS on my ECS Cluster, and then never have to actually do any management of the Instance OSes whatsoever. I would deploy containers on top of my ECS Cluster, Bottlerocket would keep itself up to date and secure, and my containers would live happily ever after. It would hopefully feel more similar to using Fargate than not, except without paying the higher price for Fargate, and having access to the wider variety of hardware configurations available to regular ECS.
Amazon has specifically said this in the Bottlerocket repo:
> Bottlerocket is architected such that different cloud environments and container orchestrators can be supported in the future.
It's mainly useful in concert with ECS or EKS right now, but it is architected to be useful in other places as well.
I'm excited about Bottlerocket as a project, and I'm glad it's open source instead of just an opaque AMI that you can use on ECS or EKS.
> It's mainly useful in concert with ECS or EKS right now, but it is architected to be useful in other places as well.
Right, this is the big question. If a community forms to support Bottlerocket off of AWS, then that's one thing, but until then, it basically just seems like a better option for ECS or EKS.
>>> I'm not entirely sure who they built it for, except as a foundational component for AWS's managed offerings.
>> <snip>
> If a community forms to support Bottlerocket off of AWS, then that's one thing, but until then, it basically just seems like a better option for ECS or EKS.
I'm failing to see why that's a problem or a source of confusion. There's a clearly defined market of "who they built it for", and there's a community option to expand Bottlerocket's target market.
The code is open source, so anyone could fork Bottlerocket and modify it to suit their needs immediately. Will Bottlerocket see success beyond ECS and EKS? Only time will tell... literally no one knows yet.
> If you're using an ECS or EKS Cluster, you still have to run some operating system on the ECS Container Instances. The containers have to run somewhere.
I literally mentioned Fargate in the same comment:
>> It would hopefully feel more similar to using Fargate than not, except without paying the higher price for Fargate, and having access to the wider variety of hardware configurations available to regular ECS.
If you're using an ECS Fargate Cluster or EKS Fargate Cluster, I don't consider those the same as an ECS Cluster or EKS Cluster. Unfortunately, there's no specific term commonly used for non-Fargate Clusters that I know of. AWS offers ECS Clusters and ECS Fargate Clusters.
If you were legitimately confused by my comment, I’m sorry. I could have said "non-Fargate" repeatedly, if it would have helped, but I thought the context made things clear, especially with the additional explicit mention of Fargate as a separate thing.
There are varying levels of lock-in. A Linux distribution to run containers is a lot easier to replace than, say, a database with a proprietary query language custom semantics.
Anytime AWS finds many of its customers using something outside of AWS, they want to build it in-house. AWS customers were using CoreOS a lot, hence this.
tough is one of the actual oss gems behind bottle rocket thats reusable in non aws contexts.. bottle rocket probably can be but likely that will always be a second class citizen, and afaics completely undocumented for usage outside of aws atm.
As far as running in non-AWS contexts, we haven't had time to head down that path yet, but as I mentioned in an earlier post, we've tried to build Bottlerocket in a way that it can be extended to work outside of AWS either as VMs or even bare metal. In fact, a few of the engineers on the team have been playing with getting it running on their RaspberryPi's at home :)
So difference between this and Firecracker would be that the latter is boot-speed and overhead optimized, and this one is a bit heavier but more capable?
If choosing between this and say Kata Containers plus Firecracker, the latter would be more secure because of VM isolation but this would be more efficient because multiple pods could go in a single VM?
Is Bottlerocket secure enough to host multi-tenant workloads within the same VM?
Firecracker and Kata containers are ways to run containers inside lightweight VMs that boot directly into the Kernel, they are not Linux distros in themselves. You are trying to compare apples to oranges. You could run Firecracker or Kata containers on top of something like Bottlerocket, however Bottlerocket is geared more towards the container-sandbox and isolation crowd while Kata/Firecracker is for those who think you can only get isolation using VMs.
multi-tenant workloads in the same vm.. thats sort of depends on your definition of 'secure', generally speaking for most enterprises the answer is no, the linux kernel is a huge surface attack space, firecracker and gvisor are different approaches at mitigating that. firecracker via a minimal hypervisor with legacy qemu stripped and a minimal exposed sys call/hypercall interface.. gvisor via reimplementing sys calls in golang intermediaries. both have tradeoffs, but my two cents would be around firecracker since its able to use more upstream kernel.
For a project similar to bottlerocket, checkout https://github.com/talos-systems/talos. It is geared for cloud, VMware, and bare metal users. We have integrations with Cluster API for each, with the bare metal provider being our implementation: https://github.com/talos-systems/sidero. Full disclosure, I am the CTO of Talos Systems.
I haven't dug into the engineering behind this yet, but my main concern with any custom Linux distribution is it often ends up as a waste of engineering.
It's pretty easy to write your own Distro and pair it down to the essentials, and pairing it down that way allows you to strip out complexity, making it easier and more reliable to patch it. But that then means you are now maintaining this custom thing forever. If you're Amazon, that might be fine, but I suspect that this will be dropped when it is no longer profitable or a competing project supplants it - meaning in 5 years this thing might be gone. (A common theme of custom Linux distributions)
And then there's troubleshooting. With a stripped-down distro, you will eventually need more tools to debug the thing, meaning you have to build and maintain packages to do that. Bottlerocket's answer to this is "run them in containers and use our API!", but I'm not sold on this. Have you ever tried to do debugging between host and container, or container to container? There's a lot of b.s. you have to hop through, and most Unix tools were not written with it in mind. I highly doubt that it will magically work for everything. If that's the case, then this "don't worry, because magic" idea is not really saving you work over maintaining a traditional OS.
Moreover, you don't need a custom distro to do live patching. There are simple tricks you can use to juggle files and processes during a live patch, to say nothing of "checkpoint process && 'mv old_file new_file' && thaw process", etc. Kernels support live patching too. So if the argument is "well it's easier to patch", i'm not sure you're not trading away "easy" in one place for "pain in the ass" in another (see above). All of this also argues that it's just as effective to treat live-patched systems as you treat immutable infrastructure, and I'm not convinced of that argument either. The former is just more complex, and complexity attracts failures.
Ultimately I think what you'll find is Bottlerocket will get a niche following, but also some people will get annoyed by it and go back to regular distros which already have well defined methods.
So, is this available as an ISO? I currently run an Ubuntu VM in bhyve on my FreeNAS home-server to host various containers for experiments, etc... could I run this instead or is it tied to AWS?
It isn't today, but we've built Bottlerocket in a way that it can be extended to work on bare metal or other places outside of AWS. There are a few issues on our github[1][2] that are similar that you're welcome to watch or contribute use cases to.
At my previous company we discarded the AWS-Linux distro and used rancherOS for container hosting because the version of yum they used was too flaky. They were unwilling to move to DNF to try and fix it. We've long badgered them for something like this (rancher style AWS-Linux dsitro) and they seem to have finally listened. Too bad, I moved to a different company and a different role to benefit from this. At least my old colleagues will be happy
Amazon Linux 2 is an AWS-optimized Linux distribution but it's a full distribution you're used to — RPM-based, generally compatible with RHEL/Centos, etc. — and that has the usual benefits and costs. You can customize it as much as any other Linux distribution but you're taking on the corresponding level of effort to secure and operate it. If all you run are containers, that's overhead for things you're not really using and it also means that you're going to be slower to update things like the kernel and system libraries since there's more to test and backwards compatibility is a big concern.
Can anyone at AWS comment on how this fits into the Fargate roadmap and Compute pricing? Presumably, a slimmer OS for things like EKS nodes could translate into some sort of Compute discount.
So, I'm a little confused; is this not what NixOS is all about, or is there a difference? (as my question probably suggests, im not all that knowledgeable about nix)
I've fallen in love with Alpine. I'm using it on my bare bone servers running from a ram disk (USB drive) hosting Docker containers and even qemu VMs. All on encrypted zfs volumes. Totally love it
Here are some links for you that I think would be useful for a related setup, however I can't speak for the user that posted that. The headless installation should follow the same principles of using a RAMDisk though.
Rancher has been around a long time and isn't cloud-specific. Until this distro can prove itself useful outside of the AWS ecosystem, then Rancher will still dominate.
> RancherOS 1.x is currently in a maintain-only-as-essential mode. That is to say, it is no longer being actively maintained at a code level other than addressing critical or security fixes.
quoting support.rancher.com, taken from the 4th comment down in the linked Github issue
Well, they use a linux kernel which is written in C and some assembly. All of the overlying components are apparently written in rust. These components still interact with the system calls exposed by the kernel.
An example of a kernel itself written in rust would be the Redox microkernel. It seems to be able to run a graphical environment but it hasn't been updated in a while.
Did you even read the blog post? This is running Linux, and only the AWS software-components running on it are largely written in Rust. It is still Linux and the OS is not written in Rust, nor the container daemon and probably none of the non-AWS software packages.
That honestly sounds terrible, and apparently only works on ECS. What a shame. I was thinking that AWS was actually going to provide something useful to the open source community.
I can see a place for managing OS updates on an instance, but that seems more like "pets" than "cattle"... and I've always treated Kubernetes nodes like cattle, not pets. Isn't that the most common approach anyway?