I have mad respect for Jeff.

I really, really, HATE Ansible.

There is a special place in hell where all the worst of humanity go. It's a room where you must build either Ansible or Jenkins infrastructure, for eternity.

Since Jeff is a professional, I am assuming his book does not detail the myriad ways in which Ansible is just fucking horrifically designed and a nightmare to use. Where the docs are usually missing or incomprehensible. Where important features seem to just not exist. Where barely anything works - but when it does, it works in a totally asinine way, that you will spend hours figuring out. Even if you've done it before. Or the nightmare that is trying to debug it, or use it with modern cloud infra and best practices.

Sometimes - rarely - Ansible is an OK solution. It's more lightweight than its competition in the configuration management space, and it can be portable, if you add all the portability yourself. Well-maintained Galaxy stuff is quite useful and can save you weeks of time and years of therapy.

But for the most part, it and its Configuration Management brethren are better left for systems where someone is manually monkeying with them and breaking them, and you need something automated that also has some dependency-tracking to fix complex things for you.

The Cloud is still, sadly, mostly not immutable. So we still need shitty "orchestration" (Configuration Management) tools to maintain it. But I really hope they go away. I have never seen a Configuration Management tool that I have liked, because their very concept is automated duct tape.

(my bona fides: 20 years operating large-scale systems, including a reimplementation of DynamoDB, which was powered by OpenStack and AWS infra scaled dynamically by Ansible playbooks. I. Still. Have. Scars.)

Ansible might not be perfect, but, like you, having done this for two decades, I seen some shit with all of Ansible's competitors (the same hate you have for Ansible I have for Chef). The stlib in Ansible is what will keep me using it as long as I have a need to ssh to anything.

Agreed - Ansible was for a time when people were SSH’ing into boxes and installing stuff and it’s great at that. Jeff still does a lot of that with all his self hosted projects on his home network. Deploying containers and using declarative tools like terraform is much more suitable for keeping cloud infrastructure in sync with your desired state.

This is the second time I've seen this today. I'm a hobbyist hacker and do small projects. I'm still SSHing into servers to install and configure things. How does it work with cloud infrastructure... Do you not SSH into your server in the cloud and install things? Do you just rely on ready-made scripts to run what you need?

It varies a bit, but here are three approaches I've seen.

1. You build custom VM images that have everything you need baked in. You don't upgrade these VMs, instead you build new ones and delete the old ones.

2. You use cluster management tools like Kubernetes or Nomad and run your software as containers. You upgrade by running a new version of the container. (If you choose to manage your own nodes in the cluster, you manage them as VMs using method 1.)

3. You don't run your software on servers you manage. You use so-called "serverless" services where your software doesn't have to think too hard about the OS it's running on.

Right, and the goal is to be able to easily deploy the same thing. Every time you SSH into a server and install something, you may not be getting the same packages due to updates or packages becoming unavailable. If you’re using an image, most likely you can easily tear stuff down and re-deploy it. Also you’ve got the code used to build the image and can modify it if necessary.

But you need to create: 1. A Custom vm image, so how do you do it?

2. A container: here is a bit simpler since you may use docer-compose...

Hashicorp Packer is the tool to build VM images with the same idea as building a container image. I used it some years ago before moving to K8s completely

Someone told me that our public facing dns server running bind has 3800+ days of uptime and despite bind is up to date, the Linux version is really old... it's running on a dl360 g4 server. Since they also proposed to have a different master/slave configuration my idea was to use ansible to have all the stuff reproducible. Then packer seemed to be necessary, then started to look at some more complex configurations and found that in many cases you just create gigantic bash script that apply the configurations. So I should go to my wiki page with all the commands to create my server and transform them into a script that I should test several times. Ok if you have 100s of identical servers, but when you have just 4?

I used to use this automation pattern when I only had 2-3 of each type of server, and it paid off massively. Regular tasks that previously took a long time like OS upgrades, audit data collections and temporary dev access only took seconds or minutes, and ticket turnaround times dropped massively.

For 1. Hashicorp Packer

For 2. You don't even need compose. Just build a container image from a Dockerfile.

I use a mix of ssh'in in and running my services as docker images / docker-compose multi images.

My "deploy script" is simply a Github CI/CD action that builds the new docker images, pushes them to the GCR, and then logs into my VPS via SSH, pulls the images in, and restarts services. Works pretty good for small scale deployments and I get all the configuration and pinned versioning stuff into the images while not needing to go fullscale kubernetes which seems imho overkill for a single vps setup.

In the 'outside' real VPS the only thing I have, if those service need HTTP(s) ingress, is set up a simple ssl enabled webserver to reverse proxy shit back to the containers, and for that I use Caddy which auto handles generating ssl certs for your internet facing domains etc.

For home labs, NixOS is the easy recommendation if you are up for it. It really does just work once you understand it and get over the initial bootstrap of a single machine. I'd say it's easier than k8s and can also manage physical hosts.

Usually no. Sometimes, when you need to debug something, e.g network problem or run some tests - you do SSH, but solely for debugging purposes, and not for configuring / installing software.

For VMs you build image with everything baked in (think packer) or use cloud-init and do that on the fly on top base Linux image. For other apps serverless or sort of k8s solution.

Then Terraform or Pulumi to configure and push above to the cloud and maintain state.

The main selling points for Ansible and similar tools is that its automated, reproducible and can be audited via version control.

In a typical cloud deployment, you can have many servers and you need to quickly configure them and deploy software. Doing it manually through SSH may not scale and can be error prone. So, automation helps and as a bonus you get to leverage existing expertise.

While many of the ready made scripts do get the work done for standard tasks, I've seen lot of custom scripts depending on the environment and business needs. So, its a mix.

While I completely agree with everything you’ve said, I guess I spent too much time with Ansible and learned to love it. It’s still love/hate relationship but I know it very well and just accept the bad design as a given. One thing I learned the hard way - sometimes if you have to deal with complex data structures and data manipulation in Ansible it’s better to decouple the logic into a plug-in or a filter, even if it’s possible in theory to get done by Ansible. Also when you know how to efficiently use it, all the inventory tricks, the proper directory and file structure - you can build runbooks that are as close to being elegant as possible with this kind of software.

I mostly use cloud-init these days, to the point where I just went and built a shim (https://github.com/rcarmo/ground-init) for desktop setups.

But if you don’t nuke & pave your infrastructure, Ansible is much better than other desired state configuration tools (Puppet was infamous for locking us out of machines, Salt was full of quirks, PowerShell DSC is… well… just not something I’ve ever liked to use in any way, etc.)

I'm not sure what you are doing but I have never had these frustrations with Ansible. I absolutely love it. I think it's great for systems you never ssh into and monkey around and just as great for ones you do.

Source: been using Ansible for 8 years and SysEng'ing for 15.

I wish there was an Ansible thing but built in a real programming language, where we'd get all the type hinting and autocompletions stuff in without much fuss. Basically what Pulumi did for IaC (versus Terraform which is just like Ansible, textual hell)

I've thought about building something like this in Rust. Maybe something that compiles a statically linked binary which gets uploaded over the wire, and that is the thing that gets run in the final system to apply your new config state.

What would be a better alternative to Ansible?

I’ve found it quite good for a self documenting ssh/bash command delivery system. Most of my homelab is set up that way. Purposefully sticking to shell rather than the specific custom modules.

Anything requiring any sort of finesse especially on timing is yeah…rough as you say

The revisions are key. It makes the book worth buying.

Many technical books are out of date within a few months of publishing and get no revisions

The fact that the majority of his revenue goes to health insurance is abhorrent

Doctors make too much money. Pharmaceutical companies make too much money. Everything in the healthcare system charges too much.

A 5 minute, yearly "wellness visit" costs my insurance company $200 (or so they say) and that's not even including any bloodwork.

Agreed.

I was very surprised to read that even avoiding traditional publishers he's only receiving 30% of retail in, at least, one case.

I have done a tiny amount of self publishing via Amazon and I've found that if you want to keep your prices "sane" across all countries; you end up having a few that get you much less than others.

And if you enable "wide distribution" you get much less for those.

Yeah it goes from like 60% (after 'fees' for printing for the paperback) to like 40%. So selling my book for $19.99 nets me about $3 or $4 per sale. If I did 'wide distribution', that would be maybe $1-2.

It's strange because that's a normal publisher's take home, from which you also subtract a given amount for printing. But the author's take is usually around 10%, which should add up to 40%.

The insight into publisher royalties is interesting for sure. Though I will say regarding ansible, I just can't like it. I rather use a python library like netmiko for configuration of vm's.

It'd be a huge success to achieve this even for a traditionally published book with a publisher pushing. For a self-published one it's extremely impressive.

I bought a copy.

Thanks, Jeff, for being interminably helpful.

I bought this! Got immediate value in that I found simpler ways to accomplish my business goals. Thanks!

I bought this and it was very helpful!