I use both lxc and docker and have uses cases for both, I think it really comes down to how stateful something is or how lazy I feel about writing a Dockerfile. I had a really hard time with learning lxd and really only got into using lxc without the daemon.
One trend with docker that I personally don't like is that a lot of projects prefer docker-compose over regular Dockerfiles (though some of them support both), and this leads to a lot of bloat with how many containers it takes to run one application and duplication where each app will need its own database, redis, webserver, etc. containers.
That's not a problem when you are at the scale to need that kind of orchestration and have the resources to run that many containers, but personally I would rather have one database host that all my containers can talk to and one reverse proxy/webserver to make them accessible to make better use of resources and get better density on one host.
One downside with lxc is that there's been some fragmentation with the image creation side of it, I had been used to using lxc-templates for years, which are just shell scripts for bootstrapping and configuring your container and pretty common between distros. I found a bug with creating containers for the latest version of Alpine that I fixed, and only then did I find out that lxc-templates are deprecated and essentially unmaintained, and that now distrobuilder is the preferred way to build lxc containers, at least per Canonical. That seems to be another Canonical-ism, since the distrobuilder docs say that snap is the only way to install it unless you build from source, so that's a huge no from me, and I also didn't feel like learning the yaml config for it.
I was actually considering moving my docker daemon into an unprivileged lxc container like the article mentions, but haven't gotten around to it.
> " [..] which are just shell scripts for bootstrapping and configuring your container and pretty common between distros. "
I find it a big advantage that this builds the containers basically from scratch and you only have to trust the distro and not any other parties. I'm always feeling uneasy with images which are hard to inspect and whose provenance is opaque and potentially involved multiple parties.
That, combined with running them unprivileged, should make a fairly secure system. Unfortunately I had great difficulty to create an unprivileged Debian LXC container with this method. If I remember correctly you have to create the privileged container first and then use a tool to fix-up uid and gid. If anyone knows an easier way to do it, I would be grateful to know.
EDIT:
I think I used the following to create the container:
I don't recall having to do any uid/gid fixup last time I made an unprivileged container. I did have to prepare the unprivileged host account, of course, by setting its subordinate uids/gids (/etc/sub?id) and virtual network interface limit (/etc/lxc/lxc-usernet).
Note that this runs the 'download' template, which (IIRC) is better suited to unprivileged containers than the 'debian' template is. The 'download' template will list its available distros if you do this:
lxc-create -t download -n <name> -- --list
Note that some versions of the 'download' template may fail with a keyserver error because sks-keyservers.net died somewhat recently. Workaround: DOWNLOAD_KEYSERVER=hkp://keyserver.ubuntu.com lxc-create ...
I use Docker in unprivileged LXC* for the mere benefit of separation of concerns [1]. I don't use VMs for this because I am limited memory wise, and LXC allow me to share all host resources. I use docker because I don't want to mess with the way authors expect their dependencies to be set up - which is easy to circumvent by settling on a docker-compose.yml. Nix [2] appears to be the best future successor for my setup, but it is not used as widely yet.
* Afaik, this only works on Proxmox, not LXC in bare Debian, because Proxmox uses a modified Debian Kernel with some things taken from Ubuntu.
One trend with docker that I personally don't like is that a lot of projects prefer docker-compose over regular Dockerfiles (though some of them support both), and this leads to a lot of bloat with how many containers it takes to run one application and duplication where each app will need its own database, redis, webserver, etc. containers.
That's not a problem when you are at the scale to need that kind of orchestration and have the resources to run that many containers, but personally I would rather have one database host that all my containers can talk to and one reverse proxy/webserver to make them accessible to make better use of resources and get better density on one host.
One downside with lxc is that there's been some fragmentation with the image creation side of it, I had been used to using lxc-templates for years, which are just shell scripts for bootstrapping and configuring your container and pretty common between distros. I found a bug with creating containers for the latest version of Alpine that I fixed, and only then did I find out that lxc-templates are deprecated and essentially unmaintained, and that now distrobuilder is the preferred way to build lxc containers, at least per Canonical. That seems to be another Canonical-ism, since the distrobuilder docs say that snap is the only way to install it unless you build from source, so that's a huge no from me, and I also didn't feel like learning the yaml config for it.
I was actually considering moving my docker daemon into an unprivileged lxc container like the article mentions, but haven't gotten around to it.