> ZFS is probably the biggest reason for me.

Maybe in the past there was an argument for that, but ever since FreeBSD started using OpenZFS implementation...what's the difference?

My ideal OS would be something like NixOS, but on FreeBSD and with better language than Nix.

ZFS is part of the OS on FreeBSD. Integrated more tightly in every respect.

ZFS is a first class part of FreeBSD. you can use it on linux, but it will always have some rough edges. How rough it open to question though, for some it works well.

Used ZFS on Debian in production for 8 years, yet to experience rough edges but always interested to learn.

Do you use ZFS for root, on Debian? (enabling "boot environments")

I've recently switched my FreeBSD setups to use that scheme, and it's been nice. Would be interested to hear if it's similarly straightforward on Debian (my second-favorite OS :)

Obviously requires support in the bootcode; I'm not sure of the state of that for Linux.

No, to be fair I use it for everything except root (which I prefer to backup as IaC, not that I wouldn't use ZFS, it's just not default so requires more effort). I use ZFS datasets to back everything except the OS, including DBs which has been the most satisfying aspect of using ZFS (being able to snapshot live DBs together with all other relevant state rather than a as a separate replication process).

While I can't comment on using it for root, I can say for other use, the only annoyance I have is compiling the kernel module on every single update. This is automatically handled by APT, but for very small servers it can be slow... at most ~15 mins. Could be solved with private distribution for a large fleet but I can't be bother with that stuff. Hoping at some point Debian will relax their strict "to the letter of the GPL" attitude at some point like they did with install media drivers. But it's not the worst experience, at least installing is automated.

zfsbootmenu https://docs.zfsbootmenu.org/en/v3.0.x/

Also nice way to recover zfs if anything goes wrong. yea, it's a linux image for just booting. But you put it as an EFI image, and works great.

I've been running zfs on root on my Debian home server, only the install was a tiny bit more involved but it was done in maybe half an hour going off of the guide and trying to understand everything it was doing with 0 experience (... And then did it again after I broke networking after an hour lol)

Well, rough edges:

- bootloader (when root is on ZFS)*

- ARC vs kernel page cache and OOM Killer

That's all in my opinion.

*: Relative to FreeBSD, but that's because bootloader in FreeBSD is part of the base system and in linux it could be one of many.

I have never had a good experience with out of tree kernel modules on linux, so I am skeptical of this report. Usually for Linux Debian is my distro of choice.

I have little experience with kernel modules. ZFS is the only one I have long term experience with on Debian.

The only issue I've had with the kernel module aspect is VPS configuration, I use Linode and they used to automatically set everything to use the hypervisor kernel, but the kernel module is obviously built against the versions installed on your OS and these will probably be different.

All you have to do is make sure the boot configuration is set to use GRUB if using a VPS. Linode seem to have switched to this by default for Debian now.

My experience with ZFS (over 10 years ago now) was awful. Still have a bad taste in my mouth though no doubt it's much improved today.

Sounds like maybe you were holding it wrong? This is year 20 for ZFS in production.

Though Linux support a decade ago with ZFS was pretty rough around the edges. That's not true anymore for many distros of Linux.

I didn't run it. But several times a year there were multi-day outages for "rebuilds" and at one point Oracle consultants were involved to get it back on its feet.

If that's Oracle's ZFS then it's not really comparable, it's essentially a very old closed fork. So on top of using a languishing outdated version you have another reliability problem (Oracle). It's a little bit like comparing OracleDB to MariaDB.

Most people use Open ZFS which is from the Illumos project, which was basically the escape hatch that the engineers who wrote ZFS used when Oracle tried to close source Solaris after the Sun acquisition. There are decades of improvements in all of the OSS versions that comprise Illumos (which Oracle has denied themselves by attempting to close source it, since they cannot feed off of downstream OSS code). i.e most of the people who wrote ZFS immediately left Oracle and worked on Open ZFS.

Open ZFS is for both FreeBSD and Linux, and is what most people are referring to when discussing ZFS. I've never used Oracle ZFS and never will.

Agreed on the larger picture. Oracle ZFS is calcified ZFS. Their last stable release it looks like was in January of 2023.

OpenZFS released 2.3.1 a few weeks ago: https://github.com/openzfs/zfs/releases It has direct support for FreeBSD and Linux.

macOS, Windows and other ports work great, but are not (yet) upstream: https://openzfs.org/wiki/Distributions

Well since FreeBSD is pulling ZFS source from Linux, I am not certain what are the rough edges. And I have ran ZFS for 7+ years on Linux with zero issues.

ZFS is not part of linux. It is a separate project that publishes both to FreeBSD and some shims so you can run it on Linux. FreeBSD pulls and directly integrates it, but for license reasons linux doesn't pull it in directly. Yes the upstream project does have some shims of linux use that they public directly, but FreeBSD is a first class user of that project while the linux shims are a hack that some in the kernel do not want to work.

> My ideal OS would be something like NixOS, but on FreeBSD and with better language than Nix.

Something nicer than nix would be nice, but otherwise perhaps keep an eye on https://github.com/nixos-bsd/nixbsd

Yeah, it's acceptable, I want to try to convert my home server to nixbsd.

My ideal OS (for a server) is Guix, NixOS without Nix-the-language. Sadly no one is packaging for it.

Ubuntu can have zfs with an install of a package.

BSD’s inability to utilize docker ecosystem had me decided to stick with Ubuntu for a decade, unless things change and BSD gets clear advantages over Linux.

Here's where I'll show my age. If you got into a unix-like OS in the '90s, the *BSD ergonomics will make more sense.

I remember when Ubuntu and docker each entered the scene and my initial impressions of both were pretty negative.

Ubuntu can have ZFS. FreeBSD just does, as long as you select "ZFS on root" for your disk layout when you install. It just works, and then you automatically get things like snapshots and easy rollback during upgrades.

Ubuntu ships an outdated version that misses a lot of bugfixes. They don't seem to be interested in either backporting fixes or shipping the current stable version. Every other day some Ubuntu user hits the ZFS GitHub issues with a problem that was fixed years ago.

After hitting this in production, I will never choose Ubuntu again.

On top of the things you mentioned, it’s basically impossible to produce a working playbook that switches from Ubuntu’s garbage package to the upstream ones - systemd wedges badly about 25% of the time, and force disables the zfs units. None of the documented systemd overrides worked.

I’m enjoying devuan and openbsd at home.

Got sick of doing ZFS on Arch Linux and lagging behind on new kernel releases. Switched to bcachefs (only for my laptop) which has the trifecta of data and metadata checksumming, compression, and encryption. It's also just simpler to use.

when do you use samba and when do you use nfsd? any iscsi?

I mainly use NFS. SMB I typically use from Windows or Android. (The latter with the app "andsmb")

iSCSI is block layer so I don't find it very useful.