Most of the issues come from the lack of enthusiasm for moving the Nix store somewhere else than /nix. Even with read-only root, macOS has some designated locations where you can write. E.g. Homebrew uses /opt/homebrew, which is fine because /opt is writable.
I understand the reasoning of avoiding this on Intel Macs, since there there are years of cached derivations which would become useless without hacks. However, Apple Silicon Macs are a clean slate and the transition would've been a good occasion to move the store to /opt/nix.
(I did suggest this in some places, but there didn't seem to be much interest in switching over, unless I missed something.)
By the way, this isn't only an issue with macOS. Nix also doesn't work on Fedora Silverblue because it uses read-only root and the Nix store path violates FHS.
Yup, this. I think a big part of this is that Eelco is really only interested in NixOS. Nix is a large community and there are plenty of people that do care about other platforms, so these things do tend to get sorted out. Still, the core devs will choose to avoid short-term, medium-painful transitions for NixOS even at the expense of killing all the other platforms.
It's not that easy to change the default store dir.
https://cache.nixos.org/nix-cache-info has it hard-coded to /nix/store. If you want another one you'll need a whole new cache. But Hydra only works with one cache, so now you're deploying a second Hydra build farm.
One of the cool features of Nix is that you can evaluate some nix code on macOS even if the target build host is Linux. And then ship the .drv over to the build host. But that only works if both hosts share the same store dir.
So now you're looking at moving the whole community to use /opt/nix. And thinking of how to upgrade the existing users to it.
And fix all of the tooling we built that assumes /nix/store as the store dir.
So far nobody had the courage to tackle this huge task.
I guess this is a valid way to frame the problem (and I personally agreed with moving it), but I'd also quibble a bit...
- IIRC, ~stakeholders weren't keen on relocating it just on macOS because it would require a separate set of build/cache infrastructure (and it sounded like the macOS+Nix community would be on the hook for supporting it).
- There did actually seem to be a fair amount of support for moving it globally (and Eelco, while skeptical, didn't sound like he was going to stand in the way), but the coordination work sounded significant to me.
- Also, I think the circular arguments around relocating /nix played their own role in the inaction/bystanding that let the problem fester (as did, to be fair, fear/uncertainty about whether Apple would later ~secure whatever new location was chosen).
I understand the reasoning of avoiding this on Intel Macs, since there there are years of cached derivations which would become useless without hacks. However, Apple Silicon Macs are a clean slate and the transition would've been a good occasion to move the store to /opt/nix.
(I did suggest this in some places, but there didn't seem to be much interest in switching over, unless I missed something.)
By the way, this isn't only an issue with macOS. Nix also doesn't work on Fedora Silverblue because it uses read-only root and the Nix store path violates FHS.