For certain definitions of the word “fun,” yes. I have a 35U (I don’t need that many slots, but at the time I did need it tall enough that my kids couldn’t reach the top, where I put the keys), with:
* 3x Dell R620
* 2x Supermicro (one X9, one X11)
* 1x APC UPS w/ external battery
* Unifi UDM Pro
* Unifi Enterprise 24-port switch
The Dells have Samsung PM863 NVMe drives which are used by Ceph (managed by Proxmox), with traffic sent over an Infiniband mesh network via Mellanox ConnectX3-Pro.
The Dells run K3OS in a VM, which is a dead project. Big mistake there.
The Supermicros have various spinners, and are in a ZFS pool. One of them is technically a backup that should power up daily to ingest snapshots, then power off, but there’s been some issue preventing that, so…
It was all very fun to set up, and has been eminently reliable, but it’s a bit much. While you can in fact make R620s relatively quiet, they’re still 1U, and those little 40mm fans are gonna whine. It’s background noise to me, but guests definitely mention it if we’re close to the rack.
Also, I’m now in the uncomfortable position of being stuck on Proxmox 7, because v8 (or more accurately, the underlying Debian release) dropped support for my HBAs, so the NAS would be dead in the water. I mean, I could compile my own kernel, or perhaps leverage DKMS, but that somewhat defeats the purpose of having a nice AIO like Proxmox. Similarly, my choose of K3OS means at some point I need to spend the time to rip everything out and start over with Talos.
Or - just maybe - I’ve done enough playing, and I should simply buy a JBOD chassis and a relatively new and quiet server (4U under light load means you can get away with much quieter fans), and just run stuff in Docker or gasp systemd. Or, hell, single-node K8s. My point is that it is fun, but eventually your day job becomes exhausting and you tire of troubleshooting all the various jank accumulating at home, and you stop caring about most of it.
I turned my supermicro 847 X10DRI+ into a JBOD by ripping out the mobo, the fans, and building a 140mm fan wall (zip ties and later 3D printed). I can highly recommend the move, it's dead silent and I have drive temps between 33 and 45C with 20 to 25 drives (I honestly don't know the total at this point).
As for your OS issues I also used to run Proxmox with Ceph (btw did you know you can use Proxmox's ceph with rook-ceph so you don't need 2 layers of storage?) including for the router and NAS, but I gave it up due to unwarranted complexity and went bare metal.
Don't know what would fit your particular use case but I can say this: I'm very happy I made a separate box with a supermicro X11 and a JBOD besides it, I can recommend this too; what benefit is there really to virualizing a NAS?
Regarding K3OS you're in luck - kubernetes manifests (you've got GitOps, right?!) are so portable you can just rebuild on a new OS. Give Talos Linux a spin. Again really think about why is it that you're virtualizing here too; maybe you genuinely need it, maybe not.
Not sure I follow here; why would I want Rook involved? I generally don’t want my orchestration layer - which is also consuming storage - to be involved with the management of said storage.
> Virtualizing vs. bare-metal
It’s partially to make upgrades via new base images easier. I bake images with Packer + Ansible, and so can push a new one out quite easily. The other part is that my NAS consumes very little compute resources, since it isn’t hosting any apps, so it would be a waste to run it on bare metal. Tbf I could run everything consuming the disk storage directly on it, but I had shied away from that initially and it stuck.
> GitOps
I have Helm templates, no ArgoCD. It’s been a TODO for me for quite some time. Same with Talos (I actually do have it running in parallel right now, just not hosting anything). My issue is that I am obsessed with getting things perfect, and for me that means bootstrapping from a bare VM to ArgoCD ingesting manifests and spinning up pods, all automated. I know this is possible, as I’ve seen it done, but I rarely have the time or energy to pursue it after work. I should probably get over myself and just manually install stuff so it’s functional.
For certain definitions of the word “fun,” yes. I have a 35U (I don’t need that many slots, but at the time I did need it tall enough that my kids couldn’t reach the top, where I put the keys), with:
* 3x Dell R620
* 2x Supermicro (one X9, one X11)
* 1x APC UPS w/ external battery
* Unifi UDM Pro
* Unifi Enterprise 24-port switch
The Dells have Samsung PM863 NVMe drives which are used by Ceph (managed by Proxmox), with traffic sent over an Infiniband mesh network via Mellanox ConnectX3-Pro.
The Dells run K3OS in a VM, which is a dead project. Big mistake there.
The Supermicros have various spinners, and are in a ZFS pool. One of them is technically a backup that should power up daily to ingest snapshots, then power off, but there’s been some issue preventing that, so…
It was all very fun to set up, and has been eminently reliable, but it’s a bit much. While you can in fact make R620s relatively quiet, they’re still 1U, and those little 40mm fans are gonna whine. It’s background noise to me, but guests definitely mention it if we’re close to the rack.
Also, I’m now in the uncomfortable position of being stuck on Proxmox 7, because v8 (or more accurately, the underlying Debian release) dropped support for my HBAs, so the NAS would be dead in the water. I mean, I could compile my own kernel, or perhaps leverage DKMS, but that somewhat defeats the purpose of having a nice AIO like Proxmox. Similarly, my choose of K3OS means at some point I need to spend the time to rip everything out and start over with Talos.
Or - just maybe - I’ve done enough playing, and I should simply buy a JBOD chassis and a relatively new and quiet server (4U under light load means you can get away with much quieter fans), and just run stuff in Docker or gasp systemd. Or, hell, single-node K8s. My point is that it is fun, but eventually your day job becomes exhausting and you tire of troubleshooting all the various jank accumulating at home, and you stop caring about most of it.