The PiKVM, mentioned about halfway through, is an absolutely extraordinary product. It's probably the only time I've ever thought "I used this Pi because it is above and beyond the best tool for the job not just because I had it on hand".
On the hardware usage side from the GPIOs over RJ45 physically to control the power/reset buttons to the USB OTG NIC/virtual disc drive/virtual USB drive/virtual serial/virtual mouse/virtual keyboard to the use of the Camera Serial Interface to the OLED status display over SPI to the exposure of the local serial as RJ45 for quick access to the use of the GPU for video encode every piece is just exactly what it needs to be not there by accident. It's built well, laid out well, and any monkey can figure out how to assemble it.
On the software side it's just prepackaged Arch Linux (which mounts in ro by default because why wouldn't it) you can actually upgrade or add packages to your liking unlike most remote access garbage with a clean and fast HTML5 interface that knocks any proprietary IPMI web interface out of the water. It even has all those macro/text pasting/shortcut related things the way you actually want them instead of "almost right". When you use the virtual drive stuff because it's not an embedded piece of crap you don't have to worry about slowly streaming bits of the OS ISO over the network as it's read by the device you just copy the whole image up and can leave as many as you want there for later.
Some things it has prebuilt software support I haven't personally taken advantage of are tailscale support, ipmi/redfish integration, multipoint hardware KVM control via GPIO, and 3d case models.
If I had to pick one thing that wasn't perfect about it it'd be the CSI limits it to 1080p50 bandwidth. Not that it really matters for KVM usage it's just not perfect.
Anyways long winded comment aside if you read this and think "that sounds like a really nifty setup/idea but more work than I'd care to put in for it" check out if you can get away with the prebuilt PiKVM.
TIL that there are adapter chips from HDMI to CSI, never thought that to be possible... might be worth a try for a project I have in mind. Thanks for pointing this out!
I had to do something similar for building out an automated hardware test cluster of ARM devices for OS testing. We settled on a bunch of varied SBCs (we were less concerned with peripherals than we were different SoC features/configs). Every SBC we found supported ARM CoreSight over some external pins (and, for better or for worse, had no way of disabling it) and so we were able to just wire them up to a test host so that it could supervise and capture detailed panics even when we had very early "dark" panics. It's a really neat setup and it's been great for productivity too since, as mentioned in the post, you can use the test cluster as remote dev hardware and test on real silicon from your desk without needing to lug around test boards, a bundle of debug cables, and wasting all your laptop's ports just to boot hardware.
This is cool and all, but I can't shake the feeling that it's a workaround for a certain kind of OS design where showstopping glitches are a fact of life. If OSs had proper fallback drivers (I.E., if the nvidia GPU driver crashes repeatedly, fall back to framebuffer), then this sort of testing wouldn't really work, because the OS would appear to keep working despite being in a failure condition.
On the hardware usage side from the GPIOs over RJ45 physically to control the power/reset buttons to the USB OTG NIC/virtual disc drive/virtual USB drive/virtual serial/virtual mouse/virtual keyboard to the use of the Camera Serial Interface to the OLED status display over SPI to the exposure of the local serial as RJ45 for quick access to the use of the GPU for video encode every piece is just exactly what it needs to be not there by accident. It's built well, laid out well, and any monkey can figure out how to assemble it.
On the software side it's just prepackaged Arch Linux (which mounts in ro by default because why wouldn't it) you can actually upgrade or add packages to your liking unlike most remote access garbage with a clean and fast HTML5 interface that knocks any proprietary IPMI web interface out of the water. It even has all those macro/text pasting/shortcut related things the way you actually want them instead of "almost right". When you use the virtual drive stuff because it's not an embedded piece of crap you don't have to worry about slowly streaming bits of the OS ISO over the network as it's read by the device you just copy the whole image up and can leave as many as you want there for later.
Some things it has prebuilt software support I haven't personally taken advantage of are tailscale support, ipmi/redfish integration, multipoint hardware KVM control via GPIO, and 3d case models.
If I had to pick one thing that wasn't perfect about it it'd be the CSI limits it to 1080p50 bandwidth. Not that it really matters for KVM usage it's just not perfect.
Anyways long winded comment aside if you read this and think "that sounds like a really nifty setup/idea but more work than I'd care to put in for it" check out if you can get away with the prebuilt PiKVM.