> I have never knew a single person put ssh host key into HSM.
You probably also never met a single person where the SSH interface sees millions of sessions as day with valuable assets (code) being transported over said sessions.
> In fact, this is not a supported option in openssh.
This definitely is supported. Though documentation for this is often HSM vendor specific, which if heavily NDA'd. So that's why you probably haven't found much information about it.
What I expect has happened here is that you've remembered that your HSM comes with instructions for how to use PKCS11 to make user authentication rely on the HSM and you've just assumed that's relevant here. While I'm sure the vendors make it seem like this is all very secret, it's just a pretty boring C library and it's probably half-arsed in real world implementations.
AIUI OpenSSH does not provide any way to use PKCS11 to protect host keys, which are the concern here.
You can use PKCS11 to sign OpenSSH certificates, so if GitHub had elected to use certificates here, it could have protected the CA keys for those certificates in an HSM, but it did not.
Correction: It was pointed out elsewhere that you can just tell sshd to use PKCS11 keys via the SSH agent mechanism, and so yes that would allow use of an HSM for host keys
Yes, but that would either be a fork of OpenSSH, private or open source (both are possible since it's BSD-licensed), or a different SSH server (which Github is of course free to use, since the protocol is standardized and their scale absolutely justifies any efforts in protecting their SSH host key). But GPs comment was about OpenSSH.
Edit: Apparently OpenSSH's sshd also supports the SSH agent protocol for host keys, and ssh-agent does support PKCS#11 – so I stand corrected!
> How can openssh documentation be vendor-specific?
I isn't, because the cryptography is (in case of HSM) not handled by OpenSSH itself. So OpenSSH's configuration has nothing to do with the HSM.
Usually, the actual cryptographic functions are not performed user-space, but handled by the kernel, which in turn can offload this to dedicated hardware. Basically if you compile OpenSSH for it to use kernel level cryptographic function, then OpenSSH can work with a HSM without it even knowing it.
Disclaimer: this is simplified explanation, there is a lot more to this, and I am by no means an expert on this matter.
Edit: meant to say kernel level cryptographic functions, not TLS.
So you‘re saying that OpenSSH has an interface for that on the host key side?
I‘m aware of the PKCS#11 integration in the OpenSSH client and have dabbled a bit with it but was not aware of any server side equivalent.
And how does TLS fit in here? SSH is a very different protocol from that, no?
Update: I can't find any OpenSSH documentation of either (server-side) PCKS#11/HSM support or kernel-mode cryptography (which also in the case of Linux only addresses symmetric encryption to my knowledge, at least the mainline kernel version I know of).
Maybe you're thinking of some other SSH implementation? The protocol definitely allows for server-side HSM usage, and Github at their scale is not bound to OpenSSH in any way.
> I can't find any OpenSSH documentation of either (server-side) PCKS#11/HSM support
OpenSSHd talks to an ssh-agent that then talks to the HSM:
> Identifies the UNIX-domain socket used to communicate with an agent that has access to the private host keys. If the string "SSH_AUTH_SOCK" is specified, the location of the socket will be read from the SSH_AUTH_SOCK environment variable.
You probably also never met a single person where the SSH interface sees millions of sessions as day with valuable assets (code) being transported over said sessions.
> In fact, this is not a supported option in openssh.
This definitely is supported. Though documentation for this is often HSM vendor specific, which if heavily NDA'd. So that's why you probably haven't found much information about it.