"* OP double-protects the SSH key. It means you need the key's passphrase and another factor (Google authenticator) to decrypt the ssh key. Then the ssh key is used to auth with the server. => the authentication with the server is still one factor auth, compromising the key at any level still grants access."

This is not correct. You can't decrypt a key with a one time password.

The OP is requiring a the second factor(the OTP) after the key is sent to the server and authenticated.