The downside of Yubikey is that if it was manufactured before April and you used OpenPGP card mode, anyone who manages to steal your Yubikey could use it to log into your servers without needing to know your PIN/password because PIN authentication was completely broken and bypassable on the earlier Yubikeys. YubiCo also heavily downplayed the importance of this issue, claiming in their announcement that it didn't matter because anyone who got hold of your Yubikey could get hold of your PIN anyway. See https://developers.yubico.com/ykneo-openpgp/SecurityAdvisory...
Every system had, has, or will have issues. That's why you have multiple layers of protection from cpu rings to user training. Sure, they had issues, but I'll still take hardened box, with ssh, with yubikey over ssh on its own. I don't trust any of those layers, but each provides more difficulty in exploitation.
(Also, this still doesn't provide the key itself. That means even with the vulnerable yubikey you can only spoof connections while it's plugged in)
It means that anyone who can swipe your Yubikey for a couple of minutes or even just wave an NFC-enabled phone past it can SSH to your server and then at their leisure convert that connection into a persistent backdoor. How often do you check your .authorized_keys? Your list of running processes? Your .bashrc for a line redirecting sudo to a small binary that logs your password for later use? Arguably this even made it less secure than a normal USB key with your SSH key on it encrypted with a strong passphrase.
