Every system had, has, or will have issues. That's why you have multiple layers of protection from cpu rings to user training. Sure, they had issues, but I'll still take hardened box, with ssh, with yubikey over ssh on its own. I don't trust any of those layers, but each provides more difficulty in exploitation.
(Also, this still doesn't provide the key itself. That means even with the vulnerable yubikey you can only spoof connections while it's plugged in)
It means that anyone who can swipe your Yubikey for a couple of minutes or even just wave an NFC-enabled phone past it can SSH to your server and then at their leisure convert that connection into a persistent backdoor. How often do you check your .authorized_keys? Your list of running processes? Your .bashrc for a line redirecting sudo to a small binary that logs your password for later use? Arguably this even made it less secure than a normal USB key with your SSH key on it encrypted with a strong passphrase.
(Also, this still doesn't provide the key itself. That means even with the vulnerable yubikey you can only spoof connections while it's plugged in)