In theory, I as the service provider know when my key database has been compromised. In theory. In practice, I will never know if a customer has been compromised, however up to a point a compromised user box can forward tokens to an attacker. So pending on whether you ever rotat the private keys, it’s a matter of ho long an attacker can retreat to a server they own to continue the attack.

In a way this reminds me a bit of SRP, which was an attempt to handle login without the server ever having your password. Which makes me think this is something to be integrated with password managers.