Assuming your local agent silently signs all requests. This attack doesn't work if e.g. your Yubikey is configured to require a physical tap for every authentication.