Direct physical access to your computer is not the only thread. Even if you have in your computer all the time it gives you better logins, easier and safer. Specially protection from phishing.

When I leave my computer out of my eyes it usually locked (unlock with password), so I don't expect somebody to quickly go in and do something. If I leave for longer, I sometimes pull the Nano out and put it somewhere I don't lose it for a while.

The advantage of your system is that you can use the U2F screen unlock instead of the password.