Except you can put a wallet on a thumb drive, and literally put it in a bank (safe deposit box).

Exactly, make sure the only copies of the wallets with your money are on non-networked, non-running hardware, preferably just straight up storage media, and secure it physically. You can make it as secure as you can possibly make anything. Hell, you could even get it insured.

Though interestingly, your bank isn't generally required to repay you for losses in your safe deposit box unless it arises from their negligence.

Probably makes sense to print it out on a sheet of paper and put it in there too.

What do you call exchanges that hold an on-site balance?

I'm also fairly sure there are several brands trying to provide a "bitcoin banking service".

not decentralized?