I had a very similar experience - dual-boot is nice but not ideal because restarting takes time and you lose all your context. Now im running a (headless) ubuntu server on the host with a windows VM with GPU passthrough on top. With a modern 6-8 core CPU + extra RAM (+ a cheap 2nd GPU for linux if you want a desktop environment on both linux and windows active at the same time) you can have both windows & linux running simultaneously with no need to dual-boot. VFIO & GPU passthrough makes the VM run pretty much at native speed although it does take some tweaking to set up.
That's an interesting idea: certainly worth experimenting with. As you say, with GPU passthrough it ought to be possible to get a near-native experience on the virtual desktop.
im using libvirt, which uses qemu, which uses KVM. There's plenty of good guides if you search for something like "VFIO VM GPU passthrough". Here is a good one:
