OK, I don't use VirtualBox, for performance reasons I use VMWare Fusion. Even with this, I find, as you, that using GUI on the VM slows the Mac much more than ssh'ing to the VM. So, the VM is a serious culprit. Certainly, if I have the VM running, I expect some slowdown and occasional swapping -- basically it means I just gave up 1 GB or memory to the VM (out of 2) plus tons of I/O scheduled by a different OS within my OS.
Other than that, I also don't run Time Machine at all -- I use Carbon Copy Cloner or Super Duper to make incremental and/or bootable backups every week or so. With iCloud, Dropbox etc for the important day to day documents, I don't see much need for Time Machine anymore.
So those would be two places to look at.
Besides those, you can try running some DTrace scripts for a more detailed look when your system starts to crap out.
Other than that, I also don't run Time Machine at all -- I use Carbon Copy Cloner or Super Duper to make incremental and/or bootable backups every week or so. With iCloud, Dropbox etc for the important day to day documents, I don't see much need for Time Machine anymore.
So those would be two places to look at.
Besides those, you can try running some DTrace scripts for a more detailed look when your system starts to crap out.