I am not aware that windows is rendering any pixel while no active desktop session is launched. You have less binaries and probably a smaller memory footprint without desktop but I would be surprised if the performance difference was material.