Do you do anything special for cooling the GPUs? I mounted a giant CPU heatsink to my gtx 1080, and it still has to turn on the fans when running ML workloads.
If you have something that produces ~300 W of heat and you want to keep it from becoming hot, you're going to need either fans or liquid cooling (which also has fans, on the radiator).