There appear to be slightly weird commercial reasons behind this, because gaming GPUs have great CUDA performance but NVIDIA won’t let you put them in a datacentre. So buying your data scientists gaming laptops (RGB and all) generally works out faster for any reasonable price point. That said, a dedicated server with a decent Xeon and MKL set up correctly generally outperforms CPU-bound stuff.
I think it really depends on your data size. All the benchmarks I can find are on massive datasets, with tens of millions of rows or thousands of columns. I’m sure there are significant performance gains in these situations. Our data just wasn’t big enough.
With larger data it really depends on the algorithm. If you must iterate over more than a few GB at a time, GPU memory capacity and bus speeds become prohibitive, while a dead-simple implementation on a single CPU with 100+ cores and TBs of RAM goes brrr.
That would only be an advantage if you had to do multiple passes over the data, otherwise the data would still go through the CPU RAM before getting loaded onto the GPU, no?
When the models get sufficiently big, even a 40GB A100 is not sufficient. Unless you can feed the core quick enough, your performance drops considerably.
GPUs are like heavy flywheels. Getting them up to speed takes some time (copy data, compile and copy the kernels, kickstart everything, etc.), so you need to start them once to get the performance benefits. Otherwise CPU is much more nimble since they're closer to RAM and made to juggle things around.
> The updated end-user license agreement (EULA) states: “No Datacenter Deployment. The software is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.” [0]
I guess it's time to invent a blockchain that trains ML models as PoW :)
As a sidenote, I have rented servers with GeForce cards from multiple providers in multiple countries, so this rule doesn't seem to be respected very much. And since it's part of the driver EULA, nvidia can't legally go after server providers, since they don't install any drivers, just build and rent out the hardware. For all they know, all their customers are running noveau.
