You can generate custom SIMD inference code for your neural net (tensor shapes, etc.) and CPU cache hierarchy (L1, L2 per thread, etc.)

For example, here is ResNet50 for a particular Skylake-X CPU:

https://nn-512.com/browse/ResNet50

I didn't know this exists, thank you!

You usually tune the software that runs on it (compiler, libraries, build options etc.). You can also play the game of overclock if you run dedicated hardware.

I wondered what those options are. The other comment mention SIMD is interesting. Thanks :)