The bottleneck is often RAM. This is especially clear when writing performance-oriented code in CUDA, where the amount of cores (threads) per one shared memory controller is in the order of thousands.