Don't need a cache? The larger your memory is, the greater the access latency will be, even if it's directly on the CPU die. That's why L1 and L2 caches tend to be around 32*2 and 256 KB, respectively. Most of the cache access time comes from the wire delays of sending signals around, and the larger the cache is, the longer the wire delays will be.

If you have a bunch of memory directly on the CPU, caching will still give significant speedups.