The issue is more that you are causing a calculated cache line to be fetched and then the timing difference between the L1 cache and L3 or MM fetch shows what the calculated value was. The only way to make those appear at the same speed is to slow L1 down on a speculative fetch to MM speed.

It isn't like a simple branch with two possible values. The cache line could be one of 256 (if scanning a byte at a time).