For almost every architecture out there the transition to 64-bits made existing code run more slowly for the reasons you describe, but on x86 the 64 bit transition also doubled the number of registers, guaranteed that SSE2 was available, and did some other stuff that actually made most code run faster.

However, I believe Google is working on some program to let people use all the goodies in x86-64 while also using 32-bit pointers.