That was my thinking too. With PageRank Google is trying to find a way to measure the perceived quality of the user experience for each page. Except there's no way to do that directly, so they look at indirect methods, like counting the number of inbound links. This is just another indirect measurement to add to (what I'm sure is) hundreds of other factors they use to rank a site.

IMHO a site that cares about the user experience would have been optimized to load quickly.

Heck, even if Google told people speed was important, and didn't incorporate any speed measurements into PageRank, I think it would have a positive effect.