Maybe this is due to their infinite caching? There was a article on HN a while back about how it impacted performance after a while[1]. Turns out checking a large cache of files (apparently with a poor choice of data structure backing the search) for visited links ends up being non-negligible a year into browsing.
I thought I saw that being debunked by a Chrome developer, I remember him saying that this was changed a while ago, and now the cache has a maximum limit. Clearing the cache doesn't seem to do anything for this problem, though (the page still reflows). It doesn't take more than a few ms, but it's annoying when other browsers do it instantly.
[1]: http://news.ycombinator.com/item?id=5281540 Unfortunately, the original article looks to have been removed.