I've heard it suggested that language made the big difference. We bummed around for 998 million years, not able to improve our lot much - but somehow, 2 million years ago we developed language, and after that we were able to rapidly improve (and even co-evolve with our improvements - I've heard that the invention of the baby sling allowed our brains to grow larger than without it).