Best and brightest at 8-15 usually do not end up best and brightest in later career. You can see it even in the fact that average salary of that cohort is only 80k/year - only slightly better than average for ALL college graduates. None of those geniuses are particularly impressive and most ended up doing the same work with the same quality as the rest of us.
There was research that showed that IQ after 120 stops being correlated with success - other factors start being more important.
Yeah, my supervisor for my undergrad thesis is a brilliant researcher who's still publishing good work 20+ years into his career. He was just a normal dude in undergrad who started in engineering but switched to math because he disliked project management, and he certainly wasn't in any sort of "young genius" program through grade school. But his PhD thesis was runner-up for best in the country, he consistently presents at all the major conferences in his field, and he has even won a few teaching awards.
It's not about what those individuals deserve, it's about the societal return on the investment of their education. The potential return is simply greater.