This is true, but it's easy to look back at historical data and draw conclusions from perfect information. When actually making an investment decision facing an unknown future, index funds are relatively low risk. This is why people actually buy them. Making the pool of companies smaller will only increase the volatility.

I would argue that psychologicaly the S&P offers the least optimistic promise of return for an investment a person would realistically make, since it is supposed to represent the market as a whole. People don't like to make an investment if they believe its quality is below-average, so expectation of average returns is really the minimum.