The original assertion of "developers working the hardware better", and your rebuttal, are both incorrect. There was a literal constraint enforcing SMB1's design: ROM space. The developers of the first game had already had several years of experience on the platform, and they were straining to squeeze in the amount of content they had. It fits in 40kB. SMB3 uses 384kB, in comparison: 9.6 times bigger. Essentially, each world in SMB3 can contain 1.2 copies of SMB1.
What changed was not a matter of experience, but that cartridge game technology grew massively more powerful and changed the whole nature of the platform within the span of a few years. This was very much a case of "more is more." You could get a few of the ideas in SMB3 in the space of SMB1, but not all of them at once. And development costs rose sharply in the latter half of the 1980's for precisely this reason - you could feasibly make a "big" game that evoked a whole world with a variety of interactions, and doing so was a huge spectacle which necessitated a team dedicated to content creation, rather than a solo dev, yet was also largely independent of the graphics fidelity. Sierra Online built its name in this period by capitalizing on exactly this quality - presenting a world just a little more fully realized than the text adventures preceding it. It was a golden era for role-playing games, too - Ultima, Wizardry, Dragon Quest, and Final Fantasy all had big entries during this period.
The picture changes as you get into the 1990's. Most of the advances are pretty firmly on the side of "graphics treadmill." Games do keep getting more complex, but by the end of the decade, with stuff like Baldur's Gate, Jagged Alliance 2, Starcraft, Counter-Strike, or Deus Ex, you can find pretty much all of the design templates that AAA leans on. Design has mostly trended towards efficiency and simplification since then.
We've only seen the tip of the iceberg, in my opinion. AAA is in the process of a slow-motion collapse, despite its best efforts, because the nature of retail has changed to favor digital distribution, triggering an erosion of price points and upending the balance sheet. (no shelf space, no distributor lock-in, etc.) This was set in motion in the 2000's, but only became a life-or-death issue more recently. Console gaming hasn't had it this bad since 1984.
And indies have always had it tough, but they've experienced a sudden shift from a private market with the most prized audiences available only to the well-positioned and connected(e.g. get on XBLA in 2008, or Steam in 2011), to a market that is saturated everywhere, with few opportunities to stand out of the crowd. In theory, this means people should be more creative and adventurous. Anyone who intends to make a business of it is obligated to shrink scope, derisk and focus on getting a great promotional story together, though, and so it's hard to remain optimistic if your approach is more technically-inclined and favors design risks. I got to be in the thick of all of it as it happened and gradually concluded in the past year that I couldn't make it work full-time, not within the context of the projects I genuinely wanted to work on.
There is still room for AAA where it feeds into the F2P business. So MMOs will definitely stick around, and competitive games like League and DOTA. But that model doesn't favor the typical single-player campaign experience. All the paths to sustainable, growing success in gaming right now lead to something that involves F2P marketing tactics, community-building, platform ownership, or preferably a combination. Anything less than that faces a huge uphill battle to not simply get lost in the shuffle.
All of that said, the market is not dead by any means. The graphics race is on its way out - it'll continue to be advanced by dedicated engine devs, but the ability to render things is entirely a commodity now, borne witness by the frenzied price wars between Unreal and Unity. That, along with the shift to digital, has shaken the industry's dynamic into something way more unpredictable. It's great if you're enterprising enough, but most devs aren't, when it comes down to it.
What changed was not a matter of experience, but that cartridge game technology grew massively more powerful and changed the whole nature of the platform within the span of a few years. This was very much a case of "more is more." You could get a few of the ideas in SMB3 in the space of SMB1, but not all of them at once. And development costs rose sharply in the latter half of the 1980's for precisely this reason - you could feasibly make a "big" game that evoked a whole world with a variety of interactions, and doing so was a huge spectacle which necessitated a team dedicated to content creation, rather than a solo dev, yet was also largely independent of the graphics fidelity. Sierra Online built its name in this period by capitalizing on exactly this quality - presenting a world just a little more fully realized than the text adventures preceding it. It was a golden era for role-playing games, too - Ultima, Wizardry, Dragon Quest, and Final Fantasy all had big entries during this period.
The picture changes as you get into the 1990's. Most of the advances are pretty firmly on the side of "graphics treadmill." Games do keep getting more complex, but by the end of the decade, with stuff like Baldur's Gate, Jagged Alliance 2, Starcraft, Counter-Strike, or Deus Ex, you can find pretty much all of the design templates that AAA leans on. Design has mostly trended towards efficiency and simplification since then.