I used to agree with you until I saw this presentation by Jim Keller, one of the biggest names in chip architecture. It really illustrates how many more options we have to improve performance.
I like that one. Building a chip involves many different manufacturing steps, each of which has room for improvement. I liked how he even mentions that advances in sandpaper allow us to make better chips.
https://www.youtube.com/live/oIG9ztQw2Gc?si=vCBGGm0tkhG-VOyd