As I understand it, reducing unsprung weight (which is not very much for a bicycle with no suspension beyond the tires) can have an outsized effect, but actually reducing weight mostly matters for ascending. For non-competitive cycling, other factors seem like they should be much more significant.
A big one, which is banned in most competitive formats, is a fairing. This is much more effective than having a human hunker down and try to be aerodynamic. Even for an upright cargo bike (which is generally extremely heavy), a fairing in front can make a dramatic difference on level ground with no wind.
Which is great if one is racing but not so much for long distance riding or riding for the joy of it. Most riders are not racing at any level where it makes a difference. A well made steel frame is a joy to ride. They are incredibly comfortable and track exceptionally well, unlike stiff frames.
carbon fiber is a lot less efficient regarding manufacturing though, compared to aluminum. The difference is even bigger with respect to steel or the life time of the materials
Clipless is also an incredibly significant change, but not as much of efficiency as the others.