Photons do not "diminish" in energy except via red shifting (where does the energy go? into the expansion of the universe!).

Light scatters, gets absorbed and re-emitted, it's true, but if distances don't change then the universe would still be much brighter than it is today.