If the problem was out-of-viewport images impeding the receipt of assets that do affect the viewport (CSS, fonts, etc.) then it would be good to postpone fetching those images just until everything else is received, rather than postponing until they're closer to the viewport.
This was the exact problem lazy loading was supposed to solve