There are (at least) two parameters that are relevant for the fabric, and latency is the critical one for HPC. Have you seen a materials science computation, of the sort that typically takes much of the time on HPC systems, running on 1GbE and SDR IB, even on three nodes? It sometimes happened on our system by mistake in the 2009 era.
HPL is simply not a good measure of the utility of a system for HPC, which generally accepted amongst HPC people, who deserve to be listened to. There's a good reason you don't see petascale DFT calculations on one of these "cloud provider"s which comprise so much of the top 500. (That might work on Azure, or something else with an Infiniband-ish fabric, but I've still not seen it.)
You can, and to some extent should, model network effects with systems like Simgrid or Dimemas.
HPL is simply not a good measure of the utility of a system for HPC, which generally accepted amongst HPC people, who deserve to be listened to. There's a good reason you don't see petascale DFT calculations on one of these "cloud provider"s which comprise so much of the top 500. (That might work on Azure, or something else with an Infiniband-ish fabric, but I've still not seen it.)
You can, and to some extent should, model network effects with systems like Simgrid or Dimemas.