This increases your attack surface area. Any breach to one of those machines and the attacker can start doing mitm attacks. It also limits auto scalability assuming newly provisioned machines require manual approval of priv key distribution (that stays in memory) via hsm, and the same goes if the process dies. One way to limit the key distribution is to embed the routing information you require in the SNI at a second lb layer that's shielded from public traffic. This way your public machines don't hold any keys and if they get compromised, limiting the damage.

I agree it's a general problem. But sometimes certain architectures would require more vulnerable approaches vs others.