I have a solution: drop the normal VM-based servers and instead use WebRTC or some other peer technology so the client (browsers or whatever) talk to eachother directly.

Unworkable because of security you say? Add encryption.

I actually think that not too many years down the line this type of thing will be common.