Since we launched the Cedar stack, we've used AWS ELBs as the front layer in our routing stack. Since we had only ever allowed regular, short-lived HTTP requests through our stack, we opted to use these in HTTP(S) listener mode [1]. When used in HTTP(S) mode, ELBs have historically been very strict about what traffic they allow through.
As the WebSocket standard is very recent, it has never been supported by ELBs in HTTP mode.
ELBs in TCP mode can support any TCP traffic. It's become clear that we need this flexibility, so we're moving to TCP-mode ELBs now. This was not trivial, though, as long-lived connections (like those used for WebSockets) have different implications for our HTTP routers. That had to be taken account.
Nonetheless, we've had a private beta for a long time that worked as described above. But we deemed it insufficient for general customer use because TCP-mode ELBs mean that you lose the client's source IP and protocol. Fortunately, ELBs now have Proxy Protocol support [2], which allows us to keep that request information that Heroku apps typically rely on.
How did you solved ELB 60s disconnections for idle Websocket connections?
I know it can be increased, but not by too much.
EDIT: also when 2 years ago I did load testing with ELB and created 1 million concurrent WebSocket connections, I did received mail from Amazon asking what the hell I doing and was asked to stop it ;)
> As the WebSocket standard is very recent, it has never been supported by ELBs in HTTP mode.
WebSocket has been around since 2010. This is just confirmation to me that Heroku isn't a good fit for apps targeting new web technologies. It's a good fit for apps targeting IE8.
