I'm starting to think that TCP is past its prime and UDP based protocols (like QUIC) are the way forward.
The problem is exacerbated by TLS encryption. It requires six packets, or three round trip times, to establish a TLS over TCP connection. That is hundreds of milliseconds before a single byte of payload can be transferred.
The QUIC protocol combines the three way handshake and the three way key exchange to half the time I takes to set up a connection and provides a fast start (0rtt) for resuming a previously configured connection.
Additionally TCP connections do not survive a changed IP address like when changing from wifi to mobile data.
The stream model of TCP with head of the line blocking is conceptually easy but ultimately the wrong model for transmitting blobs of data. Rejecting a valid packet because the order got shuffled is wasteful, when we could just as easily reassemble the packets on the receiver side (even outside the sliding window).
There are a lot of times when the stream model is a good choice (dynamically generated data) but I would argue that sending sized blobs (like images) is a larger volume of traffic in terms of bytes transferred.
But the OP is right, writing custom UDP protocol is fraught with peril. DDoS amplification is just one way to screw up.
For multihoming, MPTCP is an interesting alternative and it's full of interesting features. Once it's more spread out it'd be interesting to know which features are really used.
I haven't seen SCTP mentionned in here so I'll just say that learning and using it, first in trying to find 'a better UDP' and then in anger to solve a multihoming/high-availability problem (which ended up in TCP + checkpoint/restore anyway...) was an eye opening experience on 'what could have been', especially having it work over UDP.
Now QUIC is there, gathering a big chunk of the SCTP featureset and adding proper encryption support (where IIRC SCTP only had TLS per stream).
WebRTC uses SCTP over UDP, so there are Free implementations out there you can grab if you want to give it a try. It's a shame that the Internet has ossified to the point that we'll never have any protocols riding directly on top of IP besides TCP, UDP, and ICMP, but tunneling SCTP over UDP doesn't add ridiculous overhead.
Protocols that aren't HTTPS (like SMTP) are still alive. The reason for no new protocols is that some network infrastructure will consider non-TCP/UDP/ICMP packets badly formed and reject them.
Lol. Good one. These protocols are completely independent of the content that you can send over them.
For the most part they reuse TLS so the encryption is the same (and uses the same certificate authorities) so any man-in-the-middle proxies, like those used by corporations, are technically able to intercept them to monitor content just like they can for https. (Though personally those sorts of things make me feel ill )
Good luck banning these protocols as well. They're perfectly valid IP packets so identifying them is going to be hard.
I don't think skipping the handshake is much of a benefit to QUIC. TCP was designed to have long-lived connections. Mobile networks are moving toward keeping your external IP consistent between handoffs, handling the routing internally. Fixed networks will connect to servers once every couple of minutes at most. QUIC mostly affects time-to-first-paint, which is mostly of interest for the slowest connections, which is nice, but becoming less and less relevant over time. It's not like speeding up the connectivity time is going to matter when all gains in connectivity time will be used for additional advertising and tracking against the end-user.
The main value propositions of QUIC for me are in the encryption earlier in the connection and in the reduced infrastructure usage on the server side.
> QUIC mostly affects time-to-first-paint, which is mostly of interest for the slowest connections, which is nice, but becoming less and less relevant over time.
You give no reasoning for it being less and less relevant. It is very relevant to me, where every connection to the US has 300ms RTT.
And assuming that's because you're on another continent, that problem is intractable because it's a consequence of the speed of light. You're never going to have a 10ms round trip between Los Angeles and Australia.
You really didn't understand the point they were making. 300ms rtt is much less bad when quic merges handshake and key exchange, requiring less round trips even before 0rtt session resumption
Of course it is, that's the point. You need to reduce the number of round trips because physics doesn't allow you to reduce the round trip time no matter how sophisticated your transistors.
I mean, yes, it is but many (not all) protocols built on top of it are generally still “connection-oriented”. QUIC, for example, still results in a “connection” it’s just implemented at the application layer instead of as part of the UDP protocol itself.
That still isn't even comparable. The routers along the path of a UDP packet may not even be tracking the flow (unlike in TCP), and keeping it up. A "connection" in TCP is a very big beast, with routers and computers tracking them (conntrack and friends) to keep them alive (or not) and ensure everything runs smoothly. There is none of that with UDP. So yeah, there might be an application-level concept of a connection, but that isn't guaranteed by a single bit of infrastructure on the internet.
If the equipment does connection tracking for TCP, then it most certainly does for UDP. Raw UDP does not have the notion of stream, but it most certainly does have flows. If the equipment does it for TCP, id does it for UDP.
However most routers don't do connection tracking at all. That is way too expensive at scale for zero benefits. Take your transit provider, it has tens of terabits of data per second, and those can be handled by a single router. That's a buttload of different stream for a single device, and connection tracking is expensive to do. I can say for experience that connection tracking tables for a single hypervisor with VMs can reach gigabytes. For a single node. Now extrapolate that to a backbone router. It's not happening.
Internet routers at large are not doing connection tracking.
If a device is dropping the ball on connection tracking, it's almost certainly in your own network.
Your ISP may have carrier-grade NAT. The content provider network probably has stateful load balancers. Those devices need to keep state, but they would need to do so with QUIC and UDP quasi-connections too.
Most (if not all) ISP routers (outside your business/house) employ connection tracking. This is used to monitor bandwidth usage, implement firewalls (aka, port 25/80/443), among other things.
Yeah, 100% agree. The main place that connection tracking is happening is whenever there's NAT involved and that's going to need to happen for both TCP and UDP.
> UDP based protocols (like QUIC) are the way forward.
Sure, but QUIC was written by experts who probably had been burned in the past by thinking they could just throw away TCP and profit. Things like slow start and congestion control are there for really good reasons and you can't just casually toss them away like anachronisms. Modern networks are really big, but ultimately things like switch buffers are still finite.
Rolling your own UDP based protocol is kind of like rolling your own cryptography.
Amazon in 2001 was using lots of UDP on top of Tibco Rendezvous when I got there and it absolutely melted the fuck down in Christmas of 2003. All of the FAANG-class companies have probably rediscovered this kind of lesson independently, maybe multiple times. I certainly remember chuckling at some Facebook press release in ~2007 when they were bragging about switching to UDP and wonder how that turned out.
TCP is overkill for the vast bulk of modern data transmission by volume; "short hops" within and between data centres on backbones that have near zero error.
TCP was designed for higher error rates on less reliable paths; from the clustered data centres out to the more remote home computers today is likely better than 1980s internet.
There are good proposals for low error high bandwidth TCP-lite replacements that are gaining ground.
I'm not sure I'd want to switch to a pure UDP world just because there's lots of bandwidth in the data center. Any situation where Moore's Law works against you will run into problems unless there's back-pressure built into the protocol -- IE whenever you've got 100 clients talking to one aggregation point you'll get kicked in the teeth, regardless of if the endpoints are 100mb with the aggregation point being bonded 1gb links or the common media is 100gb and the aggregation point is ECMPed 400gb -- there are more of them than there is of you...
And -- building back-pressure into protocols is not trivial.... Getting it "for free" with TCP has been a net gain.
Error rates are still high. You quickly find that if you have latency sensitive information that you need to ship over the public internet, like live video.
Also, you can trade off bandwidth for error rate at a lower level of the networking stack, so if you know your applications will use error- correcting protocols like TCP, you can make your switches and routers talk to each other a little faster.
TLS does have session IDs / session tickets, which I believe can speed up session resumption.
There is also TCP fast open, but I don't think it is used in practice. I don't think I've ever noticed it used when looking at a TCP capture.
Though I'm sure either TLS session IDs or tickets are.
I can't remember if one is more common that the other, it's been a while since I worked on that stuff directly.
>The QUIC protocol combines the three way handshake and the three way key exchange to half the time I takes to set up a connection and provides a fast start (0rtt) for resuming a previously configured connection.
UDP is okay and has some problems. QUIC is not the universal solution for them.
The lifetime of that QUIC setup will only be ~90 days. Because with every QUIC implementation in existence use of self-signed certs is not enabled by default and TLS-less connections are not allowed in the standard. This hold especially true for QUIC based HTTP/3 libs. So as soon as something in the TLS breaks or drops you your QUIC based service is inaccessible to 99% of people on Earth. That's fine, good even, for Microsoft or Google. But that kind of built in fragility, transience, and corporate centralization/dependence is really bad for hosting a personal website or service. QUIC is for corporate persons usage and it's design reflects that. It's terrible for human persons and makes no concession for human person use cases.
Because human person traffic is a tiny percentage of actual internet traffic.
Where is most internet traffic? To places like Netflix or other streaming services.
Then you have the top X sites that serve most of the rest of the traffic, a significant percentage of that to cell phones and all of it needing encrypted.
You'll just have to accept no realistically secure protocol will be universal. This last tiny wedge of traffic is either going to disappear in the noise of bots in automated attacks, or we'll have to find some reasonably secure way of letting it co-exist.
It's really not that hard to setup a domain with letsencrypt. If you use dns verification it's even almost trivially easy to use it for internal hosts as well.
It is till it isn't. When ACME2 is sunset and ACME3 comes along. The next time LE's root cert changes. The next time any part of the incredibly complexity hidden by and within acme clients breaks because of an update on your system. The time you host your abortion clinic site and get sued in Texas. Or your scientific journal article mirror gets attacked by big publishers, they won't go after you when they can just put pressure on LE to drop you and you become unvistable.
Putting all your chips in one basket, even such a benevolent and great one as LE, is a bad idea. Even dot .org was almost lost to private equity and went nasty. LE as it becomes more important becomes a juicier target for all forms of pressure and less useful to human persons.
protocols like quic and srt (used for video) are great, forward error/erasure correction is something I would also mention as a large part of the rise of UDP over TCP based transfer/protocols.
> Rejecting a valid packet because the order got shuffled is wasteful,
This is not what happens. The data isn't dropped as long as there are only a few reordered segments. And with SACK the receiver can also acknowledge them. It's just not provided to the application layer because something in-between is still missing. This adds some latency if you're multiplexing multiple things over a single stream but it's not a loss in throughput.
> The stream model of TCP with head of the line blocking is conceptually easy but ultimately the wrong model for transmitting blobs of data.
A problem of google's own creation.
> The problem is exacerbated by TLS encryption. It requires six packets, or three round trip times, to establish a TLS over TCP connection. That is hundreds of milliseconds before a single byte of payload can be transferred.
Arguably the problem got exacerbated by the NSA and user-hostile ISPs. If they didn't push the entire internet into a low-trust equilibrium we could stick to simple protocols instead of having to expend more resources and increase complexity to ultimately get to the same state of not-being-intercepted as we had been before.
A better response to a red queen's race is to dethrone the queen, not to build better running shoes.
But yes, yes... these sure are nice shoes.
The problem is exacerbated by TLS encryption. It requires six packets, or three round trip times, to establish a TLS over TCP connection. That is hundreds of milliseconds before a single byte of payload can be transferred.
The QUIC protocol combines the three way handshake and the three way key exchange to half the time I takes to set up a connection and provides a fast start (0rtt) for resuming a previously configured connection.
Additionally TCP connections do not survive a changed IP address like when changing from wifi to mobile data.
The stream model of TCP with head of the line blocking is conceptually easy but ultimately the wrong model for transmitting blobs of data. Rejecting a valid packet because the order got shuffled is wasteful, when we could just as easily reassemble the packets on the receiver side (even outside the sliding window).
There are a lot of times when the stream model is a good choice (dynamically generated data) but I would argue that sending sized blobs (like images) is a larger volume of traffic in terms of bytes transferred.
But the OP is right, writing custom UDP protocol is fraught with peril. DDoS amplification is just one way to screw up.