I’ve been super curious about how modern SMTP handles encryption - my understanding is SMTPS was dropped in favour of STARTTLS, with a HSTS-like directive hosted in DNS.
Does the latter actually happen? Do smtp clients actively check this? What’s to stop some middleware performing a tls downgrade?
I think you are talking about MTA-STS (rfc8461) [0]. It is currently being adopted by the larger email services (Google, Microsoft).
MTA-STS is not 'hosted' in DNS, but published via a policy service over HTTPS, so it uses PKI.
I work for Mailhardener, we offer MTA-STS policy services as a service [1]. We've seen an increased interest of MTA-STS recently, likely due to Microsoft announcing MTA-STS support [2].
Yeah STARTTLS is fundamentally flawed that way. That's why Google and other SMTP interchange services allow paying customers (Google Workspace) to specify policy around TLS compliance.
I wouldn't say STARTTLS is fundamentally flawed. Given a mechanism for declaring a policy of TLS being required, a middlebox downgrade attempt to deny STARTTLS should merely cause the sender to abort the SMTP transaction. This is no different from a middlebox hijacking a pure TLS tcp connection. In both cases it's on the sender to abort if a secure connection cannot be established.
A couple of years ago, someone with a .gov address told me they couldn't subscribe to a mailing list I manage, since the mail server didn't support TLS.
Adding the appropriate option in Postfix fixed the problem, and I think there were further options to require TLS.
I wouldn't consider the initial call to starttls to be insecure. It is just a matter of a few static bytes containing an SMTP HELO and the ascii bytes "STARTTLS". This can be used to establish a secure TLS connection without compromising any private information. It is only insecure if the client decides to proceed with the rest of the SMTP transaction if the server (or middlebox) nak's the STARTTLS. Which is no different from a middlebox presenting a self-signed TLS certificate. A few extra plaintext bytes at the start of the TCP connection containing the string "STARTTLS" doesn't change anything compared to a plain TLS socket. It is always up to the client to decide if it wants to proceed in the face of bad or nonexisting TLS.
Does the latter actually happen? Do smtp clients actively check this? What’s to stop some middleware performing a tls downgrade?