Even if you enforce SSL-only on inbound mail, you're still vulnerable to the downgrade attack (or rather: the "prevent upgrade to TLS attack") - someone can MITM, and the sender will be speaking non-SSL to the MITMer, who'll speak SSL to you. To you things would look fine. Enter MTA-STS, with which the sending mail server can deduce that no, things are not OK.

Or are you saying you're not doing STARTLS at all and servers delivering mail to you have to do an SSL handshake before getting to speak SMTP to you? I'm quite surprised if that's compatible with the wider SMTP world.

MTA-STS is still vulnerable to MITM. If someone can tamper with DNS queries, it's useless. It's also apparently useless for the first message because the policy will not be in cache and only fetched afterwards.

We could have added a field so that when a server announces that they support STARTTLS, they can say that this fact should be cached for X days.

    250-AUTH LOGIN PLAIN
    250-STARTTLS
    250-STARTTLS-CACHE-90-DAYS

The vulnerability of the first message is a real concern, MTA-STS is not perfect. But it looks like the easiest to deploy tool that can strengthen security for a very large number of messages.

One challenge for your suggestion is that the mail server is often run by a different organization, on a different domain from the receiving address. The HTTPS web server, on the other hand, has a TLS certificate for the mta-sts subdomain of the receiving address. This gives confidence that the MTA-STS policy is set by the receiving domain, not the receiving mail server.

That's a hard choice. Google reports that 2% of the email they send goes out unencrypted, it's too large a percent to ignore. If you are using Postfix, would you mind sharing which setting you use? They all have challenges.