Many of the responses here saying that it's spam, not Gmail, that's killing independent email are implicitly saying that Google has no way of differentiating between a server not being well known and messages having spam content.
This is saying that the problem is unsolvable. This is patently untrue.
My email servers are configured to do all filtering on whether the connecting server is properly set up:
1) Does the HELO / EHLO name resolve in DNS to the address of the connecting server? If the answer is no, then reject as spam.
2) Is the connecting server's IP on any of a number of more conservative anti-spam DNS-based blocklists, like those that are based on dynamic IP pools, or on spam honeypots? If so, reject as spam.
3) Does the SPF for the sender's domain fail? If so, reject as spam.
The amount of spam this eliminates is tremendous, and most spam that still gets delivered comes from the big spammers: Google, Microsoft, Amazon.
I do not filter content because I'm adamantly anti-spam and and talk about and share spam with other anti-spam advocates, so content filters would be stupid.
Speaking of stupid content filtering, the number of abuse addresses which have anti-spam content filters is ridiculously high. Companies should be embarrassed that they don't know how to run email servers properly and can't accept abuse complaints properly at their abuse addresses.
Google is one of these.
Also, Google doesn't appear to do the tiniest thing with abuse complaints sent to them.
Finally, Google doesn't give people information about their spam filtering, nor ways to adjust it, so as long as Google applies arbitrary both to server reputation and to content filtering, with no ability to adjust, self hosting and smaller email servers will suffer.
Google knows this, and they COULD change this, but there's no profit, no business motivation to do the right thing. They have an interest in NOT doing the right thing, so we can't expect them to care.
What we can do is we can remind people who use Google for email that their email is non-deterministic. Nobody can say for sure whether email will be delivered or received consistently, because no regular humans know Google's rules for filtering, nor do we have access to Google's email logs.
When there are problems, we have to remind Google email users that the problems are with their choice of email hosting, and that's the price of giving up freedoms for "free" email.
I'm always fascinated about how, no matter the community, we can see downvoting without any responses that show where someone's posting is supposed to be incorrect.
I'd be curious to know if someone disagrees, and particularly what part anyone things is wrong. I have years of evidence, but the evidence is from running my own server. I'd love to hear perspectives from people who have different sources of evidence.
Downvotes without saying why just seem... emotional. I admit I downvote people who make generalizations without backing them up, but I'm not sure what's happening here.
> This is saying that the problem is unsolvable. This is patently untrue.
That is a very bold (and false) generalization. Spam is not primarily a technical problem, it's a human one. Human problems have mitigations, workarounds, not solutions.
> 1) Does the HELO / EHLO name resolve in DNS to the address of the connecting server? If the answer is no, then reject as spam.
Again, too bold.
> 3) Does the SPF for the sender's domain fail? If so, reject as spam.
Bold and incorrect.
> I do not filter content because I'm adamantly anti-spam and and talk about and share spam with other anti-spam advocates, so content filters would be stupid.
Only works on a very small scale.
> Finally, Google doesn't give people information about their spam filtering, nor ways to adjust it, so as long as Google applies arbitrary both to server reputation and to content filtering, with no ability to adjust, self hosting and smaller email servers will suffer.
Any provider with any significant size won't give you the full details. It'd be the spammers' dream.
> That is a very bold (and false) generalization. Spam is not primarily a technical problem
If you read what I wrote, you'll see I was talking about this problem: "that Google has no way of differentiating between a server not being well known and messages having spam content"
What you call too bold and incorrect I have years of data showing otherwise. From where do you get your data showing that HELO / EHLO checking and SPF failure rejection are detrimental?
What part of not using content filtering works only on a very small scale? Very small scale of what? What does the scale have to do with the kind of filtering?
So you believe that if providers shared their criteria, it'd be "the spammers' dream"... Except that hasn't happened. Many providers explicitly state their criteria, because they inform their customers what kind of spam protections are in place. Only large providers play games with random, unseen and unknowable rules.
But if you have evidence about how spammers have been in dreamland when learning providers' criteria, please do share.
> What you call too bold and incorrect I have years of data showing otherwise.
It violates standards, unless the letter doesn't pass DMARC checks an SPF fail is not sufficient. You don't have the volume necessary for proving anything either, I'm sorry. The fact that you think content filtering is not necessary proves it.
> What does the scale have to do with the kind of filtering?
It changes how accurate your tests have to be in order not to flood your customer support with complaints.
> Many providers explicitly state their criteria, because they inform their customers what kind of spam protections are in place. Only large providers play games with random, unseen and unknowable rules.
You're just excluding all providers with significant user base, of course then the problem is simple and processes (if there even are any) transparent.
The moment you have any significant userbase you're going to have a bunch of people trying to bypass your filters. Be it for spam or phishing. No reasonable provider wants to reveal all those details, don't be naive.
Handwaving, along with no data. Also, you have zero idea about my volume or the volume of my clients.
Also, you assert, again without data, that knowing a provider's criteria for acceptable email is what would justify the energy needed to try to bypass it. Imagine that!
"Spammers: you have to not use spammy servers."
Spammers: "Let's start using 100% completely legitimate email servers."
Sorry. I think you just don't understand how ridiculous that is.
> Handwaving, along with no data. Also, you have zero idea about my volume or the volume of my clients.
No, not really. I'm basing what I'm saying on best practices outlined in many places, standards and well yes, also my experience as mailop.
You're the one going against the entire industry with things like content filtering being not necessary... I won't start explaining again why that is simply wrong or just a sign of the small scale you operate at.
This is saying that the problem is unsolvable. This is patently untrue.
My email servers are configured to do all filtering on whether the connecting server is properly set up:
1) Does the HELO / EHLO name resolve in DNS to the address of the connecting server? If the answer is no, then reject as spam.
2) Is the connecting server's IP on any of a number of more conservative anti-spam DNS-based blocklists, like those that are based on dynamic IP pools, or on spam honeypots? If so, reject as spam.
3) Does the SPF for the sender's domain fail? If so, reject as spam.
The amount of spam this eliminates is tremendous, and most spam that still gets delivered comes from the big spammers: Google, Microsoft, Amazon.
I do not filter content because I'm adamantly anti-spam and and talk about and share spam with other anti-spam advocates, so content filters would be stupid.
Speaking of stupid content filtering, the number of abuse addresses which have anti-spam content filters is ridiculously high. Companies should be embarrassed that they don't know how to run email servers properly and can't accept abuse complaints properly at their abuse addresses.
Google is one of these.
Also, Google doesn't appear to do the tiniest thing with abuse complaints sent to them.
Finally, Google doesn't give people information about their spam filtering, nor ways to adjust it, so as long as Google applies arbitrary both to server reputation and to content filtering, with no ability to adjust, self hosting and smaller email servers will suffer.
Google knows this, and they COULD change this, but there's no profit, no business motivation to do the right thing. They have an interest in NOT doing the right thing, so we can't expect them to care.
What we can do is we can remind people who use Google for email that their email is non-deterministic. Nobody can say for sure whether email will be delivered or received consistently, because no regular humans know Google's rules for filtering, nor do we have access to Google's email logs.
When there are problems, we have to remind Google email users that the problems are with their choice of email hosting, and that's the price of giving up freedoms for "free" email.