Yeah I do not understand why the author waited so long to disclose and also feels that Google deserves a "stellar job" here. Sure, Google patched the bug very quickly after disclosure. But given that Google waited so long, it sure looks like they only prioritized the fix once disclosure was a risk. If anything, I think that the author should have scheduled disclosure sooner.
> I do not understand why the author waited so long to disclose and also feels that Google deserves a "stellar job" here.
Because people are afraid of megacorps. They've found the courage to disclose the issue, but they've also felt that the blow needs to be softened by praising Google's security team, despite their negligence in handling this issue.
> I think that the author should have scheduled disclosure sooner.
Yup. Ninety days is fine. More people should choose ninety days up front and not allow themselves to be strung along indefinitely.
Project Zero actually has granted two exceptions to their policy (out of well over a thousand cases), both to rival companies (Apple and Microsoft). On the whole I would say you should resist doing this, just set the policy and reap the consequences whatever they might be. If somebody's $100Bn company burns to the ground because they couldn't get their shit together for three whole months too bad.
It's not you that hurt the users, it's the company for not being able to competently route, schedule, and fix their issue.
The reporter is only to blame if they actively exploit the vulnerability in order to harm users, not if they publish it publicly, with or without advanced notice to the company.
I was thinking about how I would handle it, were I in the same situation and I think I came upon a decent idea.
1) 90-day disclosure initially
2) assuming communication, I would agree to extend for another 30 days
3) 15 days more
4) 7 days more
5) 3 days more
6) 1 day more
7) 12 hours
8) 6 hours
9) 3 hours
10) 1 hour
11) publish
More work for me, sure, but it doesn't drag out things indefinitely and i think it would have (at the later stages) created a sense of immediacy to get this fixed.
I have been on both sides of this situation. Running bug bounty programs, and submitting vulnerabilities to Google both before and after I worked there.
Often a researcher will find a bug, report it, and then weeks or months later reply with a follow up that dramatically changes the scope or severity.
Based on all of my interactions with the Google VRP program, I consider it much more likely the researcher isn't giving the whole story about the timeline. They are super responsive, take shit seriously, and push teams to get patches out.
Just a guess: There are some large existing customer depending on the current behaviour, and they were reluctant to make the email sending fail by patching it.
You can do the same thing with Fastmail. Completely indistinguishable from a legitimate email - all signatures look exactly the same. Any user can impersonate any other user. Not considered a vulnerability.
What's especially frustrating is that Fastmail have a special opaque sender header that only they can interpret, and they put a little "verified" icon on email that actually comes from them. So it's a vulnerability if I impersonate them, but not any other Fastmail user. Sigh. I'm a happy FM user, apart from that. But I'm going to keep bringing it up until they do something about it. At least let me blacklist my domain from being used by other accounts jfc.
This is not true for custom domains for some time. The DKIM signature is only added for users in the account that has the domain, and similarly mail is sent via different outbound servers that are not included in our recommended SPF record if another a user outside of this account tries to send mail from that domain. For Fastmail domains the situation is more complicated for historical reasons (we've been around over 20 years and customers have set up workflows that we need to transition) but we are working on it.
How recently was this change made? I could still do it in 2018, but I wasn't able to do it just now. As you say, no signature on the message, which makes my DMARC record do something useful. That's really great, a huge relief.
Email is an absolute disaster for security and privacy. Its a shame we will never see a real alternative because the age of open and decentralized standards is over. Matrix shows hope but I doubt it will come anywhere close to the popularity of email and SMS
This is a great find. I bet there are similar bugs hiding in many hosted mail platforms.
I always wanted to test MS365, but never got around to it. IIRC if you don't configure DKIM, they'll still sign messages using your x.onmicrosoft.com perma-ID. I remember being surprised those signatures show as valid when received in GSuite since the DKIM signing domain is different than the sending domain. Does that have something to do with ARC sealing? It should be impossible, right?
I could be totally out to lunch though since I noticed that when trying to diagnose broken DKIM signatures on a domain.
I'm also skeptical with half the planet having SPF records that point to Microsoft or Google. The dependence on domain validation after that is what makes me think there could be a lot of bugs of this class hanging around.
I bet shared hosting is a massive disaster in this area too.
If memory serves me right DKIM only mandates that the DKIM signature header is valid for the d= domain used by verifiying that the protected fields where not changed - but it makes no claims on alignment of the d= domain in the DKIM signature and the protected headers.
So anyone can sign for any domain for which they have published DKIM keys, and produce valid DKIM. It's DMARC that requires that a valid DKIM signature match the d= domain with the From domain to consider the message DMARC aligned and be awarded a DMARC pass. [or otherwise pass SPF checks and have SPF domain aligned with From domain]
I'm not sure I could resist the temptation to do some serious trolling if I discovered a bug that let me send legit looking mail as @google.com users. Just imagine emailing allusers@google.com instituting a policy of riding a unicycle to work. It would be glorious.
It seems that this kind of weakness in tenant isolation is something that's hard to avoid in a public cloud environment, yet really critical to get right, to preserve confidence in it.
Yet given the nature of SPF and DMARC, they're inherently going to be pretty open to other tenants in the absence of IP-based isolation.
Perhaps each tenant could use a unique outbound IPv6 in future (if it reaches traction on servers), and have a small pool of such personal IPv6s allocated exclusively for their own email, across the other fallback systems?
DKIM is better as it uses cryptographic keys, but you need to give a cloud email host be access to these, or realistically load its own DKIM key into your DNS.
Any other ideas on ways to stop these kinds of cross tenant issues when designing cloud services? I'm assuming nobody will provision infrastructure per tenant due to scale and cost, but this is quite a serious issue underlining the risk of relying on a tenant account with public cloud providers when it comes to isolation.
DMARC is essentially the combination of SPF and DKIM, so it's a bit misleading that you are mentioning SPF and DMARC and make it sound like DKIM is something separate. See Wikipedia [1].
If I recall correctly, DKIM setup is simply publishing the public key as a txt DNS record under your domain. Whoever has the fitting private key can then sign emails on your behalf. The cloud email host can without problem generate a new pair for every domain. So the "upload it's own DKIM key" is essentially creating a pair for you and telling you to set the public key as a certain DNS text entry. Pretty straightforward. Stopping the cloud provider to sign emails for you is also as simple as removing the DNS entry from your domain.
> I'm assuming nobody will provision infrastructure per tenant due to scale and cost
Actually, these services are available at a premium.
You can have any combination of DMARC, DKIM, and SPF. They are all distinct.
Though with DKIM there is no way for a receiver to know if a message should be signed unless there is also a DMARC policy. SPF on the other hand can be verified without DMARC at all.
If Gmail.com had a DMARC policy that required a correct dkim signature, then the attack would have been much harder.
Also, to be clear, gmail.com DOES have a DMARC and SPF policy and does not have or use DKIM.
No you cannot, like you said: DMARC passes if either SPF or DKIM passes.
But what you can do is use the 'aspf' and 'adkim' settings to force strict alignment on SPF and/or DKIM for DMARC to pass.
We see some of our [0] customers using strict SPF alignment to basically force DMARC SPF alignment to fail most of the time, thus effectively forcing a DKIM requirement for DMARC to pass. We wouldn't recommend doing this though, unless you are already closely monitoring your email traffic, and getting close to 100% DMARC alignment.
Note that none of this would have helped against the vulnerability discovered in the OP though.
I actually noticed yesterday that apparently G Suite customers can email Gmail accounts using domains with invalid SPF records (set to hardfail if not from a different mail service). I thought that was very odd. It seems that Google assumes G Suite's domain validation process is adequate, and doesn't bother checking SPF records for mail internal to Google's infrastructure. The flaw reported in this article depends on this apparent decision.
Whilst this general choice may not be the gaping security hole it feels like, because G Suite validates domain ownership, it does lead to people setting up G Suite, configuring their MX records (G Suite doesn't make SPF obvious to a new account, when I did it a few months ago for someone), and then people may email back and forth with a personal (likely Gmail) account to test it works.
Then, when they email someone with another mail server, they blame us when their SPF record is configured to block G Suite.
I can see why Google would not require G Suite customers to set up DKIM, but why do they not have it set up for google.com? It seems like it would have added another barrier of defense in the case of this bug.
It is setup for google.com but you'll need to receive a message from Google to see the selector used. It's not possible[1] to establish if domain is DKIM enabled by doing DNS queries without knowing the selector used.
[1] You'd need permissions to enumerate the entire DKIM zone of the domain which you wouldn't have for a random domain and whilst you could by to bruteforce all selector names combinations between 1 and 63 characters long I wouldn't recommend doing so.
A solution would be to extend SMTP with a new verb that allows specifying a domain-related signature prior to message delivery. This would be a hybrid of DMARC and SPF and would allow receivers to very the sender identity independent of the sending IP.
That still requires the actual sender (Google in this case) to possess the key material necessary to sign on behalf of the domain, which means you still have a "confused deputy" risk.
The sender could obtain that material from the message itself and could verify the veracity of the material against the keys stored in the DNS by the domain owner.
After it was not fixed, she publicly disclosed the issue and within 7 hours it was patched.
What broke in the process at Google? This issue allowed GSuite users to impersonate each other to send email. That is very serious.