Google wouldn't know that someone is curling it which their script said everything was ok. While the website was basically down because Chrome and Firefox will both block a site based on Google's safe browsing list.
They could use Google's safe browsing api to check if they're on that list as well as curl.
Frankly, Google's Safe Browsing list is one of those things which should be broken out of Google and ran by an independent entity, much like the Let's Encrypt model.
1. Mozilla and the Chrome clones (Edge, Brave, etc.) partner to make an (open as possible) standard for blocking and reviewing, and start maintaining their own list upstream from Google.
2. When Google adds to their blocklist, independently check it according to the consortium's own standards.
3. Cut a deal with Bing or Yandex to scan for malware as part of their crawls, to get technology independence.
I still don’t understand what the problem is here.” With Google doing this.
We don’t know that this whole thing is not invented by the author or that they were indeed not doing anything malicious.
All of this, as credible as it may seem, could just be invented bad PR against Google, which if true should make us take a hard look at whomever is behind this.
Seriously, read the Lessons Learned again and tell me for certain this really happened. How in the world would it look so staged, with the bold words and thought-out structure. I’m not going to jump on the bandwagon everytime someone makes claims.
They could use Google's safe browsing api to check if they're on that list as well as curl.