That's to some extent by design. When organic search works perfectly, the user goes directly from the search result to their desired destination. The search engine makes no money from that. This is the fundamental conflict of the search engine business. When Brin and Page originally tried to sell their technology to Yahoo, Yahoo rejected it because it was "too good" - users would immediately leave Yahoo.
Now Google has that problem for itself. About a third of Google's revenue is from AdSense (non-search ads). It's not in Google's interest for their anti-spam efforts to be too tough on made-for-Adsense sites. They crack down only on the outrageous cases. This became very clear in the "Sportsdrugs.com" case, where Google was caught in an FBI drug sting and paid $500,000,000 to avoid prosecution.
If you want to understand how Google really works, read the black hat SEO forums. Every time Google tries something new in search ranking, it's reverse engineered in a few weeks.
Trying to detect SEO with machine learning based on page contents worked for a while, but now the SEO guys use machine learning too. "Automated SEO" tries to automatically reverse-engineer Google's anti-spam algorithms, then adjusts pages and links until the SEO's model of Google says the page is under the threshold where Google will consider it spam.
It's not that hard to detect web spammers, if you're willing to do a quick background check on the company behind each web site. This can be automated; see our "sitetruth.com". Large numbers of spam domains trace back to the same source, so dealing with the problem on a per-business basis rather than a per-domain basis works.
Google has a monopoly on mind share, but only about 2/3 of US search. Bing (including Yahoo, which resells Bing) has about half the search volume Google does. That's historically about where Chrysler has been relative to General Motors. Yet Chrysler is taken seriously as an competitor to GM, while Bing is not taken seriously as a competitor to Google. It's not clear why. There's no network effect in search; it matters if your friends are on Facebook, but not if they're searching with Bing.
Bing has a very low public profile. It's hard to even find out who's in charge there. (Searching with Bing won't help.) Previously, it was Satya Nadella, who moved up to being CEO of Microsoft. There is no Bing CEO any more. Bing is under Microsoft's Online Services Division, headed by Qi Lu. Different parts of Bing are under four different parts of Online Services. (The Microsoft memo: http://www.geekwire.com/2014/internal-memo-microsoft-sets-le.... If that hadn't leaked, nobody would know who was in charge over there.)
Now Google has that problem for itself. About a third of Google's revenue is from AdSense (non-search ads). It's not in Google's interest for their anti-spam efforts to be too tough on made-for-Adsense sites. They crack down only on the outrageous cases. This became very clear in the "Sportsdrugs.com" case, where Google was caught in an FBI drug sting and paid $500,000,000 to avoid prosecution.
If you want to understand how Google really works, read the black hat SEO forums. Every time Google tries something new in search ranking, it's reverse engineered in a few weeks. Trying to detect SEO with machine learning based on page contents worked for a while, but now the SEO guys use machine learning too. "Automated SEO" tries to automatically reverse-engineer Google's anti-spam algorithms, then adjusts pages and links until the SEO's model of Google says the page is under the threshold where Google will consider it spam.
It's not that hard to detect web spammers, if you're willing to do a quick background check on the company behind each web site. This can be automated; see our "sitetruth.com". Large numbers of spam domains trace back to the same source, so dealing with the problem on a per-business basis rather than a per-domain basis works.
Google has a monopoly on mind share, but only about 2/3 of US search. Bing (including Yahoo, which resells Bing) has about half the search volume Google does. That's historically about where Chrysler has been relative to General Motors. Yet Chrysler is taken seriously as an competitor to GM, while Bing is not taken seriously as a competitor to Google. It's not clear why. There's no network effect in search; it matters if your friends are on Facebook, but not if they're searching with Bing.
Bing has a very low public profile. It's hard to even find out who's in charge there. (Searching with Bing won't help.) Previously, it was Satya Nadella, who moved up to being CEO of Microsoft. There is no Bing CEO any more. Bing is under Microsoft's Online Services Division, headed by Qi Lu. Different parts of Bing are under four different parts of Online Services. (The Microsoft memo: http://www.geekwire.com/2014/internal-memo-microsoft-sets-le.... If that hadn't leaked, nobody would know who was in charge over there.)