Google touts itself as a company that wants to catalog all the knowledge. How is it possible that WSJ content ranking falls after the introduction of the pay wall? Google has the ability to crawl behind the wall content, and yet it downgrades the WSJ content. In my opinion, this is an abuse of its search dominance position.
> Google touts itself as a company that wants to catalog all the knowledge. How is it possible that WSJ content ranking falls after the introduction of the pay wall?
Because ranking is about utility to search users, and content not accessible to general search users isn't useful to them. The knowledge (both the content itself and the meta-data regarding it's inaccessible status) is still cataloged, this is just making effective use of the meta-data for the purposes of search UX.
Google may know you are subscribed but it doesn't know what the page looks like to you. It only knows its own version of the page.
What Google sees is that many people go to WSJ because their search query matches the content of the googlebot version of the page, and then bounce back as they hit the paywall. Google may not even know about the paywall. It just sees that because many users bounce back, they must be unsatisfied, and derank accordingly.
Now, it still has a workaround. If you are logged in, it tracks your personal habits. And if it sees you are staying on WSJ, it may uprank it specifically for you. Again, it is just a matter of staying on the site or not, google doesn't really care about your subscription.
> Google has the ability to crawl behind the wall content
I thought they weren't. How does that work? Do they have a login for WSJ? That would benefit WSJ as an incumbent news provider at the expense of startups too new or small to get special treatment from Google.
