Bust up Google’s Destructive Search Monopoly

Ryan Cooper

Most everyone knows that Google overwhelmingly dominates online search. And recently, the company’s search algorithm monopoly, in which their search engine prioritizes Google’s own content in results, has rightly begun to attract significant antitrust attention in both the U.S. and Europe.

But this isn’t the only monopolistic aspect of Google’s search business. Google’s position is protected by virtue of having by far the largest and best online index.

Software engineer Zack Maril has founded an organization called the Knucklehead Club (because only a knucklehead would take on such a rich and powerful company) to research and draw attention to this fact, as Daisuke Wakabayashi writes in The New York Times. For structural reasons I’ll explain below, the activity of “crawling” the web to build such an index is a natural monopoly, and nobody is likely to ever pose a real challenge to Google’s dominance – something that is flying somewhat under the radar. “It’s generally not talked about, because people don’t know about it that much,” Maril told The Week in an interview. Governments around the world should regulate this company in the public interest, or simply take it over.

First, let me explain the mechanics. In economics parlance, a natural monopoly is a market in which there are high fixed costs but low marginal costs. A typical example is electricity, which requires huge initial investments in power plants and wires going to every house, but where the cost of delivering one additional kilowatt-hour is nearly zero. Once some company has that network built, it is going to be nearly impossible for a new company just starting out to compete.

Google’s web crawler stands in a basically similar position. Here’s why: To run a search engine, you need to make an index of the internet so there is something to search (similar to a card catalog in a library). That requires a computer to regularly crawl the internet and make copies of as many web pages as possible, so the index is as broad and current as possible.

That raises two barriers to competition. First and most obviously, the internet is huge and therefore building a crawler that can look at even a small fraction of it requires a huge investment. Google has been far out ahead for so long that it takes another tech behemoth, Microsoft, to even begin to compete. Second and more subtly, website owners have an incentive to restrict all non-Google crawlers. It may be expensive to create an internet-scale crawler, but it is cheap to create one that will just look at a few websites, and people indeed do this for many reasons. Website bandwidth costs money, and crawlers can eat up a lot of it with their automated activity. Many administrators therefore ban all crawler-like activity, but then build in an exemption for Google, because showing up in its search results matters so much for traffic. (Knucklehead research also shows that many websites give Google even more advantages over and above this.)

These two factors mean it is virtually impossible for any new company to seriously threaten the dominance of Google’s web index. Any new major crawler would be banned from most websites because it costs them money and provides little traffic in return – but the new search engine also can’t grow its traffic, precisely because it can’t develop a proper index and attract a user base. Even Microsoft, after sinking years of effort and billions into its search engine Bing, is far behind.

To the layman, this may not seem like such a big deal. After all, Google is fine, so we probably don’t need to create more web indexes or force Google to let others use its index, right? Now, even in terms of search this is not clear at all. For instance, Google has become extremely bad in certain areas like product reviews, where its search results are routinely cluttered up with tons of corrupt, algorithm-gaming garbage or its own ads. If other companies could compete, we might see search engines that beat Google at certain specific tasks, or even just ordinary search.

But the deeper problem is the enormous advantage that controlling the only good internet index provides. This massive treasure trove of data gives Google an “incredible advantage... in machine learning and artificial intelligence,” Maril told The Week. Machine translation, for instance, “only becomes effective once you reach a critical point of data.” By the same token, considered from the perspective of other websites, making sure Google can crawl looks less like some kind of favor and more like a “Google tax” that must be paid, he says. Anyone with a website simply must spend whatever time and money it takes to let Google index their pages, lest they be impossible to find. That data then gives the company an ever-more insurmountable competitive advantage.

As I have argued before, Google is so powerful and profitable primarily because of first mover advantage and network effects. It was in the right place at the right time with a search engine that was a bit better than everyone else’s, and that advantage snowballed into private control over the only good web index that exists, or can ever exist (at least under the current status quo). It’s unfair that only Google gets to use that data, but also it’s simply impossible to know how many business or research ideas have been strangled in the crib because nobody can get access to its data. Monopolists tend to get fat and lazy.

So what is to be done? The orthodox solution to a natural monopoly is either nationalization or regulation. However, those might not be quite appropriate, because unlike just about every previous natural monopoly, the internet covers the entire planet. If Google’s web index were to become owned by the federal government, for instance, they might be tempted to use it as part of some madcap imperialist crusade. It might be wiser for, say, the U.S. and the European Union to agree to an international treaty requiring Google to license out its index content for a nominal fee. Or nations could simply build their own publicly-owned internet index, free for all to use.

But before anyone can iron out any solutions, we’ve first got to investigate the problem in detail. Luckily, the House of Representatives has been on the case, and even the Trump Department of Justice has brought a lawsuit accusing Google of violating anti-trust law. That’s just the beginning of the kind of state effort that is needed to create an internet that is fair for all.