Today, Google itself is distorting the market for all three factors needed to crawl the web.
- There have been industry reports of an ongoing microchip shortage at one of the major producers, in part due to the high volume that Google and other Cloud providers have been purchasing. This suggests that, because of how much hardware Cloud providers have been buying, it’s no longer possible for new entrants and businesses to buy server hardware in sufficient quantities to compete with Google. A new entrant’s only reasonable option would be to rent computers from one of the major Cloud providers – Microsoft, Amazon or Google – two of which operate their own search engines.
- In terms of the labor involved in crawling, Google had been distorting the labor market by participating in a “no poaching” agreement with other technology firms. This made it harder to enter into the market and compete because it restricted the movement of the people with the specialized skills in building search engines.
- When it comes to finding startup capital to compete with Google, the problem is even more severe. A prominent venture capitalist has stated that there is a “kill zone” around companies like Google, such that venture capitalists will not even try to fund competitors in markets where Google has an established presence.
The prohibitively high upfront fixed costs to entry have been steadily rising. Every time a webpage is published on the web, the cost of crawling the entire web goes up just a little bit more. Since the start of the web, the size of the web has grown exponentially, forcing the costs to crawl it upwards. This means that the costs Google itself had to incur to enter into the search market in 1997 would be dwarfed by what a new market entrant today would need to spend in order to compete.
The nature of these high fixed costs — which are ever increasing — helps to explain the market dominance of Google. Google was the first company to pay enough to get past the barrier of entry to crawling the entirety of the Internet on a regular basis. It took several years for Google’s competitors to even realize that barrier was there, which only happened because they saw how superior Google’s search product was to their own in terms of comprehensiveness and freshness. By that time, it was already too late for competitors to catch up, as Google Search rapidly took over the search market ergo becoming the primary search engine online. In many senses, despite there having been many other search engines in the market, Google built the first proper search engine that could handle the magnitude of the task at hand.
Google was encouraged to be aggressive with its crawling, in part because of an algorithm called PageRank that the founders of Google developed while at Stanford. PageRank uses links between pages to get an idea of which webpages are the most relevant for particular topics. The more webpages that are fed into the algorithm, the better the results. In contrast, the algorithms that almost every other company used at the time gave worse results after more information was added to them. Unlike almost every other company at the time, Google had a stronger incentive to crawl and expand its cache and keep it fresh and updated. While other companies knew about using PageRank like algorithms to rank results, they did not focus on link analysis the same way Google did and didn’t have the pressure to crawl as much or as often, and so they got caught flat footed with stale indexes just as Google started to really take off.
So, to recap, Google Search became the world’s dominant search engine because only Google had an algorithm that incentivized the company to crawl as aggressively as it could. Over time, the costs to enter the search market have increased with the growth of the Internet, and Google has effectively pulled up the ladder by driving up entrance costs for potential competitors. Taken together, the high cost to do the necessary crawling is the first half of a natural monopoly.
The aggregate cost of being crawled is the second half of a natural monopoly. We’ve found two explanations that illustrate how the aggregate cost of being crawled limits the number of crawlers. The first is an exaggerated technical explanation and the second is a metaphorical economic explanation. After these two explanations, we will look at the specific factor of production that only Google has access to that gives them control of crawling the web.