The House recently released their report on their investigation of competition in digital markets. This report is timely, well written, and really one of the best reports we’ve ever seen come out of Congress in our lifetimes. The report details the monopolies that the large technology companies have created for themselves and the various anti-competitive measures they have taken to do so. We’re still working our way through the report but we can already tell we will be referencing the evidence collected and conclusions reached for many years to come. We congratulate all who were involved in the creation of this report and thank them for the hard work they put in to make it happen!
As part of the Subcommittee’s investigation into digital markets, we have been briefing the staff of the Antitrust Subcommittee on Google’s monopoly on crawling the web. We reached out to the Subcommittee staff last year because we felt that web crawling is a critical economic component of the search engine market that was not understood by policy makers as something worth investigating and potentially regulating. We submitted to the Subcommittee a policy paper about the dynamics of web crawling and continued afterwards to have many conversations about Google and web crawling. In fact, much of the material that can be found on the website you are reading right now was originally written for that policy paper that we sent to Congress.
So, it is with great pride that we can announce that Congress has officially recognized the importance of web crawling when it comes to regulating digital markets. On page 79 of the report, in the section discussing the online search engine market, the report includes a description of Google’s web crawling advantage, citing the author of this blog post (who is also one of the founding members of the Knuckleheads’ Club). And if you scroll on down to page 177 of the report, in the section discussing Google’s power in the search engine market, the report includes the following:
Meanwhile, steps that website owners take to block non-Google crawlers have rendered the task of creating an independent comprehensive index extremely challenging, if not effectively impossible.
The above is a perfect summation of the impact the winnowing effects of website operators choices have on the search engine market. We are thrilled that all of this made it into the report. It was very much an honor to work with the staff of the Subcommittee on this subject and we are excited that regulators have officially taken notice of the importance of web crawling in preserving open markets and the open web.
The next major step in this fight is gathering up more people who want to fight for the open web while producing research that details exactly how and why Google gained their monopoly on web crawling. So, join the club and join our fight to make something happen together and take back the fruits of the information revolution.