Eric Schmidt's Testimony Reveals, Google Tests '13,311 Precision Evaluations' Algorithm Change

Google's Executive Chairman Eric Schmidt appear before the U.S. Senate to testify and talk about Google's approach to competition -- "We welcome competition. It makes us better. It makes our competitors better. Most importantly, it means better products for our users," said Schmidt.Schmidt is speaking today at a Senate subcommittee hearing in Washington, DC, about […]

Google's Executive Chairman Eric Schmidt appear before the U.S. Senate to testify and talk about Google's approach to competition -- "We welcome competition. It makes us better. It makes our competitors better. Most importantly, it means better products for our users," said Schmidt.

Schmidt is speaking today at a Senate subcommittee hearing in Washington, DC, about Google's business practices, and answering claims that Google's business practices are anti-competitive.

Schmidt came prepared with a set of his written remarks (oral testimony) that was submitted to the subcommittee prior to his appearance, and that contains a section revealing more details about how Google tests its algorithm changes. Schmidt explains that "potential refinements to the algorithm go through a rigorous testing process, from conception to initial testing in Google's internal 'sandbox' to focused testing to final approval."


He then details how Google uses both human review and internal analysis to develop what amounted to more than 500 algorithmic changes in 2010:

To give you a sense of the scale of the changes that Google considers, in 2010 we conducted 13,311 precision evaluations to see whether proposed algorithm changes improved the quality of its search results, 8,157 side-by-side experiments where it presented two sets of search results to a panel of human testers and had the evaluators rank which set of results was better, and 2,800 click evaluations to see how a small sample of real-life Google users responded to the change.

Ultimately, the process resulted in 516 changes that were determined to be useful to users based on the data and, therefore, were made to Google's algorithm. Most of these changes are imperceptible to users and affect a very small percentage of websites, but each one of them is implemented only if we believe the change will benefit our users.

Schmidt's Testimony submitted before Senate:

Schmidt's oral testimony:

Here is the webcast: