Scott Huffman runs the evaluation team at Google explained how Google’s trying to stay ahead in search, and runs all those experiments,which include human evaluators in addition to Google’s massive computer infrastructure. “human evaluators are pretty important for us today. The more automated or user behavior/click-based things really give you complementary kinds of data. Both have noise in them: Human evaluators make mistakes. Clicks are hard to interpret; people click or don’t click for all kinds of reasons. The clicks obviously tell you what users are actually doing, and you get them at volume, at a real scale, but it’s hard to interpret. Human evaluators, there’s noise in terms of mistakes, but we can go deeper on specific examples. For this pretty rare query, we can generate examples that our ranking engineers can go look at. Where we don’t get a correlation, that’s a big red flag,” said Huffman.