Ever wondered how the search giant tests the quality of search results?
Same question was asked by Technology Review and heres Google’s director of research, Peter Norvig’s answer:
We test it in lots of ways. At the grossest level, we track what users are clicking on. If they click on the number-one result, and then they’re done, that probably means they got what they wanted. If they’re scrolling down, page after page, and reformulating the query, then we know the results aren’t what they wanted. Another way we do it is to randomly select specific queries and hire people to say how good our results are. These are just contractors that we hire who give their judgment. We train them on how to identify spam and other bad sites, and then we record their judgments and track against that. It’s more of a gold standard because it’s someone giving a real opinion, but of course, since there’s a human in the loop, we can’t afford to do as much of it.
Heres a sneak preview of the Google’s evaluation lab – courtesy Search Bristro.