Search Engine Precision and Recall

Oracle Database Tips by Donald Burleson

Google leaped to the front of the search engine products for several reasons, and many researchers are attempting to quantify those success factors with metrics defined as "precision "and "recall". Yeah, OK . . .

If it were really simple (and measurable), then there would not be such a discrepancy between the traffic (and the resulting billions of dollars in revenue) of each search engine. In a world where every search engine company invests millions of dollars in their technology, why does Google command more usage than almost of the competitors combined. Of course, a savvy web user is far like likely to use Google than Grandma who uses whatever appears on her toolbar.

See my notes on Search Engine word stemming and synonym expansion to learn more about the need to expand queries to include stems and synonyms.

Watch out for what's "relevant"

Let's explore precision and recall and examine if the definition of "relevant" might negate this research method and indicate a need to look at the search engine relevancy from a different point of view.

Precision

Precision is defined as a metrics to ensure that the query returns ALL matching pages (i.e. no lost results). In other words, precision is percentage of the statistical universe of matching results. An example of precision of search engines is this scholarly study titled "Precision and Recall of Five Search Engines for Retrieval of Scholarly Information in the Field of Biotechnology ", shows interesting academic research on the relative precision and recall of several internet search engines. It notes:

"Precision is the fraction of a search output that is relevant for a particular query. Its calculation, hence, requires knowledge of the relevant and non-relevant hits in the evaluated set of documents (Clarke & Willet, 1997)."

"In the context of the present study precision is defined as:

Precision=    Sum of the scores of scholarly documents retrieved by a search engine

                      --------------------------------------------------------------------------------

                            Total no. of results evaluated "

Recall

Recall is defined fare more loosely and it as it uses the highly-variable word "relevant", a loosely-defined term, and at the heart of the success of any search engine. Notes from Precision and Recall of Five Search Engines for Retrieval of Scholarly Information in the Field of Biotechnology, define precision as a metric that is impossible to accurately measure:

"Thus it [Precision] requires knowledge not just of the relevant and retrieved but also those not retrieved (Clarke & Willet, 1997). There is no proper method of calculating absolute recall of search engines as it is impossible to know the total number of relevant in huge databases."

Notes from Precision and Recall, show their results, with Google not coming-up #1 on either precision and recall (as defined by "relevance"):

Table 1. Mean Precision and Relative Recall of search engines during 2004

	AltaVista	Google	HotBot	Scirus	Bioweb
Precision	0.27	0.29	0.28	0.57	0.14
Recall	0.18	0.20	0.29	0.32	0.05

Ancient Precision and Recall References

Old reference research from back in the early days when the area of search engine metrics was in its infancy provide a must-read foundation to the problem of defining the "best" web search mechanism. Notes from this paper also contains a nice list of other scholarly studies on search engine precision and recall, from back in the early days when people thought that search engine ranking analysis would never morph into a multi-billion dollar a year question:

Bar-Ilan, J. (1998). On the overlap, the precision and estimated recall of search engines: A case study of the query "Erdos". Scientometrics, 42 (2), 207-208.
Chu, H., & Rosenthal, M. (1996). Search engines for the World Wide Web: a comparative study and evaluation methodology. In: Proceedings of the ASIS 1996 Annual Conference, October, 33, 127-35. Retrieved August 19, 2003 from http://www.asis.org/annual-96/ElectronicProceedings/chu.html
Clarke, S., & Willett, P. (1997). Estimating the recall performance of search engines. ASLIB Proceedings, 49 (7), 184-189.
Ding, W., & Marchionini, G. (1996). A comparative study of the Web search service performance. In: Proceedings of the ASIS 1996 Annual Conference, October, 33, 136-142.
Leighton, H. (1996, June 25). Performance of four WWW index services, Lycos, Infoseek, Webcrawler and WWW Worm. Retrieved June 10, 2005 from http://www.winona.edu/library/webind.htm
Leighton, H., & Srivastava, J. (1997). Precision among WWW search services (search engines): AltaVista, Excite, HotBot, Infoseek and Lycos. Retrieved June 11, 2005 from http://www.winona.edu/library/webind2.htm
Library of Congress (2003). Library of Congress Subject Headings (vol.s 1-5). Washington: Library of Congress, Cataloging Distribution Service.
Modi, G. (1996). Searching the Web for gigabucks. New Scientist, 150 (2024), 36-40.
Oppenheiem, C., Moris, A, Mcknight, C., & Lowley, S. (2000). The evaluation of WWW search engines. Journal of documentation, 56 (2), 190-211.
Schlichting, C., & Nilsen, E. (1996). Signal detection analysis of WWW search engines. Retrieved September 15, 2003 from http://www.microsoft.com/usability/webconf/schlichting/schlichting.htm
Scoville, R. (1996). Find it on the Net. PC World, January, 14(1), 125-130. Retrieved June 6, 2003 from http://www.pcworld.com/reprints/lycos.htm
Tague, J. (1992). The Pragmatics of information retrieval experimentation, revisited. Information retrieval experiment, 14, 59-102. Retrieved 11 June, 2005 from http://portal.acm.org/citation.cfm?id=149514