Search Engine Precision and Recall
Oracle Database Tips by Donald Burleson
Google leaped to the front of the search engine
products for several reasons, and many researchers are attempting to
quantify those success factors with metrics defined as "precision
"and "recall". Yeah, OK . . .
If it were really simple (and measurable), then
there would not be such a discrepancy between the traffic (and the
resulting billions of dollars in revenue) of each search engine.
In a world where every search engine company invests millions of
dollars in their technology, why does Google command more usage than
almost of the competitors combined. Of course, a savvy web
user is far like likely to use Google than Grandma who uses whatever
appears on her toolbar.
See my notes on
word stemming and synonym expansion to learn more about the need
to expand queries to include stems and synonyms.
Watch out for what's
Let's explore precision and recall and examine
if the definition of "relevant" might negate this research method
and indicate a need to look at the search engine relevancy from a
different point of view.
Precision is defined as a metrics to ensure
that the query returns ALL matching pages (i.e. no lost results).
In other words, precision is percentage of the statistical universe
of matching results. An example of precision of search engines
is this scholarly study titled "Precision
and Recall of Five Search Engines for Retrieval of Scholarly
Information in the Field of Biotechnology ", shows interesting
academic research on the relative precision and recall of several
internet search engines. It notes:
"Precision is the fraction of a search
output that is relevant for a particular query. Its calculation,
hence, requires knowledge of the relevant and non-relevant hits in
the evaluated set of documents (Clarke
& Willet, 1997)."
"In the context of the present study
precision is defined as:
Precision= Sum of the scores of
scholarly documents retrieved by a search engine
of results evaluated "
Recall is defined fare more loosely and it as
it uses the highly-variable word "relevant", a loosely-defined term,
and at the heart of the success of any search engine. Notes from
Precision and Recall of Five Search Engines for Retrieval of
Scholarly Information in the Field of Biotechnology, define
precision as a metric that is impossible to accurately measure:
"Thus it [Precision] requires
knowledge not just of the relevant and retrieved but also those not
& Willet, 1997). There is no proper method of calculating
absolute recall of search engines as it is impossible to know the
total number of relevant in huge databases."
Precision and Recall, show their results, with Google not
coming-up #1 on either precision and recall (as defined by
Table 1. Mean Precision and Relative Recall of search
engines during 2004
Ancient Precision and Recall
Old reference research from back in the early
days when the area of search engine metrics was in its infancy
provide a must-read foundation to the problem of defining the "best"
web search mechanism. Notes from
this paper also contains a nice list of other scholarly studies
on search engine precision and recall, from back in the early days
when people thought that search engine ranking analysis would never
morph into a multi-billion dollar a year question:
- Bar-Ilan, J.
(1998). On the overlap, the precision and estimated recall of
search engines: A case study of the query "Erdos".
Scientometrics, 42 (2), 207-208.
- Chu, H., &
Rosenthal, M. (1996).
Search engines for the World Wide Web: a comparative study and
evaluation methodology. In: Proceedings of the ASIS 1996
Annual Conference, October, 33, 127-35. Retrieved August
19, 2003 from http://www.asis.org/annual-96/ElectronicProceedings/chu.html
- Clarke, S.,
& Willett, P. (1997). Estimating the recall performance of
search engines. ASLIB Proceedings, 49 (7), 184-189.
- Ding, W., &
Marchionini, G. (1996). A comparative study of the Web search
service performance. In: Proceedings of the ASIS 1996 Annual
Conference, October, 33, 136-142.
- Leighton, H.
(1996, June 25).
Performance of four WWW index services, Lycos, Infoseek,
Webcrawler and WWW Worm. Retrieved June 10, 2005 from
H., & Srivastava, J. (1997).
Precision among WWW search services (search engines): AltaVista,
Excite, HotBot, Infoseek and Lycos. Retrieved June 11, 2005
- Library of
Congress (2003). Library of Congress Subject Headings (vol.s
1-5). Washington: Library of Congress, Cataloging Distribution
G. (1996). Searching the Web for gigabucks. New Scientist,
150 (2024), 36-40.
C., Moris, A, Mcknight, C., & Lowley, S. (2000). The evaluation
of WWW search engines. Journal of documentation, 56
C., & Nilsen, E. (1996).
Signal detection analysis of WWW search engines. Retrieved
September 15, 2003 from http://www.microsoft.com/usability/webconf/schlichting/schlichting.htm
Find it on the Net. PC World, January, 14(1),
125-130. Retrieved June 6, 2003 from http://www.pcworld.com/reprints/lycos.htm
The Pragmatics of information retrieval experimentation,
revisited. Information retrieval experiment, 14,
59-102. Retrieved 11 June, 2005 from
If you like Oracle tuning, you may enjoy my new book "Oracle
Tuning: The Definitive Reference", over 900 pages
of my favorite tuning tips & scripts.
You can buy it direct from the publisher for 30%-off and get
instant access to the code depot of Oracle tuning scripts.