Daily Archives: January 18, 2005

The Jaded Surfer

[Part 2 in a series about relevancy.]

Search engines love to tell us that they are the most relevant, and I don’t blame them. A glance through any engine’s press releases will include claims like “most relevant update”, “a dramatic increase in relevancy”, and so forth. These conflicting claims are like political rhetoric. Ultimately they have the opposite effect of what was intended, because we are all becoming jaded searchers. Here are some examples, and be sure to note how many claim to be the most relevant:

• With the largest index of websites available on the World Wide Web and the industry’s most advanced search technology, Google Inc. delivers the fastest and easiest way to find relevant information on the Internet.
  http://www.google.com/press/pressrel/aol.html
• The [MSN] index, much of which is updated weekly or even daily, provides the most relevant, timely and accurate data as quickly as possible, while minimizing frustrating dead links. http://www.microsoft.com/presspass/press/2004/nov04/11-11SearchBetaLaunchPR.asp
• [Accoona] empowers users to find the most relevant information through its Artificial Intelligence powered search engine.
  http://biz.yahoo.com/bw/041206/66005_1.html
• A pioneer in Web search technology, Inktomi provides millions of users worldwide with the freshest and most relevant search experience, and ensures that thousands of online retailers and other sites have their content constantly represented.
  http://docs.yahoo.com/docs/pr/release1110.html
• [Mamma.com]…makes it easier and faster for people to find information by gathering the most relevant results from the best search engines on the Internet. http://www.mammamediasolutions.com/corporate/pr/2004/08-11-04.html
• Yahoo! Search offers one of the most extensive search services available to ensure consumers find the most relevant results to their search.
  http://docs.yahoo.com/docs/pr/release983.html

Search companies must be allowed to say their product is relevant. Otherwise, how can they market themselves? I am not disputing any of these company’s claims or passing judgment on their right to proclaim their search product as being relevant. The issue I am emphasizing is that these claims are subjective and need to be understood as such because they can not all be the most relevant, easiest, most extensive, and fastest.

Relevancy is like Pornography
So, with all that being said, what exactly is this elusive specter called relevancy, and how is it identified? It’s like the classic question: what is pornography? We don’t know how to define it, but we know it when we see it. Similarly, search engine relevancy is subjective and means something different to everyone. And not only that, it can mean different things to the same person at different times. A relevant result can be the site that provides the exact answer to a question; it can be the authority in its topic that provides a broad selection of information; or it might be a new site in a topic that is already known very well. That is why relevancy evaluations must be comparison-based. I know that the results on one engine are bad because another engine has better results. If the other engine did not exist, then my level of expectation would be lowered and it is possible that the first engine’s results would seem relevant to me. Underlying this notion is the notion that both engines pass a minimum threshold of relevancy. It is conceivable that the results could all be not relevant, in which case the comparison does not even come in to play.

Relevancy evaluation changes based on the types of information being sought. Each and every query for information needs to be reevaluated every single time. Users must never think that the results on their favorite engine are always the most relevant. If searchers do not find what they want, they can do a few things: they can come up with a new search strategy and approach the problem from a different angle; they can stick with the same strategy, but refine the query by making it narrower or broader, or by using advanced options and syntax; and, lastly, if the engine is still not finding what they are looking for, they can go elsewhere and try a different search tool.

Next Installment - Part 3: A Call to Arms

Part 1: Defining Relevancy

Relevancy

[Part 1 in a series of postings about relevancy.]

Relevancy is subjective. Each searcher will have a different evaluation of a search tool’s relevancy, and not only that, but each searcher will change that opinion based on the specific search being done. Search relevancy is a moving target that will never be agreed upon. Novice searchers should look to experts for advice, but in the end must reach their own conclusions about relevancy. Those conclusions must be based on using a few search engines, because relevancy is contextual and can only be understood as a comparison.

This is a concept that has been discussed by countless other information professionals, many of whom will say that defining relevancy is not constructive because of its subjectivity. I disagree. I think all serious searchers need to have their own definition of relevancy in order to make judgments about search results. After all, why do we use search engines? We use them to find information. We don’t use them to be impressed by clever features, a large index, or an intriguing name. We use them to find what we are looking for, and we can only find information if the results we get for our searches are relevant. And we can only decide if results are relevant if we have a simple framework for making that decision. Relevancy is the key and the foundation for search. Without relevancy, the rest is fluff. On an engine that has good relevancy, the features that are built around it become especially valuable. On an engine that has poor relevancy, the features are useless.

There are a slew of companies offering various takes on searching electronic sources. Some companies are searching sources such as databases, archives, and home computers, while on the Web there are general search engines, visual engines, clustering engines, natural language engines, and so forth. There are also specialty search tools - tabs or advanced search on general search engines - that focus on news, blogs, images, and so forth. It is great to have these tools, but none of the bells and whistles mean a thing if the results are not relevant. Without relevancy, users will not come back no matter how many special features are available. How often will I visit a restaurant with great atmosphere, but bad food? Not often.

Definitions of Search Engine Relevancy
With relevancy being such an important part of search, how is this elusive term defined when it comes to search engines? Here are my definitions. I am sure your definition will be different. Even if your definition is similar, when it comes to actually evaluating search results people will not always agree. Even if someone agrees with everything I say, we will still often disagree in our evaluation. I may think a result is relevant, when someone else thinks it is not relevant, and because of the subjective nature of relevancy evaluation we can both be right. So, with all those caveats let me present my definitions.

Relevancy: A measure of how well a search tool finds the information being sought.
[Sound too simple? Please, I welcome any other definitions because the more complicated I tried to make my definition, the more I just kept coming back to this simple sentence.]

To break it down further, I think of relevancy in terms of three levels or grades:

Relevant: the search result provides the information I am looking for. It is that simple.

Somewhat relevant: the result is close, and may even propel me along a path that leads to the information I am looking for, but it does not exactly have what I want. A somewhat relevant result is sometimes valuable because it suggests a different way and different terms for a search.

Not relevant: the site provides no help to me. It may contain the terms I searched for, but the context is wrong. It is that simple.

Next Installment - Part 2: The Jaded Surfer