What can search engines could do better?

One person responded:

* …I’ll do a search and get results with just one of the key words in it…e.g. if I did a search on Tamil Tigers, I would probably get results with just the word Tigers, which I’m not at all interested in.

Comments: partial phrase matching is indeed an issue with search engines. Quoting searches can help with this, but can also overly limit the result set. Proximity can be a problem, as in pages that mention Tamil and Tigers, but they don’t mention them connected as a phrase (probably unlikely for this example, but you get the point). Phrase matching is really the key to relevancy. For most one-word queries it’s either easy enough to assume what the user is searching for (eBay), or ambiguous enough (jaguar) that a breadth of results is appropriate. But with a multi-term query you get into the world of concept matching, as in when a web site defines an entity as Navy Blue Shirt, but the searcher searches for Dark Blue Shirt.

But maybe the bigger challenge is the challenge of “small mentions". These days, with a two-word query like Tamil Tigers I think most results will match on both terms. However, what about pages that make only a passing reference to the term?

Another respondent wrote:

* Update blog searches quicker.

Comments: most blog search engines are near real-time right now, though I guess “near” is not real-time. I notice a lag sometimes between when I publish a posting on the Search Lounge and when it gets pushed to my RSS feeds.

What can search engines could do better?

* Perhaps be better at responding to articulated queries, i.e. understand the semantics of a plain English query.

Comments: Natural language, one of search’s holy grails. Rather than building out natural language systems, most major search engines rely on users to learn a new syntax: the language of keywords.

Generations from now people will speak a different language because of this. Instead of whole sentences people will use phrases. Instead of saying to a friend, “What is the largest city in Brazil?", people will say “Brazil largest city.” Or instead of saying at a cocktail party, “What are the ingredients in a Sazerac?” people will say, “Sazerac ingredients list?”

What can search engines could do better?

**Official Sites**

The respondent wrote:

*Always list the official site first. (So if I do a search for Nabisco I want nabisco.com listed first, not several articles about Nabisco).

Comments: A few years ago search engines weren’t very good at always getting the official page in the first position, but now they generally are. But what if users click more often on the non-official page because the official page isn’t the best source?

**Language Settings**

The respondent wrote:

* Ask me for the languages I am able to comprehend since links that are mostly written in Czech, Chinese or Icelandic do not really help me.

Comments: most engines do let users set this preference, but it’s definitely not foolproof. Results in other languages still show up often enough to be noticeable.

What can search engines could do better?

A few people wrote they want search engines to explain more about how they work. Although I understand this sentiment, do people really want that?. What I’m thinking they want is to get better results; and if they’re not getting good results they think maybe they’re doing something wrong, but they don’t know what. So rather than search engines going into the minute details of ranking algorithms and so forth, when users don’t get good search results the engines should step in and offer ways to improve their search.

But I think this issue addresses a serious topic. Namely, search engines are really complex systems, but people want to be able to enter a couple of words in the search box and get exactly what they want. How do search engines balance on the tight line between sharing information on how search works, while not overwhelming people with too much info?

Here are some of the responses on this topic:

*Offer more tips on searching.
*…providing me with a better understanding of how they [search engines] use the inputs to conduct a search would help. This info is no doubt already available, but I’m usually in convenience mode when doing searches and unless this info is up front, I usually won’t take the time to go searching for it.
* What I wish there was more (maybe there already is and i don’t know how to find it) is a “how to…” search engine. In my case, for arts and crafts, but it could be for anything else.
*Provide more intuitive search refinements than boolean language, such as dropdown menus and check boxes with refinements explained in common English.

What can search engines could do better?

I got 12 real responses to this question, which is more than I expected since it’s a tough question.
I’m going to break out the answers into separate, small posts, starting with…

*Add direct reference data. e.g. “when is labor day?” pulls up result: “September 5, 2005″ .

I love it when the SERP answers my question and I don’t have to click through to the page! Like the way Brainboost does it. Engines are experimenting more and more with this concept, like Instant Search.

When you do a search and are not satisfied with the results, what’s your next step?

8 out of 21 respondents said they try their searches on another engine when they don’t find what they want. But only 2 said that’s the only thing they do. Most people said they reformulate their query first, and then as a last resort go to another engine.

Here’s some examples:

*Alter the wording then use a different engine.
*Put different or varying words in search field (I rarely use a different search engine).
* Fine-tuning the query. I usually don’t go to another search engine until I’ve exhausted a number of possible key search words.

One respondent explained how, after unsuccessfully issuing a search on a search engine, they go directly to expert sites and look around for links to the sought-after information.

Another person said if he doesn’t find what he wants using a search engine that he’ll use a professional database (imagine that!).

It’s good that people reformulate their queries. But I’m curious to know what strategies people use other than quoting the search terms, or if they just keep hacking away hoping to find the magic keyword combination.

As to the people who don’t use other engines, or only use them as a last resort, that’s troubling. If I go to a video store and they don’t have a DVD I’m looking for, I don’t stop my search and assume it doesn’t exist. I’ll go to another store to find it. Or a book in a bookstore, or a shirt in a clothing store, etc. The same holds true for search engines.

When you do a search and are not satisfied with the results, what’s your next step?

The majority of respondents, 11 people, said they reformulate their query using the same engine. Most people do this by using quotes, but others use narrower terms. Some people wrote that they change their keywords, but I don’t know if that means narrower terms, broader terms, related terms, or what exactly.

Specific examples of what people wrote include the following:

* Refine the search, change key words, use/remove quotation marks.
* I narrow it by putting the words or phrases in quotes. Sometimes, I’ll choose a specific language or make sure I only get results in English (depending on what I’m looking for).
* I will usually try a different search with different words, perhaps a different order.
* Fine-tuning the query. I usually don’t go to another search engine until I’ve exhausted a number of possible key search words.
* Keep trying using different search phrases / narrow or expand the search, depending.

5 people specifically mention using quotes in their reformulation.

One person wrote this surprising response: This happens vary rarely. I suppose I just give up.

I’m not going to get all scientific about the demographics, but here’s a brief overview:

- None of the respondents have ever worked in the search industry.
- Most respondents are in their early to mid 30s.
- All respondents were college educated.
- 47% of the respondents have advanced degrees.
- 59% of the respondents live in the San Francisco Bay Area.

Oh, and I sent the survey to about 30 people, and received 18 responses to my 3 survey questions.

UPDATE: I now have 20 responses.

Question #1: When you need to conduct a search, what are the most important factors for you in determining which search engine to use?

Relevancy was not really that big of a stated factor in the responses to this question. Only one respondent specifically used the word relevancy, and it was only one of the factors. Here’s their answer: “Accessibility, ease of use, and relevancy.”

Although this other response doesn’t specifically use the word relevancy, the unspoken meaning is relevancy:
“It is important that the results really reflect the issues I am looking for and do not come up with all kinds of results that just mention the key words as a side matter…”

One other response was specific in terms of the type of information being sought: “If I am searching for a particular item, like a product web site, movie trailer, or phone number, I value conveninece and accuracy.”

I think the main issue here is that perceived relevance is what’s important to a lot of users. They don’t spend much time or thought exploring various search engines beyond what they already know, and so the answers to this question were generally vague. There just really are not many factors that go into most people’s decision making process on this topc. Most people don’t decide to use a particular engine for a particular type of query. They just stick with the engine they know.

Question #1: When you need to conduct a search, what are the most important factors for you in determining which search engine to use?

I purposely didn’t ask for the names of search engines that people use, yet I found out the following:

6 people only use Google.
6 different people use Yahoo and/or Google.
2 people sometimes use Dogpile.
6 people did not mention a particular engine.

A lot of people didn’t really have a reason for why they use Google, for example:
“I have no technical criteria - I use google for one-stop shopping. Brand familiarity and nothing more.”
“I go to Google. I do not consider factors.”
“Uh, no real determinants, as I always use Google.”
“Honestly, I only use Google, and not for any particular factor, except that I simply don’t have time to compare
different search engines, so I go with the default.”

One of the Dogpile users gave this insightful response:
“…If, however, I’m not looking for a specific item, and instead am trying to conduct research, I often won’t know what it is I’m looking for until I’ve found it. For these types of searches I value search engines like dogpile.com that often provide less focused searches, but that often turn up some overlooked gems.”

One respondent, who did not mention any engine names, put this very telling line: “Whatever is on my screen.”

I decided it would be interesting to ask my friends who don’t work in the search engine industry for their opinions about search, so I sent out the following email:

I’m putting together a qualitative and non-scientific survey. If you have a few minutes I’d appreciate your input. I plan to write up the findings for the Search Lounge. All references to comments will be anonymous.

(Note that I have purposefully left the questions open-ended to allow for personal interpretation.)

1. When you need to conduct a search, what are the most important factors for you in determining which search engine to use?

2. If you do a search and are not satisfied with the results, what’s your next step?

3. What can search engines could do better?

UPDATE: To read the results of the survey, check out the The Search Survey category.

Search goes like this:

1. CREATE. A new medium becomes popular.
2. SHARE. People want to find and share objects created in the new medium.
3. ORGANIZE. Manual systems are created to organize the new medium.
4. SEARCH. Manual systems become stale and can’t scale quickly enough so automated systems are built.

Looking at an example, let’s start with web search:

1. CREATE. Lots of people started creating web pages.
2. SHARE. Users tried to find information on the web.
3. ORGANIZE. Web directories were built and became popular.
4. SEARCH. Web directories go out of fashion as search engines improve and scale.
What’s next? Some people think personalization and customization, but I think we’re still working on #4.

How about another example, image search:

1. CREATE. Digital photos become popular.
2. SHARE. People want to share their photos.
3. ORGANIZE. Tagging systems like flickr become popular for organizing photos.
4. SEARCH. Search engines like Yahoo and Google search text associated with photos. What’s next for automated ways to search images?

One more example:

1. CREATE. The rise of the bloggers.
2. SHARE. Bloggers want readers.
3. ORGANIZE. People build their own blog rolls. Also blog directories are built.
4. SEARCH. Blog (RSS) search engines like Technorati do real-time indexing.

I’m anxious to see how the cycle works with video content.

Internet Librarian’s 2005 conference is coming up soon and I’m hoping to attend. It’s October 24-26 in Monterey, California. I attended last year’s conference and really enjoyed hearing people like Gary Price, Chris Sherman, Raul Valdes-Perez, Greg Notess, Rita Vine, Mary Ellen Bates and other prominent librarians and information professionals who are involved in search. I also got to reconnect with former colleagues and former library school classmates.

This year’s program again features a couple of search related tracks. On Monday, October 24th, Track A is called Information Discovery & Search. Be sure to check out Mary Ellen Bates’ Thirty Search Tips in 40 Minutes.

On Wednesday, October 26th, Track A is called Search Engines (SEs) & Beyond. It features some of the people I’ve named above as well as others such as R.J. Pittman, CEO of Groxis. I had a chance to meet him earlier this year and was impressed by his vision of clustering and visual search.

Last year’s event had a frenetic energey about blogging. As much as I’m a fan of blogging, hopefully this year’s event will not be quite so blog-heavy. I say that only because there are so many other topics to cover and I think by now most Internet Librarian attendees will have a good understanding of blogging.

My only complaint about this year’s program is about the keynote addresses. Out of five keynote addresses (including one evening session), three of them are about Google. Seems a bit Google-heavy to me, but maybe the speakers think that having Google’s name in the title of their talks will draw more attendees.

If you’re headed to IL this year, drop me a line and let me know you’ll be there.