What makes "Search Engines" like Yahoo, AOL Netfind, Lycos and many others tick?

"Search Engine" is a generic term commonly used to identify Web Sites that match users searching for information to a list of Web Sites that may meet the user's needs. While all "Search Engines" perform the same function, the methods they use to create their Web Site lists (listings) varies greatly. Some "Search Engines" use employees to review and build Web Site listings. Others generate their listings using computer programs, commonly refered to as robots. Some like Yahoo incorporate both.

Employee Generated Listings Overview

Search Engines that utilize employees accept or deny requests to add new Web Sites, generally submitted from web site administrators. The requests contain information about the site and suggestions on how the requester would like for the site to appear in the Search Engine's listings. Reviewers generally verify that Web Sites meet the "Search Engines" standards and place approved sites into categories. End users looking for information move through the categories like (Entertainment/Music/Rock and Roll) looking for web sites to meet their needs.

Robot Generated Listings

Search engines that utilize robots also accept requests to add new web pages to their listings. These requests are then passed into the robot. The robot visits the site, reads the pages, and follows all the page links and stores the information it finds in the search engines database.

Rather than browsing through categories, users looking for information through this type of search engine enter search strings. The search engine then takes the search string and tries to match the it to web site information stored the database earlier by the robot. Web Site matches are returned to the user in the order that the search engine perceives to be most relevant.

Stated another way, what the robots find when they visit your site determines if and how your site will appear when users perform searches related to your company. It is critical to understand stand how the different robots read and store information in order to utilize their features to your companies advantage.

What Robots do and don't understand

To begin with robots are similar to very older web browsers and can only understand a limited amount of the possible information that many web sites contain. For instance robots don't understand pages generated using tools like cgi scripts, database scripts, or shockwave. They may not read image maps. The use of frames may either prevent the search engine from finding pages within a web site, or it may cause the search engine to send visitors into a site without the proper frame "context" being established.

If the robot can't understand the web page it visits or only understands the wrong piece of it, the page won't be stored in the search engines database or won't be found by users trying to find the products and services your company provides.

Does all this mean that a web site has to be extremely simple so that search engines can read and record it?

No, but it does mean that the web designer needs to anticipate these problems and account for them.

Once you make your site readable by the robot it is important to give the robot the information you want it to have. Unfortunately you can't just fill out a form to tell the robot about your site, it reads tags both visible and behind the scenes to determine for it's self what information your site contains.

Have you ever performed a search for something like barbecue and gotten back stuff like this:

Bad Cat Fall Fling. October 12, 1997. Cadet. 103 lbs. David Wason (BloomingWale) Steve Truso (Hillsboro) 112 lbs. Jeffrey MaCava (WCW) Frederic Gevao...

Not likely many will click on this site over any other is it. This web site designer failed to design the site so that it spoon feed the robot quality information about the sites content. This site may be the greatest looking and most useful site ever created but it won't be of much use if no one sees it.

As important as it is give search engines the information you want displayed it is even more important to make sure that the robot has stored an accurate representation of your pages content. If the search engine holds information about your "bed and breakfast" site that causes it to come up on the search results lists of people interested in "eating breakfast in bed" it won't do your Bed and Breakfast much good. What's even worse is people that want your service won't be able to find you.

Robots collect the information contained in your sites <TITLE>, <BODY>, <META> tags and use it along with a few other factors to determine if your site matches a users search criteria and what information is displayed to the user performing the search.

More Info

This was designed as a very basic overview. There are many search engines and directories, each with it's own specific characteristics. If you're interested in learning more and have some time on your hands, the website Searchenginewatch is a great source for more indept info.

Rev 2.0 MJB/MJH Feb 2004