Sources Of Search Information

When searching the net today, results are derived from some combination of the following crawlers and directories:

• Google - The thousand pound gorilla in todays search landscape, Google employs impressive crawler-based technology which allows it to provide high quality search results to both end users and other search services. Throughout its early growth period, Google has managed to keep its focus on enhancing search results. Read more about the Google search technology.
• Open Directory Project (ODP) - The Open Directory is just that: a directory of listings built and maintained by volunteers, in a completely 'open' fashion. While submissions are free, the workload on volunteers has increased exponentially, making the time it takes to actually get a listing approved a bit extreme. Still, it's definitely worth being included, since this directory affects a good portion of search results by feeding over a half dozen services, including Google...
• Yahoo - The grand-daddy of directories. In the early years, Yahoo employed humans to partition the web into a hierarchy of categories and subcategories. This changed at the end of 2002, when Yahoo shifted to crawler-based results provided by Google. By the beginning of 2004, Yahoo had replaced out the Google results with those of its own technology. While commercial sites must pay a fee to be included in the directory, non-commercial sites can still submit for free.
• Teoma - In looking for new ways to boost the relevancy of search results, Teoma has introduced what they call "Subject-Specific Popularity", where the ranking of a page is based not on the number of links which refer back to it, but rather on the number of same-subject pages which contain a link which refers back to it. In this way, Teoma (which means "expert" in Gaelic) claims a more finely tuned search result based on a natural "clustering" of communities and topics.
• Overture - Aquired by Yahoo in October 2003, Overture offers marketing and paid inclusion services to online companies.
• Gigablast - Tiny when compared to the likes of Google and Yahoo, Gigablast serves about a half million queries a day from an index of about 200 million pages. The brainchild of Matt Wells, it's the only major search engine to do index updates and refreshing in real-time.
• MSN - The new crawler on the block, Microsoft promises to become a player, and may end up giving Google a run for its money. Stay tuned...

For a look into the latest world of search content providers and the engines they feed, refer to the Search Engine Decoder page. It's a great resource for keeping up with the ever-changing search landscape; simply select an engine to see the relationships that currently exist for that particular engine.

Remember that prior to submitting a site, one should verify that pages have been "optimized" to ensure they receive the best possible ranking. We can help with both the optimization and submission of your site; contact us to learn more.

Sources Of Search Information

The links given below represent some of the more popular search resources available at this time. For a comprehensive list of spiders, robots, crawlers, and browsers, refer to the user-agents page...