spacer blank blank blank blank blank blank blank blank blank blank blank blank
blank
blank logo blank organization name organization name organization name organization name organization name organization name organization name organization name
blank blank
issues link to access link to privacy link to defamation link to divide blank blank
One Internet with Equal Access for All
spacer blank blank blank blank blank blank blank blank blank blank blank blank
blank home link to turing link to disabled link to elderly link to health link to race link to women link to youth link to action link to research
blank blank blank blank blank
blank
blank blank blank
link to news
blank
link to about
blank
link to Internet services
blank
link to volunteer or intern
blank
link to join / donate
blank
blank
    
blank
 
Bobby Approved (v 3.2)
blank Terms of Service, Other Policies, and
Copyright ©2000-2003
Online Policy Group, Inc.
 blank
blank blank blank
blank
blank

Online Policy Group Web Site Listed in Major Search Engines

OPG logo
April 14, 2001 - The Online Policy Group web site was submitted to all the major search engines and web directories before the end of July 2000. It took two months, until September 27, 2000, to get listed in a single one of them. This article explores how this access mechanism is set up on the web and how site creators may increase access to their web sites.

Introduction
Search Results
Analysis
Ratings
Conclusions
Additional Resources
Notes

Introduction

Examining how the search results at various search engines and web directories change over time gives some insight into how web sites get access to this critical web resource. For, if a web site is not listed in search results, then only people who get a direct referral -- by word of mouth, by email advertisement, by links from other web sites, or through media reports -- will ever get access to that site.

For this purpose, we use the anecdotal example of this web site, the web site for the Online Policy Group.

Search Results

Here is a summary of the daily search results we obtained listing the percentage of search engines and web directories that listed, directly or indirectly, the OPG web site. Specific daily search results are available by clicking the link for a particular date.

Date 24Sep00 25Sep00 26Sep00 27Sep00 28Sep00 29Sep00 30Sep00
% NA NA NA 14 25 25 NA
Date 01Oct00 02Oct00 03Oct00 04Oct00 05Oct00 06Oct00 07Oct00
% 25 25 NA 32 32 36 39
Date 08Oct00 09Oct00 10Oct00 11Oct00 12Oct00 13Oct00 14Oct00
% 39 39 39 39 43 39 32
Date 15Oct00 16Oct00 17Oct00 18Oct00 19Oct00 20Oct00 21Oct00
% 32 36 36 43 43 43 50
Date 22Oct00 23Oct00 24Oct00 25Oct00 26Oct00 27Oct00 28Oct00
% 50 50 54 54 54 54 NA
Date 29Oct00 30Oct00 31Oct00 01Nov00 02Nov00 03Nov00 04Nov00
% 54 50 54 54 54 50 54
Date 05Nov00 06Nov00 07Nov00 08Nov00 09Nov00 10Nov00 11Nov00
% 54 50 54 54 50 54 54
Date 12Nov00 13Nov00 14Nov00 15Nov00 16Nov00 17Nov00 18Nov00
% 54 54 54 54 54 54 54
Date 19Nov00 20Nov00 21Nov00 22Nov00 23Nov00 24Nov00 25Nov00
% 54 54 NA 54 57 57 57
Date 26Nov00 27Nov00 28Nov00 29Nov00 30Nov00 01Dec00 02Dec00
% 73 57 57 57 57 64 64
Date 03Dec00 04Dec00 05Dec00 06Dec00 07Dec00 08Dec00 09Dec00
% 64 61 64 64 61 61 61
Date 10Dec00 11Dec00 12Dec00 13Dec00 14Dec00 15Dec00 16Dec00
% 64 61 64 64 64 61 64
Date 17Dec00 18Dec00 19Dec00 20Dec00 21Dec00 22Dec00 23Dec00
% 68 68 64 64 54 61 64
Date 24Dec00 25Dec00 26Dec00 27Dec00 28Dec00 29Dec00 30Dec00
% 71 NA NA NA NA NA NA
Date 31Dec00 01Jan01 02Jan01 03Jan01 04Jan01 05Jan01 06Jan01
% NA NA 75 71 68 68 NA
Date 07Jan01 08Jan01 09Jan01 10Jan01 11Jan01 12Jan01 13Jan01
% 61 64 61 NA NA 61 61
Date 14Jan01 15Jan01 16Jan01 17Jan01 18Jan01 19Jan01 20Jan01
% 57 61 61 64 61 61 61
Date 21Jan01 22Jan01 23Jan01 24Jan01 25Jan01 26Jan01 27Jan01
% 61 NA 61 61 61 61 NA
Date 28Jan01 29Jan01 30Jan01 31Jan01 01Feb01 02Feb01 03Feb01
% 61 61 61 61 NA 61 64
Date 04Feb01 05Feb01 06Feb01 07Feb01 08Feb01 09Feb01 10Feb01
% 61 NA NA 61 NA 61 64
Date 11Feb01 12Feb01 13Feb01 14Feb01 15Feb01 16Feb01 17Feb01
% 61 61 NA NA NA NA NA
Date 18Feb01 19Feb01 20Feb01 21Feb01 22Feb01 23Feb01 24Feb01
% NA NA 64 NA 68 68 NA
Date 25Feb01 26Feb01 27Feb01 28Feb01 01Mar01 02Mar01 03Mar01
% 68 68 71 71 71 71 71
Date 04Mar01 05Mar01 06Mar01 07Mar01 08Mar01 09Mar01 10Mar01
% NA 71 68 NA NA NA 68
Date 11Mar01 12Mar01 13Mar01 14Mar01 15Mar01 16Mar01 17Mar01
% NA 68 NA 68 68 68 71
Date 18Mar01 19Mar01 20Mar01 21Mar01 22Mar01 23Mar01 24Mar01
% 68 68 68 68 68 68 68
Date 25Mar01 26Mar01 27Mar01 28Mar01 29Mar01 30Mar01 31Mar01
% 64 68 71 71 NA NA NA
Date 01Apr01 02Apr01 03Apr01 04Apr01 05Apr01 06Apr01 07Apr01
% NA 71 71 71 NA NA 75
Date 08Apr01 09Apr01 10Apr01 11Apr01 12Apr01 13Apr01 14Apr01
% 75 75 NA NA NA NA 75

For more detail, see the specific daily search results we obtained.

Analysis

How Search Engines Share Listings

Most of the major search engines almost certainly listed the OPG site because of a listing in the Open Directory Project (ODP) where the OPG site appears as the first and only result for a search on "online policy group". ODP's listing was reproduced on AOL, CNET Search.com, Google, Netscape, and Yahoo. The entire ODP listings are reproduced wholesale on a variety of sites, including AOL, the Dutch gate99, Oingo.com, and Webhideout.com, resulting in indirect search results as well as direct ones.

About.com and iWon.com often appear to have exactly the same listings. AltaVista appears to have an entirely separate listing which was reproduced in the Ask Jeeves, CNET Search.com, MetaCrawler, and Raging Search search engines. Google's listing is sometimes reproduced in MetaCrawler results. Infoseek's listing was reproduced on the Mamma.com and MetaCrawler search engines. Lycos' listing was reproduced on CNET Search.com. NBCi's listing was reproduced on the CNET Search.com and Mamma.com search engines.

An Indirect Result Is Better Than No Result At All

Sites that listed stories about OPG, whether or not on the their front pages, such as the ACLU Cyber-Rights Project, the Free Expression Network, Queery.com, and RainbowQuery.com, helped to produce additional indirect search results for OPG, particularly on AOL and NorthernLight search engines and on the Yahoo web directory.

Search Engines That Don't Produce Results

As of the date of this report, there are still no listings for OPG in the top 10 search results at GoTo.com, Looksmart, MSN, and WebCrawler.

History of Search Engine Submissions

After the initial submissions to most of the search engines in July, OPG waited two months for the listings to start appearing. On October 2-3, 2000, OPG submitted the site address to DirectHit, Excite, FAST Search, HotBot, NBCi, and WebCrawler. DirectHit said it would take several weeks or longer to get listed. FAST Search listed the OPG site starting December 1, 2000, and identical listings started appearing on Lycos that same day. HotBot said the site would be added within 60 days to only the Inktomi database, and also gave the opportunity to add sites to the Open Directory Project listings under each specific topic category, although it did not show the OPG site listed where it had already been added to the OPD listings, implying that HotBot only synchronizes with the OPD listings on a relatively infrequent basis. NBCi requires users to register before submitting a site and claimed the site would be listed within 24 hours... and they appear to have done as they claimed as of October 4, 2000. On October 6, 2000, OPG submitted the site to Thunderstone's web crawler and Thunderstone began listing OPG as of October 18, 2000. On October 7, 2000, OPG submitted the site to Go.com (i.e. Infoseek) and the site was listed as of November 23, 2000. On October 16, OPG submitted the site to the free non-profit listing on Looksmart, which was found through a search on iWon.com. It took three months for Yahoo to list the OPG site, despite Yahoo's partnership with Google. It took more than four months for Excite to list the OPG site.

Search Engines You Have to Pay for Results

OPG declined to pay to have the OPG site listed on About.com, FindWhat.com, GoTo.com, and Kanoodle, where there was no mechanism to list free of charge, and on WebCrawler, which offered a LookSmart Network listing for $199 in partnership with Excite, or a possible Excite-only listing free of charge. Strangely, FindWhat.com started listing an OPG result on December 16, 2000, with no payment from OPG, then promptly stopped again on December 18, 2000. Similarly, Kanoodle started listing an OPG result on December 18, 2000, with no payment from OPG.

Ratings

Disclaimer
These search engine and web directory ratings are based on a very limited sample of the experience of submitting the Online Policy Group web site to the list of search engines and web directories mentioned below. We submitted the Online Policy Group web site where possible to do so without payment of a fee and we tested the search engines and web directories on a near-daily basis with one simple search on "Online Policy Group" (using a double-quoted or phrase search whenever possible). Nonetheless, the results may prove useful to readers who are submitting their web sites to search engines and web directories for the first time.

Ratings are from most to least desirable: "Hi", "Med" and "Lo", with "-" indicating not applicable.
Engine / Directory Getting Listed Result Ranking Staying Listed Others Listings Indirect Listings Ref by Others Budget Conscious
About.com (aka Sprinks) Med Hi Med Lo Hi Lo Hi
AltaVista Hi Med Hi Lo Hi Hi Hi
AOL Med Hi Hi Med Med Lo Hi
Ask Jeeves Lo Lo Lo Med Lo Med Hi
CNET Search.com Med Hi Med Hi Med Lo Hi
Direct Hit (associated with Ask Jeeves) Lo Hi Lo Med Lo Lo Hi
Excite Lo Hi - Lo Med Lo Med
FAST Search (formerly AllTheWeb) Med Hi Hi Lo Med Lo Hi
FindWhat.com (powered by Inktomi) Lo Lo Lo Lo Lo Lo Med
Go.com (powered by Infoseek) Med Hi Hi Lo Med Lo Hi
Engine / Directory Getting Listed Result Ranking Staying Listed Others Listings Indirect Listings Ref by Others Budget Conscious
Google Med Hi Med Hi Hi Med Hi
GoTo.com Lo - - - - - Lo
HotBot Lo Lo Lo Lo Med Lo Hi
iWon.com (powered by Inktomi) Med Hi Med Lo Hi Lo Med
Kanoodle Lo Hi Lo Lo Lo Lo Med
Looksmart Lo - - - - - Med
Lycos Med Hi Lo Hi Med Lo Hi
Mamma.com Med Hi Med Hi Lo Lo Hi
MetaCrawler Med Hi Med Hi Lo Lo Hi
MSN Lo - - - - - -
Engine / Directory Getting Listed Result Ranking Staying Listed Others Listings Indirect Listings Ref by Others Budget Conscious
NBCi (Snap) Hi Hi Hi Lo Med Med Hi
Netscape Med Hi Med Hi Lo Lo Hi
NorthernLight Med Hi Hi Lo Hi Lo Hi
Open Directory Project Hi Hi Hi Lo Lo Hi Hi
Raging Search Med Hi Hi Hi Hi Lo Hi
Thunderstone Hi Hi Hi Lo Lo Lo Hi
WebCrawler Lo - - - - - Med
Yahoo! Med Hi Hi Lo Hi Lo Hi
Engine / Directory Getting Listed Result Ranking Staying Listed Others Listings Indirect Listings Ref by Others Budget Conscious

Conclusions

Here are the conclusions one might reasonably draw from this example:

  • Nothing can replace the painstaking process of individual submissions to the various search engines and web directories, but prioritizing submissions to the ones that provide information to other engines, especially the Open Directory Project and AltaVista, will maximize the listings for a site with a minimum effort.
  • If a site manages to get a link listed on other sites that are indexed by a search engine, then those link sites will appear in search engine results permitting searchers to gain access indirectly to the original site.
  • As a result of the growing commercialism of the Internet, you may pay to promote a site and ensure listings of the site in more search engines. It is not clear how effective these paid listings are since many people seek out search engines that are not so commercially biased.

Additional Resources

For information about issues related to search engines and web directories, see the Search Engines section. For information on commercialization of search engines, see the Online Commercialism section.

Search Engines and Web Directories List
An excellent annotated list of search engines and web directories from SearchEngineWatch.

top of page

blank
blank