Notes
Introduction
Examining how the search results at various
search engines and web directories change over time gives some insight
into how web sites get access to this critical web resource.
For, if a web site is not listed in search results, then only
people who get a direct referral -- by word of mouth, by email
advertisement, by links from other web sites, or through media reports --
will ever get access to that site.
For this purpose, we use the anecdotal example of this web site,
the web site for the Online Policy Group.
Search Results
Here is a summary of the daily search results we obtained listing
the percentage of search engines and web directories that listed,
directly or indirectly, the OPG web site. Specific daily search
results are available by clicking the link for a particular date.
For more detail, see the specific
daily search results
we obtained.
Analysis
How Search Engines Share Listings
Most of the major search engines almost certainly listed
the OPG site because of a listing in the
Open Directory Project (ODP)
where the OPG site appears as the first and only result for
a search on "online policy group".
ODP's listing was reproduced on AOL, CNET Search.com,
Google, Netscape, and Yahoo.
The entire ODP listings are reproduced
wholesale on a variety of sites, including AOL, the Dutch
gate99, Oingo.com, and Webhideout.com,
resulting in indirect search results as well as direct ones.
About.com and iWon.com often appear to have exactly the
same listings.
AltaVista appears to have an entirely separate listing which
was reproduced in the
Ask Jeeves, CNET Search.com, MetaCrawler,
and Raging Search search engines.
Google's listing is sometimes reproduced in
MetaCrawler results.
Infoseek's listing was reproduced on the
Mamma.com and MetaCrawler search engines.
Lycos' listing was reproduced on
CNET Search.com.
NBCi's listing was reproduced on the
CNET Search.com and Mamma.com search engines.
An Indirect Result Is Better Than No Result At All
Sites that listed stories about OPG, whether or not on the their front
pages,
such as the ACLU Cyber-Rights Project, the Free Expression Network, Queery.com, and RainbowQuery.com,
helped to
produce additional indirect search results for OPG, particularly
on AOL and NorthernLight search engines and on the Yahoo web directory.
Search Engines That Don't Produce Results
As of the date of this report, there are still no listings for
OPG in the top 10 search results at
GoTo.com,
Looksmart,
MSN,
and
WebCrawler.
History of Search Engine Submissions
After the initial submissions to most of the search engines
in July, OPG waited two months for the listings to start appearing.
On October 2-3, 2000, OPG submitted the site address to
DirectHit, Excite, FAST Search, HotBot, NBCi, and WebCrawler.
DirectHit said it would take several weeks or longer to get listed.
FAST Search listed the OPG site starting December 1, 2000, and
identical listings started appearing on Lycos that same day.
HotBot said the site would be added within 60 days to only the
Inktomi database, and also gave the opportunity to add sites to the
Open Directory Project listings under each specific topic category,
although it did not show the OPG site listed where it had already
been added to the OPD listings, implying that HotBot only synchronizes with
the OPD listings on a relatively infrequent basis.
NBCi requires users to register before submitting a site and claimed
the site would be listed within 24 hours... and they appear to have
done as they claimed as of October 4, 2000. On October 6, 2000, OPG
submitted the site to Thunderstone's web crawler and Thunderstone began
listing OPG as of October 18, 2000. On October 7, 2000,
OPG submitted the site to Go.com (i.e. Infoseek) and the site was
listed as of November 23, 2000. On October 16, OPG
submitted the site to the free non-profit listing on Looksmart, which
was found through a search on iWon.com.
It took three months for Yahoo to list the OPG site, despite Yahoo's
partnership with Google. It took more than four months for Excite to
list the OPG site.
Search Engines You Have to Pay for Results
OPG declined to pay to have the OPG site listed on About.com,
FindWhat.com, GoTo.com, and Kanoodle, where there
was no mechanism to list free of charge, and on WebCrawler, which offered
a LookSmart Network listing for $199 in partnership with Excite, or a
possible Excite-only listing free of charge.
Strangely, FindWhat.com
started listing an OPG result on December 16, 2000, with no payment
from OPG,
then promptly stopped again on December 18, 2000.
Similarly, Kanoodle
started listing an OPG result on December 18, 2000, with no payment
from OPG.
Ratings
Disclaimer
These search engine and web directory ratings are based on a very
limited sample of the experience of submitting the Online Policy
Group web site to the list of search engines and web directories
mentioned below. We submitted the Online Policy Group web site where
possible to do so without payment of a fee and we tested the search
engines and web directories on a near-daily basis with one simple
search on "Online Policy Group" (using a double-quoted or phrase search
whenever possible). Nonetheless, the results may prove useful to readers
who are submitting their web sites to search engines and web
directories for the first time.
Ratings are from most to least desirable: "Hi", "Med" and "Lo",
with "-" indicating not applicable.
Engine /
Directory
|
Getting Listed
|
Result Ranking
|
Staying Listed
|
Others Listings
|
Indirect Listings
|
Ref by Others
|
Budget Conscious
|
About.com (aka Sprinks)
|
Med
|
Hi
|
Med
|
Lo
|
Hi
|
Lo
|
Hi
|
AltaVista |
Hi
|
Med
|
Hi
|
Lo
|
Hi
|
Hi
|
Hi
|
AOL
|
Med
|
Hi
|
Hi
|
Med
|
Med
|
Lo
|
Hi
|
Ask Jeeves
|
Lo
|
Lo
|
Lo
|
Med
|
Lo
|
Med
|
Hi
|
CNET Search.com
|
Med
|
Hi
|
Med
|
Hi
|
Med
|
Lo
|
Hi
|
Direct Hit (associated with Ask Jeeves)
|
Lo
|
Hi
|
Lo
|
Med
|
Lo
|
Lo
|
Hi
|
Excite
|
Lo
|
Hi
|
-
|
Lo
|
Med
|
Lo
|
Med
|
FAST Search (formerly AllTheWeb)
|
Med
|
Hi
|
Hi
|
Lo
|
Med
|
Lo
|
Hi
|
FindWhat.com (powered by Inktomi)
|
Lo
|
Lo
|
Lo
|
Lo
|
Lo
|
Lo
|
Med
|
Go.com (powered by Infoseek)
|
Med
|
Hi
|
Hi
|
Lo
|
Med
|
Lo
|
Hi
|
Engine /
Directory
|
Getting Listed
|
Result Ranking
|
Staying Listed
|
Others Listings
|
Indirect Listings
|
Ref by Others
|
Budget Conscious
|
Google
|
Med
|
Hi
|
Med
|
Hi
|
Hi
|
Med
|
Hi
|
GoTo.com
|
Lo
|
-
|
-
|
-
|
-
|
-
|
Lo
|
HotBot
|
Lo
|
Lo
|
Lo
|
Lo
|
Med
|
Lo
|
Hi
|
iWon.com (powered by Inktomi)
|
Med
|
Hi
|
Med
|
Lo
|
Hi
|
Lo
|
Med
|
Kanoodle
|
Lo
|
Hi
|
Lo
|
Lo
|
Lo
|
Lo
|
Med
|
Looksmart
|
Lo
|
-
|
-
|
-
|
-
|
-
|
Med
|
Lycos
|
Med
|
Hi
|
Lo
|
Hi
|
Med
|
Lo
|
Hi
|
Mamma.com
|
Med
|
Hi
|
Med
|
Hi
|
Lo
|
Lo
|
Hi
|
MetaCrawler
|
Med
|
Hi
|
Med
|
Hi
|
Lo
|
Lo
|
Hi
|
MSN
|
Lo
|
-
|
-
|
-
|
-
|
-
|
-
|
Engine /
Directory
|
Getting Listed
|
Result Ranking
|
Staying Listed
|
Others Listings
|
Indirect Listings
|
Ref by Others
|
Budget Conscious
|
NBCi (Snap)
|
Hi
|
Hi
|
Hi
|
Lo
|
Med
|
Med
|
Hi
|
Netscape
|
Med
|
Hi
|
Med
|
Hi
|
Lo
|
Lo
|
Hi
|
NorthernLight
|
Med
|
Hi
|
Hi
|
Lo
|
Hi
|
Lo
|
Hi
|
Open Directory Project
|
Hi
|
Hi
|
Hi
|
Lo
|
Lo
|
Hi
|
Hi
|
Raging Search
|
Med
|
Hi
|
Hi
|
Hi
|
Hi
|
Lo
|
Hi
|
Thunderstone
|
Hi
|
Hi
|
Hi
|
Lo
|
Lo
|
Lo
|
Hi
|
WebCrawler
|
Lo
|
-
|
-
|
-
|
-
|
-
|
Med
|
Yahoo!
|
Med
|
Hi
|
Hi
|
Lo
|
Hi
|
Lo
|
Hi
|
Engine /
Directory
|
Getting Listed
|
Result Ranking
|
Staying Listed
|
Others Listings
|
Indirect Listings
|
Ref by Others
|
Budget Conscious
|
Conclusions
Here are the conclusions one might reasonably draw from this
example:
-
Nothing can replace the painstaking process of individual
submissions to the various search engines and web directories,
but prioritizing submissions to the ones that provide information
to other engines, especially the Open Directory Project and
AltaVista, will maximize the listings for a site with a minimum
effort.
-
If a site manages to get a link listed on other sites that are
indexed by a search engine, then those link sites will appear
in search engine results permitting searchers to gain access
indirectly to the original site.
-
As a result of the growing commercialism of the Internet,
you may pay to promote a site and ensure
listings of the site in more search engines. It is not clear how
effective these paid
listings are since many people seek out search engines that
are not so commercially biased.
Additional Resources
For information about issues related to search engines and
web directories, see the
Search Engines section.
For information on commercialization of search engines, see the
Online Commercialism section.