Crawler FAQ

What is Discobot?
Discobot is the experimental web crawler for Discovery Engine.

Why are you crawling my web site?
We are building a large document index for a next-generation search engine.

How can I stop your crawler from visiting my website?
If you would like our crawler to stop visiting your site, please add the following lines to your robots.txt file:

User-agent: discobot
Disallow: /

From what IP addresses does your crawler run?
We are currently crawling from the following range of IP addresses:

208.96.54.67 -
208.96.54.96

Whom can I contact with a question or concern about your crawler?
Please send all questions and feedback about the crawler to Ned Rockson.

When will your new search engine be released?
A public beta of Discovery Engine will be available in 2008.

What else can you tell me about Discovery Engine?
More information about our company and search engine will be released later this year.