Crawler FAQ

What is Discobot?
Discobot is the experimental web crawler for Discovery Engine.

Why are you crawling my web site?
We are building a large document index for a next-generation search engine.

How can I stop your crawler from visiting my website?
If you would like our crawler to stop visiting your site, please add the following lines to your robots.txt file:

User-agent: discobot
Disallow: /

From what IP addresses does your crawler run?
We are currently crawling from the following range of IP addresses:

208.96.54.67 -
208.96.54.96

Whom can I contact with a question or concern about your crawler?
Please send all questions and feedback about the crawler to discobot@discoveryengine.com.

When will your new search engine be released?
A public beta of Discovery Engine will be available in 2009.

What else can you tell me about Discovery Engine?
More information about our new search engine will be released soon.