Crawler FAQ

What is Discobot?
Discobot is the experimental web crawler for Discovery Engine.

Why are you crawling my web site?
We are building a large document index for a next-generation search engine.

How can I stop your crawler from visiting my website?
If you would like our crawler to stop visiting your site, please add the following lines to your robots.txt file:

User-agent: discobot
Disallow: /

From what IP addresses does your crawler run?
We are currently crawling from the following range of IP addresses:

38.101.148.97-38.101.148.126

Whom can I contact with a question or concern about your crawler?
Please send all questions and feedback about the crawler to discobot@discoveryengine.com.

When will your new search engine be released?
A private alpha version of Discovery Engine became available in 2009. A public beta site will be launched in 2010.

What else can you tell me about Discovery Engine?
“We will sell no wine before its time.”