https://michaeljaylissner.com/2012-01-24T23:20:20-08:002012-01-24T23:20:20-08:00Mike Lissnertag:michaeljaylissner.com,2012-01-24:posts/2012/01/24/support-for-x-robots-tag-http-header-and-robots-html-meta-tag/

<p>As part of our research for <a href="/blog/respecting-privacy-while-providing-hundreds-of-thousands-of-public-documents">our post</a> on how we block search engines, we looked into which search engines support which privacy standards. This information doesn’t seem to exist anywhere else on the Internet, so below are our findings, starting with the big guys, and moving towards more obscure or foreign search engines.</p> <h2 id="google-bing">Google, Bing</h2> <p>Google (known as Googlebot) and Bing (known as Bingbot) support the <code>x-robots-tag</code> header and the robots <span class="caps">HTML</span> tag. Here’s <a href="http://support.google.com/webmasters/bin/answer.py?hl=en&amp;answer=79812">Google’s page</a> on the topic. And <a href="http://www.bing.com/community/site_blogs/b/webmaster/archive/2009/08/21/prevent-a-bot-from-getting-lost-in-space-sem-101.aspx">here’s Bing’s</a>. The <a href="http://www.bing.com/community/site_blogs/b/webmaster/archive/2009/11/04/msnbot-1-1-is-retired.aspx">msnbot is retired</a>.</p> <h2 id="yahoo-aol">Yahoo, <span class="caps">AOL</span></h2> <p>Yahoo!’s search engine is provided by Bing. <span class="caps">AOL</span>’s is provided by Google. These are easy ones.</p> <h2 id="ask-yandex-nutch">Ask, Yandex, Nutch</h2> <p>Ask (known as teoma), and Yandex (Russia’s search engine, known as yandex), support the robots meta tag, but do not appear to support the <code>x-robots-tag</code>. Ask’s page on the topic <a href="http://www.ask.com/staticcontent/about/helpcenter/about_helpcenter_webmaster#5">is here</a>, and Yandex’s <a href="http://help.yandex.com/webmaster/?id=1113833">is here</a>. The popular open source crawler, <a href="http://nutch.apache.org/">Nutch</a>, also <a href="http://nutch.sourceforge.net/docs/en/bot.html">supports the robots <span class="caps">HTML</span> tag</a>, but <a href="http://lucene.472066.n3.nabble.com/Support-for-x-robots-tag-td3678606.html">not the <code>x-robots-tag</code> header</a>. <em>Update:</em> Newer versions of Nutch now support <code>x-robots-tag</code>!</p> <h2 id="the-internet-archive-alexa">The Internet Archive, Alexa</h2> <p>The Internet Archive uses Alexa’s crawler, which is known as ia_archiver. This crawler does not seem to support either the <span class="caps">HTML</span> robots meta tag nor the <code>x-robots-tag</code> <span class="caps">HTTP</span> header. Their page on the subject <a href="http://www.alexa.com/help/webmasters">is here</a>. I have requested more information from them, and will update this page if I hear back.</p> <h2 id="blekko-baidu">Blekko, Baidu</h2> <p>Blekko does not support either the robots meta tag nor the <code>x-robots-tag</code> header, per emails I’ve had with them. I also requested information from Baidu, but their response totally ignored my question and was in Chinese. They do have some information <a href="http://wenku.baidu.com/view/ec4457d4b14e852458fb5793.html">here</a>, but it does not seem to provide any information on the noindex value for the robots tag. In any case, the only way to block these crawlers seems to be via a robots.txt file.</p> <h2 id="duckduckgo">Duckduckgo</h2> <p>I previously stated that <span class="caps">DDG</span> did not support the <code>x-robots-tag</code> header, but while that was true, it didn’t tell the entire story. The entire story is that <span class="caps">DDG</span> uses other search crawlers for their content aggregation and uses their own crawler only for maintenance-type work. You can read more about this in <a href="http://stackoverflow.com/a/24089393/64911">my answer on StackOverflow</a>.</p>