Nutch bot

Nutch is an open-source web crawler and search engine framework developed by the Apache Software Foundation. It provides the underlying technology that allows anyone to build and customize their own web crawler. When a Nutch-based crawler visits a website, it may identify itself with user-agent strings like Nutch or, if it is the official demo crawler, NutchOrg. Because it is open-source, a Nutch user-agent in your logs could be from the official project or from any number of third-party deployments.

A Nutch-based bot is crawling your website to discover, download, and index its content. Since Nutch is a framework that can be used by anyone, the specific reason depends on the operator. The official Nutch.org crawler builds a public demo search index, while other organizations might use Nutch to create specialized search engines or gather data for research. The crawl frequency varies widely depending on the purpose of the deployment. While many Nutch operators follow best practices, some may not, given the software's open-source nature.

The purpose of Nutch is to provide an open, transparent foundation for building web search and content discovery systems. Organizations use it to create specialized search engines, build research datasets, and archive web content. The data collected by Nutch deployments typically feeds into a search index. For website owners, being included in a Nutch-powered index can increase content visibility, especially within niche search engines. Unlike commercial search engines, Nutch itself does not monetize the crawled data; it simply provides the infrastructure for others to build search services.

To prevent Nutch-based crawlers from accessing your website, you can add rules to your robots.txt file. Since there are several common user-agents associated with Nutch, you may want to block them individually.

Add the following lines to your robots.txt file to block common Nutch bots:

User-agent: Nutch
Disallow: /

User-agent: NutchOrg
Disallow: /