Claude-SearchBot

Claude-SearchBot is a dedicated search crawler developed by Anthropic for its Claude AI assistant. Its sole function is to enable Claude's web search capability by systematically indexing publicly available web content. The bot identifies itself in server logs with the user-agent string Claude-SearchBot. It is designed to be an ethical crawler, respecting robots.txt directives and using reasonable crawl rates to minimize server impact. It is distinct from Anthropic's other crawlers, as its data is used for search results, not for training AI models.

Claude-SearchBot is visiting your website to index its content, making it discoverable through Claude's web search feature. It is looking for current and informational content that can help the AI provide up-to-date answers to user questions. The frequency of its visits depends on your site's content type and update schedule; sites with high-value, frequently changing information are likely to be crawled more often. This is a standard and authorized web indexing activity, similar to crawlers from traditional search engines.

The purpose of Claude-SearchBot is to power the web search feature of the Claude AI assistant. It allows Claude to overcome the limitations of its static training data by accessing real-time information from the internet. When a user activates web search, Claude uses the index built by this bot to retrieve relevant pages and synthesize answers with proper source citations. This is especially useful for queries about recent events or time-sensitive topics. For website owners, having your content indexed can lead to increased visibility and traffic from users who use Claude as a tool for information discovery.

If you wish to prevent Claude-SearchBot from indexing your website for its search feature, you can add a specific disallow rule to your robots.txt file.

To block this bot, add the following lines to your robots.txt file:

User-agent: Claude-SearchBot
Disallow: /