Claude-User

Claude-User is the user-agent identifier for Anthropic's Claude AI assistant when it utilizes its web browsing capability. It is not a traditional web crawler but an AI-powered web retrieval tool. When a human user asks Claude a question requiring current information, the AI may use this agent to access relevant web pages. In server logs, it identifies itself as Claude-User. Its behavior is distinct from indexing bots because it visits pages selectively and on-demand, based on specific user queries, rather than systematically crawling the web.

Claude-User is visiting your website because a user has prompted the Claude AI with a query, and the AI has determined that your site contains relevant information to form an answer. It does not continuously crawl or build an index. Its visits are sporadic and directly triggered by user questions. The frequency of visits is therefore tied to how often users ask Claude questions that lead it to your content. This is considered authorized web access, as it mirrors a regular user accessing public information.

The primary purpose of Claude-User is to enable the Claude AI to retrieve up-to-date information from the web to answer user queries accurately. It accesses specific web content on-demand to supplement Claude's static training data. The information it collects is used exclusively to answer the current user's question and is not stored or used for training the AI models. For website owners, this creates a new channel for content discovery; when Claude uses information from your site, it typically provides attribution, potentially driving traffic and exposure.

If you do not want the Claude AI assistant to access your website's content when prompted by a user, you can block the Claude-User agent. This is different from blocking ClaudeBot, which is used for AI model training. You can add the following rule to your robots.txt file:

User-agent: Claude-User
Disallow: /