ClaudeBot is Anthropic's web crawler that collects content that may be used to train Claude models. Anthropic runs two other bots as well: Claude-User, which fetches pages when someone asks Claude a question, and Claude-SearchBot, which supports search results. All three follow robots.txt, so you can allow or block each one separately.
What is ClaudeBot?
ClaudeBot is a crawler operated by Anthropic, the company behind Claude. According to Anthropic's support article on its crawlers, it gathers web content that may contribute to training its models. If you restrict it, Anthropic says that signals the site's future materials should be excluded from training datasets.
It is one of several AI crawlers that now show up in server logs. For the wider picture, see our guide to what an AI crawler is and how to control it.
What are Anthropic's three bots?
Anthropic describes three user agents, and they do different jobs. Treating them as one is the most common mistake.
| User agent | What it does | What happens if you block it |
|---|---|---|
| ClaudeBot | Collects content that may be used for model training | Your future content is excluded from training data |
| Claude-User | Fetches a page when a Claude user asks a question | Claude cannot retrieve your pages for those requests, which may reduce visibility in user-directed web search |
| Claude-SearchBot | Crawls to improve the quality of search results | Your pages may not be indexed for search, which may reduce visibility and accuracy in results |
The descriptions in the last column come straight from Anthropic's article. The split is the same pattern as other AI companies. Our post on GPTBot covers OpenAI's version of it.
Does ClaudeBot respect robots.txt?
Yes. Anthropic states that its bots honor robots.txt directives. It also says it supports the non-standard Crawl-delay extension, with an example of Crawl-delay: 1 for ClaudeBot to slow its requests.
Two practical points follow from Anthropic's own guidance:
- The rule goes in the robots.txt at the top level of each domain, and each subdomain needs its own file.
- Anthropic does not recommend blocking by IP address. It says that may not work correctly and can stop the bots from reading your robots.txt, which defeats the opt-out.
The same article says that from October 30, 2026, Anthropic honors a machine-readable "no AI training" signal based on an IETF draft, and that rules naming ClaudeBot directly take priority over the general setting for a path. Check the article for the exact signal before relying on it, since this is new.
How do you block ClaudeBot but stay visible?
The three-bot split lets you choose training and visibility separately. If you do not want your content used for training but you do want to be retrievable when someone asks Claude a question, block only ClaudeBot:
User-agent: ClaudeBot Disallow: /
Leave Claude-User and Claude-SearchBot without a Disallow rule. Anthropic's article gives this exact ClaudeBot rule for a full block. Whether to do it is your call. Blocking training does not remove content Anthropic has already collected.
How do you allow everything?
A missing rule means the bot may crawl. If your robots.txt has a blanket User-agent: * block, add explicit allow rules for the bots you want. Our walkthrough on allowing ChatGPT and Perplexity in robots.txt shows the pattern, and the same structure works for the Claude user agents.
How do you tell whether ClaudeBot is visiting?
- Search your server logs or CDN logs for the user-agent strings ClaudeBot, Claude-User and Claude-SearchBot.
- Check that the request is real. A user-agent string is easy to fake. Anthropic says it publishes its crawler IP addresses at claude.com/crawling/bots.json, which you can use to confirm a request.
- Note which paths it requests and how often, then compare against your robots.txt.
Our post on AI crawler monitoring covers the log method in more detail.
Should you block ClaudeBot?
It depends on what you want from AI tools. Publishers worried about training use often block the training crawler. A site that wants to be quoted in AI answers wants the retrieval bots left alone. Blocking a crawler does not change your Google rankings, since Google uses its own crawler. The decision is about AI visibility and about how you feel about training.
Write the choice down per bot, so the next person who edits robots.txt does not undo it by accident.
What mistakes do site owners make with Claude's crawlers?
- Blocking all three bots to stop training. Only ClaudeBot is described as the training crawler. Blocking the other two cuts you out of retrieval and search for no training benefit.
- Editing the wrong file. A rule on www does not cover a blog on a subdomain. Anthropic's article says to repeat the rule for each subdomain you want to exclude.
- Trusting the user-agent alone. Anyone can send a request that claims to be ClaudeBot. Verify against the published IP list before you act on log data.
- Forgetting to recheck. Bot names and policies change. Review the support article a couple of times a year and compare it with your robots.txt.
Frequently Asked Questions
Is ClaudeBot the same as Claude-User?
No. ClaudeBot collects content that may be used for training. Claude-User fetches a page when a person asks Claude something. They are separate user agents with separate rules.
Does blocking ClaudeBot hurt my Google rankings?
No. Google ranks pages using its own crawler. Blocking ClaudeBot only affects Anthropic's training collection.
Will blocking ClaudeBot stop Claude from mentioning my site?
Not necessarily. Retrieval is handled by Claude-User and Claude-SearchBot. Anthropic says blocking those two may reduce your visibility in search and user-directed queries.
Can I slow ClaudeBot down instead of blocking it?
Yes. Anthropic says it supports the Crawl-delay extension in robots.txt. Add a Crawl-delay line under a ClaudeBot user-agent entry.
Should I block ClaudeBot by IP address?
Anthropic advises against it, because that can stop the bot from reading your robots.txt. Use robots.txt rules instead.