Anthropic · User fetch crawler guide
Claude-User: who runs it and how to allow or block it
Claude-User fetches web pages when a person asks Claude a question that needs them, so Claude can answer with current content.
Claude-User at a glance
- Operator
- Anthropic
- Purpose
- User fetch It fetches a page when a person asks the assistant about it.
- Kind
- Crawler. Visits your pages under its own user agent and reads the robots.txt group that names Claude-User.
- What it feeds
- live answers in Claude
- What Anthropic says about robots.txt
- Anthropic states that disallowing Claude-User stops Claude from retrieving your content in response to user queries, which may reduce your visibility in user-directed web search.
- User agent string
- Match it in robots.txt by its token, Claude-User. Agent tokens are compared without regard to case.
- Verified
- against Anthropic's official docs (opens in a new tab).
robots.txt lines
Allow or block Claude-User.
Paste one group into the robots.txt at the root of your domain. A group that names Claude-User replaces your * group for Claude-User, so copy any * Disallow lines you still want applied into it.
User-agent: Claude-User
Allow: /
User-agent: Claude-User
Disallow: /
# AI crawler access, generated by Suede AI Agent Studio
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amzn-User
Allow: /
Opens Claude-User together with the other search and assistant crawlers in one paste.
Precedence
How Claude-User reads your robots.txt
Major crawlers apply the robots.txt standard (RFC 9309) the same way. Three rules decide every path.
- A named group beats *. When a group says
User-agent: Claude-User, Claude-User follows that group and ignores the * group entirely. With no named group, the * group applies; with neither, every path is allowed. - The longest matching rule wins. Inside the group, the Allow or Disallow line with the longest matching path decides, so
Allow: /checkout/helpoverridesDisallow: /checkoutfor the help pages.*matches any run of characters and$marks the end of the path. - Allow wins a tie. When an Allow and a Disallow match with the same length, the path is allowed.
User-agent: *
Disallow: /
User-agent: Claude-User
Disallow: /checkout
Allow: /checkout/help
Disallow: /*.pdf$
| Path | Result | Deciding rule |
|---|---|---|
/ | Allowed | No rule in the Claude-User group matches, so it is allowed |
/checkout/cart | Blocked | line 5: Disallow: /checkout |
/checkout/help | Allowed | line 6: Allow: /checkout/help |
/menu.pdf | Blocked | line 7: Disallow: /*.pdf$ |
Line 2 still blocks every crawler without a group of its own. Claude-User reads only lines 4 to 7, which is why the homepage stays open to it.
Your site
See which line decides Claude-User on your site.
FAQ
Claude-User, answered.
What is Claude-User?
Claude-User fetches web pages when a person asks Claude a question that needs them, so Claude can answer with current content. Claude-User is run by Anthropic and fetches a page when a person asks the assistant about it.
How do I allow Claude-User in robots.txt?
Add a group that names it: a line reading "User-agent: Claude-User" followed by "Allow: /". A group that names Claude-User replaces your * group for Claude-User, so repeat any Disallow lines from your * group that should still apply to it. To block it instead, use "Disallow: /" in that group.
Does blocking Claude-User remove me from live answers in Claude?
It can. Claude-User fetches your page at the moment a person asks about it, so blocking it can stop live answers in Claude from reading your page for that answer. Anthropic states that disallowing Claude-User stops Claude from retrieving your content in response to user queries, which may reduce your visibility in user-directed web search.
Related crawlers
Crawlers to set alongside it.
Every crawler, its operator and its docs are in the full AI crawler list.
Last updated .
Keep Claude-User where you want it. Every Monday.
Run the free check for the full crawler table and ranked fixes, unlock the $9.99 fix kit for paste-ready files, then let the AI Visibility Watch template read your robots.txt weekly and flag any answer bot that becomes blocked.