OpenAI · User fetch crawler guide
ChatGPT-User: who runs it and how to allow or block it
ChatGPT-User fetches pages when a person asks ChatGPT or a Custom GPT to act on them. OpenAI does not use it to crawl the web automatically.
ChatGPT-User at a glance
- Operator
- OpenAI
- Purpose
- User fetch It fetches a page when a person asks the assistant about it.
- Kind
- Crawler. Visits your pages under its own user agent and reads the robots.txt group that names ChatGPT-User.
- What it feeds
- user actions in ChatGPT and Custom GPTs
- What OpenAI says about robots.txt
- OpenAI states that because these fetches are user-initiated, robots.txt rules may not apply. OAI-SearchBot, not ChatGPT-User, decides whether content appears in ChatGPT search.
- User agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot- Verified
- against OpenAI's official docs (opens in a new tab).
robots.txt lines
Allow or block ChatGPT-User.
Paste one group into the robots.txt at the root of your domain. A group that names ChatGPT-User replaces your * group for ChatGPT-User, so copy any * Disallow lines you still want applied into it.
User-agent: ChatGPT-User
Allow: /
User-agent: ChatGPT-User
Disallow: /
# AI crawler access, generated by Suede AI Agent Studio
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amzn-User
Allow: /
Opens ChatGPT-User together with the other search and assistant crawlers in one paste.
Precedence
How ChatGPT-User reads your robots.txt
Major crawlers apply the robots.txt standard (RFC 9309) the same way. Three rules decide every path.
- A named group beats *. When a group says
User-agent: ChatGPT-User, ChatGPT-User follows that group and ignores the * group entirely. With no named group, the * group applies; with neither, every path is allowed. - The longest matching rule wins. Inside the group, the Allow or Disallow line with the longest matching path decides, so
Allow: /checkout/helpoverridesDisallow: /checkoutfor the help pages.*matches any run of characters and$marks the end of the path. - Allow wins a tie. When an Allow and a Disallow match with the same length, the path is allowed.
User-agent: *
Disallow: /
User-agent: ChatGPT-User
Disallow: /checkout
Allow: /checkout/help
Disallow: /*.pdf$
| Path | Result | Deciding rule |
|---|---|---|
/ | Allowed | No rule in the ChatGPT-User group matches, so it is allowed |
/checkout/cart | Blocked | line 5: Disallow: /checkout |
/checkout/help | Allowed | line 6: Allow: /checkout/help |
/menu.pdf | Blocked | line 7: Disallow: /*.pdf$ |
Line 2 still blocks every crawler without a group of its own. ChatGPT-User reads only lines 4 to 7, which is why the homepage stays open to it.
Your site
See which line decides ChatGPT-User on your site.
FAQ
ChatGPT-User, answered.
What is ChatGPT-User?
ChatGPT-User fetches pages when a person asks ChatGPT or a Custom GPT to act on them. OpenAI does not use it to crawl the web automatically. ChatGPT-User is run by OpenAI and fetches a page when a person asks the assistant about it.
How do I allow ChatGPT-User in robots.txt?
Add a group that names it: a line reading "User-agent: ChatGPT-User" followed by "Allow: /". A group that names ChatGPT-User replaces your * group for ChatGPT-User, so repeat any Disallow lines from your * group that should still apply to it. To block it instead, use "Disallow: /" in that group.
Does blocking ChatGPT-User remove me from user actions in ChatGPT and Custom GPTs?
OpenAI treats ChatGPT-User fetches as user-initiated, so a robots.txt block may not stop them. OpenAI states that because these fetches are user-initiated, robots.txt rules may not apply. OAI-SearchBot, not ChatGPT-User, decides whether content appears in ChatGPT search.
Related crawlers
Crawlers to set alongside it.
Every crawler, its operator and its docs are in the full AI crawler list.
Last updated .
Keep ChatGPT-User where you want it. Every Monday.
Run the free check for the full crawler table and ranked fixes, unlock the $9.99 fix kit for paste-ready files, then let the AI Visibility Watch template read your robots.txt weekly and flag any answer bot that becomes blocked.