OpenAI · Search crawler guide
OAI-SearchBot: who runs it and how to allow or block it
OAI-SearchBot surfaces websites in ChatGPT's search features. Allowing it makes your pages eligible to appear in ChatGPT search answers.
OAI-SearchBot at a glance
- Operator
- OpenAI
- Purpose
- Search It builds the search index behind AI answers.
- Kind
- Crawler. Visits your pages under its own user agent and reads the robots.txt group that names OAI-SearchBot.
- What it feeds
- ChatGPT search results
- What OpenAI says about robots.txt
- OpenAI states that sites opted out of OAI-SearchBot are left out of ChatGPT search answers, and that search picks up a robots.txt change in about 24 hours.
- User agent string
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot- Verified
- against OpenAI's official docs (opens in a new tab).
robots.txt lines
Allow or block OAI-SearchBot.
Paste one group into the robots.txt at the root of your domain. A group that names OAI-SearchBot replaces your * group for OAI-SearchBot, so copy any * Disallow lines you still want applied into it.
User-agent: OAI-SearchBot
Allow: /
User-agent: OAI-SearchBot
Disallow: /
# AI crawler access, generated by Suede AI Agent Studio
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amzn-User
Allow: /
Opens OAI-SearchBot together with the other search and assistant crawlers in one paste.
Precedence
How OAI-SearchBot reads your robots.txt
Major crawlers apply the robots.txt standard (RFC 9309) the same way. Three rules decide every path.
- A named group beats *. When a group says
User-agent: OAI-SearchBot, OAI-SearchBot follows that group and ignores the * group entirely. With no named group, the * group applies; with neither, every path is allowed. - The longest matching rule wins. Inside the group, the Allow or Disallow line with the longest matching path decides, so
Allow: /checkout/helpoverridesDisallow: /checkoutfor the help pages.*matches any run of characters and$marks the end of the path. - Allow wins a tie. When an Allow and a Disallow match with the same length, the path is allowed.
User-agent: *
Disallow: /
User-agent: OAI-SearchBot
Disallow: /checkout
Allow: /checkout/help
Disallow: /*.pdf$
| Path | Result | Deciding rule |
|---|---|---|
/ | Allowed | No rule in the OAI-SearchBot group matches, so it is allowed |
/checkout/cart | Blocked | line 5: Disallow: /checkout |
/checkout/help | Allowed | line 6: Allow: /checkout/help |
/menu.pdf | Blocked | line 7: Disallow: /*.pdf$ |
Line 2 still blocks every crawler without a group of its own. OAI-SearchBot reads only lines 4 to 7, which is why the homepage stays open to it.
Your site
See which line decides OAI-SearchBot on your site.
FAQ
OAI-SearchBot, answered.
What is OAI-SearchBot?
OAI-SearchBot surfaces websites in ChatGPT's search features. Allowing it makes your pages eligible to appear in ChatGPT search answers. OAI-SearchBot is run by OpenAI and builds the search index behind AI answers.
How do I allow OAI-SearchBot in robots.txt?
Add a group that names it: a line reading "User-agent: OAI-SearchBot" followed by "Allow: /". A group that names OAI-SearchBot replaces your * group for OAI-SearchBot, so repeat any Disallow lines from your * group that should still apply to it. To block it instead, use "Disallow: /" in that group.
Does blocking OAI-SearchBot remove me from ChatGPT search results?
It can. OAI-SearchBot builds the index behind ChatGPT search results, so blocking it keeps your pages out of that index and out of the answers drawn from it. OpenAI states that sites opted out of OAI-SearchBot are left out of ChatGPT search answers, and that search picks up a robots.txt change in about 24 hours.
Related crawlers
Crawlers to set alongside it.
Every crawler, its operator and its docs are in the full AI crawler list.
Last updated .
Keep OAI-SearchBot where you want it. Every Monday.
Run the free check for the full crawler table and ranked fixes, unlock the $9.99 fix kit for paste-ready files, then let the AI Visibility Watch template read your robots.txt weekly and flag any answer bot that becomes blocked.