What are the three kinds of AI crawlers?#
AI companies send different crawlers for different jobs, and robots.txt treats each one separately. Blocking the wrong one is the most common reason a business vanishes from AI answers.
| Role | What it does | Effect of blocking it |
|---|---|---|
| Search index | Crawls ahead of time to build the engine's search index | Your pages drop out of that assistant's search answers |
| Live fetch | Opens a page the moment a person's question needs it | The assistant cannot read your page mid-answer |
| Model training | Collects content to train future models | Future training sets leave your content out; live answers are unaffected |
Which AI crawler feeds which engine?#
Agent Studio keeps one registry of AI crawlers, checked against each operator's own documentation. The same registry powers the AI crawler list, the AI Visibility Check and the robots.txt patch, so the table below always matches what the check tests. Click any token for its guide and copyable lines.
| Token | Operator | Role | What it feeds | Recommended |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Search index | ChatGPT search results | Allow |
| ChatGPT-User | OpenAI | Live fetch | user actions in ChatGPT and Custom GPTs | Allow (user-initiated, robots.txt is advisory) |
| Claude-SearchBot | Anthropic | Search index | Claude search results | Allow |
| Claude-User | Anthropic | Live fetch | live answers in Claude | Allow |
| PerplexityBot | Perplexity | Search index | Perplexity search results | Allow |
| Perplexity-User | Perplexity | Live fetch | live answers in Perplexity | Allow (user-initiated, robots.txt is advisory) |
| Amzn-SearchBot | Amazon | Search index | search in Amazon products and services | Allow |
| Amzn-User | Amazon | Live fetch | live answers in Alexa | Allow (user-initiated, robots.txt is advisory) |
| Meta-WebIndexer | Meta | Search index | Meta AI search results | Allow |
| Meta-ExternalFetcher | Meta | Live fetch | agentic AI tasks Meta runs for users | Allow (user-initiated, robots.txt is advisory) |
| GPTBot | OpenAI | Model training | OpenAI generative AI foundation models | Your call (no effect on the score) |
| ClaudeBot | Anthropic | Model training | Anthropic's Claude model training | Your call (no effect on the score) |
| Google-Extended | Model training (control token) | Gemini model training and grounding | Your call (no effect on the score) | |
| Applebot-Extended | Apple | Model training (control token) | Apple generative AI model training | Your call (no effect on the score) |
| Amazonbot | Amazon | Model training | Amazon products, services and AI models | Your call (no effect on the score) |
| CCBot | Common Crawl | Model training | the Common Crawl open web dataset | Your call (no effect on the score) |
| Bytespider | ByteDance | Model training | ByteDance AI models | Your call (no effect on the score) |
| Meta-ExternalAgent | Meta | Model training | Meta AI model training and products | Your call (no effect on the score) |
Which AI crawlers should I allow?#
Allow every search and live-fetch bot if you want to be found in AI answers. Those are the crawlers behind ChatGPT, Claude, Perplexity, Alexa and Meta AI, and the AI Visibility Check scores your access for them. Training crawlers are a business decision: allowing them puts your content into future models, blocking them keeps it out, and neither changes your readiness score.
- Want leads from AI search? Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and the other search bots on every public page.
- Selling content itself? Block the training crawlers and keep the search bots open, so assistants still cite and link you.
- Have private areas? Keep
/admin/,/account/and checkout paths disallowed in every group.
What robots.txt lines do I paste?#
Give each answer bot its own group that allows the site and repeats your private-path rules. A crawler follows the group that names it and ignores the * group, so any Disallow you only put under * stops applying to a crawler you name. This example is built from the same registry as the check:
# AI answer bots: open the site, keep private paths closed.
# A named group replaces the * group for that crawler, so repeat your * rules in each one.
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
User-agent: ChatGPT-User
Allow: /
Disallow: /admin/
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/
User-agent: Claude-User
Allow: /
Disallow: /admin/
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
User-agent: Perplexity-User
Allow: /
Disallow: /admin/
User-agent: Amzn-SearchBot
Allow: /
Disallow: /admin/
User-agent: Amzn-User
Allow: /
Disallow: /admin/
User-agent: Meta-WebIndexer
Allow: /
Disallow: /admin/
User-agent: Meta-ExternalFetcher
Allow: /
Disallow: /admin/
# Training crawlers: your call. This example blocks one of them.
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xmlHow do I check my robots.txt is working?#
Verify AI crawler access
Step 1: Run the AI Visibility Check
The crawler table shows every bot as allowed or blocked on your homepage and key pages, with the exact line that decides it.Check my siteStep 2: Open the guide for any blocked bot
Each crawler page has copyable Allow and Block lines and an example robots.txt, tested path by path.Open OAI-SearchBot guideStep 3: Paste, publish, re-check
Update robots.txt, loadRe-run the check/robots.txtin a browser to confirm it serves, then run the check again.Step 4: Put a watch on it
The AI Visibility Watch template reads your robots.txt every Monday and alerts you, naming the bot and the rule, if an answer bot gets blocked.Open AI Visibility Watch
What else stops AI assistants from quoting me?#
Page-level directives. A noindex, nosnippet, noai or max-snippet:0 in your homepage meta robots tag or X-Robots-Tag header tells engines not to quote you even when robots.txt lets them in. The AI Visibility Check reads both and quotes the directive it found. Content hidden behind JavaScript is the other silent blocker: the check tests whether your homepage text is readable without it.
Once the bots can read you, measure whether they cite you with AI visibility, and run the free SEO audit to catch page problems that block them.
Find out which AI engines can read your site
The check tests every crawler in the registry and quotes the line that decides each one.