Apple · Training crawler guide
Applebot-Extended: who runs it and how to allow or block it
Applebot-Extended lets publishers choose whether content crawled by Applebot may be used to train Apple's generative AI models.
Applebot-Extended at a glance
- Operator
- Apple
- Purpose
- Training It collects content for AI model training.
- Kind
- Robots.txt control token. Honored by Apple's main crawler; no separate visits.
- What it feeds
- Apple generative AI model training
- What Apple says about robots.txt
- Apple states that Applebot-Extended does not crawl webpages; it only determines how data crawled by the Applebot user agent is used.
- User agent string
- Applebot-Extended has no user agent of its own. Name it in robots.txt and Apple's main crawler applies the rule.
- Verified
- against Apple's official docs (opens in a new tab).
robots.txt lines
Allow or block Applebot-Extended.
Paste one group into the robots.txt at the root of your domain. A group that names Applebot-Extended replaces your * group for Applebot-Extended, so copy any * Disallow lines you still want applied into it.
User-agent: Applebot-Extended
Allow: /
User-agent: Applebot-Extended
Disallow: /
# AI crawler access, generated by Suede AI Agent Studio
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amzn-User
Allow: /
User-agent: Applebot-Extended
Disallow: /
Opens the crawlers behind live answers in ChatGPT, Claude and Perplexity while Applebot-Extended stays out.
Precedence
How Applebot-Extended reads your robots.txt
Major crawlers apply the robots.txt standard (RFC 9309) the same way. Three rules decide every path.
- A named group beats *. When a group says
User-agent: Applebot-Extended, Apple's crawler follows that group and ignores the * group entirely. With no named group, the * group applies; with neither, every path is allowed. - The longest matching rule wins. Inside the group, the Allow or Disallow line with the longest matching path decides, so
Allow: /checkout/helpoverridesDisallow: /checkoutfor the help pages.*matches any run of characters and$marks the end of the path. - Allow wins a tie. When an Allow and a Disallow match with the same length, the path is allowed.
User-agent: *
Disallow: /
User-agent: Applebot-Extended
Disallow: /checkout
Allow: /checkout/help
Disallow: /*.pdf$
| Path | Result | Deciding rule |
|---|---|---|
/ | Allowed | No rule in the Applebot-Extended group matches, so it is allowed |
/checkout/cart | Blocked | line 5: Disallow: /checkout |
/checkout/help | Allowed | line 6: Allow: /checkout/help |
/menu.pdf | Blocked | line 7: Disallow: /*.pdf$ |
Line 2 still blocks every crawler without a group of its own. Applebot-Extended reads only lines 4 to 7, which is why the homepage stays open to it. Apple's crawler reads the Applebot-Extended group to decide how content may be used; its regular crawling follows its own group.
Your site
See which line decides Applebot-Extended on your site.
FAQ
Applebot-Extended, answered.
What is Applebot-Extended?
Applebot-Extended lets publishers choose whether content crawled by Applebot may be used to train Apple's generative AI models. Applebot-Extended is run by Apple and collects content for AI model training. It is a robots.txt control token: Apple's main crawler honors it and it makes no separate visits.
How do I allow Applebot-Extended in robots.txt?
Add a group that names it: a line reading "User-agent: Applebot-Extended" followed by "Allow: /". A group that names Applebot-Extended replaces your * group for Applebot-Extended, so repeat any Disallow lines from your * group that should still apply to it. To block it instead, use "Disallow: /" in that group.
Does blocking Applebot-Extended remove me from AI answers?
Blocking Applebot-Extended governs whether your content is used for Apple generative AI model training. Live answers come from answer bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, which follow their own robots.txt groups, so you can block Applebot-Extended and keep those open. Apple states that Applebot-Extended does not crawl webpages; it only determines how data crawled by the Applebot user agent is used.
Related crawlers
Crawlers to set alongside it.
Every crawler, its operator and its docs are in the full AI crawler list.
Last updated .
Keep Applebot-Extended where you want it. Every Monday.
Run the free check for the full crawler table and ranked fixes, unlock the $9.99 fix kit for paste-ready files, then let the AI Visibility Watch template read your robots.txt weekly and flag any answer bot that becomes blocked.