Meta · User fetch crawler guide
Meta-ExternalFetcher: who runs it and how to allow or block it
Meta-ExternalFetcher fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks for users.
Meta-ExternalFetcher at a glance
- Operator
- Meta
- Purpose
- User fetch It fetches a page when a person asks the assistant about it.
- Kind
- Crawler. Visits your pages under its own user agent and reads the robots.txt group that names Meta-ExternalFetcher.
- What it feeds
- agentic AI tasks Meta runs for users
- What Meta says about robots.txt
- Meta states that because Meta-ExternalFetcher performs fetches the user requested, it may bypass robots.txt rules.
- User agent string
meta-externalfetcher/1.1- Verified
- against Meta's official docs (opens in a new tab).
robots.txt lines
Allow or block Meta-ExternalFetcher.
Paste one group into the robots.txt at the root of your domain. A group that names Meta-ExternalFetcher replaces your * group for Meta-ExternalFetcher, so copy any * Disallow lines you still want applied into it.
User-agent: Meta-ExternalFetcher
Allow: /
User-agent: Meta-ExternalFetcher
Disallow: /
# AI crawler access, generated by Suede AI Agent Studio
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amzn-User
Allow: /
User-agent: Meta-WebIndexer
Allow: /
User-agent: Meta-ExternalFetcher
Allow: /
Opens Meta-ExternalFetcher together with the other crawlers behind live answers in ChatGPT, Claude, Perplexity, Alexa and Meta AI, in one paste.
Precedence
How Meta-ExternalFetcher reads your robots.txt
Major crawlers apply the robots.txt standard (RFC 9309) the same way. Three rules decide every path.
- A named group beats *. When a group says
User-agent: Meta-ExternalFetcher, Meta-ExternalFetcher follows that group and ignores the * group entirely. With no named group, the * group applies; with neither, every path is allowed. - The longest matching rule wins. Inside the group, the Allow or Disallow line with the longest matching path decides, so
Allow: /checkout/helpoverridesDisallow: /checkoutfor the help pages.*matches any run of characters and$marks the end of the path. - Allow wins a tie. When an Allow and a Disallow match with the same length, the path is allowed.
User-agent: *
Disallow: /
User-agent: Meta-ExternalFetcher
Disallow: /checkout
Allow: /checkout/help
Disallow: /*.pdf$
| Path | Result | Deciding rule |
|---|---|---|
/ | Allowed | No rule in the Meta-ExternalFetcher group matches, so it is allowed |
/checkout/cart | Blocked | line 5: Disallow: /checkout |
/checkout/help | Allowed | line 6: Allow: /checkout/help |
/menu.pdf | Blocked | line 7: Disallow: /*.pdf$ |
Line 2 still blocks every crawler without a group of its own. Meta-ExternalFetcher reads only lines 4 to 7, which is why the homepage stays open to it.
Your site
See which line decides Meta-ExternalFetcher on your site.
FAQ
Meta-ExternalFetcher, answered.
What is Meta-ExternalFetcher?
Meta-ExternalFetcher fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks for users. Meta-ExternalFetcher is run by Meta and fetches a page when a person asks the assistant about it.
How do I allow Meta-ExternalFetcher in robots.txt?
Add a group that names it: a line reading "User-agent: Meta-ExternalFetcher" followed by "Allow: /". A group that names Meta-ExternalFetcher replaces your * group for Meta-ExternalFetcher, so repeat any Disallow lines from your * group that should still apply to it. To block it instead, use "Disallow: /" in that group.
Does blocking Meta-ExternalFetcher remove me from agentic AI tasks Meta runs for users?
Meta treats Meta-ExternalFetcher fetches as user-initiated, so a robots.txt block may not stop them. Whether you appear in Meta search is decided by Meta-WebIndexer, which follows its own robots.txt group. Meta states that because Meta-ExternalFetcher performs fetches the user requested, it may bypass robots.txt rules.
Related crawlers
Crawlers to set alongside it.
Every crawler, its operator and its docs are in the full AI crawler list.
Last updated .
Keep Meta-ExternalFetcher where you want it. Every Monday.
Run the free check for the full crawler table and ranked fixes, unlock the $9.99 fix kit for paste-ready files, then let the AI Visibility Watch template read your robots.txt weekly and flag any answer bot that becomes blocked.