Perplexity · User fetch crawler guide
Perplexity-User: who runs it and how to allow or block it
Perplexity-User supports user actions within Perplexity, fetching a page when a person's question needs it.
Perplexity-User at a glance
- Operator
- Perplexity
- Purpose
- User fetch It fetches a page when a person asks the assistant about it.
- Kind
- Crawler. Visits your pages under its own user agent and reads the robots.txt group that names Perplexity-User.
- What it feeds
- live answers in Perplexity
- What Perplexity says about robots.txt
- Perplexity states that since a user requested the fetch, Perplexity-User generally ignores robots.txt rules.
- User agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)- Verified
- against Perplexity's official docs (opens in a new tab).
robots.txt lines
Allow or block Perplexity-User.
Paste one group into the robots.txt at the root of your domain. A group that names Perplexity-User replaces your * group for Perplexity-User, so copy any * Disallow lines you still want applied into it.
User-agent: Perplexity-User
Allow: /
User-agent: Perplexity-User
Disallow: /
# AI crawler access, generated by Suede AI Agent Studio
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amzn-User
Allow: /
Opens Perplexity-User together with the other search and assistant crawlers in one paste.
Precedence
How Perplexity-User reads your robots.txt
Major crawlers apply the robots.txt standard (RFC 9309) the same way. Three rules decide every path.
- A named group beats *. When a group says
User-agent: Perplexity-User, Perplexity-User follows that group and ignores the * group entirely. With no named group, the * group applies; with neither, every path is allowed. - The longest matching rule wins. Inside the group, the Allow or Disallow line with the longest matching path decides, so
Allow: /checkout/helpoverridesDisallow: /checkoutfor the help pages.*matches any run of characters and$marks the end of the path. - Allow wins a tie. When an Allow and a Disallow match with the same length, the path is allowed.
User-agent: *
Disallow: /
User-agent: Perplexity-User
Disallow: /checkout
Allow: /checkout/help
Disallow: /*.pdf$
| Path | Result | Deciding rule |
|---|---|---|
/ | Allowed | No rule in the Perplexity-User group matches, so it is allowed |
/checkout/cart | Blocked | line 5: Disallow: /checkout |
/checkout/help | Allowed | line 6: Allow: /checkout/help |
/menu.pdf | Blocked | line 7: Disallow: /*.pdf$ |
Line 2 still blocks every crawler without a group of its own. Perplexity-User reads only lines 4 to 7, which is why the homepage stays open to it.
Your site
See which line decides Perplexity-User on your site.
FAQ
Perplexity-User, answered.
What is Perplexity-User?
Perplexity-User supports user actions within Perplexity, fetching a page when a person's question needs it. Perplexity-User is run by Perplexity and fetches a page when a person asks the assistant about it.
How do I allow Perplexity-User in robots.txt?
Add a group that names it: a line reading "User-agent: Perplexity-User" followed by "Allow: /". A group that names Perplexity-User replaces your * group for Perplexity-User, so repeat any Disallow lines from your * group that should still apply to it. To block it instead, use "Disallow: /" in that group.
Does blocking Perplexity-User remove me from live answers in Perplexity?
Perplexity treats Perplexity-User fetches as user-initiated, so a robots.txt block may not stop them. Whether you appear in Perplexity search is decided by PerplexityBot, which follows its own robots.txt group. Perplexity states that since a user requested the fetch, Perplexity-User generally ignores robots.txt rules.
Related crawlers
Crawlers to set alongside it.
Every crawler, its operator and its docs are in the full AI crawler list.
Last updated .
Keep Perplexity-User where you want it. Every Monday.
Run the free check for the full crawler table and ranked fixes, unlock the $9.99 fix kit for paste-ready files, then let the AI Visibility Watch template read your robots.txt weekly and flag any answer bot that becomes blocked.