Common Crawl · Training crawler guide
CCBot: who runs it and how to allow or block it
CCBot collects pages for Common Crawl, a non-profit open repository of web crawl data that is widely used to build AI training datasets.
CCBot at a glance
- Operator
- Common Crawl
- Purpose
- Training It collects content for AI model training.
- Kind
- Crawler. Visits your pages under its own user agent and reads the robots.txt group that names CCBot.
- What it feeds
- the Common Crawl open web dataset
- What Common Crawl says about robots.txt
- Common Crawl documents a User-agent: CCBot group with Disallow: / as the way to keep CCBot off a site, and publishes its IP ranges so requests can be verified.
- User agent string
CCBot/2.0 (https://commoncrawl.org/faq/)- Verified
- against Common Crawl's official docs (opens in a new tab).
robots.txt lines
Allow or block CCBot.
Paste one group into the robots.txt at the root of your domain. A group that names CCBot replaces your * group for CCBot, so copy any * Disallow lines you still want applied into it.
User-agent: CCBot
Allow: /
User-agent: CCBot
Disallow: /
# AI crawler access, generated by Suede AI Agent Studio
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amzn-User
Allow: /
User-agent: CCBot
Disallow: /
Opens the crawlers behind live answers in ChatGPT, Claude and Perplexity while CCBot stays out.
Precedence
How CCBot reads your robots.txt
Major crawlers apply the robots.txt standard (RFC 9309) the same way. Three rules decide every path.
- A named group beats *. When a group says
User-agent: CCBot, CCBot follows that group and ignores the * group entirely. With no named group, the * group applies; with neither, every path is allowed. - The longest matching rule wins. Inside the group, the Allow or Disallow line with the longest matching path decides, so
Allow: /checkout/helpoverridesDisallow: /checkoutfor the help pages.*matches any run of characters and$marks the end of the path. - Allow wins a tie. When an Allow and a Disallow match with the same length, the path is allowed.
User-agent: *
Disallow: /
User-agent: CCBot
Disallow: /checkout
Allow: /checkout/help
Disallow: /*.pdf$
| Path | Result | Deciding rule |
|---|---|---|
/ | Allowed | No rule in the CCBot group matches, so it is allowed |
/checkout/cart | Blocked | line 5: Disallow: /checkout |
/checkout/help | Allowed | line 6: Allow: /checkout/help |
/menu.pdf | Blocked | line 7: Disallow: /*.pdf$ |
Line 2 still blocks every crawler without a group of its own. CCBot reads only lines 4 to 7, which is why the homepage stays open to it.
Your site
See which line decides CCBot on your site.
FAQ
CCBot, answered.
What is CCBot?
CCBot collects pages for Common Crawl, a non-profit open repository of web crawl data that is widely used to build AI training datasets. CCBot is run by Common Crawl and collects content for AI model training.
How do I allow CCBot in robots.txt?
Add a group that names it: a line reading "User-agent: CCBot" followed by "Allow: /". A group that names CCBot replaces your * group for CCBot, so repeat any Disallow lines from your * group that should still apply to it. To block it instead, use "Disallow: /" in that group.
Does blocking CCBot remove me from AI answers?
Blocking CCBot governs whether your content is used for the Common Crawl open web dataset. Live answers come from answer bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, which follow their own robots.txt groups, so you can block CCBot and keep those open. Common Crawl documents a User-agent: CCBot group with Disallow: / as the way to keep CCBot off a site, and publishes its IP ranges so requests can be verified.
Related crawlers
Crawlers to set alongside it.
Every crawler, its operator and its docs are in the full AI crawler list.
Last updated .
Keep CCBot where you want it. Every Monday.
Run the free check for the full crawler table and ranked fixes, unlock the $9.99 fix kit for paste-ready files, then let the AI Visibility Watch template read your robots.txt weekly and flag any answer bot that becomes blocked.