AI Crawler List (October 2026): Every AI Bot, Its robots.txt Token and What It Does

An AI crawler is a bot that an AI company uses to fetch web pages, and the important thing to know about any one of them is its job: collecting training data, building a search index for AI answers, or fetching a page live because a user asked. The three jobs are controlled separately. Blocking one can remove your content from model training while leaving you fully visible in that company's AI search, or the reverse, which is the mistake that costs visibility.

Below is every AI crawler we could find documented by its own operator, with the exact token to use in robots.txt, what the operator says it does, and whether the operator says it follows robots.txt. Each row links to the operator's page, and every page was read on 23 October 2026. Where an operator does not document something, we say so rather than fill the gap.

The Short Answer: AI Crawler List

Quoted cells use the operator's own wording. The sections below link to each source.

What We Found Reading 11 Operators' Documentation

We read the crawler documentation of 11 operators on 23 October 2026: OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon, Microsoft, Mistral, DuckDuckGo and Common Crawl. Four things stood out.

• 7 of the 11 separate training from search, either with two crawlers (OpenAI, Anthropic, Meta, Amazon, Mistral) or with a control token that limits AI use of what their search crawler fetches (Google, Apple). Perplexity and DuckDuckGo say their crawlers do not collect training data at all. "Block AI crawlers" is therefore not one decision but two.
• 4 of the 6 user-initiated fetchers say they may not follow robots.txt: ChatGPT-User, Perplexity-User, Amzn-User and meta-externalfetcher. Claude-User is documented as following it; Mistral does not say.
• 7 of the 11 documentation pages show no last-updated date, and Microsoft's guidance on AI answers dates from September 2023. You cannot tell from most of these pages when a token last changed, which is why every section below carries the date we read it.
• Our own content had it wrong. Checking our blog against these pages, we found 26 mentions of "Googlebot-Extended" across 13 posts, a token Google does not document, often described as the crawler behind AI Overviews. Google's page names Google-Extended, and says it does not affect Google Search. We also found the Anthropic tokens anthropic-ai and Claude-Web in 4 posts and in our robots.txt. The Anthropic tokens are fixed; some older posts still use the Googlebot-Extended name, and until they are corrected this page is the reference.

Our view, which could be wrong: most "allow these AI crawlers" advice, ours included, was written from other lists rather than from the operators' pages, which is how a nonexistent token spread across a whole category of posts. If a list does not link each token to its operator, treat it as a starting point, not a source.

What Actually Crawled rankscope.ai in the Last 30 Days

Documentation says what a bot is for. Logs say what it does. These are the AI crawler requests Cloudflare recorded on rankscope.ai in the 30 days to 23 October 2026, read from Cloudflare's AI Crawl Control report. The labels are Cloudflare's, assigned by user agent, and the counts are successful requests only (HTTP 2xx), for a reason explained below.

Cloudflare also counted Baidu (560), PetalBot (470), TikTok Spider (260) and Google-CloudVertexBot (20). In total there were 28,180 successful requests from bots Cloudflare classes as AI or search crawlers.

What we take from it, for one small B2B site over one month:

• Training crawlers fetched the most. Cloudflare's "AI Crawler" category, which is mostly training bots, made 13,270 successful requests, against 7,680 for "AI Search" and 2,700 for "AI Assistant" fetches.
• The top number is one burst. meta-externalagent's 7,190 came almost entirely from a few days in early October; in the most recent 7 days it was not in the top five. A 30-day total can describe a spike rather than a habit.
• The second-busiest crawler has no documentation. Bytespider made 3,710 successful requests, more than OAI-SearchBot, and we could not find a page from ByteDance explaining what it collects or whether it follows robots.txt.
• OpenAI's search crawler outworks its training crawler. OAI-SearchBot made 2,630 successful requests against 420 for GPTBot, and ChatGPT-User, which fetches a page when someone asks ChatGPT to read it, made 2,250. For citation purposes, the bots that matter were the busy ones.
• Many requests failed, and some were impostors. Across all status codes there were about 37,000 requests, of which about 8,000 were unsuccessful. The paths that failed most often were not broken pages but probes for secret files, such as /.env.local, /localhost.key and /.boto, sent under AI crawler names. That is why the table counts successful requests only, and why verifying a bot against its operator's published IP ranges matters. We cannot tell from this report how many failed requests under each name were genuine.

One site and one month is a sample, not a benchmark. If you run Cloudflare, the same report for your own domain takes a minute to read and will tell you more about your site than any list.

The Three Jobs, and Why They Matter More Than the Names

Almost every large AI company now runs separate bots for separate jobs. The names are inconsistent, so group them by what they do.

Training crawlers collect pages that may be used to train future models. GPTBot, ClaudeBot, meta-externalagent, MistralAI-Training and Amazonbot are in this group. Blocking one is a statement about training, and it does not by itself remove you from anything a user sees today.

Search crawlers build the index an AI assistant searches when it answers a question and cites sources. OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Amzn-SearchBot and MistralAI-Index are in this group. These are the ones that decide whether you can be cited. OpenAI's documentation is explicit: opting out of OAI-SearchBot hides your site from ChatGPT search answers, though it can still appear as a navigational link.

User-initiated fetchers load a specific page because a person asked the assistant to, for example by pasting a URL. ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, meta-externalfetcher and MistralAI-User are in this group. Several operators say these may not follow robots.txt, on the reasoning that a person, not a crawler, made the request.

If you want to stay out of model training but remain visible in AI answers, block the training crawlers and allow the search crawlers. If you block everything, you opt out of the answers too.

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot

Source: OpenAI's crawler documentation, read 23 October 2026. The page shows no last-updated date.

• GPTBot crawls content that "may be used" to improve OpenAI's foundation models. Disallowing it tells OpenAI not to use your content for training. Published IP ranges: openai.com/gptbot.json.
• OAI-SearchBot surfaces websites in ChatGPT search. It follows robots.txt, and OpenAI says robots.txt changes for search can take about 24 hours to apply. IP ranges: openai.com/searchbot.json.
• ChatGPT-User fetches pages when a user takes an action in ChatGPT or a custom GPT. OpenAI says it is not used to crawl the web automatically, that robots.txt rules "may not apply", and that it does not decide whether content appears in search. IP ranges: openai.com/chatgpt-user.json.
• OAI-AdsBot only visits pages submitted as ChatGPT ads, to check that the landing page is safe. OpenAI says its data is not used for training.

The common mistake: treating GPTBot as "the ChatGPT crawler". Allowing GPTBot is not what gets you cited in ChatGPT search; OAI-SearchBot is. Older advice, including some of ours, conflated the two.

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Source: Anthropic's help article on its crawlers, last updated 7 April 2026, read 23 October 2026.

• ClaudeBot collects web content for model training. Blocking it signals that future material from your site should be excluded from training data. It also honours the non-standard Crawl-delay directive.
• Claude-SearchBot indexes content to improve Claude's search results. Blocking it may reduce your visibility in those results.
• Claude-User fetches pages when a person asks Claude a question. Blocking it may reduce visibility in user-directed web search.

Anthropic says all three respect robots.txt, and that you need the rules on each subdomain you want covered. It also warns that blocking by IP address may not work reliably, because it stops the bots reading robots.txt in the first place.

Tokens you may still see in old guides: anthropic-ai and Claude-Web. Neither appears in Anthropic's current article. We cannot tell from Anthropic's page whether they are still honoured, so do not rely on them alone. Our own robots.txt listed both until today.

Google: Google-Extended, GoogleOther, and What Controls AI Overviews

Source: Google's list of common crawlers, last updated 15 October 2026, and Google's guide to AI features and your website, last updated 10 December 2025. Both read 23 October 2026.

• Google-Extended is not a crawler. Google calls it "a standalone product token": it has no user agent of its own, and crawling is done by Google's existing crawlers. It controls whether content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
• GoogleOther is a generic crawler that Google's product teams may use for research and one-off crawls. Rules for it do not affect any specific Google product.

What about AI Overviews and AI Mode? Google's AI features guide says that robots.txt rules for Googlebot are "the control for site owners to manage access to how their sites are crawled for Search", and describes AI as built into Search. To limit what is shown from a page, it points to nosnippet, data-nosnippet, max-snippet and noindex. The guide does not name a separate crawler for AI Overviews or AI Mode, and it does not say Google-Extended affects them.

So in practice: blocking Google-Extended is not a way to leave AI Overviews, and there is no AI Overviews token to block. Blocking Googlebot removes you from Search entirely.

Perplexity: PerplexityBot and Perplexity-User

Source: Perplexity's crawler guide, read 23 October 2026. No date shown.

• PerplexityBot surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models, and it recommends allowing it if you want to appear in results. IP ranges: perplexity.com/perplexitybot.json.
• Perplexity-User fetches a page when a user asks a question, so Perplexity can answer and link to it. Perplexity says it is not used for crawling or for training, and that because a user triggers the fetch it "generally ignores" robots.txt. IP ranges: perplexity.com/perplexity-user.json.

Perplexity notes that robots.txt changes can take up to 24 hours to take effect.

Apple: Applebot and Applebot-Extended

Source: Apple's Applebot support page, published 4 September 2026, read 23 October 2026.

• Applebot is Apple's search crawler. Its data powers features in Spotlight, Siri and Safari, and Apple says it "may also be used to help train Apple foundation models". If your robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. It does not follow Crawl-delay.
• Applebot-Extended does not crawl. It controls whether content Applebot has already fetched may be used to train Apple's foundation models. Apple says pages that disallow Applebot-Extended can still appear in search results, and that the rule is not considered in search ranking.

Meta: meta-externalagent, meta-webindexer, meta-externalfetcher

Source: Meta's web crawler documentation, read 23 October 2026. No date shown.

• meta-externalagent crawls for uses such as training foundation AI models and indexing content.
• meta-webindexer crawls to improve Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content.
• meta-externalfetcher fetches individual links at a user's request and supports agentic features. Meta says it "may bypass robots.txt rules."
• meta-externalads crawls for advertising and other business products.
• facebookexternalhit builds link previews when a page is shared on Facebook, Instagram or Messenger. Meta says it may bypass robots.txt for security or integrity checks.

Meta asks for up to 24 hours for robots.txt changes to take effect.

Amazon: Amazonbot, Amzn-SearchBot, Amzn-User

Source: Amazon's Amazonbot page, read 23 October 2026. No date shown.

• Amazonbot is used "to improve our products and services" and "may be used to train Amazon AI models."
• Amzn-SearchBot improves search experiences in Amazon products, such as Alexa. Amazon says it does not crawl content for generative AI training. If robots.txt does not mention it but allows other search bots, it follows the rules given to those bots.
• Amzn-User fetches live information for user requests, such as Alexa questions. Amazon says it does not crawl for training and "may not follow all robots.txt directives."

Amazon caches robots.txt for up to 30 days, treats each token independently, and does not support Crawl-delay.

Microsoft: Bingbot and Copilot

Microsoft does not run a separate AI crawler token that we could find. Copilot's answers draw on Bing, and Bing's control for AI answers is a meta tag rather than a robots.txt token. In a September 2023 post, written when Copilot was still called Bing Chat, Bing said:

• NOCACHE: content may still appear in chat answers, but only the URL, title and snippet are shown, and only those may be used for training.
• NOARCHIVE: content is not included in chat answers or linked from them, and is not used for training.
• Either tag leaves the page in normal Bing search results.

That guidance is three years old. Check Bing Webmaster Tools before relying on it.

Mistral, DuckDuckGo and Common Crawl

Mistral (documentation, read 23 October 2026, no date shown) runs three: MistralAI-Training builds training datasets and can be disallowed in robots.txt; MistralAI-Index crawls for search in Mistral's assistant, and Mistral says that content is not used for training; MistralAI-User fetches pages when a user asks a question. The page does not describe robots.txt handling for the last two.

DuckDuckGo (documentation, read 23 October 2026) runs DuckAssistBot, which crawls pages in real time for AI-assisted answers that cite their sources. DuckDuckGo says the data is not used to train AI models, that opting out takes 72 hours, and that opting out does not affect organic rankings.

Common Crawl (documentation, read 23 October 2026) runs CCBot, which builds an open repository of web crawl data for anyone to analyse. Its page does not mention AI training. Because the archive is open to anyone, sites that want to keep content out of datasets built by third parties often block CCBot as well.

Bots We Could Not Find Documentation For

Two names appear on most AI crawler lists without an operator page we could find on 23 October 2026:

• xAI (Grok). We found no published crawler documentation from xAI.
• Bytespider (ByteDance). We found no official documentation from ByteDance.

We have left both out of the table rather than repeat third-party claims about what they do. If you see either in your server logs, you can still disallow the name in robots.txt, but without documentation there is no published commitment that it will be honoured.

robots.txt Templates

Three common positions. Each assumes the rest of your robots.txt allows normal search crawlers.

1. Visible in AI answers, out of model training. Allow the search crawlers, block the training crawlers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: MistralAI-Training
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

OAI-SearchBot, Claude-SearchBot, PerplexityBot and the rest stay allowed by your general rules. Note that Amazonbot and Applebot both crawl for more than training, so blocking them has side effects; the -Extended tokens let you opt out of training without blocking the crawler itself.

2. Fully open to AI. If you sell to people who research in AI assistants, you probably want every search crawler and fetcher allowed. A plain User-agent: * / Allow: / does that; listing the AI tokens explicitly makes your intent clear to anyone reading the file.

3. Out of AI entirely. Disallow every token in the table. Two caveats: user-initiated fetchers may not follow robots.txt, by their operators' own description, and you will no longer be cited in AI answers from those companies.

Whatever you choose, apply it on every subdomain, wait 24 to 72 hours, and then confirm it in your server logs.

How to Check Which AI Crawlers Visit Your Site

• Server or CDN logs. Search the user-agent field for the tokens in the table. On Cloudflare, the AI Crawl Control report does this for you, as in our 30-day numbers above. This is the only way to see what is really visiting, as opposed to what your robots.txt says.
• Verify that a bot is real. Anyone can put "GPTBot" in a user-agent string. OpenAI and Perplexity publish IP ranges as JSON files, linked above; Common Crawl supports reverse DNS lookups. A request claiming to be a known bot from outside its published ranges is not that bot.
• Check what the engines actually say. Allowing the search crawlers makes you eligible to be cited; it does not mean you are. Our free AI Visibility Checker runs real prompts against ChatGPT, Google AI Overviews, Google AI Mode and Perplexity and shows whether your brand appears.

Mistakes We See Most Often, Including Our Own

• Blocking GPTBot to "stay out of ChatGPT". It keeps you out of training, not out of ChatGPT search. That is OAI-SearchBot.
• Blocking Google-Extended to leave AI Overviews. Google says it does not affect inclusion in Search. There is no AI Overviews token.
• Copying token lists from old posts. Anthropic's current article lists ClaudeBot, Claude-SearchBot and Claude-User, not anthropic-ai or Claude-Web. Until 23 October 2026, rankscope.ai's own robots.txt listed anthropic-ai, Claude-Web and a Gemini token we cannot find documented anywhere, and labelled Google-Extended as the "Google AI Mode crawler", which Google's documentation does not support. We have corrected the file and removed the same Anthropic tokens from older posts.
• Assuming robots.txt stops everything. User-initiated fetchers are the exception, by design.
• Treating llms.txt as a crawler control. It is a different file with a different purpose, and Google Search, including AI Overviews and AI Mode, does not use it. If you want one anyway, our llms.txt generator builds it.

Frequently Asked Questions

What is an AI crawler?

An AI crawler is a bot an AI company uses to fetch web pages. Each one does one of three jobs: collecting training data for future models, building a search index the assistant uses to answer and cite sources, or fetching a specific page live because a user asked. Operators publish a robots.txt token for each one so site owners can allow or block it.

What is GPTBot?

GPTBot is OpenAI's training crawler. OpenAI says content it collects may be used to improve its foundation models, and disallowing GPTBot in robots.txt tells OpenAI not to use your content for training. It is not the crawler that decides whether you appear in ChatGPT search; that is OAI-SearchBot.

Does blocking GPTBot remove my site from ChatGPT?

No. Blocking GPTBot opts your content out of training. OpenAI's documentation says OAI-SearchBot is the crawler for ChatGPT search, and opting out of OAI-SearchBot is what hides your site from ChatGPT search answers.

Does Google-Extended affect AI Overviews or AI Mode?

Google's documentation says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls use for Gemini model training and for grounding in Gemini Apps and Vertex AI. Google's AI features guide points to Googlebot's robots.txt rules and to snippet controls such as nosnippet for managing how content appears in Search.

What is ClaudeBot?

ClaudeBot is Anthropic's training crawler. Anthropic also runs Claude-SearchBot, which indexes content for Claude's search results, and Claude-User, which fetches pages when a person asks Claude a question. Anthropic says all three respect robots.txt.

How do I block AI crawlers but stay visible in AI answers?

Disallow the training crawlers and the training control tokens, such as GPTBot, ClaudeBot, meta-externalagent, MistralAI-Training, Google-Extended, Applebot-Extended and CCBot, and leave the search crawlers allowed, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. Apply the rules on every subdomain and check your server logs after 24 to 72 hours.

This page is checked against each operator's documentation and carries the date of the last read. If a token here has changed, the operator's page linked in that section is the authority.