Free AI crawler checker, robots.txt tester and validator
Enter your domain to see whether your robots.txt lets ChatGPT, Perplexity, Claude and Google's crawlers in, and the exact rule that decides each one. Syntax problems are flagged by line. Test any path, or paste a robots.txt to try rules before you publish them.
Free, no signup. Reads robots.txt, llms.txt and your sitemap.
Three kinds of AI crawler, three different decisions
Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Applebot) build the indexes AI answers draw on. Block one and that engine can't cite pages it can't crawl.
Live fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher) open a page while answering someone, for example when a user pastes your URL. Because a person started the visit, OpenAI says robots.txt rules may not apply to ChatGPT-User.
Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot, Bytespider, plus the Google-Extended and Applebot-Extended tokens) collect pages for model training. OpenAI and Google both document their training controls as separate from search, so you can opt out of training and still be found in their AI search.
Sources: OpenAI's crawler documentation and Google's list of common crawlers.
Example: allow AI search, opt out of training
One common setup. Several User-agent lines in a row share the rules under them. Paste it into the tester above (switch to "Paste robots.txt") to see how each crawler reads it.
# Let AI search engines crawl (and cite) the site User-agent: OAI-SearchBot User-agent: PerplexityBot User-agent: Claude-SearchBot Allow: / # Opt out of model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: / # Everyone else User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Note that a crawler named in its own group ignores the User-agent: * group entirely. In the example, OAI-SearchBot may crawl /admin/, because its own group allows everything. That catches a lot of sites out.
AI crawler checker FAQ
It reads your robots.txt and works out, for 17 AI crawlers, whether the rules let each one crawl a given path, and which line decides it. It also checks whether you have an llms.txt and a sitemap, and whether robots.txt points to the sitemap. You can test any path, or paste a robots.txt to test rules before you publish them.
Yes. Every line is checked, and problems are listed with their line number: rules that come before any User-agent line, missing colons, misspelt fields, paths that don't start with /, full URLs where a path belongs, sitemaps given as relative paths, and User-agent groups with no rules. It also catches a common grouping mistake: only Allow and Disallow lines end a group, so a User-agent line after a Crawl-delay line joins the group above it and shares its rules. Fields Google ignores, such as Crawl-delay and Noindex, are noted rather than flagged as errors.
If you want AI search engines to cite you, allow their search crawlers (OAI-SearchBot for ChatGPT search, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot) and the live fetchers that open a page when someone asks about it (ChatGPT-User, Claude-User, Perplexity-User). Training crawlers such as GPTBot, ClaudeBot and CCBot are a separate decision about whether your pages are used to train models. OpenAI documents each of its settings as independent of the others.
No. OpenAI documents GPTBot (model training) and OAI-SearchBot (ChatGPT search) as separate crawlers whose settings are independent. OAI-SearchBot is the one that decides search: OpenAI says sites that opt out of it are not shown in ChatGPT search answers, though they can still appear as navigational links. You can block GPTBot and allow OAI-SearchBot.
Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal there. AI Overviews and AI Mode are part of Search, which Googlebot crawls. Google-Extended is a robots.txt token, with no crawler of its own, that controls whether Google may use your content to train Gemini models and to ground answers in Gemini Apps.
robots.txt is a request, not a gate. Firewalls and CDNs can block a crawler before it ever reads robots.txt, and some of them have settings that block AI crawlers by default or with one switch. This checker reads robots.txt only, so if a crawler is allowed here but never shows up, check your CDN or firewall's bot settings.
It follows the robots.txt standard (RFC 9309). A crawler obeys the group that names it; only if no group names it does the User-agent: * group apply. Within that group the longest matching path wins, and on a tie Allow wins. * matches any characters and $ marks the end of a URL.
A 404 means there are no rules, so every crawler may crawl everything. A server error (5xx) is different: Google treats the whole site as disallowed while robots.txt keeps failing. A 403 is treated as 'no rules' by the standard, but it usually means a firewall refused the request, so the checker says so rather than guessing.
It's free with no account. We fetch your public robots.txt, llms.txt and sitemap from our server, as RankScope-AccessCheck rather than pretending to be any crawler. A result is kept for up to 5 minutes so repeat checks don't hit your site again; nothing else is stored. Pasted robots.txt files never leave your browser.
Crawlable is step one
Being allowed in doesn't mean being named. The free AI visibility checker asks ChatGPT, Perplexity, Google AI Overviews and AI Mode the questions your buyers ask, and shows who they recommend.