Free tools

AI Crawler Checker

See which AI bots — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — your robots.txt lets in, and the exact rule that decides it.

How AI crawlers read your robots.txt

AI companies send their own crawlers to collect web pages. Each one announces itself with a user-agent token — GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity — and the major operators state that they follow robots.txt. That makes robots.txt the simplest place to decide which of them may read your site.

The checker downloads your robots.txt and evaluates it the way a compliant crawler does, for each bot separately and for the exact page you entered. You get a verdict per crawler and the rule that produced it, so you can see why a bot is blocked rather than guess.

Three kinds of AI crawlers

Training crawlers such as GPTBot, ClaudeBot, CCBot and Meta-ExternalAgent collect pages that may be used to train models. Google-Extended and Applebot-Extended are tokens rather than separate crawlers: they tell Google and Apple whether content fetched by their regular bots may be used for AI.

AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so assistants can find and cite them. Blocking these is what removes a site from AI answers, and also from the referral traffic those answers send.

User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User open a page only when a person asks the assistant about it. Several operators treat these as acting for the user, so check each operator's documentation for how they handle robots.txt.

A common setup is to block training crawlers and keep AI search and regular search open, so the content is not used for training but can still be found and cited.

How the rules are matched

robots.txt is split into groups, each starting with one or more User-agent lines. A crawler obeys only the most specific group that names it; if no group names it, it falls back to User-agent: *. Inside the group, the rule with the longest matching path wins, and Allow beats Disallow on a tie. Wildcards * and the end anchor $ are supported.

Two consequences surprise people. A bot with its own group ignores everything under *, so a site-wide block for * does not apply to GPTBot if GPTBot has a separate group. And if robots.txt answers with a server error, compliant crawlers treat the whole site as disallowed until it recovers, while a 404 means no restrictions at all.

Example: block training, keep search

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Run the checker after any change: it shows at once whether the new rules do what you intended for every crawler on the list, including Googlebot and Bingbot, which should normally stay allowed.

FAQ

Does blocking GPTBot or ClaudeBot remove my site from ChatGPT or Claude answers?
Not fully. Training crawlers (GPTBot, ClaudeBot, Google-Extended) and search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are separate. Blocking a training crawler keeps future crawls out of training data, while the assistant can still find and cite your pages through its search crawler unless you block that one too.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended is a control token, not a separate crawler: it tells Google not to use your content for Gemini models. Pages are still crawled by Googlebot and ranked in Search as before.
Do AI crawlers have to obey robots.txt?
robots.txt is a voluntary standard (RFC 9309). The major operators publish their user agents and state that they follow it, but nothing technically stops a crawler that ignores it. For hard blocking you need server-side rules on the user agent or IP ranges.
Why does my result differ from what I see in robots.txt?
Crawlers use only the most specific group that names them; if none does, they fall back to the * group. Within a group the longest matching rule wins, and Allow beats Disallow on a tie. A rule for * therefore does not apply to a bot that has its own group, which is a common surprise.
Can I run this check from code?
Yes. The same check is available in the Seonar API as ai_crawlers. Create a free API key and send the URL with "checks": ["ai_crawlers"] to get the per-crawler result as JSON.

More free tools