AI Crawler Access Checker

What this checker tests

It checks 15 user-agent tokens, grouped by what blocking them changes. The groups matter more than the names: blocking a training crawler and blocking a search crawler are very different decisions.

• Search and answers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Amazonbot. Block these and you can disappear from the AI answers that cite sources.
• User-triggered fetches: ChatGPT-User, Claude-User and Perplexity-User visit a page when a person asks the assistant about it.
• Model training: GPTBot, ClaudeBot, Meta-ExternalAgent and CCBot collect content that may be used to train models.
• Usage control tokens: Google-Extended and Applebot-Extended are not crawlers. They tell Google and Apple whether content their normal crawlers fetched may be used for AI training.

Each token, and what it does, was taken from the operator's own documentation, which every result links to. The list was last reviewed on 28 October 2026.

How it decides allowed or blocked

The checker follows RFC 9309 and Google's robots.txt documentation:

1. A crawler obeys the group whose User-agent line names it, ignoring case. Only when no group names it does it fall back to User-agent: *. It never combines the two.
2. Inside that group, the rule with the longest matching path wins. If an Allow and a Disallow tie, Allow wins.
3. * matches any characters and a trailing $ marks the end of the URL.
4. A missing robots.txt (HTTP 4xx) allows everything. A server error (5xx or 429) makes Google stop crawling until the file loads, so the checker reports that instead of a list of results.
5. Only the first 500 KiB of the file counts.

Rule 1 catches people out. If you add a User-agent: GPTBot group, GPTBot stops reading your * group entirely, including any Disallow: /admin/ you put there.

Example: anobee.com, checked on 27 October 2026

This is the robots.txt anobee.com served when this tool was built:

User-Agent: *
Allow: /
Disallow: /search/

Sitemap: https://anobee.com/sitemap.xml

No group names an AI crawler, so all 15 follow the * group. The checker reports every one of them as allowed on the home page, decided by line 2 (Allow: /). Testing the path /search/ instead flips every result to blocked, decided by line 3. That is intended: internal search results are thin pages no crawler needs.

Anobee allows the training crawlers on purpose. The site publishes to be found and cited, and getting cited by ChatGPT starts with letting it read the page. Your choice can differ, and the generator under the checker writes either policy.

What the result cannot tell you

• Firewalls and CDNs. Many hosts and CDNs can block AI bots before robots.txt is ever read. A site can pass this check and still turn the crawler away. Check your host's bot settings and server logs.
• Compliance. robots.txt is a request. The crawlers listed say they honour it, but OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
• Page-level controls. Meta robots tags, X-Robots-Tag headers and login walls are not checked.
• Whether you get cited. Being allowed is necessary, not sufficient. Citations depend on the page itself.

What to do next

• New to the file? Start with the robots.txt guide for beginners.
• Wondering which bot feeds which product? Read how ChatGPT search finds and cites pages and how Google AI Overviews work.
• Crawlers allowed but still not cited? Work through how to get cited by ChatGPT and the technical SEO mistakes checklist.
• Thinking about llms.txt as well? Build or check one with the llms.txt Generator and Validator, and read how to implement llms.txt.

Frequently asked questions

Last updated 28 October 2026 by Bibek Thapa. Privacy: pasted files never leave your browser. A website check asks Anobee's server to fetch that site's /robots.txt once, with the user agent AnobeeRobotsChecker/1.0; neither the address nor the file is stored.