Free tool
AI Crawler Access Checker
Test any URL against your robots.txt for every major AI and search crawler at once.
Paste the contents of the file, not the URL.
Testing path: /docs/getting-started
Googlebot
matched "*" — no Disallow matches this path.
Bingbot
matched "*" — no Disallow matches this path.
DuckDuckBot
matched "*" — no Disallow matches this path.
Applebot
matched "*" — no Disallow matches this path.
OAI-SearchBot
matched "*" — no Disallow matches this path.
ChatGPT-User
matched "*" — no Disallow matches this path.
PerplexityBot
matched "*" — no Disallow matches this path.
Claude-SearchBot
matched "*" — no Disallow matches this path.
GPTBot
matched "gptbot" — blocked by Disallow: /.
ClaudeBot
matched "*" — no Disallow matches this path.
Google-Extended
matched "*" — no Disallow matches this path.
CCBot
matched "*" — no Disallow matches this path.
Access is stage one of three. A crawler being allowed does not mean it has visited — check your server logs for actual hits, and confirm the page renders its main content without JavaScript.
About the ai crawler access checker
Why check separately per bot
Each engine runs different crawlers for different purposes, and a rule written for one often accidentally covers another. Checking a URL against all of them at once surfaces the mismatch immediately.
Access is necessary but not sufficient
A crawler being allowed does not mean it has visited. After confirming access, check your server logs for actual hits. Allowed-but-never-crawled usually points to a discovery problem: no internal links, no sitemap entry, or an orphaned section.
The two-group distinction
Training crawlers (GPTBot, ClaudeBot, Google-Extended) affect whether your content informs future models. Live-retrieval crawlers (OAI-SearchBot, PerplexityBot) affect whether you can be cited in an answer today. They deserve separate decisions.
Questions
My robots.txt allows everything but I get no AI citations. Why?
Access is stage one of three. Check that the page renders its main content without JavaScript, that it returns a 200, and that individual passages answer questions on their own. Most failures are at the passage level, not the access level.
Does this tool crawl my site?
No. It runs entirely in your browser against text you paste. Nothing is sent to a server.
One page at a time is slow.
Aeologic runs these checks across every URL you own and tells you which twenty to fix first.
7-day trial · No card required · Cancel anytime