Free tool
Robots.txt Generator
Generate robots.txt rules for search crawlers and AI crawlers, with the distinctions that actually matter.
Googlebot ignores this. Bing honours it.
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /*?sort=
Sitemap: https://www.aeologic.ai/sitemap.xmlEach host and protocol needs its own file. A documentation subdomain has a separate robots.txt from your main site — which is the single most common cause of accidental AI invisibility we find in audits.
About the robots.txt generator
AI crawlers are not one thing
GPTBot, ClaudeBot and Google-Extended primarily gather training data. OAI-SearchBot and PerplexityBot fetch pages to answer live queries with citations. Blocking the second group removes you from AI answers entirely; blocking the first only affects training. Most brands want to allow the second group.
The most common expensive mistake
An inherited Disallow on a documentation or knowledge-base subdomain. It is the single most frequent cause of total AI invisibility we find in audits, and it is usually years old and unreviewed.
Robots.txt is not access control
It is a request, honoured by well-behaved crawlers. Never use it to hide sensitive content — and remember that disallowing a URL prevents crawling, not indexing. To keep a page out of the index, allow the crawl and serve a noindex directive.
Questions
Should I block GPTBot?
It depends on whether you want your content used for training. It does not affect whether ChatGPT can cite you in a live answer — that is OAI-SearchBot. Many brands allow both; publishers with licensing concerns often block GPTBot and allow OAI-SearchBot.
Where does robots.txt go?
At the root of each host and protocol: https://example.com/robots.txt. Subdomains need their own file, which is exactly why documentation subdomains get accidentally blocked.
One page at a time is slow.
Aeologic runs these checks across every URL you own and tells you which twenty to fix first.
7-day trial · No card required · Cancel anytime