AI-crawler policy
What robots.txt tells GPTBot, ClaudeBot and others.
The check runs on our server. We do not store the domain.
What this checks
This tool reads robots.txt and looks for rules addressed to six named AI crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, PerplexityBot, CCBot (Common Crawl) and meta-externalagent (Meta). Each one is reported allowed, blocked, or unspecified.
What to look for
"Unspecified" means that crawler has no rule of its own — it falls through to whatever User-agent: * says, which is a site-wide stance, not an AI-specific one. To opt a crawler out of training your content or answering questions from it, robots.txt needs a rule naming it directly, for example User-agent: GPTBot followed by Disallow: /.
A stance, not a score
Blocking these crawlers is a business decision, not a vulnerability, and this tool never marks either choice as good or bad.