Free agency tools / AI Crawler Checker
Can AI assistants read your site?
Your robots.txt decides whether ChatGPT, Claude, Perplexity and Gemini can crawl — and cite — your pages. We test 14 AI crawlers and show exactly which are allowed, which are blocked, and the line doing it.
Results for
How it works
- 1Enter a domain — we fetch its robots.txt and read the user-agent rules.
- 2For each AI crawler we test access to the site root and apply the same longest-match logic search engines use.
- 3Blocked crawlers show the exact robots.txt line responsible, grouped by whether they affect AI answers or model training.
Frequently asked
What's the difference between AI search and training crawlers?
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect data to train models. AI search / grounding crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User) fetch pages live to answer and cite. Blocking training is a values choice; blocking search removes you from AI answers.
Should I block AI crawlers?
It depends on your goal. If you want visibility and citations in ChatGPT, Perplexity and Google's AI answers, keep the search and user crawlers allowed. Many sites block only the training crawlers — opting out of training while staying citable.
Why does my site block a crawler I didn't mean to?
A blanket 'Disallow: /' under 'User-agent: *' blocks everything that doesn't have its own rule, and old copy-pasted robots.txt snippets often disallow bots by name. This tool shows you the exact offending line so you can fix just that.
Is robots.txt the only thing that controls this?
It's the main signal, and the one AI crawlers publicly honour. Server- or firewall-level blocks (by user-agent or IP) can also stop a crawler even when robots.txt allows it — this tool reads robots.txt, which is what the crawlers check.