Your robots.txt decides which AI systems may crawl your site, and most owners have never checked what it currently allows. This checker tests your domain against 20 known AI crawlers including GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and shows at a glance who has access and who is shut out.
How it works
Get started in 3 simple steps
Enter any public domain or URL to check.
The tool fetches your robots.txt and LLMs.txt, then tests 20 AI user-agents.
Review per-bot results showing allowed, blocked, and the matching robots.txt rule.
Best use cases
Built for teams that take AI visibility seriously
Site owners verifying they haven't accidentally blocked key AI crawlers.
AEO teams auditing crawler access policies across domains and subdomains.
Agencies reviewing client sites for AI discoverability before running AEO campaigns.
One check is a snapshot. Want to see the trend?
Start free trialFAQ
Frequently asked questions
Which AI crawlers does this check?
What if I don't have a robots.txt?
Should I block or allow AI crawlers?
What's the difference between robots.txt and LLMs.txt?
How do I fix a blocked AI crawler?
Further reading
Learn the concepts behind this tool
- Marketing for AI Agents: Your New Web DemographicAI agents are the first point of evaluation in discovery. Learn how Answer Engine Optimisation (AEO) ensures your brand is understood, trusted, and selected by AI systems.
- What Is AI Traffic? Why It Matters and How to Track ItAI traffic is website traffic from AI platforms like ChatGPT, Gemini, and Perplexity. Learn what it is, why it matters, and how to track and attribute it.