What you get
Your robots.txt decides which AI systems may crawl your site, and most owners have never checked what it currently allows. This checker tests your domain against 43 known AI and search crawlers including GPTBot, OAI-AdsBot, ClaudeBot, PerplexityBot and Google-Extended, and shows at a glance who has access and who is shut out.
- Instantly see which of 43 major AI and search crawlers can and cannot access your pages.
- Identify accidental blocks that prevent AI systems from discovering your content.
- Check for LLMs.txt presence alongside robots.txt for complete AI access coverage.
How it works
Enter any public domain or URL to check.
The tool fetches your robots.txt and LLMs.txt, then tests 43 crawler user-agents.
Review per-bot results showing allowed, blocked, and the matching robots.txt rule.
Who it's for
- Site owners verifying they haven't accidentally blocked key AI crawlers.
- AEO teams auditing crawler access policies across domains and subdomains.
- Agencies reviewing client sites for AI discoverability before running AEO campaigns.
This checks one page. PingAura checks your whole site.
| Compared | This free tool | PingAura |
|---|---|---|
| Coverage | One page at a time | Pages across your site |
| History | One-off | Saved over time |
| Follow-up | Up to you | Issues tracked |
| AI answers | Not tracked | Mentions and citations |
Used by teams at
Customer results
#1
ChatGPT rank — ahead of JLL, Cushman & Wakefield, Colliers
CBRE India
Read the case study: CBRE India#1
ChatGPT rank (from #10)
Matrix Comsec
Read the case study: Matrix Comsec7x
more AI conversations reached
Ampere by Greaves Electric
Read the case study: Ampere by Greaves Electric
Frequently asked questions
Didn't find your answer? Talk to us
Which AI crawlers does this check?
It tests 43 AI and search crawlers including GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot, Google-Extended, GoogleOther, bingbot, Bytespider (ByteDance), CCBot (Common Crawl), Amazonbot, meta-externalagent, Applebot-Extended, MistralAI-Index, Diffbot, YouBot, and more. OAI-AdsBot matters if you plan to run ChatGPT Ads: block it and your ad landing page fails review.
What if I don't have a robots.txt?
If no robots.txt is found, all crawlers are assumed to have access by default. While that means AI bots can crawl your site, having an explicit robots.txt gives you control over which bots you allow and what paths they can access.
Should I block or allow AI crawlers?
If you want your content cited by AI platforms like ChatGPT and Perplexity, allow their crawlers. Blocking GPTBot, ClaudeBot, or PerplexityBot means those platforms cannot index your content and will not cite it in AI-generated answers.
What's the difference between robots.txt and LLMs.txt?
robots.txt controls crawl access: which bots can visit which paths. LLMs.txt is a content discovery file that tells AI crawlers which pages are most important and how your content is structured. You need both for complete AEO coverage.
How do I fix a blocked AI crawler?
Find the matching Disallow rule in your robots.txt and either remove it or add a specific Allow directive for that bot. For example, adding 'User-agent: GPTBot' followed by 'Allow: /' will grant GPTBot full access to your site.
Further reading
- Marketing for AI Agents: Your New Web DemographicAI agents are the first point of evaluation in discovery. Learn how Answer Engine Optimisation (AEO) ensures your brand is understood, trusted, and selected by AI systems.
- What Is AI Traffic? Why It Matters and How to Track ItAI traffic is website traffic from AI platforms like ChatGPT, Gemini, and Perplexity. Learn what it is, why it matters, and how to track and attribute it.
