Robots.txt Generator for AI Crawlers

Control which AI crawlers can access your site

Free tool

Build your AI robots.txt

43 AI crawlers

Toggle which AI crawlers can access your site. Copy or download the generated robots.txt.

43 allowed

0 blocked

GPTBot (OpenAI)

OAI-SearchBot (OpenAI)

ChatGPT-User (OpenAI)

OAI-AdsBot (OpenAI)

ClaudeBot (Anthropic)

Claude-SearchBot (Anthropic)

Claude-User (Anthropic)

anthropic-ai (Anthropic)

PerplexityBot (Perplexity)

Perplexity-User (Perplexity)

GoogleOther (Google)

Google-CloudVertexBot (Google)

Google-NotebookLM (Google)

Google-Extended (Google)

Applebot-Extended (Apple)

Amazonbot (Amazon)

Amzn-SearchBot (Amazon)

Amzn-User (Amazon)

meta-externalagent (Meta)

meta-webindexer (Meta)

meta-externalfetcher (Meta)

FacebookBot (Meta)

MistralAI-Index (Mistral)

MistralAI-User (Mistral)

CCBot (Common Crawl)

cohere-training-data-crawler (Cohere)

Bytespider (ByteDance)

TikTokSpider (ByteDance)

AI2Bot (Allen Institute)

Diffbot (Diffbot)

ImagesiftBot (The Hive)

Timpibot (Timpi)

Omgilibot (Webz.io)

Webzio-Extended (Webz.io)

YouBot (You.com)

DuckAssistBot (DuckDuckGo)

Googlebot (Google)

bingbot (Microsoft)

YandexBot (Yandex)

Baiduspider (Baidu)

DuckDuckBot (DuckDuckGo)

PetalBot (Huawei)

Applebot (Apple)

Sitemap URL (optional)

Output

# robots.txt: Generated by the PingAura Robots.txt Generator
# Generated: 2026-11-11

# Default: allow all crawlers
User-agent: *
Allow: /

# Allowed AI crawlers
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-AdsBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: GoogleOther
Allow: /

User-agent: Google-CloudVertexBot
Allow: /

User-agent: Google-NotebookLM
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: Amzn-User
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: meta-externalfetcher
Allow: /

User-agent: FacebookBot
Allow: /

User-agent: MistralAI-Index
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: CCBot
Allow: /

User-agent: cohere-training-data-crawler
Allow: /

User-agent: Bytespider
Allow: /

User-agent: TikTokSpider
Allow: /

User-agent: AI2Bot
Allow: /

User-agent: Diffbot
Allow: /

User-agent: ImagesiftBot
Allow: /

User-agent: Timpibot
Allow: /

User-agent: Omgilibot
Allow: /

User-agent: Webzio-Extended
Allow: /

User-agent: YouBot
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

User-agent: YandexBot
Allow: /

User-agent: Baiduspider
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: PetalBot
Allow: /

User-agent: Applebot
Allow: /

See if AI engines actually notice your site

Free visibility check across ChatGPT, Gemini and Perplexity. No signup, runs on your own domain.

Run free check

What you get

This robots.txt generator builds a standards-compliant file in seconds, with individual toggles for 18 major AI bots including GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Your robots.txt is the first file a crawler reads, so it decides which AI systems may use your content long before any of them reach a page.

Toggle access for 18 AI crawlers with a single click. No manual syntax required.
Generate a valid, standards-compliant robots.txt ready to deploy at your domain root.
Pair with LLMs.txt for complete AI access control: decide who crawls and what they discover.

How it works

Toggle each AI crawler on (allow) or off (block) based on your strategy.
Optionally add your sitemap URL for crawler discovery.
Copy or download the generated robots.txt and publish it at your domain root.

Who it's for

Brands that want to appear in ChatGPT and Perplexity but block data-harvesting bots.
Publishers managing AI crawler access across multiple domains and subdomains.
AEO teams building an intentional AI access strategy as part of their optimization workflow.

Your site changes. Keep these files accurate automatically.

Start free trial

Customer results

#1
ChatGPT rank — ahead of JLL, Cushman & Wakefield, Colliers
CBRE India
Read the case study: CBRE India
#1
ChatGPT rank (from #10)
Matrix Comsec
Read the case study: Matrix Comsec
7x
more AI conversations reached
Ampere by Greaves Electric
Read the case study: Ampere by Greaves Electric

Frequently asked questions

Didn't find your answer? Talk to us

Where do I put the robots.txt file?

Place it at the root of your domain (e.g. yoursite.com/robots.txt). AI crawlers check this path automatically before indexing your content.

Can I allow some AI bots and block others?

Yes. Each bot gets its own User-agent block with Allow or Disallow rules. You might allow GPTBot and ClaudeBot for AI citation while blocking data-harvesting bots like CCBot or Bytespider.

Does blocking a bot remove existing content from their index?

Not immediately. Blocking a bot in robots.txt prevents future crawling. To request removal of already-indexed content, you need to use each platform's specific content removal process.

Should I include a sitemap in robots.txt?

Yes. Adding a Sitemap directive helps both traditional search engines and AI crawlers discover your content structure more efficiently.

How does this relate to LLMs.txt?

robots.txt controls crawl access (who can visit). LLMs.txt controls content discovery (what to prioritise). Use both together: robots.txt manages the gate, LLMs.txt guides the crawlers once inside.