An AI crawler is an automated client that fetches web content for an AI provider. The purpose matters: some bots gather material that may be used in model training, others build a search index, and some fetch a page because a user asked an assistant to visit it.
OpenAI separates GPTBot for potential training material, OAI-SearchBot for ChatGPT search discovery, and ChatGPT-User for certain user-initiated fetches. Anthropic similarly distinguishes ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity lists PerplexityBot for search indexing and Perplexity-User for user requests. These names describe different traffic, even when the requests come from the same company.
Server logs can show which user-agent a request claims to use, along with the URL and response status. A user-agent string alone does not verify identity; IP checks or signed requests can provide stronger evidence. Keep bot requests separate from human referrals when reporting AI traffic.
Robots.txt can express access preferences for named crawlers, but the applicable rules vary by provider and by bot purpose. Google-Extended and Applebot-Extended are control tokens used in robots.txt, not standalone crawlers that appear in logs. A site owner should review each provider's current documentation before deciding which traffic to allow or block.