An AI crawler is a bot that an AI company runs to fetch and read web pages, either to train a model or to pull fresh content into a live answer. The four you will actually see in server logs are GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), and Google-Extended, the user-agent token that controls whether Google can use your content to ground Gemini and AI Overviews.
For a marketing buyer, this matters because a page that these bots cannot read might as well not exist to the AI system in question. Traditional SEO tolerated a slow, JavaScript-heavy page because Googlebot would eventually render it and a human would wait a few extra seconds. AI answer generation does not offer that patience. If GPTBot hits a blank shell because your content loads client-side, nothing from that page reaches ChatGPT's answer, no matter how well the content itself is written. Pipeline that used to come from a page ranking on page one now depends on a bot successfully parsing that page in milliseconds.
In practice, each crawler identifies itself with a distinct user-agent string, and site owners can allow or block each one individually in robots.txt. Most AI crawlers do not execute JavaScript, so they need server-rendered HTML to see anything beyond a loading spinner. Training crawlers like GPTBot and ClaudeBot sweep the web periodically to build or refresh a model's knowledge base. A separate category, sometimes called fetchers, works differently: OpenAI's ChatGPT-User and Perplexity's user-triggered fetch requests hit a page on demand, the moment a real person asks a question that needs current information from that URL. Both categories show up as "AI crawler" traffic in a server log, but one builds a dataset months in advance and the other retrieves your page in the same second someone is reading the answer.
Take a mid-sized SaaS company with roughly 800 published articles. A 30-day pull from its CDN logs showed 118,000 GPTBot requests, 42,000 ClaudeBot requests, and 9,000 PerplexityBot requests. When the team checked which pages actually got hit, 30% of requests landed on URLs that returned an empty client-rendered shell rather than full HTML, because those templates relied on a browser-side data fetch. After the engineering team switched those templates to server-side rendering, citations of the brand in ChatGPT answers roughly tripled over the following two months, even though not a single word of the underlying content changed.
People commonly mix up AI crawlers with the search engine crawlers they already know, Googlebot and Bingbot. They are not the same bots, and blocking one does not touch the other. Googlebot exists to build a search index that ranks pages against a query. An AI crawler like GPTBot exists to feed a language model, either during training or at answer time, and it has no concept of ranking at all; a page is either usable material or it is invisible. A robots.txt rule written years ago to manage Googlebot's crawl budget will do nothing to stop or allow GPTBot, ClaudeBot, or PerplexityBot. Each user-agent needs its own explicit directive, and a site that never reviewed its robots.txt since the AI crawlers arrived is often blocking, or accidentally exposing, content without anyone deciding to.