AI crawlers: the complete user-agent list (2026)

AI crawler list 2026: every major bot in one table

The table below lists every major AI crawler in 2026, what it is for, and what its operator says about robots.txt. Each row was checked against the operator's own documentation on 5 November 2026. Purpose falls into three jobs: training (collecting text for future models), search (building the index an AI answer engine retrieves from) and user fetch (reading a page live because someone asked a question).

Two names that often appear on AI crawler lists are missing from the table on purpose. Bytespider (ByteDance) and the various xAI or Grok user-agents had no official operator page we could find in November 2026. What is known about them comes from third-party bot directories, and those directories disagree about whether the bots obey robots.txt. Treat any claim about them as unverified.

Training vs search vs user fetch: what blocking each does

Blocking an AI training bot keeps your own pages out of future training sets, while blocking a search or user-fetch bot removes you from live AI answers. These are different decisions, and most operators give each job its own user-agent so you can make them separately.

• Training bots (GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot, MistralAI-Training, plus the Google-Extended and Applebot-Extended tokens). Blocking stops your future pages from being collected for new models. It does not remove what others write about you. A large audit of training corpora (Longpre et al., "Consent in Crisis", NeurIPS 2024; graded strong) found robots.txt restrictions covered about 5% of all tokens in C4, RefinedWeb and Dolma, but 25% or more of the most important domains, between April 2023 and April 2024. GPTBot was the most-blocked crawler.
• Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Meta-WebIndexer, Amzn-SearchBot, MistralAI-Index). Blocking removes your pages from the index the engine retrieves from, so you stop being cited. OpenAI's documentation is explicit that blocking GPTBot does not take a site out of ChatGPT search, because OAI-SearchBot is a separate setting.
• User-fetch bots (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User, MistralAI-User). These read a page when someone asks about it. Several operators say robots.txt may not apply to them, so a block here is unreliable, and a firewall block simply means the assistant can't read your page when a buyer asks.

Google is the special case. Google-Extended controls whether Google may use crawled content for Gemini training and for grounding in Gemini Apps and the Vertex AI API. Google says it "does not impact a site's inclusion in Google Search". AI Overviews and AI Mode come from the Search index, so blocking Google-Extended keeps you in them. To limit what Search shows, Google's controls are snippet rules such as nosnippet, not Google-Extended.

Two findings from fingerprinted test pages (RESONEO, 2026; one research group, graded strong) show why the split matters for ChatGPT. OAI-SearchBot writes the snippet index, and ChatGPT-User writes a separate full-page Markdown cache that is shared across accounts and refreshed about 30 minutes after a read. In the same tests, GPTBot followed none of 8 URLs listed only in a sitemap, but all 8 URLs that were linked from other pages. That single test suggests internal links matter more than sitemaps for discovery. The guide on how ChatGPT chooses sources covers the snippet index in detail.

AI crawlers by company: what each operator documents

Each AI company documents its crawlers differently, and the details below are the ones that change what you should put in robots.txt. All of it comes from the operator pages linked in the table.

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI says GPTBot and OAI-SearchBot settings are independent, and that if both are allowed it may use one crawl for both purposes. Changes take about 24 hours to reach search. A site that opts out of OAI-SearchBot can still appear as a navigational link. OpenAI also lists OAI-AdsBot, which checks landing pages submitted as ChatGPT ads and is not used for training. OpenAI publishes IP lists for each bot: gptbot.json, searchbot.json and chatgpt-user.json at openai.com.

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic says all three bots honor robots.txt and support Crawl-delay, and that they "will not attempt to bypass CAPTCHAs". Blocking Claude-SearchBot keeps pages out of Claude's search index; blocking Claude-User stops Claude reading them for a user. Anthropic advises against IP blocking, which it says "impedes our ability to read your robots.txt file". It publishes its IP list at claude.com/crawling/bots.json. Its help page also describes a newer "no AI training" preference signal based on an IETF draft, and says a robots.txt rule that names ClaudeBot takes priority over it.

Perplexity: PerplexityBot and Perplexity-User

Perplexity recommends allowing PerplexityBot, which builds its search index and is not used for training. Perplexity-User fetches pages live for a question and "generally ignores robots.txt rules". Both have published IP lists at perplexity.ai (perplexitybot.json and perplexity-user.json).

Google: Googlebot and the Google-Extended token

Googlebot rules affect Google Search and the products built on it. Google-Extended has no HTTP user-agent: Google crawls with its normal agents and reads the token only as a usage rule. Google has also said it intends to support an AI-training preference signal from the same IETF work in early 2027.

Microsoft: Bingbot

Bingbot builds the Bing index that Microsoft Copilot grounds its answers in, so blocking Bingbot also removes a site from Copilot. Bing's guidance says the noarchive and nocache meta rules limit how Copilot uses a page, which is the finer-grained control.

Meta, Apple, Amazon, Common Crawl, DuckDuckGo, Mistral

• Meta: Meta-ExternalAgent trains models, Meta-WebIndexer feeds Meta AI search, and Meta-ExternalFetcher handles user requests and may bypass robots.txt. Meta says its crawlers may cache robots.txt for up to 24 hours. Its crawler page publishes no IP ranges.
• Apple: Applebot crawls for Spotlight, Siri and Safari, and its data may also train Apple's models. Applebot-Extended is only a usage token: blocking it opts out of training while pages stay in Apple search.
• Amazon: Amazonbot may train Amazon AI models; Amzn-SearchBot and Amzn-User are not used for training. Amazon says robots.txt changes take about 24 hours to apply, and it treats an unreachable file as if it did not exist.
• Common Crawl: CCBot builds a free public archive that many labs train on. Filtered Common Crawl was 60% of the GPT-3 training mix (Brown et al., 2020).
• DuckDuckGo: DuckAssistBot fetches pages for AI-assisted answers, and DuckDuckGo says the data is not used to train AI models. Opting out does not affect organic rankings.
• Mistral: MistralAI-Training collects training data, MistralAI-Index builds a search index, and MistralAI-User fetches pages for user questions in Vibe.

Sample robots.txt rules for AI crawlers

The most common robots.txt policy for AI crawlers blocks training bots and leaves search and user-fetch bots alone. Under the robots.txt standard (RFC 9309), a bot follows the group that names it and ignores the * group, and several User-agent lines can share one group. Put these lines in robots.txt, one per line, in this order:

• User-agent: GPTBot
• User-agent: ClaudeBot
• User-agent: Google-Extended
• User-agent: Applebot-Extended
• User-agent: Meta-ExternalAgent
• User-agent: CCBot
• User-agent: MistralAI-Training
• Disallow: /

Any bot not named in a group follows your * group, so search bots stay allowed as long as * doesn't block them. If your * group disallows a lot, add an explicit group for the AI search bots so they keep access to public pages:

• User-agent: OAI-SearchBot
• User-agent: Claude-SearchBot
• User-agent: PerplexityBot
• User-agent: Meta-WebIndexer
• Allow: /
• Disallow: /account/

How WAF and CDN bot rules block AI crawlers by accident

A firewall or CDN bot rule can block an AI crawler that your robots.txt allows, and the bot never tells you. Allowing search and retrieval bots, with no firewall challenge, is a binary gate for live AI answers: the research we rely on grades it confirmed, because a page the bot can't fetch can't be read live. Common causes:

• One-click "block AI bots" settings that catch search and user-fetch bots along with training bots. Check exactly which user-agents a CDN preset covers.
• JavaScript or CAPTCHA challenges. Anthropic says its bots won't bypass CAPTCHAs, and AI fetchers don't run JavaScript, so a challenge page is all they see.
• Rate limits and geo-blocks tuned for abusive traffic. AI crawlers often come from cloud data-center ranges in a few countries.
• robots.txt itself behind the firewall. If the bot gets a challenge or a 5xx error for robots.txt, rules differ by operator. Amazon, for example, treats an unreachable file as if it did not exist.
• IP blocks. Anthropic warns that blocking its IPs stops it reading robots.txt, so your opt-out may not register.
• Subdomains. robots.txt applies per host, so blog.example.com and shop.example.com each need their own file.

Content hidden behind client-side JavaScript is a related gate: the bot gets in but finds an empty shell. Check that your main text is in the server HTML. The Optimz site audit tests robots.txt rules for each AI crawler and flags firewall challenges, grading each finding by its evidence.

How to verify an AI crawler is real

To verify an AI crawler, match the request's IP address against the operator's published IP list, or run the reverse-then-forward DNS check where the operator documents one. User-agent strings are easy to fake, so a log line that says GPTBot proves nothing on its own.

The DNS check has two steps. First look up the hostname for the IP (host 66.249.66.1 on Linux, nslookup on Windows). Then look up that hostname and confirm it returns the same IP. Microsoft's own guidance warns against hard-coding Bingbot IPs, because they change, so refresh any IP list you allow-list on a schedule.

How to see AI crawler visits in server logs

AI crawler visits show up in your web server or CDN access logs as requests whose user-agent contains the bot's name. A search such as grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log is enough to start. Then check four things:

1. Status codes. A run of 403, 429 or 503 responses to an AI bot means a firewall or rate limit is blocking it, whatever robots.txt says.
2. robots.txt fetches. OpenAI marks its robots.txt requests in the user-agent, which tells you the bot is checking your rules.
3. Which URLs each bot reads. Search bots should reach your key pages; user-fetch hits on a page mean someone asked an assistant about it.
4. Whether the IP verifies. Filter out spoofed hits with the checks in the previous section before you count anything.

Watch for one gap: if your CDN serves cached pages, those requests never reach your origin server, so origin logs undercount. Use CDN logs where you can. Optimz Real AI data reads server logs, keeps only AI crawler fetches whose IPs verify, and shows them next to AI-referred visits.

AI crawler checklist for 2026

The AI crawler checklist below covers what to confirm on every site you manage. Run it again whenever robots.txt, the CDN or the firewall changes.

• Decide on training separately from search: block GPTBot, ClaudeBot, CCBot and the -Extended tokens only if the client wants out of training.
• Keep OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Meta-WebIndexer allowed on public pages.
• Don't rely on robots.txt to stop user-fetch bots; several operators say it may not apply.
• Repeat private paths inside every named User-agent group, not just under *.
• Check CDN bot presets, JavaScript challenges, rate limits and geo-blocks against each AI user-agent.
• Serve main content in the HTML, not only after JavaScript runs.
• Give each subdomain its own robots.txt.
• Verify bots by published IP list or reverse DNS before you trust a log line.
• Read CDN logs for 403, 429 and 503 responses to AI bots every month.
• Link new pages from existing ones instead of relying on the sitemap alone.