AI Crawlers and Bots: What They Crawl, Why and What It Means for SEO

Pull your server logs for the last seven days and sort the user agents by volume. Between Googlebot and the scrapers you already know you will find GPTBot, ClaudeBot, PerplexityBot and a few names that mean nothing to you. Most of them never appear in your analytics, because they do not run your tracking script, yet all of them are reading your catalog, your docs and your blog. These AI crawlers are now a visitor class of their own.

That traffic is large enough to plan around. AI crawlers have become a second class of visitor, and they behave differently from the first.

This guide explains how AI crawlers and bots work: the four kinds that visit, why they crawl, how much they take, how they differ from search engine bots and what to do about them. Where a section raises a question about your own site, a short recipe shows how to answer it from your logs.

JetOctopus is a cloud-based technical SEO platform built for large websites, e-commerce catalogs and the agencies that manage them, and its log analyzer classifies more than 40 crawler types. The recipes in this guide use it, and each one names the report to open.

The Evolution of Crawlers: From Search Bots to AI Bots

For most of the web’s history, bots sorted themselves into two camps. Good bots, such as search engine crawlers and uptime monitors, read your pages and sent visitors back. Bad bots, such as scrapers and spam tools, took what they wanted and gave nothing in return. The good bots vs bad bots split was never tidy, but a robots.txt file and a firewall covered most of it.

Bots are no longer a side issue: Imperva’s 2026 Bad Bot Report puts them at over 53% of all web traffic. AI crawlers fit neither camp. Most of them announce themselves and read robots.txt, which is how good bots behave, yet they collect content for products that can answer your customers’ questions without sending anyone to your site, which looks more like a bad bot’s bargain. The numbers later in this guide show how lopsided that trade can be.

The Types of AI Bots

An AI crawler is a bot that collects web content for an AI system, and the most useful way to sort the ones in your logs is by what the visit is for. Four groups cover almost everything you will see.

The list below gives each token the way its operator documents it, so you can match what you see in your logs.

The groups overlap more than the table suggests. Google-Extended is a token rather than a crawler: Google states that it has no separate user agent string, and crawling happens with its existing user agents. Applebot serves search and may also help train Apple’s models, which is why Apple offers Applebot-Extended as a separate opt-out. Treat the tables as a map and each operator’s documentation as the final word.

Two cautions before you write rules from this list. Older guides also name Claude-Web and anthropic-ai, but Anthropic’s current crawler page lists only ClaudeBot, Claude-User and Claude-SearchBot, so review any rule that still depends on the old names. And Crawl-delay support is patchy: Anthropic says it supports the directive, Amazon says its crawlers do not, Google does not support it either and the OpenAI and Perplexity pages we reviewed do not mention it.

Expect surprises when you first look. In JetOctopus’s case study of a large marketplace, the biggest AI crawler turned out to be Meta-ExternalAgent, ahead of both OpenAI and Anthropic.

Which AI bots visit my site, and what type are they?

Where: the AI Bots report in the log analyzer

Steps:

1. Open the AI Bots report and set a period of several weeks, because a few days will not show a pattern.
2. Check that the report counts only verified requests. JetOctopus matches claimed AI bots against the IP ranges their operators publish, so impersonators do not inflate the numbers (setup details are in our guide to tracking AI crawler activity).
3. Sort the bots into the four groups from the table above: training, search, user and agent.
4. Find the top three bots by requests and note which site sections each one visits.
5. Save the AI bot filter as a segment, for example “AI Crawlers”, so every later report starts from it.

What to look for: the share of search and user bots compared with training bots, unknown user agents and any bot that is active in a section you have closed.

What to do next: set an access policy per bot type (see the section on blocking below) and check unknown user agents against the operator documentation in the list above.

Why AI Bots Crawl: Content Hunger and Live Answers

AI bots crawl for two reasons, and both are about content. Models need huge volumes of text to learn from, and assistants need current pages to answer from.

The first reason feeds training. OpenAI describes GPTBot as the crawler used to make its generative AI foundation models more useful and safe, and Anthropic says ClaudeBot collects web content that can contribute to training. Some of that collection never touches your server, because the nonprofit Common Crawl publishes an open archive of web crawl data that anyone can analyze, and its CCBot crawler keeps the archive growing.

The second reason is live answers. Search bots such as OAI-SearchBot and PerplexityBot build an index that an assistant can search, and user bots fetch a page at the moment someone asks about it. Google describes a related pattern in its own products, where AI Overviews and AI Mode may use a query fan-out technique that splits one question into several related searches. Both reasons lead to the same place: the pages an AI system can reach and read are the pages that can shape what it says about you.

How Much Are AI Bots Crawling?

On the networks that publish numbers, AI crawlers now make up a share of requests close to Googlebot’s, and the share is growing. Between 1 January and 2 December 2025, AI bots other than Googlebot accounted for 4.2% of HTML request traffic on Cloudflare’s network, while Googlebot alone accounted for 4.5%. On Vercel’s network, GPTBot, Claude, AppleBot and PerplexityBot together made nearly 1.3 billion fetches in the month Vercel measured, against 4.5 billion for Googlebot.

Two measurements from two networks: AI crawlers take a share of requests that sits close to Googlebot’s.

So what percentage of your website traffic is AI bots? Across Cloudflare’s network the answer was a few percent of HTML requests, and averages across a network hide the differences between individual sites, which is why your own ratio matters more than anyone’s average.

The trend points up. Cloudflare saw GPTBot requests rise 305% between May 2024 and May 2025, and crawling for user actions, such as ChatGPT-User visits, grew more than 15 times during 2025. Most of the volume is still training: over the 12 months to August 2025, 80% of AI crawling was for training, 18% for search and 2% for user actions.

What comes back is the part that matters to a business. Cloudflare’s crawl-to-refer ratio compares how many pages a platform crawls with how often it sends visitors to a website. In July 2025, Anthropic crawled about 38,000 pages for every referred visit and OpenAI about 1,091, against 5.4 for Google.

Crawl-to-refer ratio, July 2025: the gap between crawling and sending visitors back is far wider for AI platforms.

Treat every number here as a floor. Logs only show bots that identify themselves, and in August 2025 Cloudflare reported that it had observed Perplexity using undeclared crawlers that changed user agents and IP addresses. That is Cloudflare’s account of what it saw, and it is a reminder that real volumes can be higher than any log shows.

Is AI crawling growing, and how does it compare with Googlebot?

Where: the Bot Dynamics report

Steps:

1. Choose a period of at least several weeks, because a few days do not show a trend.
2. Add the “AI Crawlers” segment from the previous recipe next to Googlebot.
3. Read the change in crawl frequency for each bot family over the period.
4. Mark the dates of releases and robots.txt changes, so a jump has an explanation.
5. Compare the curve with your impressions and clicks in Google Search Console.

What to look for: AI bots growing faster than Googlebot, sharp changes after a site change and a single bot that dominates the picture.

What to do next: divide AI bot requests by Googlebot requests and compare your ratio with the figures above, then set up monitoring (see our guide to tracking AI crawler activity for the alerts).

How AI Bots Differ From Search Bots

AI bots differ from search engine bots in three ways that change how you manage a site. Most of them do not run JavaScript, the vendor documentation we reviewed offers no way to request a re-crawl, and each purpose has its own bot with its own robots.txt rules.

Google’s own AI features sit on the other side of that line, because they draw on the Googlebot you already manage. For AI Overviews and AI Mode, a page must be indexed and eligible to appear with a snippet, and Google lists no additional technical requirements.

Re-crawling works differently too. Search Console gives you URL Inspection to ask Google for a fresh look, and the AI vendors offer nothing comparable. OpenAI notes that it can take about 24 hours for a robots.txt update to be reflected in search results, so a fix needs to land before the bot’s next visit.

Finally, the bots are split by purpose. OpenAI and Anthropic each run separate bots for training, search and user requests, and Perplexity separates search from user fetches. The operators treat them differently: OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores them. One robots.txt rule therefore rarely covers all of them.

Do AI Bots Crawl JavaScript?

Most AI crawlers do not render JavaScript. In Vercel and MERJ’s measurement, OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider and CCBot did not render it, while Gemini, which runs on Googlebot’s infrastructure, and AppleBot did. ChatGPT and Claude requested JavaScript files in 11.5% and 23.84% of their fetches but did not execute them.

The consequence is easy to picture. A client-side rendered product page can reach Googlebot complete and reach GPTBot as an empty shell, and prices, reviews and internal links that arrive through script do not exist for a crawler that never runs the script.

The same product page before and after JavaScript runs. A non-rendering crawler only receives the left-hand code.

That measurement dates from late 2024 and none of the vendor pages we reviewed describe rendering, so re-test rather than assume. Our JavaScript SEO page covers the fixes, from server-side rendering to pre-rendering.

Do AI bots see my content without JavaScript?

Where: JavaScript crawling in JetOctopus, or the free JS SEO tool for a single URL

Steps

1. Pick one URL for each template: category, product and article.
2. Run the page without JavaScript and with JavaScript.
3. Compare the title, H1, canonical, main text, prices and internal links between the two versions.
4. Write down every template where content or links appear only after rendering.
5. Estimate how many URLs each of those templates covers.

What to look for: a difference in content or links between the two versions, and an empty shell in the raw HTML.

What to do next: queue the affected templates for server-side rendering or pre-rendering, then run the same test again.

What AI Bots Crawl: Depth, Links and Crawl Budget

AI bots crawl what they can find and fetch, which means pages that are linked from places they already visit and that answer with a 200. The vendors publish little about how they pick URLs, so your own logs are the best evidence, and they tend to show wasted effort. In Vercel’s measurement, 34.82% of ChatGPT’s fetches and 34.16% of Claude’s hit 404 pages, against 8.22% for Googlebot. The same data shows that the mix of files differs by bot: 57.70% of ChatGPT’s fetches were HTML, while 35.17% of Claude’s were images.

How AI Bots Discover Content

How do AI crawlers like GPTBot and ClaudeBot discover content? No vendor publishes its URL selection logic, so the answer combines what the documentation says with what crawling mechanics imply. The documentation says that search bots matter for visibility: OpenAI recommends allowing OAI-SearchBot and Perplexity recommends allowing PerplexityBot and its published IP ranges. The mechanics say that a crawler which follows links finds pages through links, which makes internal linking the main discovery path and your website architecture the thing that decides what sits within reach.

You cannot submit URLs to the AI vendors, because the documents we reviewed describe no submission option. For Bing, IndexNow lets you notify participating search engines when content changes, although its documentation says a 200 response only confirms that the URL was received.

Crawl Budget

AI bots draw on the same server capacity as Googlebot, so crawl budget becomes a shared resource once they arrive. Every visit to a dead URL, a redirect chain or a filtered duplicate is a visit that did not reach a page you want read. Our crawl budget optimization guide covers the mechanics.

Three examples show what finding the leaks looks like, and all three are Googlebot results. On TemplateMonster, crawl and log data together exposed 3 million orphaned pages that Googlebot kept visiting. At Swiss Marketplace Group, the case study credits crawl budget optimization based on crawl data and log analytics for +350% Googlebot visits. JetOctopus’s CTO Serge Bezborodov reported that after a new internal linking scheme, the share of pages Googlebot crawled on a test set rose from 40% to 70%. The AI bot version of the work starts with the same question: which valuable pages do the bots never reach?

Which pages do AI bots never reach?

Where: the AI Efficiency report (logs, crawl and Google Search Console data together), the crawl reports for depth and orphan pages and the AI Internal Linker

Steps:

1. Make sure you have a fresh crawl and logs for several weeks.
2. In the AI Efficiency report, filter for indexed pages that received no AI bot visits.
3. Split the list by site section and by depth from the index.
4. Add a filter for orphan pages and for pages with very few internal links.
5. Sort what is left by page value, using impressions and clicks from Google Search Console.

What to look for: valuable pages with no visits, deep sections and parameters or pagination that soak up bot attention.

What to do next: in the AI Internal Linker, choose the weak and orphaned pages as acceptors and the pages that already earn crawls and impressions as donors, then review the suggested links.

Site Speed

How does site speed affect AI crawler indexing? No AI vendor publishes a response time threshold, so any number presented as a requirement is a guess. What you can measure is whether bots receive fast 200 responses. web.dev treats a time to first byte of 0.8 seconds or less as good and anything above 1.8 seconds as poor, thresholds set for visitors that also make a reasonable yardstick for bots.

Slowness costs the most where bot attention concentrates. A slow template that bots request thousands of times drains more capacity than a slow page nobody visits, and redirect chains and 5xx errors belong in the same bucket.

Are bots getting errors or slow responses?

Where: the Bot details report

Steps:

1. Open one of your top three bots.
2. Read the split of status codes: 200, 3xx, 404 and 5xx.
3. List the 404 URLs the bot requests, since bots often ask for pages that do not exist.
4. Check response time by section and use 0.8 seconds as a rough yardstick.
5. Pick out the slowest templates.

What to look for: spikes of 5xx errors, redirect chains and slow sections that receive many bot visits.

What to do next: fix the problems in priority order, starting with the sections that bots visit most, then check the same report again a week later.

What Makes a Page Source-Worthy for AI Bots

A page is source-worthy for an AI system when the system can fetch it, read it and trust it enough to use it. Fetching and reading are technical, and the earlier sections cover them: a 200 response, content in the raw HTML and a path of links that leads there. Google’s documentation says its generative AI models use publicly accessible, crawlable content and that llms.txt and similar files neither harm nor help visibility in Google Search.

Trust is editorial. It comes from a clear answer near the top, specific facts such as prices and dates in plain text, a named author or brand behind the page and a visible update date. No vendor publishes a checklist, so treat those as informed judgment. Our AI crawlability guide walks through the details.

AI User Bots and AI Bot Crawl Analysis: A New Kind of Impression

A visit from a user bot means someone asked an assistant a question and the assistant went to your page for the answer. OpenAI says ChatGPT-User visits pages when a user asks ChatGPT or a custom GPT a question, and Anthropic describes Claude-User the same way for Claude. That makes user bot hits the nearest thing to an AI impression you can count in your own data, with one caveat: a fetch confirms that a page was read and says nothing about whether it was quoted or linked.

AI bot crawl analysis comes down to three checks on the logs: who the bot is, verified against the operator’s published IP ranges, which pages it requested and what your server answered. The setup is in our guide to tracking AI crawler activity, and the wider metrics, such as the share of pages that AI bots visit, are in our piece on measuring AI visibility beyond traditional analytics.

Is my content used in AI answers?

Where: the AI Efficiency report and Query Fan-Outs Detection in AI Search Visibility

Steps:

1. In the logs, filter for user bots and sort pages by the number of visits.
2. Open Query Fan-Outs Detection and review the long queries that are tied to your pages.
3. Open your Google Search Console data and compare the pages that earn impressions with the pages user bots fetch.
4. Mark pages that have fan-out queries but no user bot visits, and the reverse.
5. List separately the pages that show up in neither place.

What to look for: pages with impressions but no user bot visits, which have a different kind of exposure, and pages with visits but no matching queries.

What to do next: update the content on those pages to answer the questions behind the fan-out queries, then check the visits again after the next crawl.

Should You Block AI Bots?

The answer depends on the bot’s type and on how your business earns money, and for most commercial sites it differs by type. Blocking a search bot is costly: Anthropic says that disabling Claude-SearchBot prevents its system from indexing your content for search, so you drop out of the answers that product gives. Blocking a training bot is a narrower step that asks the operator to leave your future content out of training data. Plenty of sites already choose it: Cloudflare found that AI crawlers were the most frequently fully disallowed user agents in robots.txt files.

What each kind of AI bot visit leads to, and whether robots.txt controls it.

A retailer or marketplace usually wants search and user bots on its product pages, because being named in an answer is the point. A publisher whose revenue depends on ads or subscriptions may welcome search bots, decide section by section for user bots and block training. Those are starting points rather than rules, and your contracts, content licenses and appetite for risk come first.

Editorial defaults for an AI bot access policy by bot type and business model.

Three details change the plan:

• User and agent bots may ignore robots.txt. OpenAI, Perplexity, Meta and Google say so for their user-triggered fetchers, so blocking them needs a firewall rule rather than a robots.txt line.
• Google has no separate AI Overviews switch. Google-Extended does not affect inclusion in Google Search, and for AI Overviews and AI Mode robots.txt directives for Googlebot are the control, which also govern regular Search.
• Your CDN may already have decided. Cloudflare changed its defaults in July 2025 so that every new domain starts with AI crawler controls in place, which means some sites block AI bots without anyone on the SEO team choosing to. Check your CDN settings.

Law adds another layer. RFC 9309 states that robots.txt rules are not a form of access authorization, and the EU’s 2019 copyright directive says that for content published online, rights should be reserved for text and data mining by machine-readable means. Neither tells you whether a given training run is lawful, and that question belongs with a lawyer.

Did my access policy work, and will I know if it breaks?

Where: the AI Bots report and Log Alerts

Steps:

1. After a robots.txt or firewall change, compare status codes by bot before and after.
2. Confirm that the bots you blocked have disappeared and that the bots you allowed receive 200 responses.
3. Check that Googlebot was not affected.
4. Create a Log Alert for the conditions that matter, such as AI bots crawling non-indexable pages or failing to reach the homepage.
5. Choose where alerts go: email, Slack or MS Teams.

What to look for: blocked bots that still request pages and allowed bots that receive 403 responses.

What to do next: adjust the rules and repeat the check. The technical setup is covered in the same AI crawlability guide linked above.

What to Do Now

Five steps cover most of what AI crawlers change for a site, in the order that tends to pay back fastest. The full 2026 plan is in our technical SEO playbook for AI crawlers.

• Find out who is visiting. Run the first recipe and sort what you find into the four groups.
• Put your content in the raw HTML. Test one URL per template without JavaScript and fix the templates that come back empty.
• Stop wasting visits. Clear dead URLs, redirect chains and unlinked pages out of the paths that bots follow.
• Set a policy per bot type. Allow what earns visibility, restrict what only takes and verify the result in your logs.
• Measure user bot visits over time. A rising count is the closest early sign that assistants are using your pages.

The goal is to be the page an assistant chooses to read, and the recipe below turns the findings from all the others into a fix order.

What should I fix first?

Where: the AI SEO Recommender

Steps:

1. After each crawl, open the recommendations.
2. Read the ranked list of the biggest problems. Each entry shows how many pages it affects and the related impressions from Google Search Console.
3. Switch the ranking between number of pages and number of impressions.
4. Assign the top three problems to owners.
5. Repeat after the next crawl and confirm in the logs that bots now receive 200 responses.

What to look for: the problems that block the largest number of pages or the most impressions.

What to do next: turn the top items into backlog tickets and check the logs after each release. The Recommender is built on Claude and reflects the last 30 days of your crawl, Google Search Console and log data.

FAQ

Is It Legal for AI Companies to Crawl My Website?

There is no single answer, because the law differs by country and courts are still deciding how it applies to AI training. Robots.txt is a technical convention that signals what you permit and does not work as an access control. For anything beyond a general understanding, such as a licensing dispute, ask a qualified lawyer in your jurisdiction.

Do AI Bots Crawl Pages Behind a Login or Paywall?

Bots read only what a plain request returns, so pages behind a login usually stay out of reach, while a paywall that sends the full text in the HTML can be read in full. None of the major operators we reviewed publishes a paywall rule. Check your logs for 200 responses on gated URLs.

Do AI Bots Crawl Images, PDFs and Other File Types?

Yes, measurements show that AI crawlers request more than HTML, including images and JavaScript files, and the mix differs by bot. Operators rarely say how they use those files, so your logs are the only reliable view. Filter them by content type to see which formats each bot requests on your site.

Do AI Bots Respect the Crawl-Delay Directive?

Some do and some do not. Anthropic says it supports Crawl-delay, Amazon says its crawlers do not and Google ignores it, while OpenAI and Perplexity do not document it. If a bot strains your server, a firewall rate limit works whatever the bot supports, so test any robots.txt delay in your logs before you rely on it.

How Can I Get AI Bots to Crawl My Key Pages, and Can I Submit My Site?

You cannot submit a site to the AI vendors, because the operator documentation we reviewed describes no submission option. You can allow the right bots in robots.txt, link key pages from shallow crawlable pages, keep an accurate XML sitemap and notify Bing through IndexNow. Then confirm in your logs that the bots arrive.

What Are Common Crawl and CCBot, and Do They Feed AI Models?

Common Crawl is a nonprofit that publishes an open archive of web crawl data, and CCBot is the crawler that builds it. Anyone can analyze the archive, so any AI developer can use it, and whether a given model did depends on its maker. You can ask CCBot to skip your site with a Disallow rule in robots.txt.

What Signals Show a Bot Is Crawling for Training Rather Than for Live Answers?

The user agent is the clearest signal. Operators label GPTBot, ClaudeBot and Meta-ExternalAgent as training or general crawlers, and ChatGPT-User, Claude-User and Perplexity-User as fetchers that act when a person asks. Verify the claim against the operator’s published IP ranges, then check behavior: training crawlers sweep many pages, user bots request single URLs.

Make AI Crawlers a Measured Audience

AI crawlers are already a visitor class on your site, and the useful first move is to measure them before deciding anything about them. Find out who visits, check that your content survives without JavaScript, stop the waste and set a policy per bot type. Your own numbers will show whether the effort belongs in access, structure or content.