164 AI Chatbot Traffic Statistics October 2026

AI chatbot traffic statistics for 2026 describe two flows that point in opposite directions. Visits from assistants to websites are small and still under 1% of most sites' traffic. Requests from AI bots to those same sites are large, and they are growing faster than human traffic. For a typical publisher, bots take many more pages than assistants return visits.

Every number on this page links to the organisation that collected it: CDN and security networks, analytics vendors that measured their own customers, publisher panels, a wiki foundation and academic researchers. Vendor data is labelled as vendor data, so you can weigh it. For the search side of the same shift, see our AI search statistics hub, and for the wider organic picture see SEO statistics.

How big is the flow of visits from AI assistants to websites?

Very little next to search, and one assistant sends most of it. Every large dataset below puts the total between a fraction of 1% and about 1% of visits.

What share of website visits comes from AI assistants?

Less than 1% in every large dataset. Search and direct visits still carry nearly all the traffic.

• Direct visits made up 39.27% of the sample and paid visits 15.48% (Ahrefs, Sep 2026).
• Across the Chartbeat network of publishers, AI chatbots account for less than 1% of total pageviews (Chartbeat, Jun 2026).
• AI platforms made up less than 1% of sessions across 30 billion sessions in the 2,000 Clarity projects that get the most LLM traffic (Microsoft Clarity, Apr 2026).
• AI referrals were less than 1% of overall traffic on more than 1,200 publisher and news sites (Microsoft Clarity, Nov 2025).
• AI apps supplied 0.12% of referrals to TollBit publishers on average over the last three quarters of 2025, while Google supplied about 80.5% (TollBit, Feb 2026).
• In the first half of 2026, AI apps supplied 0.16% of external referrals to North American publishers and 0.05% to European ones (TollBit, Jul 2026).
• Google supplied about 84% of external referrals to European publisher sites and 80% in North America (TollBit, Jul 2026).
• Google's share of visits across 74,752 tracked sites fell from 35.11% in June 2025 to 30.53% in March 2026, a drop of 4.58 points (Ahrefs, May 2026).

Which chatbots send the most visits?

ChatGPT, by a wide margin. Meta AI is the surprise at number two in one dataset, and Claude and Gemini are the assistants gaining share.

• ChatGPT sent 4.3 million visits in September 2026, a 0.34% share of the tracked sites' traffic (Ahrefs, Sep 2026).
• Meta AI sent 577.1 thousand visits (0.05%), ahead of Gemini at 265.6 thousand (0.02%), Claude at 79.4 thousand and Perplexity at 64.0 thousand (0.01% each) (Ahrefs, Sep 2026).
• In December 2025 ChatGPT held about 84% of standalone assistant referrals, Perplexity 8.9%, Gemini 4.5%, Copilot 2.1% and Claude 0.6% (Previsible, Jul 2026).
• Claude's monthly sessions grew 64 times, from 133 in November 2024 to 8,528 in May 2026, and passed Perplexity in March 2026 (Previsible, Jul 2026).
• Gemini grew 3.2 times in the same period, from 5,598 sessions to 18,119 (Previsible, Jul 2026).
• Perplexity peaked at 17,507 sessions in March 2025 and fell 61% to 6,788 by May 2026 (Previsible, Jul 2026).
• Copilot peaked at 8,651 sessions in August 2025 and fell 96% to 339 by May 2026 (Previsible, Jul 2026).
• Gemini overtook Perplexity in referral share with a 25% lead, after Gemini referrals grew 33% in December 2025 (BrightEdge, Jan 2026).
• Claude referrals grew 153.5% from February to March 2026, which Ahrefs ties to Anthropic widening its web search (Ahrefs, May 2026).

How quickly are referral visits from AI assistants rising?

Fast, with sharp reversals whenever a platform changes how it links out.

• ChatGPT referral sessions grew 12.8 times, from 47,606 in November 2024 to 610,910 in May 2026 (Previsible, Jul 2026).
• ChatGPT sessions fell from 448,412 in October 2025 to 213,345 in November, then recovered to 442,609 in December (Previsible, Jul 2026).
• Traffic from AI platforms grew 155.6% over eight months on Clarity's publisher sites, against 24.0% for search, 21.5% for social and 14.9% for direct (Microsoft Clarity, Nov 2025).
• Clarity measured a further 22% rise in AI-referred traffic over the six months to March 2026 (Microsoft Clarity, Apr 2026).
• ChatGPT referrals to Chartbeat publishers grew more than 200% year over year (Chartbeat, Jun 2026).
• Across the tracked sites, Gemini referrals grew 40.4% from August to September 2026, ChatGPT 14.1% and Perplexity 3.5%, while Claude fell 17.2% (Ahrefs, Sep 2026).
• Average growth rates over Ahrefs' tracked period were 30.7% for Claude and 12.8% for Gemini, against 2.9% for Perplexity and 1.4% for ChatGPT (Ahrefs, May 2026).

Which regions, industries and pages get the most AI referrals?

AI referrals are concentrated: a few regions, a few industries, and often a page the site owner would not have picked.

Which regions get the most AI referrals?

Oceania, Africa and Europe lead on ChatGPT's share of traffic, and Meta AI beats ChatGPT in South America.

• ChatGPT's share of site traffic was 0.54% in Oceania, 0.49% in Africa and 0.48% in Europe (Ahrefs, Sep 2026).
• It was 0.31% in Asia, 0.30% in North America and 0.24% in South America (Ahrefs, Sep 2026).
• Meta AI sent 0.30% of South American traffic, more than ChatGPT's 0.24% (Ahrefs, Sep 2026).
• Google's share of site traffic was 39.12% in Europe, 26.17% in North America and 20.91% in South America (Ahrefs, Sep 2026).
• The AI app share of external referrals to a fixed cohort of European publishers fell 40% in a quarter, from 0.063% to 0.038%, or about 1 in 2,634 referrals (TollBit, Jul 2026).

Which industries get the most AI referrals?

Small business, insurance and finance lead on share. News and publishing sit near the bottom.

• ChatGPT sent 1.0% of the traffic of Beauty and Fitness sites and Meta AI 0.5%, the highest shares of any industry (Ahrefs, Sep 2026).
• ChatGPT sent 0.4% of traffic in both Jobs and Education and Real Estate, and 0.1% in News, Finance and Shopping (Ahrefs, Sep 2026).
• Meta AI sent 0.4% of Travel and Transportation traffic, ahead of ChatGPT at 0.3% (Ahrefs, Sep 2026).
• Small and medium business sites went from 0.40% of sessions from AI assistants in December 2024 to 1.71% in May 2026 (Previsible, Jul 2026).
• Insurance grew 18.9 times to 1.51%, the highest penetration after small business (Previsible, Jul 2026).
• Finance rose from 0.56% to 1.19% and legal reached 0.75% (Previsible, Jul 2026).
• E-commerce grew 37 times from near zero to 0.37%, almost entirely on ChatGPT (Previsible, Jul 2026).
• Education grew 5.4 times to 0.54%, and health was the only industry to decline, from 0.23% to 0.17% (Previsible, Jul 2026).
• Consumer electronics sites received 17.1 million AI referral visits a month, up 215.4% in a year (Similarweb, Sep 2026).
• Beauty sites went from 1.0 million AI referral visits in May 2025 to 4.0 million in May 2026 (Similarweb, Sep 2026).

Which pages do AI assistants send visitors to?

Often not the page you would choose. Assistants often link a domain without choosing the page, so a large share of visitors land on search results or hub pages.

• ChatGPT sends 28.8% of its referral traffic to internal search results pages (Previsible, Jul 2026).
• Across all industries, about 25% of AI-referred traffic lands on internal search pages (Previsible, Jul 2026).
• Product pages take 43% of all e-commerce traffic from assistants (Previsible, Jul 2026).
• News pages capture 54% of AI referrals to publisher sites (Previsible, Jul 2026).
• Course pages are the entry point for 52% of education traffic from assistants (Previsible, Jul 2026).
• 13% of Perplexity's classified referrals land on blog pages, against 8.7% in the overall mix (Previsible, Jul 2026).
• ChatGPT and Claude referrals skewed toward software and developer tool downloads and B2B professional topics in the second half of 2025 (TollBit, Feb 2026).
• Perplexity drove the most referrals to shopping pages, while news topics made up at least 15% of referrals from every assistant TollBit measured (TollBit, Feb 2026).

How much of the web is now AI bot traffic?

A few percent of requests on the largest networks, but growing faster than people. One company's bots account for most of it, and training is giving way to bots that fetch pages while a user waits.

What share of requests comes from AI bots?

A few percent of HTML requests on Cloudflare's network, rising quickly, and a growing multiple of human visits on publisher sites.

• AI bots made up 4.2% of HTML requests to Cloudflare customers on average in 2025, from a low of 2.4% in early April to a high of 6.4% in late June (Cloudflare, Dec 2025).
• Googlebot alone made up 4.5% of HTML requests, slightly more than all other AI bots together (Cloudflare, Dec 2025).
• On 2 December 2025, humans generated 47% of HTML requests and non-AI bots 44% (Cloudflare, Dec 2025).
• Daily requests from AI agents on Cloudflare's network grew by more than 1,700% over the year to September 2026 (Cloudflare, Sep 2026).
• Monthly AI-driven traffic grew 187% between January and December 2025 across HUMAN's customers, peaking at 3.61 times January's level (HUMAN Security, Apr 2026).
• AI requests grew 30% between January and May 2026 across Fastly's network, 6.5 times faster than human traffic (Fastly, Jun 2026).
• TollBit's publishers saw AI bot scrapes grow 29% from the second to the third quarter of 2025 and 20% from the third to the fourth, while human traffic fell 5% in the fourth quarter (TollBit, Feb 2026).
• AI bot traffic on TollBit's publisher sites grew a further 25% over the first half of 2026 (TollBit, Jul 2026).
• The median European site saw 1 AI bot visit for every 33 human visits in the first half of 2026, against 1 for every 93 in North America (TollBit, Jul 2026).

Which bots and companies send the most requests?

OpenAI's bots lead among the AI labs, but Google's crawler is larger than all of them.

• Meta-ExternalAgent accounted for about 16% of AI-driven traffic and Anthropic's bots about 11%, with dozens of other bots sharing less than 5% (HUMAN Security, Apr 2026).
• ChatGPT-User made up 31.8% of AI-driven traffic, OAI-SearchBot 19.5% and GPTBot 17.6% (HUMAN Security, Apr 2026).
• ClaudeBot made up 10.6% and PerplexityBot 2% (HUMAN Security, Apr 2026).
• Googlebot was responsible for over 28% of Verified Bot traffic on Cloudflare in 2025, GPTBot for about 7.5% and Bingbot for 6% (Cloudflare, Dec 2025).
• Search engine crawlers made up 40% of Verified Bot traffic, AI crawlers 20% and SEO bots over 13% (Cloudflare, Dec 2025).
• GPTBot's share of AI crawler traffic rose from 5% to 30% between May 2024 and May 2025, and Meta-ExternalAgent entered at 19% while Bytespider fell from 42% to 7% (Cloudflare, Jul 2025).
• By the end of 2025, ClaudeBot crawling was about 10% above its January level and PerplexityBot's was about 3.5 times higher (Cloudflare, Dec 2025).
• OpenAI averaged about 4.5 million scrapes per European site in the first half of 2026, roughly 3.5 times the next platform (TollBit, Jul 2026).
• Amazonbot and ClaudeBot each made about 960,000 scrapes per European site, against about 3.16 million for ChatGPT-User (TollBit, Jul 2026).
• Over two months, Googlebot accessed about twice as many unique pages as ClaudeBot or GPTBot and 167 times as many as PerplexityBot, and crawled roughly 8% of the unique URLs Cloudflare sampled (Cloudflare, Jan 2026).

What are the bots doing: training, search or answering?

Training still leads in total volume, but the share of bots that fetch pages for a live answer is growing fastest.

• Training crawlers made up about 67.5% of AI bot volume in 2025 and AI scrapers 31.9% (HUMAN Security, Apr 2026).
• Training crawlers fell from about 90% of AI-driven traffic in January 2025 to 74% in December, while scrapers rose from 10% to 24% and AI agents reached 1.7% (HUMAN Security, Apr 2026).
• AI scraper traffic grew 597% over 2025, and training crawler volume grew 136% (HUMAN Security, Apr 2026).
• At peak, training crawling ran at 7 to 8 times search crawling and 32 times user-action crawling on Cloudflare in 2025 (Cloudflare, Dec 2025).
• Training accounted for 80% of AI crawling over the twelve months to July 2025, against 18% for search and 2% for user actions (Cloudflare, Aug 2025).
• RAG bots, which retrieve pages to answer a prompt, grew 33% per quarter on average between the second and fourth quarters of 2025, while training bots fell 15% on average and AI search indexers grew 59% (TollBit, Feb 2026).
• In the fourth quarter of 2025, a RAG bot fetched the average page about 10 times for every request from a training bot (TollBit, Feb 2026).
• RAG bots made 49% of AI scrapes on North American sites in the first half of 2026, while training bots made 41% of those on European sites (TollBit, Jul 2026).
• 85% of AI requests on Fastly's network came from scheduled crawlers and 15% from fetchers triggered by a user prompt (Fastly, Jun 2026).
• Retail and e-commerce sites took 62.5% of training crawler traffic in 2025, while media sites took 40.9% of AI scraper traffic (HUMAN Security, Apr 2026).

Are AI bots who they say they are?

Mostly yes for the big labs. The risk is impostors borrowing their names.

• Less than 2% of the scrapes carrying OpenAI, Anthropic, Amazon or Perplexity user agents were spoofed, at 0.67% for OpenAI, 1.96% for Anthropic, 0.87% for Amazon and 0.23% for Perplexity (TollBit, Jul 2026).
• Claude-User was spoofed 12.70% of the time and Claude-SearchBot 5.87% (TollBit, Jul 2026).
• In one 24-hour window in September 2026, Fastly saw about 6.3 million requests from nearly 10,200 IP addresses falsely claiming to be AI crawlers and fetchers (Fastly, Oct 2026).
• Roughly 1,600 of those addresses sent 94% of the impostor requests, and the campaign reached more than 100,000 sites (Fastly, Oct 2026).
• Each of the ten most impersonated bot identities drew around 500,000 requests in the day, spread across OpenAI, Anthropic, Perplexity, DuckDuckGo and Common Crawl (Fastly, Oct 2026).
• Common Crawl scrapes appeared on about 74% of TollBit's sites in the first half of 2026, and the data reseller Diffbot on more than 33% (TollBit, Jul 2026).

How many pages do AI bots read for each visit they send back?

Many more than they return, and the ratio is widening. This is the clearest number for why publishers see AI as a cost before it is a channel.

What is the scrape-to-referral ratio for publishers?

Between dozens and hundreds of fetches for each human visitor, and worse in Europe than in North America.

• The European cohort's ratio rose from 150 scrapes per referral in the first quarter of 2026 to 227 in the second, 51% more (TollBit, Jul 2026).
• OpenAI's ratio worsened from 165 to 265 scrapes per human visitor, and Meta needed over 1,500 (TollBit, Jul 2026).
• On TollBit's chart, Anthropic's ratio was 1 then 2, Perplexity's 4 in both quarters and Mistral's 24 then 33, though TollBit warns that Anthropic caches content for longer and that undisclosed user agents flatter some ratios (TollBit, Jul 2026).
• Google sent a European publisher about 1,681 human visitors for every one an AI application sent (TollBit, Jul 2026).
• In the second half of 2025, Google delivered over 678 human visitors for every AI application visitor (TollBit, Feb 2026).
• Sports content took 218 RAG scrapes per human visitor in the second half of 2025, for example (TollBit, Feb 2026).
• RAG bots fetched each unique European page about 41 times in the first quarter and 50 in the second, against 5 for training crawlers (TollBit, Jul 2026).

How do crawl-to-referral ratios differ by AI company?

Anthropic crawls the most per referral, Perplexity the least among AI labs, and the search engines run far lower than all of them.

• Anthropic's crawl-to-refer ratio on Cloudflare fell 86.7%, from 286,930 to 38,065 crawls per referral between January and July 2025 (Cloudflare, Aug 2025).
• OpenAI's fell 10%, from 1,217 to 1,091 (Cloudflare, Aug 2025).
• Perplexity's rose 256.7%, from 54 to 195, as its crawling grew faster than its referrals (Cloudflare, Aug 2025).
• Microsoft's ratio moved from 38.5 to 40.7 and Yandex's from 15.5 to 21.4 over the same months (Cloudflare, Aug 2025).
• Anthropic's ratio peaked at 500,000 to 1 in early 2025, a spike Cloudflare attributes to sparse referral traffic (Cloudflare, Dec 2025).
• Perplexity started 2025 below 100 to 1, spiked above 700 to 1 in late March and stayed under 200 to 1 from September (Cloudflare, Dec 2025).
• Microsoft's ratio peaked between 50 and 70 to 1 and followed a weekly cycle, lowest on Thursdays and highest on Sundays (Cloudflare, Dec 2025).
• DuckDuckGo's ratio stayed below 1 to 1 for the first three quarters of 2025 (Cloudflare, Dec 2025).

Do people click when an AI answer cites a page?

Rarely, and less each quarter.

• AI apps' click-through rate to publishers was over 13 times lower than that of pre-AI Google results pages (TollBit, Feb 2026).
• Sites with a direct licensing deal saw click-through from AI apps fall from 8.8% in the first quarter of 2025 to 1.33% in the fourth (TollBit, Feb 2026).
• The click-through advantage of a licensing deal narrowed from over 15 times that of sites without a deal in the first quarter of 2025 to 7 times in the fourth (TollBit, Feb 2026).
• More than half of consumers read summaries in search results, and they are over 40% more likely to end their search after reading one (Cloudflare, Sep 2026).
• Cloudflare estimates that for every hour people spend searching for information online, only 15 minutes goes to the open web (Cloudflare, Jul 2026).
• Newer agents such as Claude Code and Codex do not include citation links in their output at all, so they offer a publisher no click (TollBit, Jul 2026).

Do visitors from AI assistants convert and engage more?

Yes on conversion, from a small base. They arrive further along in the decision and spend their time on fewer pages.

Do AI visitors convert better than search visitors?

On every comparison here, yes: at least double, and often three times or more.

• On Clarity's sample, direct visitors signed up at 0.13% and social visitors at 0.46% (Microsoft Clarity, Nov 2025).
• Subscription conversion was 1.34% for LLM referrals, 0.55% for search, 0.41% for direct and 0.37% for social (Microsoft Clarity, Nov 2025).
• 52% of the 1,277 domains Clarity analysed converted AI traffic into sign-ups or subscriptions in the month studied (Microsoft Clarity, Nov 2025).
• Consumers referred by AI search convert at between three times and over five times the rate of those referred by traditional search (Cloudflare, Sep 2026).
• On financial services sites, conversion pages such as enrollment and sign-up show some of the highest AI densities in Previsible's dataset, with LLM-to-total ratios above 2% (Previsible, Jul 2026).

Which AI platforms convert best?

Copilot, Perplexity and Gemini on Clarity's publisher sample. Claude and ChatGPT produce the most sign-ups on Ahrefs' own site.

• Copilot referrals converted to subscriptions at 17 times the rate of direct and 15 times the rate of search (Microsoft Clarity, Nov 2025).
• Perplexity converted at 7 times the rate of both direct and search, and Gemini at 4 times direct and 3 times search (Microsoft Clarity, Nov 2025).
• Perplexity and Gemini produced the highest sign-up conversion rates (Microsoft Clarity, Nov 2025).
• In March 2026, Ahrefs counted 2,836 new users who credited Claude at onboarding, 1,978 who credited ChatGPT, 619 Gemini and 67 Perplexity (Ahrefs, May 2026).

How do AI visitors behave on a site?

They stay longer, view fewer pages and bounce back less.

• AI-referred sessions had 1.60 quick backs, against 2.04 for organic search, 21.6% fewer (Microsoft Clarity, Apr 2026).
• The average AI session lasted about 943 seconds, against 849 for organic search (Microsoft Clarity, Apr 2026).
• AI sessions covered about 10.6 pages, against 16.8 for organic search (Microsoft Clarity, Apr 2026).
• Average scroll distance was 2,554 pixels for AI sessions and 2,769 for organic search, the only channel ahead of AI (Microsoft Clarity, Apr 2026).
• AI sessions logged 0.009 rage clicks on average, against 0.006 for organic search (Microsoft Clarity, Apr 2026).
• News and media sites got just 4.8 pageviews per article from AI platforms, against 10.6 for home and garden, 8.6 for health and 8.5 for science and education (Chartbeat, Jun 2026).
• Economy, business and finance articles drew more than 7 million pageviews from AI sources, ahead of science and technology at 1.97 million and environment at 1.92 million (Chartbeat, Jun 2026).

What do AI answers cost publishers, and can sites block the bots?

Publishers pay twice, once in lost visits and once in requests they still have to serve. Blocking is common, partial and unenforced.

How much traffic have publishers lost?

Total pageviews fell modestly, but search traffic fell sharply, and small publishers took the worst of it.

• Average weekly pageviews across the Chartbeat network fell from about 7.64 billion in 2024 to 7.19 billion in 2025, about 6% (Chartbeat, Jun 2026).
• Pageviews from Google Discover fell 16% between December 2024 and December 2025 (Chartbeat, Jun 2026).
• In Chartbeat's data, small publishers saw search traffic fall 60%, medium publishers 47% and large publishers 22% (Chartbeat, Jun 2026).
• Internal traffic rose from about 38% of pageviews in 2024 to about 41% since November 2025, and dark social from about 7% to 10% in January 2026 (Chartbeat, Jun 2026).
• Human pageviews to Wikipedia ran roughly 8% below the same months of 2024 once bot traffic was reclassified, which the Foundation links to generative AI and social media (Wikimedia Foundation, Oct 2025).
• Google referrals to news sites in Cloudflare's data were 9% lower in March 2025 than in January and 15% lower in April (Cloudflare, Aug 2025).
• 75% of publisher executives expect agentic tools to have a large or very large impact on the news industry (Reuters Institute, Jan 2026).

Who blocks AI crawlers?

A growing minority. Sites block training far more often than they block search.

• More than 2.5 million websites had chosen to disallow AI training through Cloudflare's managed robots.txt or managed rule by August 2025 (Cloudflare, Aug 2025).
• Between July 2025 and January 2026, sites on Cloudflare's AI Crawl Control blocked other AI crawlers such as GPTBot and ClaudeBot nearly seven times as often as they blocked Googlebot and Bingbot (Cloudflare, Jan 2026).
• 546 of the 3,816 top-10,000 domains with a robots.txt file, about 14%, had allow or disallow rules aimed at AI bots in June 2025; GPTBot was disallowed by 312 domains and explicitly allowed by 61 (Cloudflare, Jul 2025).
• 60.0% of reputable news sites disallow at least one AI crawler, against 9.1% of misinformation sites (Steinacker-Olsztyn et al., Oct 2025).
• Reputable sites disallow an average of 15.5 AI user agents, and their AI blocking rose from 23% in September 2023 to nearly 60% in May 2025 (Steinacker-Olsztyn et al., Oct 2025).
• About 75% of top publishers blocked an OpenAI-related crawler at some point from mid-2023 (Zhao and Berman, Apr 2026).
• 63% of European sites on TollBit disallow CCBot, against 46% in North America, and 60% disallow GPTBot, against 41% (TollBit, Jul 2026).
• Only 9% of European sites disallow Claude-User, against 26% in North America, and 13% disallow Perplexity-User, against 26% (TollBit, Jul 2026).

Do AI bots obey robots.txt?

Often not, and least of all the bots that fetch pages for a live answer.

• The median European site's robots.txt instructions were ignored 2.8 times as often as the median North American site's (TollBit, Jul 2026).
• ChatGPT-User ignored explicit robots.txt rules on 54% of its scrapes, Bytespider on 48% and PerplexityBot on 42% (TollBit, Jul 2026).
• 30% of all AI bot scrapes in the fourth quarter of 2025 did not abide by explicit robots.txt permissions (TollBit, Feb 2026).
• 42% of ChatGPT-User's scrapes in that quarter reached content on sites that had blocked it (TollBit, Feb 2026).
• Deals and shopping sites received over 400 bypassing scrapes for every 100 permitted ones, and entertainment sites almost 250 (TollBit, Feb 2026).
• A controlled test of 130 self-declared bots over 40 days found that bots comply less with stricter directives and that AI search crawlers rarely check robots.txt at all (Kim et al., May 2025).
• Cloudflare observed an undeclared Perplexity crawler rotating IP addresses and networks across tens of thousands of domains, at millions of requests a day (Cloudflare, Aug 2025).
• Wikimedia now blocks or throttles about 30% of automated requests from crawlers that do not follow its policies, and about 1.5 billion of its daily error responses are blocked or throttled requests (Wikimedia Foundation, Mar 2026).

What do the bots cost, and what does blocking change?

The bots cost bandwidth and compute whether or not they send anyone back. Blocking has a measured price in traffic.

• Bandwidth for downloading Wikimedia's multimedia files grew 50% from January 2024, driven mostly by scrapers rather than readers (Wikimedia Foundation, Apr 2025).
• More than 50% of crawl traffic from well-behaved bots on Cloudflare goes to re-fetching pages that have not changed (Cloudflare, Jul 2026).
• More than 51% of AI requests on Fastly's network need a trip to the origin server, against less than 9% of human traffic (Fastly, Jun 2026).
• Scrapers take about 16 seconds on average to retrieve a publisher page, against under 0.7 seconds through a licensed route, and advanced scraping services charge up to $22.50 per 1,000 pages (TollBit, Jul 2026).
• A typical organisation found nearly one in five of its site visits was a scraping attempt in 2025, nearly double the 2022 rate (HUMAN Security, Apr 2026).
• Visits by AI bots to TollBit's bot paywall grew 999% in 2025, to 1 billion (TollBit, Feb 2026).
• More than 50 publisher and AI deals have been signed since 2023, nearly all bespoke and bilateral (Cloudflare, Sep 2026).
• 20% of publisher executives expect substantial revenue from AI deals, 49% expect a minor contribution and 20% expect none (Reuters Institute, Jan 2026).
• Sites that block Google's AI crawler are significantly less likely to be retrieved by AI Overviews, and AI Overviews appeared above the organic results for 51.5% of the real-user queries studied (Grossman et al., Apr 2026).

Methodology and Sources

We collected these figures on 14 October 2026 and fetched every link that day to confirm that it loads, carries the number and shows its date. Each number comes from the organisation that produced it: bot-management and CDN networks (Cloudflare, Fastly, HUMAN Security, TollBit), analytics vendors reporting on their own customers (Ahrefs, Chartbeat, Microsoft Clarity, Previsible, Similarweb, BrightEdge), a publisher survey (Reuters Institute), a wiki foundation (Wikimedia) and academic papers. Roundups and "statistics" pages that repeat other people's numbers were not used. Vendor datasets cover their own customers or panels, so shares differ from source to source: Ahrefs, Previsible and Chartbeat measure site analytics, while Cloudflare, Fastly, HUMAN and TollBit measure requests at the network edge. Where two sources disagree, both are shown with their sample. We refresh the page monthly.

Sources

• Ahrefs, AI Chatbot Traffic: What It Is, and How to Get More
• Ahrefs, AI vs Search Traffic Analysis (chatgpt-vs-google.com)
• BrightEdge, Gemini's December Surge: What Citation Data Reveals About Where It's Sending Traffic
• Chartbeat, Pageviews are down, but AI's impact is complicated
• Cloudflare, Content Independence Day, one year on
• Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025
• Cloudflare, Google's AI advantage: why crawler separation is the only path to a fair Internet
• Cloudflare, Have it both ways: stay discoverable in search while disallowing AI training
• Cloudflare, Making AI search smarter
• Cloudflare, Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
• Cloudflare, The 2025 Cloudflare Radar Year in Review
• Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals
• Cloudflare, The Internet has a second audience
• Fastly, AI Bot Cosplay
• Fastly, AI Bots, Web Crawlers, and the Future of High-Tech: Every Bot Wants a Bite
• Grossman et al., How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews
• HUMAN Security, 2026 State of AI Traffic and Cyberthreat Benchmark Report
• Kim et al., Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study
• Microsoft Clarity, AI Traffic Converts at 3x the Rate of Other Channels (Study)
• Microsoft Clarity, AI Visitors Show Stronger Intent Signals Than Traditional Channels
• Previsible, 2026 State of AI Discovery report
• Reuters Institute, Journalism, media, and technology trends and predictions 2026
• Similarweb, AI Referral Traffic by Industry: 2026 Data
• Steinacker-Olsztyn, Gosain and Dao, Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web
• TollBit, State of the Bots 2025 Q3 and Q4: The Leaky Pipes
• TollBit, State of the Bots 2026 Q1 and Q2: The Bad Bots
• Wikimedia Foundation, How crawlers impact the operations of the Wikimedia projects
• Wikimedia Foundation, New User Trends on Wikipedia
• Wikimedia Foundation, Quo Vadis, Crawlers? Progress and what's next on safeguarding our infrastructure
• Zhao and Berman, Strategic Response of News Publishers to Generative AI (Rutgers Business School and Wharton)

Glossary

• AI referral traffic: visits that arrive from a link inside an AI assistant's answer.
• AI bot: an automated program run by an AI company that fetches web pages, whether to train a model, build a search index or answer a user's prompt.
• Crawler: a bot that follows links to read many pages in bulk, for example to train a model or build an index.
• Fetcher (user agent): a bot that retrieves a page on demand while a person waits for an answer, such as ChatGPT-User.
• RAG: retrieval-augmented generation, where an assistant fetches live pages and writes its answer from them.
• Crawl-to-refer ratio / scrape-to-referral ratio: the number of pages an AI platform fetches for each human visit it sends back to a site.
• robots.txt: a text file at a site's root that tells bots which pages they may fetch. It is a voluntary standard with no enforcement.
• Disallow / bypass: a robots.txt rule that bars a named bot, and a request that goes ahead anyway.
• Spoofing: sending requests under another bot's name or a browser's name to get past access rules.
• Verified Bots: bots on Cloudflare's list of known, authenticated automated programs.
• Quick back: a visit where the user opens a page and goes straight back to the previous one.
• Penetration rate: AI-referred sessions as a share of a site's total sessions.
• LLM: large language model, the technology behind ChatGPT, Gemini, Claude and similar assistants.