AI Scraping Goes Industrial, and Europe Bears the Brunt

Written By Tiago Palladino

Welcome back, embedders.

This week we lead with a new TollBit report on who is scraping publishers, at what scale, and what they hand back: AI bots taking content at industrial volume, Europe hit hardest, OpenAI far out front, and most of it returning nothing to the site. Around it, we cover OpenAI building an ad business as its enterprise revenue passes consumer, Anthropic's surge to a rare profit, Wall Street souring on ad tech, and the growing friction between AI platforms and publishers.

- Vas

The Research

What the Bots Take, and What Publishers Get Back

A new report from TollBit, the content-licensing platform, puts numbers on how one-sided AI scraping has become. It is vendor data with a stake in the answer, so read it as directional. Across its network, AI bot traffic rose 25 percent in the first half of 2026 and more than doubled per site on the busiest days, and European sites are scraped about four times as hard as comparably sized US ones, with their robots.txt ignored 2.8 times more often. Every one of those requests costs the publisher, in bandwidth, server load, and the cybersecurity spend to detect and block the bots, whether or not a human ever arrives.

Not all of that scraping is the same, and most of it was never going to send a reader back. Training crawlers, which take content once to teach a model, make up 41 percent of Europe's scrapes and never return a visitor. Indexing crawlers build the index behind AI search and do not send visitors directly either. Only RAG crawlers, which re-fetch a page each time someone asks about it, can cite a source and send a human, and Europe skews away from those, toward training.

Even RAG, the one type that can pay a visit back, barely does. It takes 179 RAG scrapes to earn a single human visit to a European site, against 53 in North America, and the exchange worsened from 150 to 1 in Q1 to 227 to 1 in Q2.

One company is out front. OpenAI averages about 4.5 million scrapes per European site, roughly 3.5 times the next platform; Meta, Amazon, and Anthropic each sit near a quarter of its rate. OpenAI's ChatGPT-User bot alone averages 3.16 million scrapes per site, more than three times any other bot, and it is the least well behaved: it ignores an active robots.txt block on 54 percent of its scrapes.

For the AI companies, scraping is not cheap either: up to $22.50 per 1,000 pages, likely tens of millions a year for a large chatbot before legal fees, and a scrape that takes about 16 seconds on average and sometimes fails, against under 0.7 seconds for a licensed route. TollBit argues licensing is the better deal, but it sells licensing, and a license is a fee that scraping avoids, so read the cost case as interested. The speed and reliability edge is the part that is hard to argue with.

Signals

OpenAI builds its ad machine as enterprise passes consumer

OpenAI is assembling the pieces of a real ad business: product carousels and an AppsFlyer attribution tie-in that about 40 brands including Grubhub are testing, ads now live across the UK, Mexico, Brazil, Japan, and South Korea, and a new small-business ad unit staffed with hires from Meta, Google, and TikTok (MarTech, Digiday) The ad line is nearing a $1 billion annualized run rate, still small against the core: CFO Sarah Friar told shareholders the company is at a $40 billion run rate and that enterprise revenue has now passed consumer, months ahead of plan (TNW). The commercial org is also in flux, with CRO Denise Dresser out after eight months, replaced by ex-Wiz operator Dali Rajic, the tenth senior departure since April (TNW, Adweek). Anthropic, meanwhile, keeps surging: $11.5 billion in Q2 revenue, roughly a $46 billion annualized run rate, up 14 times year over year, and, unusually for the sector, a positive adjusted operating income (TNW).

Wall Street turns on ad tech

Q2 earnings were brutal for independent ad tech, and growth did not save anyone. The Trade Desk grew revenue 3 percent and still lost about 22 percent of its stock value, with CEO Jeff Green admitting the result was below the standard the company holds itself to; Criteo (down 24 percent), Taboola (down 27.5 percent), and Teads (down 24 percent) fell too, and even AppLovin's 53 percent growth drew a 19.7 percent drop. The backdrop: Amazon, Google, and Meta took roughly 56 percent of US ad spend last year, on track for 58 percent this year. The message from investors is that the walled gardens are compounding and the independents are consolidation targets. (Digiday)

Perplexity blocks Time's ads for AI agents

Perplexity blocked the brand messages Time embeds in the machine-readable versions of its pages, calling them "deceptive advertising" and warning publishers it could cut their trust scores in its index. It acted less than two weeks after the practice was reported. (Digiday)

Sources

Tiago Palladino
Previous

The pendulum has shifted on AI
Next

Your Brand Is Renting Space Inside AI