Cloudflare now blocks AI agents on any page that shows an ad

The default flipped on 15 September

Since 15 September, Cloudflare's defaults for ad-supported domains let search crawlers through, refuse AI training, and block user-directed agents on any page that shows an ad. Chat-fetch bots and browser-use agents fall under it; there is no opt-out directive for agents yet, only a block ¹². Cloudflare set that deadline in July↗, pressing mixed-use crawlers to split search from training ²³.

The same week, Amazon barred Meta's Muse shopping agent from its store over credential risks, and Shopify wired Muse into Shop Pay checkout across its merchants↗: one front of the agent war opened, the content front closed ⁴.

AI training is now 52% of crawler traffic

AI training alone is 52% of crawler traffic on Cloudflare's network, up from 22% a year earlier, and CEO Matthew Prince calls most traffic on the internet non-human now ¹³. Site owners are drawing lines: 17% restrict training, fewer than 1% block search↗ ⁵.

Mixed-use crawlers, built for search and training at once, carry another 36.6% of verified crawler traffic ⁶. The money moves unseen: 132 million Americans have shopped with AI's help, but only 23% of merchants can see that traffic or its sales, the same agent traffic ad pages now block by default ¹.

Three meters, one page, no price

Google's invitation-only pilot↗ pays when content "contributes significantly" to an answer in Gemini, AI Overviews or AI Mode, and Google alone judges what counts ⁵⁷. Participants get a monthly figure in Search Console with no arithmetic behind it ⁸. One executive briefed on the program calls it "quite black box"; another said the offers were too low to opt in ⁵.

Cloudflare's Pay Per Crawl is mechanical: sites set the price from $0.001 a fetch, and a paying crawler gets the page, HTTP 200 and all ⁵. Microsoft licenses content to Copilot on "delivered value" it has not defined publicly ⁵. The payers are AI labs and agent builders, in cash at the meter or in lost inventory when their agents are blocked. Whichever meter wins will set the price of the open web for AI.

The toll collector's books

Cloudflare sells both the blockade and the meter. Revenue has risen ten straight quarters, to $696.1 million in Q2 2026↗, up 35.9% on the year, and the quarter still lost $170 million ⁹⁶. The market has paid up anyway: $115.2 billion of market cap after a 43.9% run in 90 days ⁶.

Google's exit ramp, and what to watch

Google points to Google-Extended, a robots.txt token it says never touches Search ranking↗, and calls the pilot a learning phase ³⁵¹⁰. That token opts out of Gemini training only; a separate Search Console setting governs AI Overviews and AI Mode ¹⁰.

Cloudflare's framing still holds: an ad marks a page built for a person ¹. The exposed party is the small ad-funded publisher, because an agent reads, buys nothing, and never sees the ad.

The next readings are dated: Google's URL-level Google-Extended tools land within weeks, Cloudflare's single control over how much content appears in AI summaries arrives by early next year, and Microsoft's Bing no-training support is targeted for early 2027 ⁵¹⁰. The storefronts opened their checkouts to agents; the content layer just installed a toll booth.