Cloudflare now blocks AI agents on any page that shows an ad
Shopify opened its checkout to AI agents this week; Cloudflare closed the ad-funded page instead. Google, Cloudflare and Microsoft are each building the meter that sets what a page of the open web costs an AI.
Vincent Jiangthree-minute read
The default flipped on 15 September
Since 15 September, Cloudflare's−4.68% — Cloudflare, down 4.68 percent today defaults for ad-supported domains let search crawlers through, refuse AI training, and block user-directed agents on any page that shows an ad. Chat-fetch bots and browser-use agents fall under it; there is no opt-out directive for agents yet, only a block 12. Cloudflare set that deadline in July, pressing mixed-use crawlers to split search from training 23.
The same week, Amazon barred Meta's Muse shopping agent from its store over credential risks, and Shopify+0.28% — Shopify, up 0.28 percent today wired Muse into Shop Pay checkout across its merchants: one front of the agent war opened, the content front closed 4.
AI training is now 52% of crawler traffic
AI training alone is 52% of crawler traffic on Cloudflare's network, up from 22% a year earlier, and CEO Matthew Prince calls most traffic on the internet non-human now 13. Site owners are drawing lines: 17% restrict training, fewer than 1% block search 5.
Mixed-use crawlers, built for search and training at once, carry another 36.6% of verified crawler traffic 6. The money moves unseen: 132 million Americans have shopped with AI's help, but only 23% of merchants can see that traffic or its sales, the same agent traffic ad pages now block by default 1.
Three meters, one page, no price
Google's invitation-only pilot pays when content "contributes significantly" to an answer in Gemini, AI Overviews or AI Mode, and Google alone judges what counts 57. Participants get a monthly figure in Search Console with no arithmetic behind it 8. One executive briefed on the program calls it "quite black box"; another said the offers were too low to opt in 5.
Cloudflare's Pay Per Crawl is mechanical: sites set the price from $0.001 a fetch, and a paying crawler gets the page, HTTP 200 and all 5. Microsoft−1.02% — Microsoft, down 1.02 percent today licenses content to Copilot on "delivered value" it has not defined publicly 5. The payers are AI labs and agent builders, in cash at the meter or in lost inventory when their agents are blocked. Whichever meter wins will set the price of the open web for AI.
The toll collector's books
Cloudflare sells both the blockade and the meter. Revenue has risen ten straight quarters, to $696.1 million in Q2 2026, up 35.9% on the year, and the quarter still lost $170 million 96. The market has paid up anyway: $115.2 billion of market cap after a 43.9% run in 90 days 6.
Cloudflare revenue has risen ten straight quarters while still losing money
Data
| Revenue | |
|---|---|
| Q4 '23 | $362.47M |
| Q1 '24 | $378.6M |
| Q2 '24 | $401M |
| Q3 '24 | $430.08M |
| Q4 '24 | $459.95M |
| Q1 '25 | $479.09M |
| Q2 '25 | $512.32M |
| Q3 '25 | $562.03M |
| Q4 '25 | $614.51M |
| Q1 '26 | $639.76M |
| Q2 '26 | $696.06M |
Google's exit ramp, and what to watch
Google points to Google-Extended, a robots.txt token it says never touches Search ranking, and calls the pilot a learning phase 3510. That token opts out of Gemini training only; a separate Search Console setting governs AI Overviews and AI Mode 10.
Cloudflare's framing still holds: an ad marks a page built for a person 1. The exposed party is the small ad-funded publisher, because an agent reads, buys nothing, and never sees the ad.
The next readings are dated: Google's URL-level Google-Extended tools land within weeks, Cloudflare's single control over how much content appears in AI summaries arrives by early next year, and Microsoft's Bing no-training support is targeted for early 2027 510. The storefronts opened their checkouts to agents; the content layer just installed a toll booth.
Get Platforms by email, free
The cloud, data and developer software AI is built on. In our daily email, every weekday.







Reader comments
0 comments