Cloudflare's AI block now stops everyone but Google, Apple and Microsoft
The week Cloudflare set for converting its customers' AI-training blocks into robots.txt preferences has run. Google, Apple and Microsoft are now merely asked to honor those preferences; everyone else's training crawlers still get stopped at the edge.
Vincent Jiang · 3 min read
Cloudflare's customer email of 16 September 2026 said legacy Block AI Bots settings would migrate automatically over the following week, and that the switch would vanish from the dashboard once the migration was complete 1. That week has run. What publishers clicked as a block is now, for the three companies that matter most, a polite note in a text file.
What a block became
The mapping is fixed. A legacy toggle lands on Search Allow, Training Disallow AI Training, and Agent blocked on pages with ads 12. Disallow AI Training writes a no-training directive into robots.txt while Googlebot, Applebot and Bingbot keep fetching, because all three companies run one crawler for both search indexing and model training 12. The gap it addresses is real: 17% of sites on Cloudflare's network block training in some form, fewer than 1% block search bots, and the network sits in front of more than a fifth of the web 13.
Blocking training is common on Cloudflare's network; blocking search is rare
Data
| Value | |
|---|---|
| Sites blocking AI training | 17% |
| Sites blocking search crawlers | <1% |
| Web behind Cloudflare's network | >20% |
Enforcement now has two tiers
The setting still stops someone. Training crawlers run by OpenAI, Anthropic, Meta and Amazon are blocked outright at Cloudflare's edge 4. Google, Apple and Microsoft keep fetching under Cloudflare's Accountable label, and what happens to the page afterwards rests on the operator honoring the file 14. Microsoft cannot even read the new rule until its robots.txt support arrives, targeted for early 2027 12.
Accountable is Cloudflare's own designation, not an outside certification 4. The label was granted on capabilities each operator already has, paired with time-bound commitments for the rest 12. And in August the fourth condition required an operator to "show publicly" that disallowing training does not hurt search results; the September version asks for assurance 51.
The manual still promises the opposite
Cloudflare's own documentation, last updated 1 July 2026, still says mixed-purpose crawlers "will also be blocked by all configurations to block AI training, including the legacy Block AI bots option" 6. That page lists three options for each control: Allow, Block on pages with ads, and Block 6. The shipped product carries a fourth, Disallow AI Training, which the manual never mentions 1. The behavior that shipped is the inverse of the page Cloudflare still publishes.
The only way back to a hard block is Block, which now cuts a site off from Google, Apple and Microsoft search altogether 12. Blocking Bingbot takes a site out of Bing, Yahoo and Copilot at once 1.
The receipt arrives in weeks, not now
URL-level reporting, the mechanism meant to reveal which disallowed pages were used anyway, is weeks away at Google and due next year at Apple 124. Until it lands, the ledger reads one way: publishers keep their search traffic, the giants keep the corpus, Cloudflare keeps both sides as customers, and the only crawlers still stopped by force belong to the giants' AI rivals 4. Summaries are next, with central controls targeted for early 2027 13.
How this brief was made
01Gathered & sourced272 channels · 1,762 articles▾
Agents swept 272 channels and ingested 1,762 articles, then de-duplicated and ranked them for signal.
02Verified & cross-validated6 claims · 22 data feeds▾
Every one of 6 load-bearing claims was checked against primary sources, with 22 live data feeds reconciling the figures and charts.
- 1PPC Land, "Cloudflare drops planned Googlebot block for sites refusing AI training", 27 September 2026
- 2Search Engine Journal, Matt G. Southern, "Cloudflare Lets Sites Disallow AI Training Without Blocking Googlebot", 15 September 2026
- 3Best Media Info, "Cloudflare separates AI training opt-outs from search crawling for publishers", 16 September 2026
- 4heise online, Moritz Förster, "Google search yes, model training no: New Cloudflare rules", 18 September 2026
- 5Cloudflare blog, Jin-Hee Lee, "Say it once: introducing Bot Preference Sync", 21 August 2026, updated 15 September 2026
- 6Cloudflare documentation, "Block AI Bots", last updated 1 July 2026
03Reviewed & edited1 human editor▾
One editor read the draft against the evidence, tuned the framing, and signed off before it shipped.
Become a contributor
Reporting on the business of AI and want it read? We take pitches from outside contributors who bring primary sources and a number worth arguing about.
Deepdive
AI-generated from this story and its cited sources. Not investment advice.



