---
title: "Cloudflare now blocks AI agents on any page that shows an ad"
description: "Shopify opened its checkout to AI agents this week; Cloudflare closed the ad-funded page instead. Google, Cloudflare and Microsoft are each building the meter that sets what a page of the open web costs an AI."
publisher: "The Inference"
section: "Platforms"
published: 2026-09-23T19:55:11.433Z
modified: 2026-09-23T19:55:11.433Z
canonical: https://theinference.org/article/cloudflare-now-blocks-ai-agents-on-any-page-that-shows-an-ad
language: en
keywords: "Cloudflare, AI Agents, Web Crawling, Google, Digital Advertising, Shopify"
---

# Cloudflare now blocks AI agents on any page that shows an ad

> Shopify opened its checkout to AI agents this week; Cloudflare closed the ad-funded page instead. Google, Cloudflare and Microsoft are each building the meter that sets what a page of the open web costs an AI.

## The default flipped on 15 September

Since 15 September, [Cloudflare](https://theinference.org/markets/companies/cloudflare)'s defaults for ad-supported domains let search crawlers through, refuse AI training, and block user-directed agents on any page that shows an ad. Chat-fetch bots and browser-use agents fall under it; there is no opt-out directive for agents yet, only a block [1][2]. [Cloudflare set that deadline in July](https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/), pressing mixed-use crawlers to split search from training [2][3].

The same week, [Amazon barred Meta's Muse shopping agent from its store](https://theinference.org/article/amazon-banned-muse-from-your-card-shopify-already-has-it) over credential risks, and [Shopify](https://theinference.org/markets/companies/shopify) [wired Muse into Shop Pay checkout across its merchants](https://www.pymnts.com/commerce/ecommerce/2026/shopify-brings-shop-pay-checkout-solution-to-metas-muse-ai-agent/): one front of the agent war opened, the content front closed [4].

## AI training is now 52% of crawler traffic

AI training alone is 52% of crawler traffic on Cloudflare's network, up from 22% a year earlier, and CEO Matthew Prince calls most traffic on the internet non-human now [1][3]. Site owners are drawing lines: [17% restrict training, fewer than 1% block search](https://www.searchenginejournal.com/googles-ai-payment-pilot-vs-cloudflare-and-microsoft-models/589636/) [5].

Mixed-use crawlers, built for search and training at once, carry another 36.6% of verified crawler traffic [6]. The money moves unseen: 132 million Americans have shopped with AI's help, but only 23% of merchants can see that traffic or its sales, the same agent traffic ad pages now block by default [1].

## Three meters, one page, no price

Google's [invitation-only pilot](https://9to5google.com/2026/09/17/google-ai-contribution-pilot-tests-paying-websites-when-theyre-used-in-ai-results/) pays when content "contributes significantly" to an answer in Gemini, AI Overviews or AI Mode, and Google alone judges what counts [5][7]. Participants get a monthly figure in Search Console with no arithmetic behind it [8]. One executive briefed on the program calls it "quite black box"; another said the offers were too low to opt in [5].

Cloudflare's Pay Per Crawl is mechanical: sites set the price from $0.001 a fetch, and a paying crawler gets the page, HTTP 200 and all [5]. [Microsoft](https://theinference.org/markets/companies/microsoft) licenses content to Copilot on "delivered value" it has not defined publicly [5]. The payers are AI labs and agent builders, in cash at the meter or in lost inventory when their agents are blocked. Whichever meter wins will set the price of the open web for AI.

## The toll collector's books

Cloudflare sells both the blockade and the meter. Revenue has risen ten straight quarters, to [$696.1 million in Q2 2026](https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0001477333), up 35.9% on the year, and the quarter still lost $170 million [9][6]. The market has paid up anyway: $115.2 billion of market cap after a 43.9% run in 90 days [6].

*Chart: **Cloudflare revenue has risen ten straight quarters while still losing money** Line chart of Cloudflare quarterly revenue from Q4 2023 to Q2 2026, rising from $362.5M to $696.1M*

*Quarterly revenue in USD millions as reported in Cloudflare's SEC filings via Sharadar, Q4 2023 through Q2 2026; annotation notes the Q2 2026 net loss from the same filings. [9]*

## Google's exit ramp, and what to watch

Google points to [Google-Extended, a robots.txt token it says never touches Search ranking](https://www.searchenginejournal.com/cloudflare-lets-sites-disallow-ai-training-without-blocking-googlebot/589559/), and calls the pilot a learning phase [3][5][10]. That token opts out of Gemini training only; a separate Search Console setting governs AI Overviews and AI Mode [10].

Cloudflare's framing still holds: an ad marks a page built for a person [1]. The exposed party is the small ad-funded publisher, because an agent reads, buys nothing, and never sees the ad.

The next readings are dated: Google's URL-level Google-Extended tools land within weeks, Cloudflare's single control over how much content appears in AI summaries arrives by early next year, and Microsoft's Bing no-training support is targeted for early 2027 [5][10]. The storefronts opened their checkouts to agents; the content layer just installed a toll booth.

## Takeaway

Cloudflare has flipped the default on the ad-funded web: AI agents are now blocked on any page that shows an ad, while Google and Microsoft race to set their own prices for the same content. Whoever's meter wins decides what the open web is worth to an AI.

## Sources

1. [PYMNTS, Cloudflare Blocks AI Agents From Ad-Supported Pages, 17 September 2026](https://www.pymnts.com/news/artificial-intelligence/2026/cloudflare-blocks-ai-agents-from-ad-supported-pages/)
2. [Help Net Security, Cloudflare changes AI crawler access rules, 2 July 2026](https://www.helpnetsecurity.com/2026/07/02/cloudflare-ai-crawler-controls/)
3. [TechCrunch, Cloudflare's new policy pushes AI companies to pay for publishers' content, 1 July 2026](https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/)
4. [PYMNTS, Shopify Brings Shop Pay Checkout Solution to Meta's Muse AI Agent, 22 September 2026](https://www.pymnts.com/commerce/ecommerce/2026/shopify-brings-shop-pay-checkout-solution-to-metas-muse-ai-agent/)
5. [Search Engine Journal, Google's AI Payment Pilot Vs. Cloudflare & Microsoft Models, September 2026](https://www.searchenginejournal.com/googles-ai-payment-pilot-vs-cloudflare-and-microsoft-models/589636/)
6. [Analytics Insight, Cloudflare Adds AI Crawling Controls as Stock Valuation Draws Focus, September 2026](https://www.analyticsinsight.net/news/cloudflare-adds-ai-crawling-controls-as-stock-valuation-draws-focus)
7. [9to5Google, Google 'AI contribution pilot' tests paying websites when they're used in AI results, 17 September 2026](https://9to5google.com/2026/09/17/google-ai-contribution-pilot-tests-paying-websites-when-theyre-used-in-ai-results/)
8. [Nieman Journalism Lab (citing Digiday), Google is quietly rolling out a pay-per-value AI licensing program to publishers, 14 September 2026](https://www.niemanlab.org/reading/google-is-quietly-rolling-out-a-pay-per-value-ai-licensing-program-to-publishers/)
9. [Sharadar quarterly fundamentals, from Cloudflare's SEC filings, retrieved 23 September 2026](https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0001477333)
10. [Search Engine Journal, Cloudflare Lets Sites Disallow AI Training Without Blocking Googlebot, 15 September 2026](https://www.searchenginejournal.com/cloudflare-lets-sites-disallow-ai-training-without-blocking-googlebot/589559/)


---

Cloudflare now blocks AI agents on any page that shows an ad — The Inference. Canonical: https://theinference.org/article/cloudflare-now-blocks-ai-agents-on-any-page-that-shows-an-ad
