--- title: "OpenAI, Google, Microsoft and Anthropic adopt a free robots.txt opt-out; Cloudflare's margin fell to 71.8%" description: "OpenAI, Google, Microsoft and Anthropic adopted a shared robots.txt opt-out on October 30, and Microsoft says it creates no payment arrangements. Cloudflare, which wants every agent visit billed at its edge, reports November 5 with gross margin already sliding." publisher: "The Inference" section: "Platforms" published: 2026-10-31T14:14:06.922Z modified: 2026-10-31T14:14:06.922Z canonical: https://theinference.org/article/openai-google-microsoft-and-anthropic-adopt-a-free-robots-txt-opt-out-cloudflare-s-margin-fell-to-71-8 language: en keywords: "Cloudflare, OpenAI, Anthropic, Microsoft, Google, AI" --- # OpenAI, Google, Microsoft and Anthropic adopt a free robots.txt opt-out; Cloudflare's margin fell to 71.8% > OpenAI, Google, Microsoft and Anthropic adopted a shared robots.txt opt-out on October 30, and Microsoft says it creates no payment arrangements. Cloudflare, which wants every agent visit billed at its edge, reports November 5 with gross margin already sliding. ## One line of text, four labs, zero tolls Publishers gained a one-line veto over four AI companies on October 30 [1]. [OpenAI](https://theinference.org/markets/companies/openai), Google, [Microsoft](https://theinference.org/markets/companies/microsoft) and [Anthropic](https://theinference.org/markets/companies/anthropic) said they will honor a shared "policy token" inside robots.txt, the plain-text file that has steered web crawlers since the 1990s, so a site can bar all four from training on its content at once [1]. The token, named __usage_ai-training, was drafted by Kevin Kelley, an Anthropic standards lead, and [submitted to the Internet Engineering Task Force](https://thenextweb.com/news/ai-training-opt-out-robots-txt-google-openai-microsoft-anthropic) [2]. Anthropic already honors it. OpenAI promises the coming months, [Google](https://theinference.org/markets/companies/alphabet) and Microsoft arrive in 2027, Microsoft on February 1 [1][2]. Microsoft was blunt about what the signal is not: the new preferences "do not establish licensing or payment arrangements" [2]. ## The free signal is aimed at a meter That line lands on a business. [Cloudflare](https://theinference.org/markets/companies/cloudflare), which carries more than 20% of the web, blocks AI crawlers by default and runs a [gateway launched in September](https://theinference.org/article/cloudflare-s-bouncer-now-runs-the-bank-for-ai-agents) where software agents pay small fees per page [1][3]. Chief executive [Matthew Prince](https://theinference.org/markets/people/matthew-prince) predicts agent traffic will reach "a thousand times human traffic" within five years, and the network already returns more than 2 billion pay-required replies a day, most ending without payment [3]. ## Eighteen million attacks no token stops What no token governs is traffic that never asks permission. Cloudflare [mitigated 18.2 million network-layer DDoS attacks in the third quarter](https://blog.cloudflare.com/ddos-threat-report-2026-q3/), one every 0.44 seconds, after 13.2 million in Q2 and 10.0 million in Q1 [4][5]. The polite camp's record is imperfect too: research by licensing startup TollBit documented AI agents bypassing robots.txt signals, and reporting that followed named OpenAI and Anthropic among them [6]. The polite crawlers promised to behave. The impolite ones sent 18.2 million attacks last quarter [6]. *Chart: **The web no token governs grew every quarter this year** Column chart of network-layer DDoS attacks Cloudflare mitigated per quarter in 2026, rising from 10.0 million in Q1 to 18.2 million in Q3* *Network-layer (L3) DDoS attacks autonomously mitigated by Cloudflare, millions per quarter, 2026. Sources: Cloudflare Q3 2026 DDoS Threat Report; The Hacker News, October 31, 2026. [4][5]* The flood shows in the filings. [GAAP gross margin slid from 77.8% in mid-2024 to 71.8% last quarter](https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0001477333), even as revenue growth accelerated to 36% [7]. On the non-GAAP basis management guides on, margin was 73.1% in Q2, down from 76.3% a year earlier [8]. *Chart: **Gross margin has slid for two years while revenue growth accelerated** Line chart of Cloudflare GAAP gross margin across ten quarters, declining from 77.5 percent in Q1 2024 to 71.8 percent in Q2 2026* *Cloudflare GAAP gross margin, percent of revenue, by quarter, from its SEC filings; non-human traffic passed human traffic on its network in May 2026. Sources: Cloudflare quarterly SEC filings; Yahoo Finance, October 30, 2026. [7][3]* ## The margin decides on November 5 The labs' side is real: publishers get control free, with no intermediary, and Microsoft keeps opted-out sites in Bing results [1][2]. Whether that blunts a paid gateway, nothing yet shows. NET closed the week down 14% at $280.70 with no bad news of its own, still near [190 times forward earnings](https://theinference.org/article/cloudflare-is-buying-the-whole-stack-ai-agents-build-on-and-the-stock-trades-at-49-5-times-sales) against a 173 average since mid-2023 [8]. So November 5 turns on gross margin at or above 73.1%: hold it and the dip case stands, slip again and Guggenheim's $160 Sell gains ground against the $345 consensus [8]. [The guide is $736 to 737 million of revenue, 31% growth, and $0.34 of adjusted EPS](https://finance.yahoo.com/markets/stocks/articles/cloudflare-sinks-14-datadog-13-104409899.html) [8]. February is management's own deadline for the first agent-payment revenue [3][8]. Until then the race is OpenAI's coming months against Prince's meter. ## Takeaway Publishers can now bar OpenAI, Google, Microsoft and Anthropic from training with one free line of robots.txt, Microsoft says that creates no payment rights, and Cloudflare, which wants agent visits billed at its edge, reports November 5 with GAAP gross margin down to 71.8% from 77.8% two years ago. ## Sources 1. [The Verge, "The biggest names in AI are backing a new way to opt out of scraping," 30 October 2026](https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out) 2. [The Next Web, "Google, Microsoft, OpenAI and Anthropic back one AI training opt-out," 31 October 2026](https://thenextweb.com/news/ai-training-opt-out-robots-txt-google-openai-microsoft-anthropic) 3. [Yahoo Finance (TIKR), "Cloudflare CEO: Agent Traffic Will Be 'a Thousand Times Human Traffic.' Its Pricing Has to Change," 30 October 2026](https://finance.yahoo.com/technology/articles/cloudflare-ceo-agent-traffic-thousand-180910976.html) 4. [Cloudflare blog (Cloudforce One), "Cloudflare DDoS Threat Report 2026 Q3: More swarms, fewer giants," 29 October 2026](https://blog.cloudflare.com/ddos-threat-report-2026-q3/) 5. [The Hacker News, "Cloudflare Reports 18.2 Million Network-Layer DDoS Attacks in Q3, Up 38%," 31 October 2026](https://thehackernews.com/2026/10/cloudflare-reports-182-million-network.html) 6. [Tech Times (via MSN), "AI scraping opt-out bypassed by OpenAI and Anthropic, ABC tells Parliament," 6 October 2026](https://www.msn.com/en-us/news/other/ai-scraping-opt-out-bypassed-by-openai-and-anthropic-abc-tells-parliament/ar-AA2dH0aX) 7. [Cloudflare, Inc., quarterly revenue and GAAP gross margin from its SEC filings, retrieved 31 October 2026](https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0001477333) 8. [Yahoo Finance (TIKR), "Cloudflare Sinks 14% and Datadog 13% in a Week as Software Selloff Hits Cloud Stocks," 31 October 2026](https://finance.yahoo.com/markets/stocks/articles/cloudflare-sinks-14-datadog-13-104409899.html) --- OpenAI, Google, Microsoft and Anthropic adopt a free robots.txt opt-out; Cloudflare's margin fell to 71.8% — The Inference. Canonical: https://theinference.org/article/openai-google-microsoft-and-anthropic-adopt-a-free-robots-txt-opt-out-cloudflare-s-margin-fell-to-71-8