OpenAI, Google, Microsoft and Anthropic adopt a free robots.txt opt-out; Cloudflare's margin fell to 71.8%

One line of text, four labs, zero tolls

Publishers gained a one-line veto over four AI companies on October 30 ¹. OpenAIPVT$500B — OpenAI, private, latest valuation $500B, Google, Microsoft▼−1.38% — Microsoft, down 1.38 percent today and AnthropicPVT$350B — Anthropic, private, latest valuation $350B said they will honor a shared "policy token" inside robots.txt, the plain-text file that has steered web crawlers since the 1990s, so a site can bar all four from training on its content at once ¹. The token, named __usage_ai-training, was drafted by Kevin Kelley, an Anthropic standards lead, and submitted to the Internet Engineering Task Force↗ ².

Anthropic already honors it. OpenAI promises the coming months, Google▼−0.82% — Google, down 0.82 percent today and Microsoft arrive in 2027, Microsoft on February 1 ¹². Microsoft was blunt about what the signal is not: the new preferences "do not establish licensing or payment arrangements" ².

The free signal is aimed at a meter

That line lands on a business. Cloudflare▼−9.41% — Cloudflare, down 9.41 percent today, which carries more than 20% of the web, blocks AI crawlers by default and runs a gateway launched in September where software agents pay small fees per page ¹³. Chief executive Matthew Prince predicts agent traffic will reach "a thousand times human traffic" within five years, and the network already returns more than 2 billion pay-required replies a day, most ending without payment ³.

Eighteen million attacks no token stops

What no token governs is traffic that never asks permission. Cloudflare mitigated 18.2 million network-layer DDoS attacks in the third quarter↗, one every 0.44 seconds, after 13.2 million in Q2 and 10.0 million in Q1 ⁴⁵. The polite camp's record is imperfect too: research by licensing startup TollBit documented AI agents bypassing robots.txt signals, and reporting that followed named OpenAI and Anthropic among them ⁶. The polite crawlers promised to behave. The impolite ones sent 18.2 million attacks last quarter ⁶.

The flood shows in the filings. GAAP gross margin slid from 77.8% in mid-2024 to 71.8% last quarter↗, even as revenue growth accelerated to 36% ⁷. On the non-GAAP basis management guides on, margin was 73.1% in Q2, down from 76.3% a year earlier ⁸.

The margin decides on November 5

The labs' side is real: publishers get control free, with no intermediary, and Microsoft keeps opted-out sites in Bing results ¹². Whether that blunts a paid gateway, nothing yet shows. NET closed the week down 14% at $280.70 with no bad news of its own, still near 190 times forward earnings against a 173 average since mid-2023 ⁸.

So November 5 turns on gross margin at or above 73.1%: hold it and the dip case stands, slip again and Guggenheim's $160 Sell gains ground against the $345 consensus ⁸. The guide is $736 to 737 million of revenue, 31% growth, and $0.34 of adjusted EPS↗ ⁸. February is management's own deadline for the first agent-payment revenue ³⁸. Until then the race is OpenAI's coming months against Prince's meter.