Platforms

OpenAI, Google, Microsoft and Anthropic adopt a free robots.txt opt-out; Cloudflare's margin fell to 71.8%

OpenAI, Google, Microsoft and Anthropic adopted a shared robots.txt opt-out on October 30, and Microsoft says it creates no payment arrangements. Cloudflare, which wants every agent visit billed at its edge, reports November 5 with gross margin already sliding.

Vincent JiangVincent Jiangthree-minute read
Share
Matthew Prince, co-founder and chief executive of Cloudflare, speaking on Centre Stage at Web Summit 2025 in Lisbon
1 / 6Slide 1 of 6
Cloudflare co-founder and chief executive Matthew Prince, whose agent-traffic meter the free robots.txt opt-out is aimed at, at Web Summit 2025 in Lisbon.

One line of text, four labs, zero tolls

Publishers gained a one-line veto over four AI companies on October 30 1. OpenAI$1.18T — OpenAI, private, latest valuation $1.18T, Google, Microsoft−0.74% — Microsoft, down 0.74 percent today and Anthropic$350B — Anthropic, private, latest valuation $350B said they will honor a shared "policy token" inside robots.txt, the plain-text file that has steered web crawlers since the 1990s, so a site can bar all four from training on its content at once 1. The token, named __usage_ai-training, was drafted by Kevin Kelley, an Anthropic standards lead, and submitted to the Internet Engineering Task Force 2.

Anthropic already honors it. OpenAI promises the coming months, Google−3.94% — Google, down 3.94 percent today and Microsoft arrive in 2027, Microsoft on February 1 12. Microsoft was blunt about what the signal is not: the new preferences "do not establish licensing or payment arrangements" 2.

The free signal is aimed at a meter

That line lands on a business. Cloudflare−4.16% — Cloudflare, down 4.16 percent today, which carries more than 20% of the web, blocks AI crawlers by default and runs a gateway launched in September where software agents pay small fees per page 13. Chief executive Matthew Prince predicts agent traffic will reach "a thousand times human traffic" within five years, and the network already returns more than 2 billion pay-required replies a day, most ending without payment 3.

Eighteen million attacks no token stops

What no token governs is traffic that never asks permission. Cloudflare mitigated 18.2 million network-layer DDoS attacks in the third quarter, one every 0.44 seconds, after 13.2 million in Q2 and 10.0 million in Q1 45. The polite camp's record is imperfect too: research by licensing startup TollBit documented AI agents bypassing robots.txt signals, and reporting that followed named OpenAI and Anthropic among them 6. The polite crawlers promised to behave. The impolite ones sent 18.2 million attacks last quarter 6.

The web no token governs grew every quarter this year

0M5M10M15M20MQ1 2026Q2 2026Q3 2026one every 0.44 seconds
Data
L3 DDoS attacks mitigated
Q1 202610M
Q2 202613.2M
Q3 202618.2M
Network-layer (L3) DDoS attacks autonomously mitigated by Cloudflare, millions per quarter, 2026. Sources: Cloudflare Q3 2026 DDoS Threat Report; The Hacker News, October 31, 2026.4,5

The flood shows in the filings. GAAP gross margin slid from 77.8% in mid-2024 to 71.8% last quarter, even as revenue growth accelerated to 36% 7. On the non-GAAP basis management guides on, margin was 73.1% in Q2, down from 76.3% a year earlier 8.

Gross margin has slid for two years while revenue growth accelerated

70%72%74%76%78%Q2 '24Q4 '24Q2 '25Q4 '25Q2 '2671.8%bots passed humans in May
Data
GAAP gross margin (%)
Q1 '2477.5%
Q2 '2477.8%
Q3 '2477.7%
Q4 '2476.4%
Q1 '2575.9%
Q2 '2574.9%
Q3 '2574%
Q4 '2573.6%
Q1 '2671.2%
Q2 '2671.8%
Cloudflare GAAP gross margin, percent of revenue, by quarter, from its SEC filings; non-human traffic passed human traffic on its network in May 2026. Sources: Cloudflare quarterly SEC filings; Yahoo Finance, October 30, 2026.7,3

The margin decides on November 5

The labs' side is real: publishers get control free, with no intermediary, and Microsoft keeps opted-out sites in Bing results 12. Whether that blunts a paid gateway, nothing yet shows. NET closed the week down 14% at $280.70 with no bad news of its own, still near 190 times forward earnings against a 173 average since mid-2023 8.

So November 5 turns on gross margin at or above 73.1%: hold it and the dip case stands, slip again and Guggenheim's $160 Sell gains ground against the $345 consensus 8. The guide is $736 to 737 million of revenue, 31% growth, and $0.34 of adjusted EPS 8. February is management's own deadline for the first agent-payment revenue 38. Until then the race is OpenAI's coming months against Prince's meter.

Get Platforms by email, free

The cloud, data and developer software AI is built on. In our daily email, every weekday.

Loading NET

Reader comments

0 comments

    More in Platforms

    All in this topic →

    Platforms

    Pump.co's claimed spend under management jumped from $600 million to $10 billion in six weeks

    The Y Combinator-backed startup charges customers $0, says cloud providers pay it, and is hiring account executives and partnership leads anyway. For FinOps rivals that bill monthly, and for anyone underwriting the next round, the homepage's new $10B+ figure is the first claim to test.

    Platforms

    LangChain raises $300 million at $3 billion, and the cap table now includes its own customers

    LangChain's agent-governance layer just repriced 2.4 times in 13 months on a usage number its own lead investor calls apples to oranges, and two of its new backers sell competing agent controls. Whose money moves: anyone pricing AI application dependencies against what the model labs will bundle next.

    Platforms

    Parcha's Greptile bill passed $7,000 a month, so it built an 18-cent reviewer and open-sourced it

    The bill is the first customer-published measure of what usage-priced AI code review costs when agents write the pull requests, and it landed the week Cohere bought the reviewer ranked above Greptile on an independent leaderboard. The money at stake is Greptile's $750 million mark and every seat-priced review vendor's renewal.

    Platforms

    DXC booked 80 cents of new orders for every dollar of revenue last quarter

    DXC spent last week selling OASIS, the agent platform it says cuts incident-resolution time 95% inside banks and hospitals. Its September order book, 80 cents of new work per dollar billed, says the money has not followed the demos, and the core it must replace is still running off.

    Platforms

    Palantir's Karp says a wealth tax 'is going to need surveillance' as the stock sits at 219 times earnings

    Recorded on November 10 for Walter Isaacson's podcast and carried to more than 14,000 Reddit points, the interview landed in a week that took 17.55 percent off Palantir. At 219 times earnings, holders own the founder's story as much as his contract book.

    Platforms

    Amtrak's booking site failed during the FAA's four-hour national ground stop

    Amtrak blamed a "systemwide update", the same failure mode that grounded the country's flights: a change pushed into a system that is never allowed to stop. The money Congress is asked to add will buy change discipline on live systems, and the vendors selling staged rollouts and tested failover are the layer positioned to be paid for uptime.

    Subscribe to TheInference for free