Cloudflare steered OpenAI's crawler to hundreds of millions of pages with 1% of its network, named no price
Signals from 1% of Cloudflare's network steered OpenAI's crawler to hundreds of millions of pages no link-following spider finds. The post names no money for it, and Q3 results are now dated for November 5.
Richard Tang · 3 min read
Cloudflare's−6.30% — Cloudflare, down 6.30 percent today first results from the discovery pilot it launched with OpenAI in July settle the capability question and leave the money question exactly where it was 1. Signals drawn from about 1% of Cloudflare's network helped OpenAI's$1.18T — OpenAI, private, latest valuation $1.18T search crawler discover and index hundreds of millions of web pages that had never been in its index 1. Nowhere does the post name a price; where one would go, it says the two companies are still working out how discovery translates into "real business outcomes" 1.
The signal comes from traffic, not links
A link-following crawler never finds a page nobody links to, which is why the manual for your exact washing machine can stay invisible to AI search 1.
Cloudflare's edge network sits in front of the traffic itself, across more than a fifth of the web, so it sees which addresses real people actually request and when a page's cache behavior shows it changed 2. Fed into OpenAI's crawl scheduling, that metadata aims the crawler at pages worth fetching and skips the rest 2. The far end is a ChatGPT answer that reaches the exact manual, with fewer wasted fetches for the site hosting it 1.
Access has a price. Discovery does not.
Cloudflare already runs the other side of this market. AI Crawl Control, available on every plan, gives site owners the gate: see which AI services visit and allow or block them crawler by crawler 3. Pay Per Crawl, still in beta, adds the till, letting owners set a per-crawl price that AI companies pay through Stripe 3. The pilot stays on the far side of that gate: metadata about pages, never their content, and a signal grants no right of access 1.
So a site owner on Cloudflare can already charge an AI company each time it fetches a page, while the pilot hands the signals that guide that crawler to OpenAI with no price attached 1.
Publishers feel the difference. ChatGPT referrals were still under 1% of publisher page-view traffic in Chartbeat's March 2026 data, and the pilot's next phase is measuring what being found actually does for the sites that get indexed 2.
The checkpoint is November 5
OpenAI is private, so the listed company here is Cloudflare (NYSE: NET), which dated its third-quarter results on October 13 for after the close on November 5 45. The stock prices none of this. NET has fallen four sessions in a row from its October 6 record of $370 and is still up more than half this year 6.
Those shares trade at roughly 49.5 times sales, the figure at the center of the case we made on October 14: that Cloudflare keeps buying every layer of infrastructure AI agents get built on, and that buyers at this price are paying for all of it before the newest layers carry any revenue.
A Simply Wall St analysis published on Yahoo Finance already treats the crawler-control layer as a moat between publishers and AI companies and names "practical, monetizable services" as the execution test 7. The pilot is the first working evidence for the layer beneath those controls: the signals demonstrably work, and the post shows nothing about what anyone would pay for them. Whether Cloudflare ever puts a price on discovery is the question the November 5 call can begin to answer.
Deepdive
AI-generated from this story and its cited sources. Not investment advice.
