--- title: "Cloudflare steered OpenAI's crawler to hundreds of millions of pages with 1% of its network, named no price" description: "Signals from 1% of Cloudflare's network steered OpenAI's crawler to hundreds of millions of pages no link-following spider finds. The post names no money for it, and Q3 results are now dated for November 5." publisher: "The Inference" section: "Platforms" published: 2026-10-16T18:17:46.946Z modified: 2026-10-16T18:17:46.946Z canonical: https://theinference.org/article/cloudflare-steered-openai-s-crawler-to-hundreds-of-millions-of-pages-with-1-of-its-network-named-no-price language: en keywords: "Cloudflare, OpenAI, AI Crawling, Web Publishing, Stock Valuation" --- # Cloudflare steered OpenAI's crawler to hundreds of millions of pages with 1% of its network, named no price > Signals from 1% of Cloudflare's network steered OpenAI's crawler to hundreds of millions of pages no link-following spider finds. The post names no money for it, and Q3 results are now dated for November 5. [Cloudflare](https://theinference.org/markets/companies/cloudflare)'s first results from [the discovery pilot it launched with OpenAI in July](https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/) settle the capability question and leave the money question exactly where it was [1]. Signals drawn from about 1% of Cloudflare's network helped [OpenAI](https://theinference.org/markets/companies/openai)'s search crawler discover and index hundreds of millions of web pages that had never been in its index [1]. Nowhere does the post name a price; where one would go, it says the two companies are still working out how discovery translates into "real business outcomes" [1]. ## The signal comes from traffic, not links A link-following crawler never finds a page nobody links to, which is why the manual for your exact washing machine can stay invisible to AI search [1]. Cloudflare's edge network sits in front of the traffic itself, across more than a fifth of the web, so it sees which addresses real people actually request and when a page's cache behavior shows it changed [2]. Fed into OpenAI's crawl scheduling, that metadata aims the crawler at pages worth fetching and skips the rest [2]. The far end is a ChatGPT answer that reaches the exact manual, with fewer wasted fetches for the site hosting it [1]. ## Access has a price. Discovery does not. Cloudflare already runs the other side of this market. [AI Crawl Control, available on every plan](https://developers.cloudflare.com/ai-crawl-control/), gives site owners the gate: see which AI services visit and allow or block them crawler by crawler [3]. Pay Per Crawl, still in beta, adds the till, letting owners set a per-crawl price that AI companies pay through [Stripe](https://theinference.org/markets/companies/stripe) [3]. The pilot stays on the far side of that gate: metadata about pages, never their content, and a signal grants no right of access [1]. So a site owner on Cloudflare can already charge an AI company each time it fetches a page, while the pilot hands the signals that guide that crawler to OpenAI with no price attached [1]. Publishers feel the difference. ChatGPT referrals were still under 1% of publisher page-view traffic in [Chartbeat's March 2026 data](https://www.msn.com/en-us/news/other/chatgpt-search-gained-hundreds-of-millions-of-pages-link-crawlers-never-found/ar-AA2em8Jk), and the pilot's next phase is measuring what being found actually does for the sites that get indexed [2]. ## The checkpoint is November 5 OpenAI is private, so the listed company here is Cloudflare (NYSE: NET), which [dated its third-quarter results on October 13](https://www.nasdaq.com/press-release/cloudflare-announces-date-third-quarter-2026-financial-results-2026-10-13) for after the close on November 5 [4][5]. The stock prices none of this. NET has fallen four sessions in a row from its October 6 record of $370 and is still up more than half this year [6]. Those shares trade at roughly 49.5 times sales, the figure at the center of [the case we made on October 14](https://theinference.org/article/cloudflare-is-buying-the-whole-stack-ai-agents-build-on-and-the-stock-trades-at-49-5-times-sales): that Cloudflare keeps buying every layer of infrastructure AI agents get built on, and that buyers at this price are paying for all of it before the newest layers carry any revenue. [A Simply Wall St analysis published on Yahoo Finance](https://finance.yahoo.com/technology/ai/articles/cloudflare-net-ai-crawler-controls-060920372.html) already treats the crawler-control layer as a moat between publishers and AI companies and names "practical, monetizable services" as the execution test [7]. The pilot is the first working evidence for the layer beneath those controls: the signals demonstrably work, and the post shows nothing about what anyone would pay for them. Whether Cloudflare ever puts a price on discovery is the question the November 5 call can begin to answer. ## Takeaway Cloudflare's discovery pilot with OpenAI proves the signals work, indexing hundreds of millions of pages link crawlers never find, but the post names no price; the November 5 earnings call is the first checkpoint for whether discovery becomes revenue. ## Sources 1. [Cloudflare Blog, 'Cloudflare and OpenAI: Making AI discovery better for everyone,' 15 October 2026](https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/) 2. [Tech Times via MSN, 'ChatGPT search gained hundreds of millions of pages link crawlers never found,' 15 October 2026](https://www.msn.com/en-us/news/other/chatgpt-search-gained-hundreds-of-millions-of-pages-link-crawlers-never-found/ar-AA2em8Jk) 3. [Cloudflare documentation, 'AI Crawl Control: Overview,' updated 14 August 2026](https://developers.cloudflare.com/ai-crawl-control/) 4. [TMCnet, 'Cloudflare Announces Date of Third Quarter 2026 Financial Results,' 13 October 2026](https://www.tmcnet.com/usubmit/2026/10/13/10462358.htm) 5. [Nasdaq, 'Cloudflare Announces Date of Third Quarter 2026 Financial Results,' 13 October 2026](https://www.nasdaq.com/press-release/cloudflare-announces-date-third-quarter-2026-financial-results-2026-10-13) 6. [Yahoo Finance (Schaeffer's), 'Software Stock Pullback Signaling Bulls to Move In,' 15 October 2026](https://finance.yahoo.com/markets/stocks/articles/software-stock-pullback-signaling-bulls-180912837.html) 7. [Yahoo Finance (Simply Wall St), 'Do Cloudflare's (NET) New AI Crawler Controls Quietly Strengthen Its Data-Moat Narrative?,' 25 September 2026](https://finance.yahoo.com/technology/ai/articles/cloudflare-net-ai-crawler-controls-060920372.html) --- Cloudflare steered OpenAI's crawler to hundreds of millions of pages with 1% of its network, named no price — The Inference. Canonical: https://theinference.org/article/cloudflare-steered-openai-s-crawler-to-hundreds-of-millions-of-pages-with-1-of-its-network-named-no-price