Industry NewsAI Business & Ecosystem
Cloudflare and OpenAI pilot helps ChatGPT discover hundreds of millions of web pages
Article content
Cloudflare and OpenAI have shared early findings from a joint research pilot designed to help AI-powered search discover useful content across the web more efficiently.
The initiative, first announced in July 2026, explores whether signals from Cloudflare's global network can help OpenAI identify web pages that may be valuable to people searching through ChatGPT.
According to the companies, the pilot has enabled OpenAI to discover and index hundreds of millions of previously unknown web pages, expanding the sources available to ChatGPT when responding to questions and helping users find relevant, timely information.
The research addresses a growing challenge for publishers, businesses and website owners: ensuring that useful information can be discovered by AI systems as people increasingly use conversational interfaces to research products, solve problems and make decisions.
Moving beyond traditional search visibility
For years, website discoverability has largely been associated with search engine rankings, backlinks and the ability to attract visitors from conventional search results.
AI-powered discovery introduces a different set of considerations. Systems need to locate relevant pages, assess their usefulness and connect information to the questions people are asking.
This can be particularly important for specialised content that may not attract many external links, including product documentation, technical guides, local business information and first-hand product experiences.
Cloudflare's network can provide signals to help OpenAI's search crawler identify potentially useful pages. OpenAI then independently evaluates and ranks discovered content according to its relevance and usefulness.
The distinction matters: helping a page become discoverable does not guarantee that it will rank highly or appear in a particular answer. It creates another route through which relevant content can enter the search process.
A pilot covering approximately 1% of Cloudflare's network
The research pilot used a participating cohort representing approximately 1% of Cloudflare's total network.
Within that scope, the companies report that OpenAI discovered and indexed hundreds of millions of new web pages. The expanded index gives ChatGPT Search a broader pool of sources from which to retrieve information and link users to original websites.
The approach uses network-level signals to support discovery rather than relying exclusively on conventional indicators such as links from other websites.
This could be particularly relevant for smaller publishers, specialist businesses and websites containing detailed information that is useful to a specific audience but difficult for traditional discovery mechanisms to surface.
However, inclusion in an index does not automatically translate into traffic, visibility or commercial outcomes. The research's next phase will examine how improved discovery translates into meaningful results for participating websites.
Improving crawling efficiency and content quality
Finding potential URLs is only one part of the process. AI search systems must also determine whether a page exists, contains distinct information and is accessible.
Cloudflare and OpenAI say the pilot has helped them identify patterns that can waste crawling resources, including requests for non-existent pages, redirects to homepages and duplicate content.
Filtering these cases can reduce unnecessary requests and allow crawlers to focus more of their resources on pages that may offer useful information.
The companies are also testing signals that help identify potentially harmful pages. For example, a compromised page containing a fraudulent login form may need to be excluded from an index without treating the entire legitimate website as unsafe.
These signals are intended to improve discovery quality while supporting a more targeted approach to safety.
Balancing AI visibility with website control
As AI systems become another channel through which audiences discover information, website owners face new questions about access, crawling frequency and control over their content.
Repeatedly fetching pages that have not changed can create unnecessary server load without improving the information available to users.
The pilot explores how change signals can help crawlers determine when a page is worth revisiting. Combined with filtering for non-existent, redirected or duplicate pages, this can make crawling more efficient while helping search systems keep their indexes current.
Cloudflare says website owners can use its AI Crawl Control tools to understand and manage how AI systems access their sites.
Importantly, a signal that helps identify a page does not grant permission to access it. The companies also state that the research uses web metadata for search and discovery rather than sharing page content for model training.
The next phase of AI discovery
Cloudflare and OpenAI plan to build on the initial findings by improving page selection, testing faster and more reliable signals for detecting content changes, and measuring what happens after pages are discovered.
A key question is whether better technical discovery can lead to tangible outcomes for website owners, including more relevant referrals and stronger connections with audiences searching for specific information.
For businesses and publishers, the findings highlight an emerging aspect of digital visibility. As AI-powered search develops, being discoverable by an AI system may become an increasingly important consideration alongside conventional search optimisation.
The pilot is an early step towards understanding how that discovery process can become more useful, efficient and controllable for both AI platforms and the wider web.
About Cloudflare
Cloudflare is a connectivity cloud company providing services designed to help businesses make their websites, applications and networks more secure, reliable and performant. Its offerings include content delivery, application security, network services and tools for managing automated traffic, including AI crawlers.
About OpenAI
OpenAI is an artificial intelligence research and deployment company that develops AI systems and products, including ChatGPT and ChatGPT Search. Its work includes developing AI tools that help people access information, complete tasks and interact with digital content.