Cloudflare Opens AI Training Opt-Out Settings
Cloudflare has opened the "opt-out of AI training" switch to all customers, allowing sites to continue being indexed while rejecting content from entering model training, thus separating the trade-off between search and training for the same crawler. Apple and Google have complied or committed to this setting, while Microsoft has provided a clear timeline. Choosing "block" will prevent mixed crawlers from Apple, Bing, and Google from accessing search; to keep search while stopping training, the new switch must be used.
The company has also established a "accountable" crawler identification with four criteria: providing an opt-out for training, providing an opt-out for AI summaries, disclosing indexing and training usage by URL, and publicly committing that opting out of training will not affect traditional search rankings. Apple, Google, Microsoft, as well as Amazon, Anthropic, Meta, and OpenAI have received this designation. The latter four have separated their search crawlers from training crawlers, ensuring that blocking training does not affect search. Bing's domain-level support is targeted for early 2027.
Data from the network side indicates that approximately 36.6% of verified crawler traffic is of mixed use, making it the largest category. About 17% of sites have enabled some form of training blocking mechanism, while blocking search crawlers is less than 1%. Newly added domains that rely on advertising for monetization will default to prohibiting training and proxies on ad pages while allowing search; the remaining sites will have three items default to allow. There is currently no separate "prohibit" category for proxies, which will be discussed once preference standards mature.
The switch is synchronized via robot preferences written into robots.txt and is available for free accounts. Since July, traffic has been divided into three categories: search, training, and proxies; from that day, the old "block AI crawlers" and "ad page block" settings take effect for mixed crawlers. Cloudflare aims to allow site owners to control how much their content is used in one place, rather than negotiating with each entity.
In market mechanisms, buyers are model companies that need both corpus and search entry, while sellers are sites that rely on advertising and clicks for survival. This is a default change in infrastructure: changing "not training" from a threat that must sacrifice indexing to a preference that can be selected independently. Traffic is shifting from training crawlers to still-allowed search crawlers; beneficiaries are publishers who can demand both ranking and refuse training, while pressured parties are crawler operators whose training and search have not yet been separated, as well as search engines that must prove that opting out of training does not affect rankings.
Source: Public Information
ABAB AI Insight
Cloudflare has upgraded content rights from a courtesy agreement to a default switch. For thirty years, the exchange has been crawling your site and sending back traffic; generative retrieval turns the same crawl into training corpus, leaving publishers with an all-or-nothing choice. The asymmetry of 17% to less than 1% indicates that site owners fear being written into weights, not just being indexed. The accountable designation brings Google, Apple, Microsoft, and four labs into the same compliance table; those who do not separate crawlers will disappear from search when "prohibit training" is activated.
The capital path is the edge network charging model factories for passage rights. Cloudflare does not own content but controls the first hop for most sites. Writing preferences into free accounts effectively pressures training parties to split products: search bots remain, while training bots must pay or be blocked. Amazon and OpenAI have separated crawlers first because they can afford two teams; smaller crawlers cannot afford this and will disappear under default blocking. The default prohibition on training on ad pages legally defines the purpose of attention currency for display ads.
This is not a birth of robots.txt but a cookie consent wall: platforms first offer a one-click refusal, then force collectors to prove they can separate. Google-Extended has already tried opting out of training; Cloudflare has made it the default language across the network. The industry is moving from "indiscriminate crawling" to "usage must be declared." The lack of a prohibition category for proxy bots indicates that the next battle is real-time agents, not offline training.
Structural judgment indicates a transfer of pricing power. The mechanism is: the price of corpus was once zero plus bandwidth, now it is a switch that can be turned off. Whoever controls the switch can change training from tacit approval to permission. The promise of search ranking is collateral that can be sold with the switch—once rankings drop due to opting out of training, the collateral is void, and the site will reopen the floodgates.
ABAB News · Cognitive Laws
The same crawler doing two things can only be turned off once by the site owner.
A default switch is closer to property rights than a courtesy agreement.
If opting out of training does not harm rankings, refusal can have a price.