Major AI companies, including OpenAI, Google, Microsoft and Anthropic, are backing a shared system that allows publishers to communicate their preferences about whether their online content can be used to train artificial intelligence (AI) models through a single website instruction.
The initiative will let website owners communicate their content-use preferences through a shared policy token in their robots.txt files, reducing the need to configure separate restrictions for individual AI developers. The companies announced their support on 30 October 2026, following discussions within the Internet Engineering Task Force (IETF).
AI Companies Back a Shared Content Protection System
The proposed mechanism introduces a shared policy token called __usage_ai-training that website operators can add to their robots.txt files. This standard file communicates instructions to automated crawlers that access websites and collect information. The new token allows publishers to express their preferences about whether their material can be used to train generative AI models.
Previously, website owners had to identify individual AI crawlers and establish separate restrictions for each provider. OpenAI said the shared mechanism would reduce this work by allowing publishers to communicate the same preference to multiple participating companies. Existing crawler controls will remain available alongside the new system.
The proposal builds on discussions within the IETF's AI Preferences (AIPREF) working group, which is developing standards for communicating website owners' preferences to technology companies. Kevin Kelley of Anthropic submitted the technical proposal outlining the shared signal. The document uses existing robots.txt structures rather than introducing an entirely new website configuration system.
OpenAI, Google and Microsoft Set Different Rollout Dates
The companies have announced different implementation schedules for the shared mechanism. Anthropic has already started developing support for the signal, although it has not announced a specific launch date. OpenAI said it aims to implement the mechanism in the coming months and will provide further details as its rollout progresses.
Moroever, Google plans to support the shared control in 2027, with its documentation identifying early 2027 as its target. The company will interpret a publisher's decision to disallow the shared training token as a restriction on Google-Extended, provided the website does not explicitly specify otherwise. Its existing Google-specific controls will continue to function.
Meanwhile, Microsoft plans to begin honouring the shared AI training preference on 1 February 2027. It is also introducing separate page-level controls that allow publishers to manage AI training and AI grounding, which refers to using website content to inform AI-generated responses. Microsoft said these preferences will not affect a website's visibility in Bing search results.
Publishers Still Face Challenges Over AI Content Use
Despite the agreement, the new mechanism is voluntary and depends on participating providers recognising and respecting the shared signal. The announcement does not mean that every AI company will automatically follow the preference.
Furthermore, the mechanism does not automatically prevent AI companies from accessing website content. The proposed signal communicates how publishers want their material used rather than directly controlling whether automated crawlers can visit their websites. It is also a preference mechanism, not a technical enforcement system.
The initiative comes amid continuing disputes between publishers and AI developers over copyright, licensing and compensation. The New York Times filed a lawsuit against OpenAI and Microsoft in December 2023, alleging that the companies used millions of its articles without permission to train AI models. Both companies have disputed the claims, citing fair use protections.
