One robots.txt Line for AI Training Opt-Outs: What Four AI Labs Agreed To

Sources checked and dated on publication. See our editorial policy.

On October 30, 2026, OpenAI, Google, Microsoft and Anthropic each published or endorsed a plan to honour one shared robots.txt signal for generative-AI training preferences. The idea is simple: a website states its choice once, using the policy token __usage_ai-training, instead of keeping a separate rule for every AI crawler. Nothing is enforced yet. The underlying specification is an individual IETF Internet-Draft, not an adopted standard, and the four providers gave different timelines ranging from the coming months to 2027.

What actually happened on October 30

Related guide: The UK can test frontier AI models again: what the 17-day freeze and the U-turn actually changed

On October 30, 2026, three companies published announcements on the same day. OpenAI said it "will honor a common signal" in robots.txt that lets websites express preferences about the use of their content for foundational generative AI training. Google's Search Central blog announced plans to support "a shared AI training control" based on work from the IETF's AI Preferences (AIPREF) working group. Microsoft Bing went furthest, publishing an implementation guide for separate AI Training and AI Grounding preferences. The document all three link to is an individual IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, by Kevin Kelley of Anthropic, dated the same day.

Anthropic's role is slightly different, and stating it precisely matters. Kelley announced the draft on the IETF ai-control mailing list, writing that "Anthropic has already started implementing support for this draft" and that it "complements our existing crawler controls rather than replacing them." The Verge and The Next Web reported the four-company backing that day. The accurate description is therefore three corporate blog posts plus one standards engineer's mailing-list announcement, not four identical press releases.

What they converged on is narrow: one vendor-neutral way to say "do not use this content to train generative models." Today a publisher who wants that has to maintain separate rules for named crawlers such as GPTBot, Google-Extended, ClaudeBot and Applebot-Extended, and keep that list current as new bots appear. OpenAI's post says the shared signal reduces "that work for operators"; Google's says it lets websites express preferences "across multiple AI providers in a single step." Both describe a voluntary, opt-in mechanism rather than a regulation.

How the token works

Related guide: Claude Dashboards and Motion: Live Data Dashboards and Animated Explainers

The draft defines a single policy token: __usage_ai-training. A site writes an ordinary robots.txt group whose User-Agent line names that token, then uses the usual Disallow and Allow path rules to say where the preference applies. The draft's own example expresses a preference against training for everything except a press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The design deliberately adds no new directives and no new grammar. Path matching and longest-match precedence come straight from RFC 9309, the robots.txt standard, so existing parsers should simply ignore a group for a user agent they do not recognise. Kelley's mailing-list post gives the reasoning: the mechanism "uses only existing syntax, so parsers and crawlers that don't implement it but comply with RFC 9309 should gracefully ignore preference expressions and still receive other signals."

Three details are easy to miss and matter more than the headline. First, wildcard groups do not count: the draft says User-Agent: * groups MUST NOT be interpreted as matching the policy token, so a site's existing blanket rules do not accidentally become a training opt-out. Second, silence is not a preference: if no group names __usage_ai-training, the draft says no preference has been expressed at all. Third, use and crawling are independent. The document states plainly that "this policy token is not a directive for crawling behavior"; rules for a crawler's own product token still govern fetching. A training preference is about what happens to content after it is retrieved, not about whether a bot may retrieve it.

Microsoft documents the same syntax in its implementation guide, including a whole-site disallow, a path-level disallow and an optional explicit allow, and adds page-level meta tags and HTTP headers for training and grounding. Bing's FAQ defines AI Training, quoting the IETF vocabulary, as "using an asset to modify the learned parameters of a generative AI model," and notes that models used exclusively for non-generative tasks fall outside the category.

The timelines do not line up

Related guide: Microsoft Copilot's 2026 Overhaul: Inside Home, Code, and Persistent Autopilot

Nothing changed on October 30 in terms of crawler behaviour. The Bing FAQ answers "Does anything change today?" with a flat "No. October 30, 2026 is an announcement." Each provider published its own schedule, and they differ by months.

Microsoft is the most specific. Its new preferences begin taking effect on February 1, 2027, as Bing crawls pages. Publishers still using the older NoArchive and NoCache tags must publish their AI Training preference in robots.txt or on pages by March 1, 2027; after that date those tags are no longer read as training or grounding preferences. Temporary grounding transition settings open in Bing Webmaster Tools on December 15, 2026 for eligible legacy users, and any temporary preference expires on September 30, 2027. Microsoft also separates AI Grounding, whether content can inform generated responses in Copilot, AI Mode, AI Overviews and Microsoft 365, from training, and adds a middle option, DisallowExtended, that allows grounding in Microsoft-operated experiences but not through grounding services sold to third parties.

Google says it "aim[s] to support this new control along with other AI providers in 2027, and will provide updates on our progress." Its crawler documentation is quoted by The Next Web as saying early 2027; neither page gives a month. Google's post adds an important mapping: when a site opts out with the shared token and does not mention Google-Extended separately, Google plans to treat that "as if the publisher had disallowed Google-Extended," its own training control offered since 2023. It also says sites that specifically allow Google-Extended remain eligible to appear in Gemini app responses and receive clicks.

OpenAI says it "aim[s] to implement the signal in the coming months" and will share implementation details later. Its post is explicit that the new signal "will not replace the controls we already provide," so its existing crawler controls, such as GPTBot, continue to apply. Anthropic, via Kelley, says support has started but "it'll take us some time to add support across our services and work through the inevitable complications and edge cases," and that interpretation may change as standards evolve; The Next Web quotes the same statement and describes the implementation as begun but not yet complete.

What the signal does not do

Related guide: A Frontier AI Standards Body Is Taking Shape: What the Labs Actually Agreed On

Reading the limits is more useful than reading the announcement headlines. The draft is an individual Internet-Draft, not an adopted IETF standard. The IETF Datatracker lists it as "Active Internet-Draft (individual)" with no RFC stream; the standard boilerplate says Internet-Drafts are working documents valid for a maximum of six months and "inappropriate to use ... as reference material or to cite them other than as 'work in progress.'" This one was published October 30, 2026 and expires 3 May 2027. The Next Web notes the Datatracker says the IETF has "not endorsed it and has no formal standing in the standards process." Bing's FAQ anticipates the syntax may change as the standard is finalised and promises advance notice.

It is a preference, not enforcement. Participation is voluntary: it binds only providers that choose to honour it, and scrapers that ignore robots.txt today will still ignore this. The draft says a consumer MAY use preferences to inform crawling, but does not require it. Microsoft states its preferences "do not establish licensing or payment arrangements" and that allowing training or grounding "does not guarantee inclusion, traffic, or payment." It also warns that choosing DisallowExtended or Disallow for grounding "will significantly reduce your content's visibility in AI-generated experiences," while leaving eligibility for ordinary Bing search results unchanged.

Finally, training and visibility are separate questions. Google's post is careful to say company-specific controls "will continue to function alongside this new joint signal," and Microsoft's FAQ says indexing and ranking in traditional search results is not grounding. A publisher that wants to stay out of training but stay citable in AI answers has to make two different decisions, and the shared token only addresses one of them. The Verge also reports that Meta communications head Andy Stone signalled support for a common signal without saying whether Meta would honour it, a reminder that four labs are not the whole industry.

What publishers should do now

Related guide: Genie 4: Google's Interactive World Model Now Generates Sound

The practical answer is to prepare without assuming protection. Keep the per-crawler rules you already have: OpenAI says its existing controls stay in force, Google says Google-Extended keeps working, and Bing's older tags are only retired in 2027. Those are the rules that are actually being read today.

Adding a __usage_ai-training group now is low-risk, because a compliant parser that does not recognise the token simply ignores the group, an explicit design goal of the draft. But adding it is not the same as being protected: no provider in these announcements claims to be enforcing it yet. If your Bing setup relies on NoArchive or NoCache, put the March 1, 2027 deadline on a calendar and read Microsoft's transition steps, since its FAQ says old restrictions do not carry over automatically. If you use neither, Microsoft says no transition action is needed.

Decide training and grounding separately, because they now have separate controls and different consequences. Opting out of training does not remove your pages from search, and opting out of grounding can reduce citations in AI answers. And watch for the implementation updates each provider promised: OpenAI says it will publish details, Google says it will post progress, and Microsoft has already published syntax that it says may change. Until a provider documents what its crawler actually reads and when, treat the shared token as a direction of travel rather than a switch you have flipped.

Bottom line: A single robots.txt policy token now has published backing from OpenAI, Google, Microsoft and Anthropic, which would eventually let a website state an AI-training preference once instead of per crawler. The specification is still a draft, the controls are not active, and the start dates range from the coming months to 2027. Credit the direction, keep your current rules, and wait for each provider's implementation notes before relying on it.

Prices and benchmark figures are standard published rates as of November 6, 2026 and can change without notice. Some outbound links may be affiliate links — see our disclosure. Found an error? Tell us and we'll correct it in public.