🔍 Read the full analysis: How To Navigate The New AI Training Opt-Out From Four Companies on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The signal is voluntary, does not cover every use of web content, and will take effect on different schedules; Meta expressed support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared website preference asking them not to use content to train generative AI models. The proposal gives publishers one robots.txt rule to communicate that preference, but it is voluntary, does not stop all AI uses of a site, and will be implemented on different timelines.
The proposed rule uses the user-agent name __usage_ai-training with Disallow: / in a site’s robots.txt file. It appears in an Internet-Draft titled “Machine-Readable Training Preference Signal,” published by Kevin Kelley of Anthropic. The draft builds on work within the Internet Engineering Task Force’s AI Preferences working group, but it is not an adopted internet standard.
Company plans differ. Anthropic says it has started implementing support, though it expects rollout across its services to take time. OpenAI says it aims to adopt the signal “in the coming months.” Google says it aims to support it in early 2027; where a publisher has no separate Google-Extended rule, Google says it plans to treat the shared signal as a disallow for that crawler.
Microsoft plans to honor the training signal from February 1, 2027, as it crawls sites. It is also introducing a separate page-level preference for AI grounding—using web content to inform answers in Bing, Copilot and Microsoft 365. Microsoft says its existing NoArchive and NoCache tags will stop counting as preferences for training or grounding after March 1, 2027. Meta’s communications chief voiced support for a common signal but did not commit the company to honoring it.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
For website operators, the proposal could reduce the work of maintaining separate training opt-outs for each company. Instead of writing individual instructions for crawlers such as GPTBot, Google-Extended and ClaudeBot, publishers can add a shared rule for participating services. That makes the preference easier to express, especially for organizations managing many sites.
But the signal is a request, not a technical barrier. It does not prevent a crawler from accessing a page, guarantee that every company will comply, or remove material from models that have already been trained. It also does not automatically cover AI answers based on live web content. Microsoft’s separate grounding preference shows that companies may treat training and answer-generation as distinct uses requiring distinct controls.
The change does not itself create a payment arrangement or promise traffic, attribution or inclusion in AI products. Microsoft states that neither choice guarantees inclusion, traffic or payment. Publishers therefore need to decide what they want to permit across training, AI answer services and search, rather than treating one new line as a universal content switch.
Popular right now
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
Dyson V8 Cyclone Cordless Vacuum
TCL QM64L 85-Inch QD-Mini LED 4K Smart Fire TV
As an affiliate, we earn on qualifying purchases.
How the Opt-Out Proposal Emerged
The shared signal is set out in an individual Internet-Draft, a proposal circulated for discussion rather than a finalized standard. The draft was published on October 30 and is associated with the IETF’s AI Preferences working group. Its existence and the companies’ announcements establish a common proposal; they do not mean the standards process has approved it.
The working group’s status remains relevant. At a meeting on October 21, one of its chairs said, “We haven’t declared consensus on anything yet.” The draft’s author works at Anthropic, which has disclosed that it is implementing the proposal. The companies’ public statements and the draft provide the basis for their stated plans, while actual compliance will depend on each company’s rollout.
Before this proposal, publishers commonly relied on company-specific crawler instructions. The new rule is intended to provide a common training preference while leaving those existing controls in place. OpenAI says its current crawler controls will continue to apply. Google says Google-Extended does not affect Google Search, and Microsoft says its preferences do not change eligibility for Bing search results. Search access and AI training preferences remain separate questions.
““has already started implementing support for this draft””
— Anthropic, in its announcement of the draft
Adoption and Coverage Still Vary
It is not yet clear how many other AI companies will honor the signal or whether the proposal will gain formal standing in the IETF process. The source material reports no implementation commitment from Meta, and no statement from Apple, Amazon, xAI or Perplexity among the sources reviewed. Their absence from those reports does not establish what they may do later.
The announced dates are plans, not evidence that every service has already changed its behavior. Anthropic says rollout will take time, OpenAI has not supplied a calendar date, and Google and Microsoft point to 2027 schedules. The signal’s voluntary nature also leaves open how compliance will be monitored or what recourse a publisher would have if a company does not follow it.
Scope is another limitation. The shared token concerns model training; it does not by itself block crawling, erase previously learned material, cover every form of AI answer generation, or settle how content may be used under other policies. Publishers will need to check each company’s documentation as implementation details develop.
Publishers Should Track Rollouts
Site operators considering the preference can add the shared user-agent group and Disallow: / to robots.txt, while keeping existing company-specific rules until each provider confirms its transition. They should check that the rule is correctly placed and does not conflict with instructions they intend to retain. The shared rule is for website content, not for opting out of training on a person’s private conversations with an AI assistant.
Microsoft’s stated dates call for particular attention: its new training and grounding preferences take effect from February 1, 2027, while NoArchive and NoCache stop counting for those purposes after March 1, 2027. Publishers using those tags should review Microsoft’s guidance before that change. Google’s target is early 2027, and OpenAI says implementation is expected in the coming months; Anthropic says support is in progress.
The next milestones are company documentation updates and any further discussion in the IETF working group. Until those arrive, the practical position is one shared request, uneven adoption, and separate controls for separate uses. Publishers should treat the rule as an additional way to state a preference—not as a guarantee that their content will be excluded from every AI product.
Key Questions
What should a website add to robots.txt?
The proposed group is User-Agent: __usage_ai-training, followed by Disallow: /. It expresses a preference against using the site’s content for generative AI training by companies that honor the signal.
Does the rule block AI companies from crawling a website?
No. The source describes it as a training preference, not a technical block on crawling. It also does not remove content already used to train a model.
Does the signal stop a site’s content from appearing in AI answers?
Not necessarily. The shared rule concerns training and does not automatically cover AI answers based on live web content. Microsoft has announced a separate AI grounding preference for that use.
When will the four companies support the signal?
Anthropic says implementation has started; OpenAI aims for the coming months. Google targets early 2027, while Microsoft says its preference takes effect on February 1, 2027, as it crawls sites. These are company plans, not a single common launch date.
Will Meta honor the new preference?
Meta’s communications chief expressed support for publishers’ ability to state preferences, but the statement described in the source did not commit Meta to honoring the signal.
Source: ThorstenMeyerAI.com
