← Blog

Law & security

Your new AI opt-out cannot erase what was already scraped

Hand und rote Markierung auf Website-Unterlagen neben einem geschlossenen Laptop in einem kleinen Büro.

You have written an expert article, published a how-to guide or explained your services on your website. Now a robots.txt entry is meant to tell AI providers not to use that content for training. It sounds like a simple switch. On 30 October 2026, OpenAI announced it would respect a shared signal. Microsoft, however, plans to take the preference into account only from February 2027, and to do so gradually as Bing crawls sites again. So what does the entry actually do, and what can it no longer change?

One entry, four providers, no instant protection

The first benefit is less administration. Until now, website owners have had to tell different AI providers about their preferences using different rules. A shared signal in robots.txt is intended to let you communicate the same decision to several participating companies. OpenAI said it would respect the signal. According to a report, Google, Microsoft and Anthropic are also among the providers planning to recognise it.

For a small business, that makes things easier. If you run a garage, a consultancy or an online shop, you are unlikely to have someone on hand to manage every technical change straight away. A shared rule could reduce the number of separate adjustments you need to make. It does not yet tell you when each provider will take it into account, or which uses it covers.

OpenAI did not give a precise launch date, saying it aimed to implement the signal over the coming months. The shared initiative is also still a technical draft submitted to the IETF, not an approved standard. If you change the file today, do not assume every service will behave differently tomorrow.

What the robots.txt rule actually says

The first entry names the group the rule is meant to apply to. The second means every page in this example. The exact format matters. An entry in the wrong place or with a typo may fail to communicate what you intend. If an agency or external provider manages your website, ask them to check the change.

The key point is that this is a preference. The signal is not, according to its description, a command that automatically stops a crawler. It tells providers how you would like your content to be used, and participating providers have said they will take that into account. OpenAI also stresses that the new signal does not replace its existing robots.txt controls for its own crawlers. One rule will not automatically cover every setting already in place.

  • `User-Agent: __usage_ai-training`
  • `Disallow: /`

What has already been collected remains an open question

A new entry cannot turn back the clock. If providers have already crawled your content and used it for training, a signal added later will not remove it from a model retroactively. Inc’s article explicitly notes that the shared signal does not solve the problem of scraping that has already happened.

This is particularly tangible for small businesses. Perhaps a detailed guide draws on your own experience. Or you have published case studies on your website that represent many hours of unpaid work. A rule can state your preference for future use. It cannot tell you which older content has already been processed or how to remove it from that processing.

That is why this is worth treating as more than a technical question. Which pages should remain open to as many people and search services as possible? Which content would you rather not make available for training? This is a business decision about knowledge, visibility and control. Make it before an external provider applies a blanket block.

Training and answers are two different uses

If you are thinking only about AI training, you may be looking at the wrong setting. During training, content contributes to a model’s parameters, or what it learns from examples. With grounding, a system retrieves external information to produce a specific answer. That is a different use, even though both involve AI.

Microsoft treats these preferences separately. The training rule covers learning from content. Grounding can affect whether a page is used in answers from Bing, Copilot or Microsoft 365. For grounding, Microsoft lists the options `Allow`, `DisallowExtended` and `Disallow`. Which one suits your website depends on whether you want that kind of visibility through references or AI-generated answers.

For example, a small consultancy might not want its own copy used to train models, while still wanting a search engine to point to its current services page. There is no contradiction. These are separate decisions, and you should consider them separately before restricting the reach of your content.

Microsoft’s timetable makes crawling matter

Even a correctly added rule may not take effect straight away. Microsoft names 1 February 2027 as the date from which it plans to take the training preference into account. It will roll this out gradually as Bing crawls each website again. That means your change has to be picked up on the next crawl before it can be recognised.

The report says Google plans to implement the signal in 2027 alongside Google-Extended and its own priority rules. OpenAI, meanwhile, announced plans to implement it over the coming months, without giving a precise date. These different timetables show why a shared signal can bring the wording into line without giving every provider the same start date.

Microsoft also names 1 March 2027 as a transition deadline for website owners currently using `NoArchive` or `NoCache` to restrict training and grounding. If your website includes these rules, ask the person responsible to check how they work alongside the new options. A rushed change could unintentionally alter an existing setting.

What this means for your business

The sensible first step is to take stock, not add a rule blindly. Ask whoever manages your website for the current robots.txt file and a short explanation of its existing rules. Check whether it already contains instructions for individual crawlers, and whether you want to distinguish between training and discoverability.

You can then make a decision for each area of your site. A public contact form, a current product page and a detailed expert article all serve different purposes. A blanket block across the whole website may be technically simple, but it could also affect content you still want available to answer engines or search services.

Write down who will change the rule and when it will be reviewed. After the change, check that the file is still accessible and that the entries look as intended. Then ask your website provider how often search engines and other crawlers revisit your pages. Without that final step, you will not know when providers are likely to recognise your preference.

Next Monday, this could be very straightforward. You do not need to open a new AI console. Instead, go through the existing file with your agency. You will then know which content you want to protect and which visibility you want to retain. The important decision comes before the entry, not after an unexpected change.

  • Ask to see the current robots.txt file.
  • Make separate decisions about training and grounding.
  • Find out when participating providers are likely to crawl your site again.

Frequently asked questions

How can I stop AI providers using my website for training?

You can add the shared training preference to your robots.txt file. The draft gives `User-Agent: __usage_ai-training` and `Disallow: /` as the entries to use if you want the rule to apply to every page. The signal tells participating providers your preference. It is not a technical command that immediately stops every crawler. Ask someone to check that the rule is in the right place and works with your existing settings.

Does robots.txt stop AI crawlers immediately?

No. A robots.txt entry does not guarantee an immediate effect. A crawler first has to recognise the change when it revisits your website. Microsoft plans to take the training preference into account gradually from 1 February 2027 as Bing crawls sites again. Providers will not all implement the shared signal at the same time either.

Does an AI opt-out remove website content that has already been used?

No. The signal states your preference for how your content is used. It does not apply retroactively to scraping or training that has already taken place, and it cannot guarantee that older content will be removed from existing models. It is primarily a rule for how participating providers should take your preference into account in future.

What is the difference between AI training and grounding?

During training, content contributes to what a model learns from examples. With grounding, an AI system retrieves external information to answer a specific question. Microsoft treats these uses separately. You can therefore object to your website being used for training while also deciding whether you want its pages to appear in answers from Bing, Copilot or Microsoft 365.

Is the shared AI signal already an official IETF standard?

No. According to the information available, the signal has been submitted to the IETF as an Internet-Draft. It is a technical draft, not an approved IETF standard. OpenAI and other providers have said they plan to respect the signal, but their implementation plans and start dates differ.

Interested?

Want to know what this means for your company?

Leave your details. We will get back to you personally, no sales pressure and no newsletter sign-up.

Sources

  1. Making it easier for websites to express AI training preferences · openai.com, 2026-10-30
  2. The 4 AI Giants Just Agreed to Let Websites Opt Out. They’ve Already Taken the Data · inc.com, 2026-10-31
  3. AI, una sola regola per opporsi al training: la proposta delle big tech agli editori · ai4business.it, 2026-10-31

This article was researched and written with AI assistance and automatically checked against the sources listed before publication. More on our AI transparency

Free guide · PDF

The AI Compass for small and mid-sized businesses

9 jobs your team can hand over by Monday.

Get it free

Klarframe AI Readiness Check

How ready is your company for AI?

About ten minutes, instant result with your stage, a 90-day plan, suitable solutions and notes on the EU AI Act.

Start the check

More articles