Anyone who runs a website and wants to keep it out of AI training has had a tedious job. Every AI company sends its own crawler under its own name, and each one needs its own line in the site's robots.txt file. Miss a new bot, and the rule has a gap.
That is about to get simpler. On October 30, Google, Microsoft, OpenAI and Anthropic said they will all respect one shared signal that asks them not to use a site's content to train generative AI models.
What actually changed?
The signal is a new entry for robots.txt, the plain text file websites already use to tell crawlers what they may fetch. Instead of naming a specific bot, the site addresses a "policy token" called __usage_ai-training and lists the parts of the site it wants kept out of training.
The idea comes from a draft submitted to the Internet Engineering Task Force (IETF), the body behind many of the internet's technical standards. It was written by Kevin Kelley, who works on standards at Anthropic. Google, OpenAI and Microsoft announced their support the same day.
The important shift is that the rule describes a purpose rather than a company. A crawler launched next year under a name nobody has heard of yet would be covered too, as long as its owner has agreed to honour the signal.
When does it start?
Not everywhere at once:
- Anthropic says it has already started implementing it.
- OpenAI says support will follow in the coming months, without a firm date.
- Microsoft plans to honour it from February 1, 2027.
- Google is aiming for early 2027. It will treat the new rule as a block on Google-Extended, its existing control for Gemini training.
What it does not do
The promise is narrower than the headlines suggest.
It covers training only. It does not stop a crawler from visiting a page, it does not affect search rankings, and it does not stop an AI assistant from reading a page live to answer a question. Google's AI Overviews and AI Mode have their own separate controls.
It is not retroactive. Content that went into a model before the rule was added stays there.
It is a request, not a lock. Like everything in robots.txt, it depends on companies choosing to follow it. Meta, Apple and Amazon have not committed, and scrapers that ignore robots.txt today will ignore this too.
It is not yet an official standard. The draft is an individual submission that the IETF has not endorsed.
There is no money attached either. Microsoft says the new preferences do not create licensing or payment deals, and warns that blocking its AI from drawing on a page will make that content far less visible in AI-generated answers.
Why this matters for a website owner
Consider a small publisher with a paid archive. Today, protecting that archive from AI training means tracking every new crawler and updating the file by hand. With the shared rule, a short group addressed to __usage_ai-training with Disallow: /archive/ could cover all four companies once they have switched it on.
One detail catches people out: an old catch-all rule for every crawler (User-agent: *) does not count. A site that has not added the new entry has expressed no preference at all.
The trade-off is visibility. A site that wants to appear in AI answers, and win clicks from them, may choose to keep public pages open while protecting premium content.
What to check before changing your robots.txt
Start by listing the AI crawler rules already in the file. They remain valid, so there is no reason to remove them while the new signal rolls out. Decide which sections genuinely need protecting, add the shared entry for those paths, and keep the bot-specific rules as a backstop until each company confirms support.
For content that would be costly to see copied, a paywall or login is still the only real lock. The new rule makes your wishes easier to state. It does not enforce them.
