AI News · 7 min read
How do you opt your website out of AI training with robots.txt?
Short answerOn October 30, 2026, OpenAI, Google, Microsoft and Anthropic said they will honor one shared robots.txt setting, __usage_ai-training, that tells AI companies not to train on your website's content. It is voluntary, it only covers training (not AI search answers), and most of the support arrives in 2027. For a small business, the job this week is to decide what you want and check your robots.txt file.
What happened this week?
On Friday, October 30, 2026, four of the biggest AI companies agreed to respect the same signal in robots.txt, the small text file that sits at the root of most websites and tells automated visitors what they may do. The signal is a single token, __usage_ai-training. A site that disallows it is asking AI developers not to use its pages to train their models (PPC Land, October 31, 2026).
The wording comes from an early-stage proposal at the Internet Engineering Task Force, the group that writes many web standards. It was written by Anthropic's Kevin Kelley and submitted the same day. It is not yet an official standard, and its draft status runs only until May 3, 2027 (AI Chat Daily, October 30, 2026).
The rollout is staggered. Google's own crawler documentation, updated October 30, says it intends to support the signal in early 2027 (Google Search Central). OpenAI says it will honor the signal "in the coming months" (Born City, October 30, 2026). Microsoft plans to start honoring new preferences on February 1, 2027 (PPC Land, above).
What does the shared opt-out actually do?
Think of it as a sign on your shop door. A "no photography" sign does not lock anything. Polite visitors read it and comply, and the sign is still useful because the big, reputable ones do. The robots.txt token works the same way: it is a request, not a lock. Google's Gary Illyes has been quoted as saying robots.txt is not a security control (PPC Land).
In practice, the draft uses the same simple rules robots.txt already has. A site would add a group that names the token and disallows everything, and it can carve out exceptions for specific folders. A group written for all crawlers with an asterisk does not count as a training preference, and a file with no group for the token states no preference at all (AI Chat Daily).
It also does not reach backward. Content already collected into existing datasets or models is not removed by a new line in your file.
Will opting out hurt my Google rankings or my AI visibility?
Training and search are different things, and mixing them up is the most common mistake. Google states that its existing Google-Extended setting does not affect inclusion in Google Search or act as a ranking signal (Google Search Central). Microsoft likewise says traditional Bing rankings are unaffected by the new settings (PPC Land).
The trade-off is more subtle. If you also block the controls that govern AI answers, you can show up less often in AI experiences, and Google's training token covers training only, so blocking answers means reading a second set of documentation (PPC Land). A business that wants to be recommended by AI assistants may reasonably allow search and answer crawlers while declining training. A business that sells original photos, courses or written expertise may choose the opposite.
Should a small business opt out?
There is no universal answer. Here is a simple way to decide.
- Opt out of training if your website's value is its original content: paid guides, a photo library, detailed pricing research, or members-only material.
- Stay open if your site mainly exists so new customers find you, and you want AI assistants to describe your services accurately.
- Split it if some folders are public marketing pages and others are premium. The format lets you allow one folder and block the rest.
Whichever you pick, keep it consistent with what your own tools do. If you paste customer data into AI tools, the bigger privacy question is inside your business, not on your public website. Our guide to AI chatbots for customer service covers how to keep customer conversations under control.
What should you do this week?
You do not need to rush, because most support begins in 2027. A short checklist is enough.
- Open your robots.txt. Type your domain followed by /robots.txt into a browser. If you see an error, you may not have one, which means you currently express no preference.
- Write down your goal. Training opt-out, visibility in AI answers, or both. A one-line answer is enough.
- Keep existing rules. The new token is meant to sit alongside bot-specific rules, not replace them, since companies that have not adopted it can ignore it (AI Chat Daily).
- Ask your web provider. Whoever manages your site can add the line in minutes. If you use Bing's NoArchive or NoCache tags, note that Microsoft's deadline for new preferences is March 1, 2027 (PPC Land).
- Add a reminder for early 2027. The draft can still change in wording and syntax before it is adopted, so re-check once Google and OpenAI publish final details.
If your website is out of date or hard to edit, this is a good prompt to fix that. Our web development service can handle small technical changes like this, and our AI integration services help when you want AI tools connected to your business data safely. For a broader plan, see which business tasks to automate first, and what AI costs a small business.
What is still uncertain?
Quite a lot. The proposal is a draft that could change. Google and OpenAI committed to support the signal, not to one fixed definition of what "training" means, and Google's own timeline is only "early 2027" (PPC Land). Critics also say disputes remain, including over how user-initiated actions are treated (AI Chat Daily). We will update this guide as final documentation appears.
The bottom line: this is a useful, low-cost control, not a guarantee. It gives honest crawlers a clear instruction, and it gives you a reason to look at a file most owners never open.
Frequently asked questions
What is the robots.txt AI training token?
It is a shared setting, written as __usage_ai-training, that lets a website say it does not want its content used to train AI models. OpenAI, Google, Microsoft and Anthropic said on October 30, 2026 that they will honor it, with most support arriving in 2027.
Does blocking AI training remove my site from Google Search?
No. Google says its Google-Extended setting does not affect inclusion in Google Search or act as a ranking signal. The shared training token is meant to control training only. Rules for AI answers are separate.
Can robots.txt stop every AI company from using my content?
No. It is a request that reputable companies choose to follow, not a security control. It also does not remove content that was already collected. Sensitive material should sit behind a login.
Sources
- PPC Land — ppc.land
- AI Chat Daily — www.aichatdaily.com
- Google Search Central — developers.google.com
- Born City — borncity.com
Written and reviewed by the INTELVYRA team. Prices and figures were checked on the date above and can change. Always confirm current pricing with each provider.