Google, Microsoft and AI giants agree to a shared website scraping rule, but webmasters still have no real control over how their content is used

Google and Microsoft among those signing up to OpenAI’s plan for a new website crawling standard — including opting-out of internet scraping.

Google and Microsoft have joined OpenAI and Anthropic in promising to honor a single line in robots.txt that tells AI developers whether a website’s content can be used to train their models.

Until now, a site that wanted to say no had to write a rule for each company’s crawler, and sometimes for several crawlers run by the same company. The shared signal is meant to replace that sprawl with a single entry that every participating provider reads the same way.

That is genuine progress for overworked webmasters, but the promise is narrower than much of the coverage suggests: The new line states a preference about how content may be used rather than stopping bots from fetching it; it is not an approved standard, does nothing about content already collected, and none of the four major players fully honor it yet.