Google and Microsoft have joined OpenAI and Anthropic in promising to honor a single line in robots.txt that tells AI developers whether a website’s content can be used to train their models.
Until now, a site that wanted to say no had to write a rule for each company’s crawler, and sometimes for several crawlers run by the same company. The shared signal is meant to replace that sprawl with a single entry that every participating provider reads the same way.
That is genuine progress for overworked webmasters, but the promise is narrower than much of the coverage suggests: The new line states a preference about how content may be used rather than stopping bots from fetching it; it is not an approved standard, does nothing about content already collected, and none of the four major players fully honor it yet.
If you got something out of this, please chip in to keep this site running, or subscribe to go ad-free.
46 views