AI firms back new standard to let publishers block scraping

Summarised from 2 outlets · Updated 31 Oct, 10:51 · Archive

OpenAI, Google, Microsoft and Anthropic will adopt a new robots.txt standard letting publishers block AI training scraping across all four companies at once.

OpenAI, Google, Microsoft and Anthropic have committed to adopting a new robots.txt standard that will let publishers block scraping for AI training across all four companies at once, rather than setting preferences individually with each provider.

The proposal, written by Anthropic technical lead Kevin Kelley for the Internet Engineering Task Force, adds a new user-agent directive to websites' robots.txt files. Publishers can use this to prevent AI crawlers from gathering their content for training data, whilst still allowing search indexing and other uses. The directive also allows publishers to opt out of scraping on their whole site or specific sections such as a press kit.

Anthropologic has already started supporting the new standard. Microsoft said it will honour the directive from February 1, 2027. Meta's communications head Andy Stone signalled support for the approach but stopped short of confirming the company will implement it. The announcement comes as publishers face increasing volumes of AI scraping; some have attempted to counter it through licensing standards or blocking tools.

How it is being reported

• Google joins OpenAI and Anthropic in promising new scraping controlsGoogle, OpenAI, Anthropic and Microsoft pledge to support a new common signal for robots.txt that blocks scraping for AI training.Engadget · 31 Oct, 10:34
• The biggest names in AI are backing a new way to opt out of scrapingOpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually. The shared "policy token" signal […]The Verge · 30 Oct, 20:26

In this story: OpenAI · Google · Microsoft · Anthropic · Kevin Kelley · Internet Engineering Task Force · Meta

This summary was written by AI from the headlines and standfirsts above, and states as fact only what at least two outlets report. How we use AI · Report a problem · Corrections log