Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

As reported by TechRadar · Updated 8 Nov, 17:05 · Archive

One robots.txt line could soon tell Google, Microsoft, OpenAI, and Anthropic not to train on your site, but it won't stop their bots from crawling it.

Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping — Image: TechRadar

Comments (0)

Read the report

Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scrapingOne robots.txt line could soon tell Google, Microsoft, OpenAI, and Anthropic not to train on your site, but it won't stop their bots from crawling it.TechRadar · 8 Nov, 17:05