OpenAI, Google, Microsoft and Anthropic say they will restrict their AI crawlers from sites carrying a shared publisher signal, first reported by The Verge. That would give publishers one place to signal whether AI companies can use their sites for training, replacing the separate company-by-company settings they currently need to manage for each provider.
The proposed “policy token” would sit in a website’s robots.txt file, which has long told web crawlers whether they may scrape a site for search indexing, archiving or AI training. Anthropic technical lead for standards Kevin Kelley proposed the signal as part of the Internet Engineering Task Force (IETF).
Publishers currently have to signal AI-training preferences separately to each company. The four firms say they will restrict their AI crawlers from accessing a site if they encounter the shared token, consolidating those separate settings into one signal.
The proposal arrives as publishers face a continuing influx of AI crawlers scraping their sites and feeding content into models. Other efforts include the Really Simple Licensing (RSL) standard, which lets publishers charge crawlers that visit their sites, and Cloudflare’s default blocking of AI crawlers.
Microsoft says it will offer granular controls to opt out of AI training or AI grounding, the process of pointing an AI model to a verified source to reduce hallucinations. Microsoft says websites will still appear in Bing search results regardless of those preferences; Google has similar controls for AI search features and training that do not affect search visibility. Those examples separate a site’s AI-use preferences from its presence in search results.
Meta communications head Andy Stone also signaled support for a common setting, but did not say whether Meta would honor it. Anthropic has already adopted the signal; Google and Microsoft plan to support it next year, while OpenAI says it will implement it in the coming months.