Anthropic’s models are improving roughly twice as quickly as a year ago, according to an 11-week METR investigation first reported by The Verge. The researchers attributed most of the acceleration as of October to AI, while Anthropic’s extensive redactions left the public with less evidence to judge the findings and their uncertainties.
METR’s benchmark results suggested Anthropic had made roughly two years of historical progress over the last year. Researchers said they were confident the acceleration was not simply the result of scaling training compute, historically a major driver of AI progress.
Informal Slack polls and employee interviews supported the assessment. Employees using Anthropic’s most capable internal model self-reported being, on average, around 12 times as productive at their respective work as they would have been without AI assistance. That was around double what staff had reported a few months earlier while using Mythos 5.
Much of that acceleration appeared to come from AI assisting human researchers rather than AI working independently, METR said. Researchers found no clear signs of diminishing returns or observable bottlenecks that could prevent even faster acceleration; they also noted signs that fully autonomous contributions could play a larger role soon.
Anthropic completed training three unreleased models over the past six months: Model 2 in July, Model 3 in August-September, and Model 4 in September-October. METR also identified an unreleased model dating to 2022 called Unreleased 2022.
Anthropic shared confidential research, model-performance data and employee interviews with METR, but did not grant unrestricted access. Anthropic redacted a majority of the evidence METR considered most informative, and a small number of data requests were denied.
The assessment puts benchmark progress and employees’ self-reported productivity figures alongside a public account missing most of the evidence METR considered most informative. Anthropic cofounder Sam McCandlish said the redactions were meant to protect commercially sensitive information and avoid giving rivals a roadmap to move faster. He said Anthropic plans to publish its own report on AI progress in the coming weeks, along with new safety steps.