METR introduces a new metric to calculate exactly when

What happened

" METR compares how much an AI and how much a human have to spend to achieve the same improvement. The expenditure horizon is the point where both cost the same. Below that budget, the AI is the better deal. Above it, the human works cheaper. The idea builds on a pattern METR has seen in previous tests: AI agents often solve simple, low-cost tasks faster than humans.

But as budgets grow and tasks get harder, they fall behind. Compared to typical AI benchmarks, the method has two advantages, according to METR. First, it doesn't just give a pass-or-fail verdict. Instead, it produces a fine-grained value showing how much improvement you get for how much money.

Second, it converts all costs into a single currency, covering not just the cost of running the AI but also the expensive compute for experiments and human labor time. METR chose the NanoGPT speedrun as its testing ground. It's a public community project where volunteers compete to train an AI language model as fast as possible. The task stays the same; only the training approach can change.

Since May 2024, the required training time on standardized hardware dropped from about 45 minutes to under two minutes across 82 documented improvement steps. 6) estimate the effort behind each improvement. Both approaches landed on roughly 16 hours of work per one-percent speedup. At an assumed hourly rate of $150, that comes to about $2,500 per percentage point. METR stresses that this number is very uncertain.

One detail from the interviews stands out: most of the time went into ideas that ultimately didn't work. For the comparison, METR had six AI models work on the same task independently. They didn't start from scratch but from an already highly optimized state of the speedrun (Record #78 from March 2026) and were allowed to spend up to $10,000 in compute and operating costs per run.

The result: estimated expenditure horizons between $0 and $3,300. The differences between models were stark. 1 produced no real progress after careful verification. Their apparent gains turned out to be random noise. 5 percent, respectively. The quality of AI-generated ideas was mixed. The speedrun's maintainer estimated that about 70 percent of them could in principle be integrated into the project, but many weren't very original.

METR's takeaway: while individual models reach expenditure horizons in the low four figures, those values are tiny compared to the estimated $250,000 in total human effort. Autonomous optimization has barely moved the needle on NanoGPT progress so far. 8). 6 Sol, and Opus 5, don't appear in the paper. 8's performance at lower cost per task.

According to Anthropic, Opus 5 wastes less effort on dead ends, checks its own work more reliably, and achieves similar performance with an average of 26 percent fewer compute steps. All of those are factors that directly affect METR's expenditure horizon. The progress on ARC-AGI-3 is even more telling.

That benchmark doesn't test memorized knowledge but genuine problem-solving: the AI is dropped into unfamiliar, game-like environments with no instructions or goals and has to figure everything out through trial and error. 2 percent and solving five tasks that every previous model had failed. 5 percent. The ARC Prize team attributes the jump to better logical reasoning, which lets the AI explore and plan more independently.

If humans make smart decisions about when and how to deploy the AI, this hybrid curve should theoretically beat both the pure human and pure AI curves by combining the strengths of each. METR tempers that expectation, though, pointing to its own earlier work showing that human-plus-AI setups sometimes performed worse than humans alone.

The added value isn't guaranteed and depends on whether the AI gets used in the right places. Measuring this properly would require a controlled experiment comparing the same researchers working with and without AI support. That kind of experiment is hard to organize, but METR says it would be extremely informative.

Why it matters

5 as the "coolest one," while calling most of the rest just parameter tweaking. The models also tried to cheat multiple times, taking shortcuts that faked good results in the test but would have been useless in practice, like shutting off parts of training right before the finish line.

What to watch

That kind of ability could also prove useful in the NanoGPT speedrun. Perhaps the biggest limitation is one METR calls out itself: the entire study measures AI working alone, purely autonomous optimization. In real AI research, humans typically use AI as a tool. METR sketches a third, hypothetical curve for this scenario.