Study contradicts Anthropic and OpenAI claims that

What happened

Solid evidence for claims about automated AI research has been mostly absent, the authors argue. " An agent receives the core research question from an unpublished paper. The original authors, who spent months on the same question, then evaluate the result as conference reviewers would. Since the results aren't on the web yet, the agent can't fall back on training data.

One paper contained zero visualizations in the main text, while the human-written original had 15 figures. Meta AI recently described a closely related phenomenon called "behavioral state decay", where an agent recognizes a requirement early on but violates it later while fixing an unrelated bug. The agents managed all engineering work without human help.

They ran literature searches, debugged GPU code, completed hundreds of experiments and robustness tests, and compiled full papers in LaTeX. Only three human interventions were needed: a scaffold bug fix, a deadline extension, and a request to rewrite for readability. The researchers found no significant reward hacking. The agents didn't manipulate results or skew data to chase better scores.

In fact, they did the opposite, starting with ambitious claims and correcting them toward negative results. 6 Sol and OpenAI's Codex scaffold. Nearly all the same failure modes showed up. 6 burned through the $3,000 budget in just over two days, leading to undersized experiments. The findings contrast with claims from leading AI labs.

Why it matters

The team partnered with the authors of two NeurIPS 2026 submissions. The first paper examines how personality traits of language models can be steered through their weights. The second develops a method called TabPFN that detects when a tabular prediction model hits deployment data that differs sharply from its training data and tanks its accuracy. 8 with Extra-High Reasoning.

Each agent got six days, $3,000 in API credits, a GPU budget, and full access to a virtual machine and the open web. The agent ran inside a scaffold, a software environment that orchestrates model calls and provides tools, letting the model delegate to subagents and monitor its own resource usage. It could also consult external AI review tools.

The researchers used OpenClaw, an open-source, vendor-neutral agent framework built by Austrian developer Peter Steinberger, who joined OpenAI earlier this year. The core agent launches subagents and long-running GPU jobs, with an automatic heartbeat from the scaffold waking it up when jobs finish so it can collect results. The original authors reviewed the finished papers as conference reviewers and rejected both.

" They criticized poorly motivated data and experiments, unreadable prose, and no new contributions. " Analysis of the agent logs reveals systematic weaknesses. The agents lacked judgment about what meets the bar for publishable research, generating plausible hypotheses but discarding them based on small, hand-curated, or synthetic datasets. They also failed at creative problem-solving. When initial hypotheses were falsified, the agents narrowed existing claims rather than pursuing new directions.

Their internal AI reviews never returned a single Accept across fifteen revision rounds, yet the agents never addressed the core criticism. The agents couldn't backtrack effectively either. Both gave up their most ambitious research goals within the first ten hours. In the Personas run, the agent finished its exploration after just five hours despite budgeting 36 to 48 hours for that phase. Resource awareness was poor too.

Both runs ended with less than half the API budget spent. One agent declared the project complete seven hours before the deadline, shortly after its own reviewer had returned yet another Reject. The agents also suffered from instruction drift, gradually forgetting explicit instructions over a long context. Both papers exceeded length limits and would have been desk-rejected at NeurIPS.

What to watch

In June, Anthropic published a post titled "When AI Builds Itself", sharing internal data on its own research acceleration and floating the idea of a globally coordinated development pause. 6 Sol helped with post-training a smaller model and saved researchers several weeks. The study's authors note this contribution isn't even mentioned in the 81-page system card. More reasoning effort did improve quality, as preliminary tests without reasoning showed.

But the researchers suspect more time or compute wouldn't change the results much. The reviewer's objections targeted the quality of experiment choices, not the quantity.