How enabling two settings tripled our scores on the

What happened

6 Sol has solved longstanding open problems in mathematics like the cycle double cover conjecture⁠(opens in a new window) and beaten games like Pokémon FireRed. 4%. Were 2D puzzle games unusually difficult for our models? Or was something else going on? Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting.

The next improvement came from replacing rolling truncation with compaction⁠(opens in a new window), another setting in the Responses API. The ARC-AGI-3 harness addresses context limits with rolling truncation. When the conversation context exceeds 175,000 characters, the oldest messages are discarded. Rolling truncation has two drawbacks. First, the model loses earlier observations and actions.

Second, it spends much of the tasks operating with a fuller context window, which can slightly impair performance. 6 Sol was better able to preserve what it had learned about each game across longer runs, and achieved a higher score with fewer output tokens. 6 Sol’s 175K context window as it solves a series of ARC-AGI-3 puzzles with each harness.

6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens. We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting.

This isn’t the first time we’ve been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages.

Why it matters

In the case of ARC-AGI-3, we discovered that turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set. ARC-AGI-3 is a benchmark designed to measure how well AI agents learn and reason. Agents explore unfamiliar 2D games and infer how they work without explicit instructions. org/tasks⁠(opens in a new window).

ARC-AGI-3 uses an intentionally generic harness, without tools or special features. ARC’s reasoning was that a simple harness makes model shortcomings more visible and makes model comparisons more fair. Commercial developers, by contrast, optimize harnesses for each model’s features and quirks. But as we looked deeper, we discovered much of the model’s confusion was not inherent to the model itself, but due to settings in the harness.

First, we noticed that after each game action, all private reasoning was discarded. 6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.

Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. 6 Sol unable to remember its past thinking, it was losing memory of its past actions too. 6 Sol was struggling to learn over time. Our models are trained to think with private reasoning messages before they output replies or tool calls.

These private thinking messages are retained as part of the conversation history. If a conversation grows too long, we summarize it and continue. This is how our models are trained, and also how they are deployed in ChatGPT and Codex. To better match our production setup, we implemented the ARC-AGI-3 harness with our Responses API⁠(opens in a new window).

6, passing the previous response ID automatically retains reasoning across tool calls and turns. With reasoning retained, we noticed two big changes. 6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. 6 Sol was much better at learning over time and employing coherent strategies.

What to watch

If you’re an API developer trying to maximize performance, we recommend using the same settings that we deploy in our own products: And if you’re comparing models, we recommend relying on evals that use the settings above, which best match real-world use in ChatGPT and Codex.

We are grateful to ARC for their years of creative work on AGI evaluation, and for their analysis that inspired us to take a closer look here.