How Much Memory Does Your Agent Actually Need?

What happened

In our previous post, we compared ALTK-Evolve with ACE and showed that how you deliver an agent's self-distilled guidelines — a few retrieved per task vs. the whole set injected — drives both accuracy and cost. This post steps back to the question that comes before it: how much should you give it?

The number of guidelines a model mines depends on its own capability, so we report configurations by strategy — "full guideline set" vs. "curated retrieval" — rather than by raw counts, which aren't comparable across models. Representative models from the eight-model sweep, measured by task completion (TGC) on test_normal: Figure 1. Representative models in the three observed patterns. Bars show TGC on AppWorld test_normal for baseline vs.

the best-memory configuration; the x-axis begins at 40% to make differences visible. TGC alone understates the larger SGC gains — see the SGC columns in the table below. 5pp TGC gain — because good guidelines especially help an agent clear every variant of a scenario, not just the average case. 1pp SGC respectively. Memory keeps paying off as long as a model has a remaining failure mode to target.

A practical concern: injecting a full guideline set inflates every ReAct step's input, because the guidelines are re-sent each turn. Here's what we observed: Table 1. Average token use per task, accumulated across agent steps, measured against the no-memory baseline. The real efficiency lever in production is prompt caching: the static portion of the guideline set is identical across steps and can be cached, cutting effective cost substantially.

Why it matters

Equipping an agent with agentic memory sounds simple: distill lessons from its past work, put them back in context, and more experience should mean better performance. It doesn't always work that way. When we scaled the evaluation to eight models — from a 30B dense model to frontier proprietary systems — one finding stood out: Not every model benefits from the same amount of memory.

Across eight models spanning the capability spectrum, we saw three recurring patterns: What puts a model into one pattern rather than another isn't simply parameter count. Benchmark headroom, context-window size, architecture, guideline quality, and task distribution all appear to shape where a model lands, and separating those factors is ongoing work.

The practical takeaway holds either way: the right dose of memory depends on the model, and we can calibrate it. "Memory" here doesn't mean replaying a past transcript. It means a guideline set — strategies that worked, mistakes to avoid, and edge cases — distilled from the agent's own prior trajectories. The loop is straightforward: No model weights are updated.

The learning loop changes the guidance available to the agent, not the underlying model — which is exactly why it's cheap to adopt and portable across the eight models we tested. We evaluated on AppWorld — 585 multi-step tasks (168 test_normal + 417 test_challenge) across 9 simulated apps (calendars, messaging, payments, and so on).

Tasks are scored two ways: whether the agent fully completes each task (TGC — Task Goal Completion) and whether every variant of a scenario passes (SGC — Scenario Goal Completion, a stricter, all-or-nothing bar). Full definitions are in the appendix. Because the confusing part of any memory study is what's actually in the context window, we define the configurations up front.

Both memory configurations draw from the same guideline set, mined once (via the loop above) from AppWorld's training split only. What changes between them is only how that one set is delivered — the full guideline set injects all of it every step, while curated retrieval delivers a selected subset — never how the guidelines were produced, and no test-split data ever goes into building it.

What to watch

Cache-aware prompt design — keeping the shared guideline-set prefix stable so it stays cacheable — is worth engineering for. We also hypothesize that context-window size plays a role: models with larger windows may absorb the full guideline set more effectively, while smaller-context models benefit more from retrieval that keeps injected content compact. We have not yet run controlled experiments isolating this factor.

The lesson isn't to give an agent everything it has learned. It's to give it the amount of experience it can actually use. The gains are real across the board — automatic, leakage-free, and requiring no human annotation — but only when the dose fits the model. This is a starting point, not the finish line: AppWorld tasks are graded by two metrics, both reported as percentages (higher is better):