Thinking of ACE? We Can Do It with Fewer Tokens

When an LLM-based agent falters during a complex, multi-step workflow—such as reconciling orders across a suite of simulated applications or navigating intricate API pagination—the failure rarely stems from a lack of raw intelligence. More often, the agent simply hasn't internalized the reliable, repeatable patterns required to execute those tasks flawlessly. It knows the tools, but it lacks the "muscle memory" of how to use them without error.

Two emerging systems, ACE (Agentic Context Engineering) and our own ALTK-Evolve, aim to solve this by turning an agent’s past trajectories into reusable lessons. Both frameworks function as a form of agentic memory, feeding historical successes and failures back into the model at inference time without requiring weight updates or human-labeled datasets. While they share a fundamental philosophy, their approaches to delivery create a massive divergence in operational costs.

The Philosophy of Non-Compression

Both ACE and ALTK-Evolve are built on a shared conviction: do not compress.

In the world of agentic memory, there is a constant temptation to summarize past experiences into a tidy, concise set of rules. However, both systems identify two primary failure modes that arise from such compression:

• Brevity Bias: The tendency for optimization to collapse toward short, generic, and ultimately unhelpful instructions.
• Context Collapse: The result of forcing a model to rewrite its entire context at every step, which inevitably leads to the loss of critical, granular detail.

ACE addresses this by maintaining a comprehensive, evolving playbook where every bullet point is tracked with a helpful/harmful counter. ALTK-Evolve arrives at the same conclusion from a different angle. We assign a "support count" to every distinct guideline, representing the number of independent episodes that generated it.

"A lesson five different tasks discovered is a different object from one that appeared once, and both are worth keeping. So on the core question—should you compress an agent's hard-won lessons into a tidy summary?—ACE and ALTK-Evolve give the same answer: no."

Divergent Paths: Building and Delivering Memory

While the two systems agree on the importance of keeping a rich, itemized record of experience, they part ways on how that memory is constructed and, more importantly, how it is delivered to the model.

Consolidation: How the Store is Built

ACE manages its playbook through a Generator → Reflector → Curator loop. It applies incremental delta updates and utilizes embedding-based de-duplication to keep the playbook organized.

In contrast, ALTK-Evolve clusters near-duplicate lessons and merges them within a cluster while strictly conserving support counts. When multiple lessons are merged, the survivor inherits the combined count of its predecessors. This allows our store to shrink in size without losing the vital record of how much experience backs each specific guideline. Furthermore, we categorize these into typed guidelines—strategy, recovery, and optimization—with full causal attribution back to the source trajectory. This ensures that a lesson learned in one application can be effectively transferred to another.

Delivery: The Token Bill

The most significant differentiator between the two systems is the delivery mechanism. ACE injects its entire, comprehensive playbook into the context window at every single step of the agent’s execution. It is a "one-size-fits-all" approach that remains constant regardless of the model’s capacity or the task’s complexity.

ALTK-Evolve treats delivery as a dial rather than a constant. We utilize a small, fixed core of high-support guidelines, which is then extended per task with a curated selection of relevant lessons. When a model possesses the computational headroom to handle it, we can scale up to the full consolidated set. By sending only what a given model can actually utilize, we drastically reduce the token overhead.

The Performance Breakdown

To measure the impact of these different strategies, we ran both systems in-house using the AppWorld benchmark, utilizing the same base ReAct agent. The results highlight a clear trade-off between raw context injection and intelligent, selective retrieval.

Performance Metrics (AppWorld)

| Model | System | TGC | SGC | Tokens/task | | :--- | :--- | :--- | :--- | :--- | | DeepSeek-V3.2 | ACE | 80.4 | 73.2 | 634K | | DeepSeek-V3.2 | ALTK-Evolve | 89.3 | 80.4 | 263K | | gpt-oss-120b | ACE | 54.8 | 35.7 | 777K | | gpt-oss-120b | ALTK-Evolve | 56.0 | 37.5 | 116K |

TGC: Task Goal Completion; SGC: Scenario Goal Completion.

On the stronger DeepSeek-V3.2 model, ALTK-Evolve outperforms ACE on both accuracy metrics while consuming only about 40% of the inference cost. On the weaker gpt-oss-120b model, we achieve a slight edge in accuracy at roughly one-seventh the token cost.

Why Selective Retrieval Wins

The by-difficulty breakdown reveals why these two systems behave differently across the capability spectrum.

On the gpt-oss-120b model, ACE’s full playbook actually performs well on "Easy" and "Medium" tasks. For these simpler problems, the model benefits from the sheer volume of generic instruction-following provided by the full prompt. However, on "Hard" tasks, the model struggles to navigate the noise of a massive playbook. Here, ALTK-Evolve’s curated retrieval pulls ahead, allowing the model to focus on the specific lessons required for the task at hand.

The story flips for the stronger DeepSeek-V3.2 model. Because this model has more capacity to spare, it can absorb the full playbook effectively. Even so, our method of delivering lessons—keeping the most relevant ones at the forefront—allows the agent to maintain higher accuracy without the crowding effect that often hampers performance when context windows are flooded.

Key Takeaways

The fundamental difference between these two approaches is the philosophy of calibration.

1. Efficiency through Calibration: ACE’s fixed-injection strategy is straightforward, but it forces the model to process a massive amount of data at every step. ALTK-Evolve’s calibrated delivery ensures that the model is only provided with the guidance it can effectively use. 2. Accuracy at Scale: By avoiding the "context overload" that occurs when weaker models are fed massive playbooks, we ensure that guidance actually helps the agent rather than getting in the way. 3. Cost-Effectiveness: The reduction in token usage is not merely a secondary benefit; it is the direct result of a more intelligent retrieval pipeline. Serving only the necessary guidelines per task significantly lowers the barrier to deploying agentic memory in production environments.

We believe that the future of agentic memory lies in this type of intelligent, task-aware delivery. Whether you are working with a massive, high-capacity model or a more constrained, efficient one, the goal should be to provide the right lesson at the right time—not just every lesson, all the time.

*

For those interested in exploring these methods, the ALTK-Evolve library is available, including the full extraction, consolidation, and retrieval pipeline used in these tests. You can also review our full technical report for a deeper dive into our methodology and ablations.

Method Notes

• Benchmark: AppWorld test_normal (168 tasks).
• Agent: ReAct code agent (Python-based environment).
• Evaluation: Results are single runs (pass@1).
• Comparison: ACE numbers were generated via our own in-house runs to ensure a controlled comparison against our base models and harness. Both systems utilized the same base ReAct agent, with differences limited to the prompt template.