Are We Ready For An Agent-Native Memory System? | Summary

This article explains the key points of Are We Ready For An Agent-Native Memory System?.

• 2026-06-23 (arXiv)
• Zhou, Wei, Zhou, Xuanhe, Han, Shaokun, Xu, Hongming, Li, Guoliang, Li, Zhiyu, Xiong, Feiyu, Wu, Fan.
• Shanghai Jiao Tong University, Tsinghua University, MemTensor (Shanghai) Technology Co., Ltd
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Are We Ready For An Agent-Native Memory System? examines agent memory as persistent data-management infrastructure rather than a retrieval component judged only by final answer quality. The central problem is that end-to-end metrics conceal failures in evidence preservation, temporal updates, retrieval, and operational efficiency.
• The paper decomposes memory systems into representation and storage, extraction, retrieval and routing, and maintenance. It reports comparisons of 12 representative systems and two reference baselines across five benchmark workloads spanning 11 datasets, supplemented by controlled component variants; experimental coverage varies across workloads.
• The results favor workload-specific architecture selection: linked or hierarchical memory helps retrieve dispersed evidence, while aggressive extraction and summarization can discard details needed later. Systems using localized maintenance offer favorable utility–latency trade-offs in the reported runs, but inconsistencies between some plotted results and narrative claims limit confidence in particular rankings.

3.1 Memory Representation and Storage

The analytical framework formalizes an agent memory system as \(M_{sys}=\langle R,S,Q,U\rangle\), separating the memory lifecycle into four modules. Representation and storage distinguish the logical organization exposed to retrieval and reasoning from physical persistence and indexing.

• The preliminaries distinguish short-term from long-term memory and identify episodic, semantic, procedural, and preference information. Unlike the paper’s characterization of conventional RAG as read-only retrieval over a static corpus, agent memory manages persistent, revisable agent-specific state.
• Agent workloads combine semantic access, uncertain or conflicting observations, and heterogeneous tasks such as episodic recall, temporal reasoning, and procedural execution. Context engineering curates an inference window; memory infrastructure governs state across windows and sessions. For Context engineering, see Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models | Summary.
• Figures 1 and 2 introduce execution workflows and representation choices; Table 1 maps representative systems to combinations of all four modules.

The workflows expose different stages at which systems extract, retrieve, and maintain memory. Streaming reflection, tier movement, temporal graph processing, and hybrid storage routing explain why an end-to-end answer score cannot identify which module caused a failure.

Flat text and latent vectors encode content without explicit relationships, while graphs and trees expose links or abstraction levels. The composite memory object shows that metadata, text, embeddings, and graph structure can coexist rather than requiring a single representation.

3.1.1 Logical Representation.

Logical representations fall into token-level sequences, graph and tree-based topologies, and heterogeneous composites. These choices determine whether memory exposes readable facts, latent vectors, explicit relationships, hierarchical summaries, or coordinated views of the same information.

• Token-level memory includes discrete statements and continuous representations such as embeddings or KV-cache tensors. These support text processing or semantic access but do not themselves expose explicit relationships for predicate-level filtering and targeted updates.
• Zep models temporal knowledge graphs, while MemTree places granular observations at leaves and broader summaries at ancestor nodes.
• Composite objects combine text, metadata, embeddings, and links; MemOS’s MemCube includes plain-text, activation, and parametric memory payloads.

The taxonomy maps individual systems to module combinations instead of treating broad architecture labels as complete explanations. It includes systems beyond those appearing in every benchmark, so taxonomy membership does not imply full experimental coverage.

3.1.2 Physical Storage and Indexing.

Physical storage ranges from transient in-context registers to specialized single-engine stores and heterogeneous multi-engine backends. Figure 3 separates these deployment choices, which affect capacity, persistence, access paths, and coordination overhead independently of the logical representation.

• MemoChat keeps structured memos in the LLM input context, whereas external stores can use files, relational databases, vector engines, or graph databases.
• SimpleMem uses LanceDB with dense embeddings, sparse BM25 indices, and SQL predicates; MemoryOS combines dense cosine similarity with discrete Jaccard similarity.
• Multiple indexing views expand retrieval options but introduce synchronization and routing work whose cost must be measured.

The diagram separates operation without an external store from a single specialized backend and a routed collection of heterogeneous stores. This distinction identifies where persistence and cross-store coordination enter the execution path.

3.2 Memory Extraction

Memory extraction transforms dialogue and tool traces into the units that will be stored. Figure 4 distinguishes raw sequence concatenation, schema-free semantic extraction, and schema-constrained structured extraction.

• Raw concatenation avoids a separate parsing stage and retains interaction sequences or transient state summaries.
• Schema-free extraction distills standalone facts or latent representations, as in Mem0’s natural-language factual statements. Greater selectivity can also remove context needed for later questions.
• Schema-constrained extraction produces predefined JSON fields or typed entity–relation triplets; Zep additionally applies verification intended to suppress hallucinated triplets.

The same conversational information can remain as raw turns, become a free-form fact or vector, or be parsed into a structured record and topology. Each transformation changes the evidence available for later queries.

3.3 Memory Retrieval and Query Routing

Retrieval and routing cover the complete path from a query to historical evidence. Figure 5 organizes the alternatives into native attention, dense semantic retrieval, topological traversal, autonomous agentic routing, and multi-stage hybrid execution.

• Dense retrieval selects nearby embeddings, while graph traversal gathers connected facts; A-MEM combines dense anchor selection with traversal of linked notes.
• Agentic routing includes explicit function calls, such as archival_storage.search(), and generated query expansions, as in SimpleMem.
• Sequential hybrid routing filters before semantic ranking; parallel ensembles combine candidate sets from vector search, BM25, and graph traversal before fusion or reranking.

The retrieval paths range from implicit attention to explicit planning and multi-engine execution. The sequential pipeline filters candidates before ranking, whereas the parallel pipeline gathers complementary candidates before fusion, illustrating distinct ways to manage evidence coverage and execution cost.

3.4 Memory Maintenance

Maintenance governs revised facts, bounded capacity, consolidation, and forgetting. Figure 6 distinguishes timestamp-based multi-versioning, capacity-driven physical eviction, LLM-driven semantic consolidation, and continuous parametric optimization.

• Versioning preserves history while marking obsolete relationships invalid or applying chronological precedence. Physical eviction instead removes or overwrites entries under FIFO, token-limit, or priority policies.
• Semantic consolidation can merge redundant observations inline or issue explicit Create, Read, Update, and Delete operations through tools. MemTree recursively updates parent summaries, while Mem0 uses tool-driven mutations.
• The taxonomy also includes offline parameter optimization, illustrated by MemoRAG’s Reinforcement Learning with Generation Feedback (RLGF), although the reported comparisons primarily concern context-based and external-store memory systems.

Invalidating an obsolete fact preserves historical versions, whereas FIFO or score-based eviction removes stored content. Consolidation and CRUD operations modify semantic state, while offline optimization changes parameters; these are distinct maintenance mechanisms rather than interchangeable forms of forgetting.

4 End-to-End Assessment

The end-to-end assessment asks five questions: whether memory improves task effectiveness, retrieves gold evidence, handles revised facts, remains stable over longer horizons, and provides acceptable operational cost. Long Context and Embedding RAG serve as reference baselines, but system coverage differs between experiments.

• LoCoMo evaluates long-conversation question answering; LongMemEval evaluates multi-session recall and reasoning; DB-Bench draws database-operation tasks from LifelongAgentBench.
• LongBench adds context-length stress testing, while evidence-distance and session-history bins expose different forms of long-horizon difficulty. The assessment separates answer quality, evidence access, temporal grounding, and operational latency rather than reducing them to one score.
• The testbed is identified as https://github.com/OpenDataBox/MemoryData, and the taxonomy resource is https://github.com/OpenDataBox/awesome-agent-memory.

4.1 Overall Effectiveness (RQ1)

Figure 7 shows that the leading configuration changes with the workload and metric. Zep Local reaches 48.0 LLM Judge Accuracy on LongMemEval, Cognee reaches 35.3 ROUGE-L F1 there, and MemOS reaches 11.5 EM on LoCoMo.

• LoCoMo scores are unweighted means of category-level EM and Answer F1 over four query categories; LongMemEval also reports Substring EM, ROUGE-L Recall, and GPT-5.4-based judge accuracy.
• The figure reports Letta at 61.6 for both DB-Bench EM and Task Success Rate, exceeding Long Context at 48.2 and MemoChat’s Task Success Rate at 55.4. This contradicts the accompanying narrative that Long Context has the best DB-Bench EM.
• MemoChat’s DB-Bench EM is 36.8 while its Task Success Rate is 55.4, demonstrating why lexical agreement and executable success require separate measures. The authors’ workload-alignment finding links dispersed reasoning to relational and temporal retrieval, dialogue grounding to coarse-to-fine filtering, and stateful execution to trace preservation; these are design interpretations, not isolated causal tests.

The eight panels show that rankings depend on workload and metric. They also reveal a narrative inconsistency: Letta has the highest plotted DB-Bench EM and Task Success Rate at 61.6, despite the prose identifying Long Context as the EM leader.

4.2 Memory Retrieval Fidelity (RQ2)

Retrieval fidelity is evaluated independently of answer generation using LoCoMo’s annotated source evidence. A hit requires the top-k retrieved source-id groups to contain the gold evidence; Recall@10 is also measured across six evidence-distance bins from 1–5 to 26–31 sessions, using the distance between the query’s final session and its earliest supporting evidence.

• SimpleMem has the highest Recall@1 at 39.0%, whereas A-MEM leads Recall@5 and Recall@10 at 69.5% and 85.9%; MemTree reaches 59.7% and 80.5%.
• Figure 8 shows substantially stronger long-distance evidence coverage for linked or hierarchical systems than for Embedding RAG, which reaches only 17.7% aggregate Recall@10.
• The results support separating early evidence localization from broader evidence assembly. Recall@1 remains a recall measure, not a direct measurement of retrieval precision, and the whole-system comparison does not isolate structure from other pipeline differences.

SimpleMem leads at the smallest retrieval budget, but A-MEM and MemTree provide stronger coverage as the budget expands. The distance curves show that linked or hierarchical systems retain distant supporting evidence much better than the flat Embedding RAG baseline, although the comparison does not isolate the effect of representation alone.

4.3 Memory Evolution Robustness (RQ3)

Update robustness is assessed on LongMemEval’s Knowledge Update and Temporal Reasoning slices and LoCoMo’s Temporal slice. Table 2 shows different leaders for direct revisions, temporally distributed evidence, and exact temporal grounding rather than a universal update winner.

• Zep leads Knowledge Update with 44.4 Substring EM and 36.8 ROUGE-L F1.
• Cognee leads Temporal Reasoning with 18.7 Substring EM and 35.8 ROUGE-L F1; on LoCoMo Temporal, MemOS leads EM at 8.9 while Cognee leads Answer F1 at 28.1.
• These slice-level results are consistent with benefits from relation-aware and temporally selective retrieval, but they do not directly measure the correctness of individual database update operations.

Direct knowledge revision, temporal reasoning, and temporal question answering favor different systems. Zep's strongest Knowledge Update scores and Cognee's strongest Temporal Reasoning scores support distinguishing access to revised facts from reconstruction of time-distributed evidence.

Figure 9 varies the LLM backbone among Qwen3-8B, DeepSeek-Chat, GPT-5.4-mini, and GPT-5.4. MemOS retains the highest Answer F1 among the plotted configurations at 32.2, 41.2, 38.6, and 41.2, although stronger generators do not improve every system monotonically.

• A-MEM overtakes MemTree under GPT-5.4-mini and GPT-5.4, showing that the relative ranking is broadly stable but not invariant. LightMem remains above MemTree and A-MEM across the four backbones.
• Embedding RAG and LightMem retain identical plotted Recall across the four backbones, while MemOS’s Recall changes substantially. MemTree’s Recall stays within 38.2–39.6, while A-MEM varies from 45.7 to 56.8.
• The figure contains five memory settings despite the experimental description referring to six. Its Recall values should not be assumed identical to Figure 8’s Recall@10 results, and it does not establish that only final answer realization changes with the backbone.

4.4 Long Horizon Memory Stability (RQ4)

Long-horizon stability is tested through LongBench context-length buckets, LongMemEval historical-session bins, and LoCoMo evidence-distance bins. Figure 10 generally favors relational or hierarchical organization over flat matching when supporting evidence becomes distant, but individual trajectories are not uniformly monotonic.

• From Short to Medium on LongBench, SimpleMem changes from 35.2 to 34.9 Accuracy, while Long Context falls from 42.6 to 19.0. This comparison does not establish stability across every context bucket or system; LightMem and MemTree also show workload-dependent changes rather than constant performance.
• Embedding RAG’s LoCoMo Answer F1 falls from 37.1 to 7.4 as evidence distance increases, whereas Cognee, MemOS, and MemoryOS remain substantially stronger in distant bins. These results support evaluating distant-evidence access separately from total input length.
• LongMemEval’s session bins contain 9, 24, 30, and 12 examples, respectively. Their unequal, small sample sizes and the absence of uncertainty estimates warrant caution about fine-grained trends.

The panels distinguish longer input contexts, more historical sessions, and more distant supporting evidence. Structured systems generally perform better on distant evidence, but non-monotonic curves and unequal session-bin sample sizes discourage interpreting every local change as a scaling law.

4.5 Memory Operation Cost (RQ5)

Operation cost includes memory construction and query time, interpreted as amortized per-query overhead for cumulative or bursty writes. Normalized Utility averages six min–max-normalized answer-quality metrics from LoCoMo and LongMemEval, while the cross-workload view reports outlier-filtered average total latency per query.

• LightMem achieves 48.3 Normalized Utility at 3.67 s per query, and MemTree achieves 63.5 at 15.9 s; MemoryOS reaches 82.0 at 28.6 s.
• Cognee and Zep exceed 84 utility at 116.5 s and 155.1 s, respectively. The authors attribute this higher overhead to broader reorganization and coordination, but the whole-system comparison does not isolate maintenance scope as the sole cause.
• On LongBench, LightMem requires 17.3 s per query, compared with 116.7 s for MemTree and 2867 s for Zep Local. These are workload-specific, outlier-filtered averages, not tail-latency guarantees or monetary costs.

The utility–latency plot gives LightMem and MemTree favorable lower-cost trade-offs, while several higher-utility structured systems require substantially more amortized operation time. The heatmap shows that latency differences widen sharply on LongBench, so one aggregate cost cannot characterize all workloads.

5 Fine-Grained Component Comparison

The component study changes one module at a time within selected systems to examine representation, extraction, routing, and maintenance. These experiments provide more targeted evidence than whole-system comparisons, but their conclusions apply to the tested variants rather than every implementation of each architectural family.

• The ablations use LoCoMo and LongMemEval, with maintenance evaluated on LoCoMo.
• Several ablation scores differ markedly from the corresponding end-to-end configurations. Cross-table differences should therefore not be interpreted as effects of a single change without matched experimental conditions.

5.1 Memory Representation and Storage (M1)

Table 3 shows substantial performance losses when LightMem replaces original content with summaries. User-Only Raw reaches 38.9 LoCoMo Answer F1 and 26.0 LongMemEval Substring EM, compared with 15.6 and 11.7 for User-Only Summary.

• User-Only Compressed remains close to raw storage on LoCoMo Answer F1 at 38.6, but its LongMemEval Substring EM falls to 10.7.
• MemTree’s Deeper Tree improves LongMemEval ROUGE-L F1 from 29.9 to 30.9 over Flat-biased, a much smaller change than the LightMem representation differences.
• Mem0’s Graph Store does not consistently outperform Default: LoCoMo Answer F1 changes from 6.2 to 6.5, but LongMemEval ROUGE-L F1 changes from 16.5 to 15.9. Added topology alone does not establish a general benefit.

Raw LightMem content outperforms summaries across all four reported metrics, while light compression preserves LoCoMo performance but loses much of LongMemEval exactness. Deeper trees and a graph store offer smaller or mixed changes, supporting evidence retention more clearly than structural complexity alone.

5.2 Memory Extraction (M2)

Table 4 favors coverage-preserving extraction as a balance across the two evaluated workloads, while also showing gains from finer extraction on some factual-retrieval metrics. The clearest trade-off appears in MemOS, where Fast Memorize and Fine Memorize use the same tree_text backend.

• Fine Memorize improves LongMemEval ROUGE-L F1 from 26.1 to 30.2 but reduces LoCoMo Answer F1 from 40.8 to 5.0.
• MemoChat Heuristic Topic exceeds LLM Topic on LongMemEval Substring EM, 10.7 versus 7.3, while LoCoMo Answer F1 is slightly lower, 33.5 versus 34.4.
• LightMem Hybrid Raw, which includes user and assistant turns, slightly improves LoCoMo Answer F1 from 38.9 to 39.7 while leaving LongMemEval ROUGE-L F1 at 31.4. The benefit is modest and workload-dependent.

MemOS Fine Memorize improves LongMemEval overlap while sharply reducing LoCoMo answer quality, exposing a workload-specific extraction trade-off. The MemoChat and LightMem variants also show that more selective or elaborate extraction does not uniformly outperform broader evidence retention.

5.3 Memory Retrieval and Routing (M3)

Table 5 favors balanced dense–sparse fusion over sparse-heavy fusion for A-MEM and explicit planning over direct retrieval or planning with reflection for SimpleMem. These results support targeted retrieval structure, not an unrestricted increase in reasoning steps.

• A-MEM Hybrid-Balanced reaches 24.6 LoCoMo Answer F1 and 27.5 LongMemEval Substring EM, compared with 23.0 and 24.3 for Hybrid Sparse-Leaning.
• SimpleMem Planning Only improves over No Planning from 18.7 to 20.7 Answer F1, 86.4 to 90.6 Recall, and 22.9 to 27.9 LongMemEval ROUGE-L F1.
• Planning + Reflect reduces the corresponding results to 20.0 Answer F1, 88.6 Recall, and 26.1 ROUGE-L F1. The table establishes no quality gain from reflection in these runs; it does not separately quantify reflection latency.

Balanced fusion outperforms sparse-leaning fusion for A-MEM on every reported metric. SimpleMem improves with planning but loses some of those gains when reflection is added, identifying a benefit from planning rather than additional deliberation in general.

5.4 Memory Maintenance (M4)

Figure 12 compares conservative merging, delayed flushing, and summary granularity on LoCoMo. MemoryOS Conservative-Merge modestly improves Answer F1 from 23.2 to 23.5 and Substring EM from 22.4 to 22.8, while Delayed-Flush reduces them to 20.6 and 19.5.

• MemoChat Topic1, which forces a single-topic summary per window, underperforms Default at 16.2 versus 16.6 Answer F1 and 16.8 versus 18.4 Substring EM.
• Long Context retains the highest Substring EM in this ablation at 23.7, but its Answer F1 is 21.3, below default and conservative-merge MemoryOS. Maintenance can improve answer quality without improving exact lexical agreement.
• The maintenance finding favors selective consolidation over delayed organization or overly coarse summaries; the small conservative-merge gains remain descriptive because no significance tests are reported. The conclusion synthesizes the taxonomy, end-to-end assessment, and component comparisons as guidance for choosing workload-appropriate memory architectures.

Conservative merging produces small improvements over default MemoryOS, while delayed flushing and single-topic MemoChat summaries reduce both plotted metrics. Long Context remains strongest on Substring EM but not Answer F1, showing that maintenance gains in answer quality need not imply better lexical agreement.

Appendix

• The supplied PDF contains no appendix; it ends with 6 Conclusion and References. Implementation details absent from the main text cannot be recovered from an assumed supplementary section.

Brief Thoughts

The most useful contribution is the separation of memory quality into recoverable content, evidence access, temporal validity, and maintenance cost. The representation and extraction ablations provide concrete support for preserving information at write time, while the cost comparisons show that richer structure can carry substantial operational overhead.

The paper is better read as comparative design evidence than as a definitive system leaderboard. Figure 7 contradicts the prose about the DB-Bench winner, several stated configuration counts do not match plotted coverage, and the PDF does not fully specify hardware, shared default backbones, context-bucket thresholds, outlier filtering, or repeated-run uncertainty; these gaps limit reproducibility and latency generalization.