Gorio Tech Blog search

Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute | Summary

한국어로 읽기

07 Oct 2026 | Paper Review KV Cache LLM Serving Long-Context Language Models
Contents

This article explains the key points of Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute.

Read this article in Korean

Summary

  • Language-model serving can repeat computation for previously processed text and loses direct access to material outside its context window, the tokens available for attention in a request. On one NVIDIA H100, galahad-kv restored all 100 probed blocks from a 50,000,000-token store for each of two tested models, with reported speedups of 2.8× and 4.25× over full processing of a previously unseen 16k-token block with the same request shape.
  • The system saves key-value (KV) state, the attention state computed from processed tokens, to encrypted local NVMe storage, a local disk tier that retains blocks outside GPU memory. Only one block is resident at a time; restoring historical blocks neither creates simultaneous attention over 50 million tokens nor provides learned retrieval, the selection of passages by meaning.
  • Gemma 4 12B and Gemma 4 31B answered 82/100 and 98/100 planted-code questions correctly, respectively, while neither returned a code absent from its restored block. The reported GPU energy reduction factors were 8.8× and 12.3× relative to recomputation, but the stores required 1.86 TB and 6.25 TB; byte-exact logit equivalence, equality of the model’s output scores after restoration and recomputation, is attributed to companion work rather than tested again here.

1 Introduction

The introduction identifies two problems: losing access to tokens outside the configured 16k-token context window and rebuilding previously computed KV state. The proposed solution processes each block once, retains its state on disk, and restores it when requested. Its claims concern durable block reuse at historical depths with constant GPU memory; single-pass attention remains limited to the active window.

The paper compares its system with prefix caching, which reuses shared initial prompt state, and KV offloading, which moves that state outside GPU memory. Retrieval-augmented generation (RAG) retrieves text and processes it again inside the context window, whereas cache-augmented generation precomputes document state for a fixed set that still fits the window. Agent memory systems such as MemGPT select which text enters that window.

  • Gemma 4 uses sliding-window attention, which restricts most layers to approximately 1,024 nearby tokens and reduces the stored state footprint. The memory layer benefits from this existing model property.
  • Earlier reports covered an integrated memory layer on 97k tokens, a frozen-model 6M-token memory, and reuse of verified knowledge. This experiment increases the corpus to 50M tokens, uses the vLLM serving engine, encrypts stored state, and adds the anti-cheat protocol; Blaise, the learned document-retrieval component, is excluded.

3 System under test

The tested engine is galahad-kv==1.31.4, implemented through libgalahad.so and loaded by vLLM 0.31.0 using the GalahadConnector KV-transfer plug-in. During deposit, the operation that saves computed state, it writes each block in MRLNCRY1 storage format using AES-256-GCM, encryption with authentication, with one key per organisation, then frees the resident state. During restore, it inserts stored state into the running model so computation can continue without processing the block’s tokens again; only this KV reuse mode is evaluated.

4 Anti-cheat protocol

The anti-cheat protocol combines runtime mutation of source facts, ingestion without questions, planted records, same-format decoys, and physical storage audits. Replacing names, dates, amounts, and e-mails with random values is intended to prevent answers from pretraining or static caching. Withholding questions until all 50M tokens are deposited addresses query lookahead, access to evaluation questions during ingestion.

  • A needle is a planted named record containing a four-digit code; decoys are similar records containing other codes. Section 4 specifies one needle per block, whereas Section 5 specifies 100 needles, one per 500k tokens, leaving their exact distribution unclear.
  • The engine metric checks zero-recompute restoration using cached_tokens ≈ block size. The separate model metric checks whether the emitted code exactly matches the answer key, which is withheld from the engine.
  • Storage audits check the MRLNCRY1 header and expected byte footprint. Energy comes from the NVIDIA Management Library (NVML), the GPU hardware-counter interface, after subtracting idle consumption; negative controls ask for absent records and expect refusal.

5 Experimental setup

Experiments use one NVIDIA H100 80 GB with local NVMe and temperature 0, the generation setting used throughout. Gemma 4 12B uses bf16, a 16-bit floating-point format, and Gemma 4 31B uses FP8, an 8-bit floating-point format. Each model runs through vLLM with its own tokenizer, which converts text into model tokens; software includes galahad-kv==1.31.4, vLLM 0.31.0, transformers, datasets, and nvidia-ml-py.

  • The corpus contains 50,000,000 tokens = 3,125 blocks × 16,000 tokens: WildChat contributes 40%, permissively licensed source code 25%, arXiv papers 20%, and educational web text 15%. Runtime mutations, 100 needle records, and same-format decoys are then applied.
  • The paper states that the method can run on a single NVIDIA GPU with ≥24 GB, but the reported performance measurements use the H100 80 GB configuration.

5.1 KV footprint

For the 12B model, the reported state footprint is 36.4 KiB/token, compared with approximately 160 KiB/token for full attention, yielding a reported 1.86 TB store instead of approximately 8 TB. For the 31B FP8 model, the reported figures are 134 KiB/token and 6.25 TB. The author reports auditing all 3,125 blocks in each store, finding the expected headers and no missing or evicted blocks; the unit convention behind the 31B storage calculation is unclear.

6 Results

The results separate successful state restoration from successful reading of restored content. They report restoration and answer accuracy, restore-versus-recompute latency and GPU energy, and the initial deposit’s throughput, GPU memory use, and disk footprint.

6.1 Byte-exact restore at every depth

At 100 probed depths between 0 and 49,488,000 tokens, both models restored 100/100 blocks without recomputation. Exact-code recall was 82/100 for Gemma 4 12B and 98/100 for Gemma 4 31B. Neither model emitted a code absent from the restored block, and both refused all 20 absent-record negative controls.

6.2 Restore is faster and cheaper than recompute

The baseline performs a full prefill, the initial processing of prompt tokens, on a previously unseen 16k-token block with the same request shape. Median restore versus recompute measurements are 0.266 s / 59 J versus 0.760 s / 521 J for 12B, and 0.347 s / 80 J versus 1.475 s / 976 J for 31B. The table reports speedups of 2.8× and 4.25× and energy reduction factors of 8.8× and 12.3×.

  • The 90th-percentile restore time to first token (TTFT), the delay before the first output token, is 0.284 s for 12B and 0.579 s for 31B.
  • Low-third to high-third depth TTFT changes from 0.193 to 0.274 s for 12B and from 0.343 to 0.375 s for 31B. Restoration remains fast at distant depths, but these values do vary, and the paper provides no uncertainty estimates for the differences.
  • The larger tested model shows greater relative savings. Two model sizes using different numerical precisions provide limited evidence for a general relationship between model size and reuse savings.

6.3 Deposit (the one-time cost)

Depositing 50,000,000 tokens achieves 20,839 tok/s for Gemma 4 12B and 10,700 tok/s for Gemma 4 31B. Reported peak video RAM (VRAM), the GPU’s resident memory, remains flat at 70,864 MB and 75,111 MB, respectively; the corresponding encrypted stores occupy 1.86 TB and 6.25 TB.

  • The paper describes resident memory as an O(1) moving window, meaning its size remains constant as deposited history grows. Deposit is a one-time cost; the reported latency and energy ratios apply to subsequent reuse.

6.4 What the numbers say

The author interprets identical 100/100 restoration rates and different recall scores as a separation between storage reliability and the model’s ability to read restored content. Every reported recall failure selected a decoy or another number actually present in the block. Serving with a cheaper model and escalating to a stronger model is a proposed application; the experiments do not evaluate an escalation policy or demonstrate transfer of one model’s KV state to another.

7 Why this is long-term memory and not a cache

The paper motivates the term long-term memory through four properties: persistence across restarts, encryption at rest, byte-exact restoration, and reuse of material beyond the active context window. Model weights remain unchanged; restoring state makes previously processed facts available for a request. These properties explain the author’s terminology, while this experiment directly measures block restoration, recall, and resource use.

8 Limitations and scope

Only one block is resident, so the system does not attend to all 50M tokens simultaneously. Learned passage selection is outside scope, and combining it with restoration at this scale remains future work. The results therefore do not establish end-to-end question answering over an automatically searched 50M-token history.

  • Restores take approximately 0.3 s, approaching the resolution of the NVML energy counter. The author considers relative energy ratios reliable but acknowledges greater uncertainty in absolute per-probe joules.
  • The stores require 1.86 TB and 6.25 TB for the tested models. The reported latency and energy figures require local NVMe and do not apply to network volumes.

9 Reproducibility (single GPU, public software)

The reproduction instructions specify pip install galahad-kv==1.31.4 vllm==0.31.0 datasets transformers nvidia-ml-py, obtaining a Gemma 4 model, and obtaining a free Galahad licence on the GPU host. Serving uses GalahadConnector with kv_role set to kv_both, –max-model-len 16640, and –enable-prompt-tokens-details.

  • The four-stage harness, the scripts that run the evaluation, builds the mutated corpus, deposits it without query access, audits storage, and probes restoration and recall. The paper describes it as a public HTTP client using transformers, datasets, and urllib, and states that raw artifacts and a SHA-256 manifest, a list of file-integrity checksums, support the reported measurements.

Appendix

  • A Harness lists build_hardened.py for corpus construction, mutations, needles, decoys, and the answer key; deposit_blind.py for blind ingestion and NVML memory/energy measurement; audit_storage.py for MRLNCRY1 and footprint checks; and probe_watertight.py for the two metrics, recompute baseline, negative controls, and miss classification.
  • B Data and manifest identifies RESULT_12b_probe.json, RESULT_31b_probe.json, deposit logs, the answer key, and SHA256SUMS_31b.txt. It reiterates that the local NVMe stores contain 3,125 MRLNCRY1 blocks each, occupying 1.86 TB for 12B and 6.25 TB for 31B. This appendix names the artifacts without supplying a download URL.

Brief Thoughts

The strongest evidence is the separation of engine restoration from model recall, supported by 100 probes per model and absent-record controls. The demonstrated benefit is cheaper reuse of a selected historical block; automatic retrieval, simultaneous reasoning across the full history, and byte-exact output equivalence are outside the experiment’s direct evidence. Reproduction needs clarification of the conflicting needle-placement descriptions and the conversion from 134 KiB/token over 50,000,000 tokens to the reported 6.25 TB store. The claim of flat restore latency needs assessment alongside the reported low-third and high-third TTFT values. Uncertainty estimates would help determine whether those differences are meaningful. The tested models use different precisions and belong to the same model family. Additional configurations are needed to assess broader claims of model independence or savings that scale with model size.

Related Posts