Agentic Learner with Grow-and-Refine Multimodal Semantic Memory | Summary

This article explains the key points of Agentic Learner with Grow-and-Refine Multimodal Semantic Memory.

• 2025-11-26 (arXiv)
• Bo, Weihao, Zhang, Shan, Sun, Yanpeng, Wu, Jingjing, Xie, Qunyi, Tan, Xiao, Chen, Kunbin, He, Wei, Li, Xiaofan, Zhao, Na, et al.
• Nanjing University of Science and Technology, Baidu Inc, AIML, Adelaide University, Singapore University of Technology and Design
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Multimodal large language models (MLLMs), models that answer questions using images and text, can repeat visual interpretation and reasoning errors when they solve each question independently. With step-by-step prompting in both comparison conditions, ViLoMem’s separate visual and logical memory banks raise GPT-4.1 accuracy on MathVision, a benchmark of visually grounded mathematical reasoning, from 47.47% to 53.95%, a gain of 6.48 percentage points measured as pass@1, the accuracy of one generated answer per question.
• The framework generates corrective memory after an incorrect answer is detected against a reference answer, merges similar guidelines, and adds distinct ones. Visual retrieval first selects memories by image similarity and then filters their guidelines for question relevance, while logical retrieval uses an analysis of the question’s subject and key concepts.
• Removing either memory stream reduces GPT-4.1 accuracy on the two reported component-ablation benchmarks, where ablation means disabling a component to measure its contribution, and memory from other models further improves the smaller Qwen3-VL-8B-Instruct solver. Benefits depend on memory relevance and task alignment: transfer between benchmarks sometimes lowers accuracy, attention maps (heatmaps highlighting regions associated with retrieved visual cues)can harm mathematical reasoning, and the scalability experiment does not directly measure retention on previously encountered tasks.

1. Introduction

The paper, titled “ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory,” argues that storing textual reasoning summaries alone loses the perceptual cues needed to solve later image-based questions. Its semantic memory, persistent structured knowledge abstracted from experience, separates instructions about where to look from instructions about how to reason.

• A visual mistake can propagate into a logical error: selecting the wrong diagram feature can invalidate an otherwise appropriate formula.
• Figure 1 illustrates why remembering the triangle-area formula must be paired with correctly identifying the base and corresponding height.
• Project page: https://weihao-bo.github.io/ViLoMeo-page/. The supplied PDF identifies itself as arXiv:2511.21678v2, dated 2 May 2026.

The triangle examples distinguish remembering a formula from identifying the visual quantities needed to apply it. They motivate separate corrective knowledge for theorem use and diagram interpretation, but provide a conceptual illustration rather than measured performance.

2. Related Work

The related-work discussion connects two approaches: changing the context supplied to a model and maintaining persistent external memory. ViLoMem represents visual error patterns explicitly alongside logical guidelines and retrieves each stream with a different matching procedure.

2.1. Context Engineering

Context engineering modifies input prompts to improve behavior without changing model weights. The paper discusses ReAct, which combines reasoning and actions; Reflexion, which uses verbal feedback from failures; TextGrad, which generates gradient-like textual feedback for prompt refinement; and GEPA, which uses evolutionary prompt optimization based on execution traces. It identifies transient context and brevity bias, the loss of useful details through repeated shortening, as limitations motivating persistent structured memory.

2.2. Long-term Memory

Dynamic Cheatsheet stores strategies from previous inferences, while ACE, Agentic Context Engineering, incrementally maintains a context playbook through generation, reflection, and curation. The paper argues that these approaches emphasize logic and motivates separate visual and logical representations through a hub-and-spoke account of human semantic memory, in which a central hub integrates modality-specific representations. This cognitive analogy motivates the architecture without establishing biological equivalence.

3. Method

ViLoMem maintains a logical bank of textual reasoning guidelines and a visual bank of guidelines paired with source images. Its Memory Cycle retrieves from both banks, supplies the retrieved guidance to a solver, verifies the resulting answer against ground truth, and triggers memory generation when the answer is incorrect.

• Matching uses cosine similarity, the normalized dot product of embeddings: \(\operatorname{Sim}(u,v)=\frac{u\cdot v}{\|u\|\|v\|}\), where embeddings are vector representations of text or images.
• A large language model analyzes the question and reasoning trace to identify the subject and key concepts; the original question and this analysis form an enriched retrieval query.
• The operational update rule is error-driven, despite the abstract’s broader description of learning from successful and failed experiences.

3.1. Memory Generation

Memory Generation analyzes an incorrect answer for visual and logical errors in parallel. In “3.1.1. Visual Memory Generation,” the analyzer receives the image, question, erroneous reasoning, and reference answer; in “3.1.2. Logical Memory Generation,” the analyzer receives textual inputs without the image.

• Visual analysis identifies misinterpretations such as object confusion, missed symbols, or incorrect spatial relationships and produces a corrective observation guideline.
• Logical analysis classifies errors as Logical or Non-Logical and generates a guideline only for a logical error, such as a calculation mistake or formula misapplication.
• Text similarity determines whether a guideline is merged with the nearest existing entry or added as a new entry. The equations use similarity thresholds, but the supplied experimental details do not specify their numerical values.

3.2. Memory Retrieval and Utilization

In “3.2.1. Visual Memory Retrieval,” image embeddings first select candidate memories, then text embeddings rank and threshold their guidelines against the enriched question. “3.2.2. Logical Memory Retrieval” directly matches logical guidelines to that enriched question, and “3.2.3. Solution Generation with Dual Memory” conditions the solver on the original image, question, and both retrieved streams.

• The image stage restricts candidates to visually similar examples; the text stage checks whether their corrective guidance addresses the current question.
• Optional question-aware attention maps are supplied as auxiliary images alongside the original image to highlight regions associated with retrieved visual cues.
• The paper expresses solution generation as ỹᵢ = Gen(Iᵢ,qᵢ,Rᵢᴸ,Rᵢⱽ), where Iᵢ and qᵢ are the image and question, Rᵢᴸ and Rᵢⱽ are the retrieved logical and visual guidelines, and Gen is the solver. Numerical retrieval counts are not given in the supplied implementation details.

The architecture separates the solver–verifier cycle, parallel error analysis, and specialized retrieval. The visual branch combines image search with question filtering, while the logical branch uses text matching; similarity checks determine whether memory is merged or created.

4. Experiments

The experiments examine six benchmarks spanning visually grounded mathematics, hallucination (answers inconsistent with the supplied evidence) and real-world robustness, and knowledge requiring visual understanding. Additional analyses test individual memory components, memory usage, transfer between solvers and benchmarks, and growth of a mathematical memory pool.

4.1. Experimental Setup

The main solvers are GPT-4.1, Qwen3-VL-235B-A22B-Instruct, and Qwen3-VL-8B-Instruct. The comparisons use Baseline with default prompting, Step with an explicit step-by-step prompt, and +ViLoMem with step-by-step prompting and retrieved memory; Step provides the closer comparison for assessing the added memory system.

• The benchmarks are MMMU (val), described as 1,050 college-level questions across six domains; MathVista (mini) and MathVision (mini), which test visually grounded mathematics; HallusionBench, which tests hallucination and visual illusion through 1,129 control-paired questions; MMStar, which tests vision-dependent reasoning with 1,500 samples; and RealWorldQA, which assesses spatial reasoning over 765 natural scenes.
• Logical memory generation uses Qwen3-235B-A22B-Instruct, visual memory generation uses Qwen3-VL-235B-A22B-Instruct, and matching uses Qwen3-Embedding and Qwen2.5-VL-Embedding.
• Evaluation uses VLMEvalKit, an evaluation toolkit, with a language-model judge for ambiguous answer matching. Table 1 combines externally reported baseline scores with starred self-evaluated scores, and the supplementary tables label MMMU as dev rather than val.

4.2. Main Results on Multimodal Benchmarks

Against Step, GPT-4.1 improves on all six benchmarks: MMMU 74.16→77.26, MathVista 74.27→76.88, MathVision 47.47→53.95, HallusionBench 74.44→75.29, MMStar 70.43→72.43, and RealWorldQA 72.03→74.38. Qwen3-VL-8B-Instruct also improves on all six, although its MathVista gain is only 77.80→77.87.

• Qwen3-VL-235B-A22B-Instruct improves on five benchmarks relative to Step but declines on RealWorldQA, 78.66→77.22; its MMStar score of 78.31 also remains below the reported default baseline of 78.40.
• The qualitative examples show visual guidelines addressing surface reflectivity, diagram digits, and background-induced color illusions, while some retrieved logical guidelines are irrelevant to the visual question.
• The authors identify attribution failures when the solver’s reasoning omits visual evidence or describes complex diagrams poorly, making visual errors difficult to distinguish from logical errors.

The Step rows provide the closest prompting comparison for ViLoMem. GPT-4.1 and the 8B solver improve on all six benchmarks, while the 235B solver declines on RealWorldQA; comparisons against default prompting also combine externally sourced baseline scores with starred self-evaluations.

The cases show visual guidance targeting material appearance, exact diagram digits, and background-induced color perception. Several logical entries are unrelated to the immediate visual task, while the geometry case contains no visual guideline.

4.3. Ablation Study

For GPT-4.1, removing logical memory changes MMMU/MathVista accuracy from 77.26/76.88 to 76.64/75.59; removing visual memory changes it to 76.88/75.66. These results support complementary contributions on the two tested benchmarks, with a larger logical-memory contribution on MathVista than on MMMU.

• Adding attention maps raises MMMU to 78.21 but changes MathVista to 76.87, a slight decrease from 76.88.
• The attention variant is an additional configuration, so its results should be distinguished from standard ViLoMem.

Both single-stream removals lower accuracy relative to full ViLoMem on MMMU and MathVista. Attention adds 0.95 percentage points on MMMU but changes MathVista by −0.01, showing that the auxiliary heatmaps have different effects across the two benchmarks.

4.4. Memory Usage Analysis

Visual memory accounts for 59%–93% of generated memory across the six benchmarks, whereas retrieved memories are more evenly divided between visual and logical streams. Retrieval events also exceed generation events, indicating reuse of previously stored guidelines.

• These statistics reflect the framework’s own error attribution and memory operations; they are not an independently annotated census of all model errors.
• Similar retrieval patterns across the three solvers show use of both banks, but counts alone do not establish the value of every retrieved entry.

Generated memory is predominantly visual, with shares from 59% to 93%, while retrieval draws more evenly from both streams. The pattern shows memory reuse and differing generation frequencies under the system’s own error attribution.

4.5. Cross-Model Memory Transfer

Cross-Model Memory Transfer replaces each solver’s own memory with memories produced by the other two solvers on the same benchmark. Qwen3-VL-8B improves from 69.90 to 71.26 on MMMU and from 77.87 to 79.20 on MathVista, gains of 1.36 and 1.33 percentage points without weight fine-tuning.

• GPT-4.1 improves on MMMU, 77.26→78.21, but declines on MathVista, 76.88→76.58.
• Qwen3-VL-235B declines slightly on both tested benchmarks, 79.40→79.26 and 84.98→84.21. Transfer benefits are strongest for the smaller solver in this experiment.

Replacing a solver’s own memory with memory from the other two models benefits the 8B solver on both tested benchmarks. GPT-4.1 shows mixed changes and the 235B solver declines slightly on both, limiting the evidence for universally beneficial cross-model transfer.

4.6. Cross-Benchmark Memory Generalization

Cross-Benchmark Memory Generalization tests Qwen3-VL-8B-Instruct using memories from all other benchmarks while excluding the target benchmark’s memory. Relative to Step, this raises MathVision 48.35→50.00, MMStar 70.22→70.93, and RealWorldQA 70.85→71.63, but lowers MMMU 65.52→65.14, MathVista 77.80→76.10, and HallusionBench 73.08→70.66.

• Compared with task-specific ViLoMem memory, cross-benchmark memory performs worse on five of six benchmarks; MathVision is the exception, 49.34→50.00.
• The results show task-dependent transfer and are consistent with interference from mismatched memories.

Excluding target-benchmark memory produces heterogeneous transfer: accuracy rises over Step on MathVision, MMStar, and RealWorldQA but falls on MMMU, MathVista, and HallusionBench. Task-specific memory performs better on five of the six benchmarks.

4.7. Memory Scalability

Memory Scalability accumulates memory in the sequence MathGlance, a mathematical-diagram perception benchmark, →MathVista→MathVision→MathVerse, a visual mathematics benchmark, and evaluates on unseen WeMath, a mathematical reasoning benchmark. Accuracy rises from 72.53% at 15k tokens (0.1k samples) to 72.82% at 50k (0.5k), 73.91% at 87k (1k), 74.33% at 105k (2k), and 74.58% at 150k (3k), exceeding 73.85% with memory generated directly on WeMath.

• The experiment shows increasing utility of this progressively accumulated mathematical memory pool on WeMath.
• Because evaluation remains on WeMath, these results do not directly test catastrophic forgetting, loss of performance on earlier tasks after later experience.

WeMath accuracy increases at every reported memory-size checkpoint, reaching 74.58% with 150k tokens and exceeding the Direct condition’s 73.85%. This supports the utility of accumulated mathematical memory on WeMath without directly establishing retention on earlier tasks.

5. Conclusion

The conclusion attributes ViLoMem’s gains to separate visual and logical guidelines, specialized retrieval, and incremental merge/create updates. The Step comparisons, component ablations, and transfer to the smaller solver support these mechanisms, while the broader continual-learning claim is limited by the absence of direct measurements of retention on earlier tasks.

Appendix

• The Supplementary Material’s “6. Additional Results and Ablation Study,” beginning with “6.1. Integration with more models,” extends evaluation to GLM4.1v, InternVL3-38B, and Gemini2.5-flash. All three improve over Step on all six reported benchmarks, but GLM4.1v and Gemini2.5-flash remain below their default baselines on RealWorldQA. “6.2. Attention Mechanism Ablation” documents mixed effects: GPT-4.1 MathVision falls from 53.95 to 50.66 with attention, while HallusionBench rises from 75.29 to 75.73; the authors attribute mathematical-task limitations to heatmap precision and interpretation. “6.3. Additional Case Study” illustrates traffic-light inspection, ruler alignment, spatial reference frames, chart reading, and optical illusions.
• “6.4. Comparison with Existing Memory Methods” evaluates a retrieval-based Dynamic-Cheetsheet adaptation using text-only matching instead of its original cumulative context. On MathVista, ViLoMem scores 76.88% versus 73.87%, with retrieval latency of 120 versus 325 ms, storage of 6.12 versus 18.44 MB, and memory size of 73K versus 221K tokens. The paper attributes the smaller memory pool to generating entries only for incorrect answers, whereas the comparison method updates memory for every case. These measurements cover retrieval and storage, without accounting for the full cost of memory generation, question analysis, attention-map generation, and solving.
• The two-stage retrieval analysis reports WeMath accuracy of 74.58% and latency of 61.6 ms per case at 150k memory tokens, compared with 73.41% and 115 ms for text-only retrieval. The prose claims a 22.3× latency reduction, but the tabulated values imply approximately 1.87×. At 87k tokens, two-stage retrieval scores 73.91% at 20.4 ms, versus 73.22% at 22.8 ms for text-only retrieval.
• “6.5. Failure Case Analysis” identifies 66 regressions on 1,041 MMMU samples with GPT-4.1: 22 cases (33.3%) involve generic visual guidance and 44 (66.7%) involve empty retrieval. Its reported net gain of +8.86% is not reconciled with the main table’s 74.00→77.26 result. “6.6. Visual Memory Characterization” analyzes 835 subject-matched samples: visual-memory contribution, measured by removing visual memory from the full system, is +9.8 in Tech & Engineering; logical-memory contribution, measured by removing logical memory, is +8.1 in Health & Medicine; and both contributions are negative in Business. “6.7. Results on Visual Perception Benchmarks” reports 166 visual and 16 logical entries on MathGlance with +1.87% average accuracy improvement, while noting that the current binary error-driven update mechanism does not directly support OCRBench and DocVQA, perception benchmarks evaluated through character-matching scores.
• “7. Additional Experimental Details” specifies temperature 0.7, maximum solver output of 8,192 tokens, problem-analysis output of 1,024 tokens, and memory-generation output of 2,048 tokens. Most locally deployed models use vLLM, a batched inference engine; Qwen3-VL-235B-A22B-Instruct and proprietary models use provider application programming interfaces (APIs), whose rejected requests introduce reported score noise. Attention maps use Qwen2.5-VL-3B to produce token-level saliency, indicating which image regions are emphasized, and ambiguous answer matching uses Math-Verify, an answer-verification tool, and Qwen3-8B-Instruct as a judge. “8. Prompt Templates” provides the Step, problem-analysis, logical-memory, visual-memory, and answer-verification templates; verification evaluates final-answer agreement independently of reasoning quality.

Brief Thoughts

The component ablations and qualitative examples support encoding corrective visual inspection strategies alongside reasoning rules. Assessing deployment would also require an explicit separation between memory-building and evaluation examples, numerical retrieval settings, reference-answer availability, and total pipeline cost; the supplied protocol leaves these points unresolved. Benchmark exceptions, negative Business contributions, and regression cases make memory relevance a necessary part of evaluation.