MemCoRe: Recovering Evidence from Progressively Compressed Factual Knowledge for Agent Memory

arXiv:2602.07885 · cs.AI, cs.LG · Submitted 2026-02-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "MemCoRe: Recovering Evidence from Progressively Compressed Factual Knowledge for Agent Memory".

Jane: The paper was written by the authors from.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary/Abstract: Tom: The abstract for "MemCoRe: Recovering Evidence from Progressively Compressed Factual Knowledge for Agent Memory" really zeroes in on a core conflict that most current methods struggle with. They show that traditional retrieval methods, while preserving verbatim details, end up accumulating massive amounts of redundancy without ever consolidating those concepts.

Jane: And on the flip side, they point out that memory-augmented approaches try to solve this redundancy by compressing the data using LLMs. But as they do that, they often lose that fine detail—the precise information needed for complex reasoning—in the process of trying to achieve efficiency.

Lu: The paper suggests a unified theory for this challenge by framing it as an information-theoretic optimization problem. Instead of just picking one side, we are looking at a principled approach to determining exactly what info is worth keeping and what should be discarded.

Meng: That sounds like the right kind of systematic approach. If they are tackling that dilemma head-on, I’m curious about the initial scope of their solution—did they find a way to handle both high-level thematic organization and low-level entity tracking within the same system?

Lalam: The implications here suggest we're moving beyond just managing tokens; we are building a coherent narrative. We are looking at how agents can handle long, complex histories without losing the thread of what was actually important in those early interactions.

Tom: It’s a necessary correction to existing methods, making sure that the AI isn't overwhelmed by redundant data while also ensuring it sets up the stage for a complex task. This leads directly into how they achieve this remarkable structure in memory management.

The Methodology/Improvements: Tom: To solve that bottleneck, the authors propose some highly sophisticated architectural improvements using "MemCoRe." They introduce a very structured, stratified hierarchy built from three distinct layers: Notes, Keywords, and Topics.

Jane: That layered structure is the real core innovation here. It’s not enough to just have a high-level topic; you need those specific keywords and the individual notes that anchor that topic in reality. It provides both a macro view of knowledge and those crucial micro-level retrieval points.

Lu: I see the brilliance in how they use those keywords as symbolic anchors, which are essentially stabilizing the semantic space between raw data and abstract concepts. They act like waypoints, giving the system a much firmer scaffolding than just relying on simple vector databases alone.

Meng: And then they layer on the method for optimization: utilizing an "information bottleneck" approach driven by an LLM-based optimizer. This suggests they can handle extremely complex logical updates without needing to calculate massive backpropagations across the entire memory graph every time new data arrives.

Lalam: This layered approach fundamentally changes how AI understands context. It allows the agent to understand not just *what* was said, but exactly where that statement fits within a broader conceptual structure—is it a core topic? Is it merely an outlying note? That architectural understanding is what makes the knowledge truly useful for long-term memory.

Jane: It moves far beyond simple keyword matching and into true structural reasoning. The Notes-Keyword-Topic hierarchy provides the necessary granularity for both human comprehension and machine efficiency at scale, which is vital for large deployments.

Tom: So, if this framework works as advertised, it fundamentally changes what an AI agent's memory should be—it’s not a hard drive; it’s a living, architecturally managed knowledge system that has the potential to sustain complex reasoning.

Mechanism Details and Practical Application: Tom: The authors describe three specific ways this architecture works: they have macro-semantic navigation through Topics, micro-symbolic anchoring via Keywords, and then topological expansion using associative links.

Jane: That’s a brilliant way to handle complex queries because you don't just search one spot. You are simultaneously looking at the big picture (Topics), checking specific entities (Keywords), and also following logical connections that might be scattered throughout the memory structure.

Lu: I think the use of those keywords is particularly powerful for stabilizing semantic space. They provide a robust, distributionally stable feature set that lets us find things based on shared symbolic features rather than just relying on potentially noisy vector correlations.

Meng: And to make sure this works in real-time, they implement an "Iterative Evidence Refinement" protocol. This means the AI doesn't stop after the first search; it checks if the answer is complete, and if not, it generates a new sub-query to find the missing piece of evidence.

Lalam: That iterative process suggests that our future AI companions won’t just give us an immediate answer; they will genuinely evolve their understanding by actively searching for gaps in knowledge over time. This changes the conversation about how much "memory" an AI actually possesses.

Jane: It allows the system to build a complete, coherent narrative piece by piece, rather than guessing based on one large block of context that might be too noisy or incomplete.

Tom: The combination of these three pathways means the AI is intelligently assembling its answer using evidence from Notes, Keywords, and those logical connections. This leads us into how this system performs compared to other state-of-the-art methods in the next segment.

Conclusion/Wrap up: Tom: So, to wrap up our discussion on "MemCoRe: Recovering Evidence from Progressively Compressed Factual Knowledge for Agent Memory," it’s clear that this represents a major paradigm shift in how we think about artificial memory architecture itself.

Jane: Exactly. We've seen how the concept of progressively compressing knowledge while retaining verifiable evidence fundamentally changes the scale and reliability of AI agents. It moves them from being mere pattern matchers to something much closer to genuine, cumulative reasoners.

Lu: And what’s exciting is that this isn't just a theoretical improvement; it addresses a tangible bottleneck in deploying truly autonomous systems that need to remember months of interaction without the memory degrading or becoming useless.

Meng: It really emphasizes the difference between merely storing data and actually structuring knowledge. That structural understanding is what makes the whole system so powerful when it allows for practical, large-scale deployment in complex operational environments.

Lalam: I think this ultimately means that our future AI companions will feel less like tools we query and more like actual collaborators who genuinely retain context and grow over time, making them feel much more dependable.

Tom: It’s a massive leap forward, encapsulating years of research into one cohesive, actionable framework under the title "MemCoRe: Recovering Evidence from Progressively Compressed Factual Knowledge for Agent Memory."

Jane: It really grounds the science in utility, which is always what we want to see. This isn't just academically interesting; it’s immediately practical for enterprise applications where reliable memory matters.

Lu: I hope this opens up new avenues for exploration in how we structure digital knowledge bases across entire industries, not just conversational data streams.

Meng: My final thought is that this feels like the kind of practical architectural innovation that allows large-scale agent deployment to truly succeed in diverse operational settings where memory matters most.

Lalam: We can’t wait to see the future where these AI agents are operating with such deep, coherent memory—it genuinely changes the conversation around digital intelligence.

cs.AI, cs.LG

Submitted: 2026-02-08

Updated: 2026-09-04

Comments: An earlier version of this work, titled "MemFly: On-the-Fly Memory Optimization via Information Bottleneck," was accepted by the ICLR 2026 MemAgents Workshop

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 81/100

The gist: The paper introduces MemFly, a novel framework designed to address the fundamental dilemma in large language model (LLM) agents where efficient compression of long-term memory conflicts with

Key concepts

The Memory Dilemma
Current retrieval methods either accumulate excessive redundancy without consolidating concepts or use LLMs to compress data, which often causes the loss of fine detail needed for complex reasoning. This creates a core conflict that limits current AI performance.
MemCoRe Architecture
This solution uses a structured, stratified hierarchy built from three layers: Notes, Keywords, and Topics. This layered structure provides both a macro view of knowledge (Topics) and crucial micro-level retrieval points (Notes/Keywords), ensuring both broad context and specific data anchoring.
Iterative Evidence Refinement
This mechanism allows the AI to actively search for gaps in its knowledge rather than stopping after the first query. If an initial answer is incomplete, the system generates a new sub-query to find missing evidence, allowing it to build a complete narrative piece by piece.

Terminology

Summary

The paper introduces MemFly, a novel framework designed to address the fundamental dilemma in large language model (LLM) agents where efficient compression of long-term memory conflicts with maintaining the precise fidelity required for complex reasoning. By formalizing agentic memory as an Online Information Bottleneck (IB) problem, MemFly provides a unified, principled objective that minimizes representational complexity while maximizing task-relevant information. This approach is crucial for enabling LLMs to tackle complex tasks through historical interactions by moving beyond the limitations of traditional retrieval-centric or compression-heavy memory systems.

How it works: Information Bottleneck Optimization

MemFly models the construction of agentic memory as an Information Bottleneck (IB) optimization problem, seeking a policy pi that minimizes the Lagrangian cost: min LIB (Mt) = I(X 1:t; Mt) - beta I(Mt; Y). This framework utilizes an online greedy strategy, adapting the Agglomerative Information Bottleneck (AIB) algorithm to handle continuous input streams. The system maintains a stratified Note-Keyword-Topic hierarchy, where the goal is to compress redundant information and discard irrelevant details while preserving critical evidence.

How it works: Stratified Memory Structure

The memory state Mt is organized into three distinct layers, each serving a specific purpose in ensuring structural integrity and retrieval efficiency:

  1. Fidelity Layer (Notes N): These are the atomic units, defined as tuples containing raw observational data (ri) and an augmented context (ci). This layer preserves raw observational fidelity, mitigating hallucination risks.

  2. Anchoring Layer (Keywords K): Keywords serve as intermediate symbolic anchors, bridging continuous embedding spaces and discrete symbolic reasoning, which helps stabilize the semantic space and mitigate vector dilution.

  3. Navigation Layer (Topics T): Topics aggregate keywords into semantic centroids that partition the memory latent into navigable regions, enabling O(1) macro-semantic localization during retrieval.

How it works: On-the-Fly Consolidation

The system employs a computation-on-construction mechanism involving three primary operations driven by an LLM policy pi: Merge, Link, and Append. The LLM acts as a gradient-free (Yang et al., 2024) policy that approximates the Jensen-Shannon divergence through semantic assessment. These decisions are governed by two critical scores:

  • Redundancy Score (sred): Quantifies semantic overlap, where high redundancy triggers a Merge operation to integrate details from the New Node into the candidate’s context.

  • Complementarity Score (scomp): Measures logical or topical connections between distinct information, triggering a Link.

If neither threshold is met, the content is Appended, preserving distributional diversity.

How it works: Tri-Pathway Retrieval and Refinement

To access the optimized memory structure, MemFly utilizes a tri-pathway hybrid retrieval strategy that decomposes queries into semantic signals. This includes macro-semantic navigation via Topics (h topic), micro-symbolic anchoring via Keywords (H keys), and topological expansion along the E related edges. The final evidence pool is constructed using Reciprocal Rank Fusion (RRF) to prioritize consistent evidence. For complex, multi-hop queries, the system employs an Iterative Evidence Refinement (IER) protocol:

  • The system first evaluates if the current evidence pool E(i) is sufficient.

  • If not, a refined sub-query q(i+1) is synthesized to target missing information.

  • This process continues until the sufficiency predicate is met or the maximum iteration count (I) is reached, ensuring sufficient information is gathered for accurate reasoning.

Improvements for AI systems

Based on a thorough analysis of the paper MemFly: On-the-Fly Memory Optimization via Information Bottleneck, I have identified several critical, high-impact improvements that can be implemented immediately to enhance current AI agents and LLM systems.

These improvements are highly specific and move beyond simple retrieval augmentation into active, principled memory management.

Instead of relying on fixed token limits or simple static storage, the system should implement a continuous, on-the-fly memory evolution policy guided by the IB principle (I(X 1:t; M t - beta I(M t; Y)).

  • What this achieves: The system actively monitors incoming information streams (X 1:t) and dynamically decides whether to store, consolidate, or discard data to ensure maximum relevance for future tasks (Y) while minimizing memory redundancy.

  • Specific Mechanism: Implement an LLM-based gradient-free policy (pi) that approximates Jensen-Shannon divergence. This allows the system to calculate a Redundancy Score (sred) and a Complementarity Score (scomp) for any two pieces of information, enabling structural decisions based on informational content rather than mere semantic proximity.

The system must replace static storage with a dynamic update mechanism that applies three distinct structural operations based on the calculated scores:

  • A. Merge Operation (sred > tau m): When high redundancy is detected, merge the content into a unified context (c'i) while preserving all distinct information. This drastically reduces representational complexity and eliminates noise.

  • B. Link Operation (scomp > tau l): When information is complementary but distinct, establish an explicit associative edge between two memory units (e.g., Related to: [Keyword]). This preserves critical logical dependencies necessary for multi-hop reasoning without merging the content.

  • C. Append Operation (otherwise): When information is entirely novel or distinct, store it as an autonomous unit.

  • What this achieves: The system moves beyond simple summarization (which sacrifices fidelity) by intelligently deciding how to integrate new knowledge, ensuring that crucial contextual links are preserved even when the raw data is consolidated.

The memory structure must be organized into a three-layer hierarchy grounded in the Double Clustering Principle, providing multiple, distinct pathways for retrieval:

  • Layer 1: Notes (N, Fidelity): Store raw observational data (r i) alongside a semantically denoised summary (c i). This dual representation mitigates hallucination risks by preserving the original verbatim content.

  • Layer 2: Keywords (K, Anchoring): Extract symbolic anchors (keywords) from the Notes. These serve as distributionally robust feature spaces that stabilize semantic proximity and prevent vector dilution, providing precise, entity-centric retrieval.

  • Layer 3: Topics (T, Navigation): Aggregate keywords into macro-level semantic centroids (Topics). This allows for O(1) macro-semantic localization, enabling fast navigation through large memory spaces.

  • What this achieves: The AI system gains three independent avenues to find evidence—the precise entity match (Keywords), the general theme (Topics), and the logical connection (Topological expansion)—significantly increasing retrieval robustness.

The retrieval process must be decentralized and iterative, rather than a single vector search:

  • A. Tri-Pathway Search: Execute parallel traversals guided by the query's decomposed intent:

  1. Macro-Semantic Localization: Find relevant Topic centroids (T*).

  2. Micro-Symbolic Anchoring: Match query entities against the keyword index (K*).

  3. Topological Expansion: Traverse the explicit associative edges (ER ELATED) established during consolidation.

  • B. Iterative Evidence Refinement (IER): If the initial evidence pool is deemed insufficient by an LLM-based Reflector Agent, the system must synthesize a targeted sub-query to expand its search, repeating this process until sufficiency is achieved or maximum iterations are reached.

  • What this achieves: The system can solve complex multi-hop reasoning tasks that require synthesizing evidence across disparate memory units, even if the initial query vectors are weak. It ensures that the agent doesn't stop at the first close enough match, but actively seeks necessary context.

The improved AI system will be capable of:

  1. Maintaining high-fidelity long-term memory by actively consolidating redundant information (Merge) while preserving critical contextual links (Link).

  2. Achieving superior multi-hop reasoning by utilizing the structured, tri-pathway retrieval mechanism to synthesize evidence from different parts of a complex history.

  3. Minimizing retrieval noise and hallucination risk by maintaining raw verbatim data alongside semantically denoised summaries, and by using symbolic anchors (K) for precise entity matching.

  4. Adapting to continuous interaction streams without the limitations of fixed context windows, effectively treating long-term memory as a dynamic, evolving knowledge graph guided by the Information Bottleneck principle.

Sources

Related papers