MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

arXiv:2609.40195 · cs.CV, cs.AI, cs.CL · Submitted 2026-09-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories".

Tom: Long-term egocentric video enables personalized AI assistants to reason about daily life, but reprocessing raw clips for every query becomes computationally prohibitive.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So we’ve talked about the title and authors of MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories, which points directly at building a system that organizes and understands personal video history. The team behind it is Meta Reality Labs, the University of Virginia, and the University of Illinois Urbana-Champaign.

Jane: It’s important to understand that this paper is proposing a multimodal memory system designed specifically for long-term egocentric video processing where reprocessing raw clips for every single question becomes impossible. Essentially, they are addressing the "how do we remember everything without running out of time or computing power?" problem.

Lu: The authors clearly set out to solve the scaling issue that current memory systems have encountered, moving away from just compacting videos into text representations which often fail on practical benchmarks regarding evidence preservation and retrieval accuracy.

Meng: So they are proposing a system where the memory construction and the memory access are separated, which is a smart architectural move if we’re looking at how to make this work in an operational environment.

Lalam: I think that separation is crucial because it lets them focus on making sure the memory itself is high quality, and then they have a specialized reader to pull out exactly what the user needs without wading through everything.

The paper's summary: Tom: Moving into the summary of MemLife, we see that the system constructs entity-grounded, first-person text episodes from these video segments and then retrieves them using a timeindexed agentic reader. This means it’s not just storing data; it’s building stories about what happened in a specific place or with specific people.

Jane: The core idea here is that the system uses two main principles during memory writing: first, anchoring memories in time and grounding entities across modalities so they connect spoken references to real people and places; second, narrating in the first person, which matches how users actually phrase questions about their own lives.

Lu: That first-person narration is a clever way to narrow the gap between what the user thinks they want to ask and what the memory system can actually find through agentic retrieval.

Meng: From an engineering standpoint, this means we aren't relying on simple keyword matching anymore; we’re aiming for a richer semantic understanding embedded in the text representation itself during construction.

Lalam: I see this as a way to make the AI feel more intuitive and personalized because it’s not just spitting out facts; it’s recalling an experience from your perspective, which is really powerful for building trust.

The paper's improvements: Tom: The paper highlights several improvements they suggest, particularly around how the memory writer and reader work together. They propose that MemLife’s reader combines agentic semantic search with time-scoped memory fetching and presents those retrieved episodes in chronological order for reasoning over long histories.

Jane: That chronological ordering is key because it allows the AI to reason about sequences of events, not just isolated facts, which is something standard retrieval methods often struggle with when dealing with long histories.

Lu: They also move away from relying solely on heuristic prompt engineering for memory construction and instead focus on optimizing memory generation using task feedback, which I think points toward a more robust training methodology.

Meng: I’m paying attention to the reader's action space; having actions like Rewrite, SearchMemory, FetchMemory, and FetchVideo gives it enough flexibility to handle complex queries that require multiple steps.

Lalam: It sounds like they are creating a very deliberate retrieval process rather than a passive search, which should significantly improve the precision of what the AI pulls up when you ask it something specific.

Conclusion: Tom: To wrap things up with MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories, the authors show that their multimodal memory system can perform better than prior state-of-the-art systems on long-horizon benchmarks without needing query-time video access.

Jane: The overall implication is that we can build scalable alternatives to constantly reprocessing raw video for every question, provided we design the memory construction and retrieval around entity grounding and first-person narrative principles.

Lu: They demonstrate that learning what to remember is an effective approach because they move toward optimizing the memory writer using a reinforcement learning framework focused on faithfulness, informativeness, and retrievability.

Meng: The combined system, MemLife plus the optimization framework, shows gains of four point zero to seventeen point zero percent over prior baselines and actually reduces memory size by up to thirty-one times in some cases; that level of compression is what we need for real deployment.

Lalam: It really shows that optimizing the memory writer alone can yield significant gains across different systems and distributions, which suggests a very generalizable approach for improving how AI assistants learn from experience.

Tom: So, MemLife provides the structure for creating persistent episodic memory, and MemOpt provides the mechanism to optimize that construction through multi-granular feedback on faithfulness, informativeness, and retrievability. It’s a solid foundation for personalized long-term video question answering.

Guangzhi Xiong, Xinyuan Zhang, Xiao Yang, Hyokun Yun, Kai Zhang, Shiun-Zu Kuo, Hyeonjeong Ha

Meta Reality Labs

cs.CV, cs.AI, cs.CL

Submitted: 2026-09-30

Updated: 2026-09-30

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Long-term egocentric video enables personalized AI assistants to reason about daily life, but reprocessing raw clips for every query becomes computationally prohibitive.

Key concepts

MemLife System
This system constructs persistent episodic memories from egocentric videos. It generates textual descriptions for each segment by anchoring entities across modalities and narrating in the first person. A reader then uses agentic semantic search and time-scoped fetching to retrieve these episodes for reasoning.
Entity Grounding
This involves aligning spoken references in the video with specific people and places mentioned in the memory segment. It ensures that when a user asks about a memory, the system correctly identifies which entities are being discussed, improving accuracy by linking speech to concrete entities.
MemOpt Framework
This is a reinforcement learning framework that optimizes how MemLife writes memories. It uses the FIRM reward system to train the writer to produce memories that are faithful (not hallucinated), informative (contain key facts), and retrievable (easy to find later).
FIRM Reward System
This is a set of rewards used in MemOpt training. It specifically rewards three qualities: faithfulness, which checks if claims are supported by the source video; informativeness, which checks if key facts are preserved; and retrievability, which ensures the memory is easily found during retrieval.

Terminology

Summary

Long-term egocentric video enables personalized AI assistants to reason about daily life, but reprocessing raw clips for every query becomes computationally prohibitive. This work introduces MemLife, a multimodal memory system that constructs entity-grounded, first-person text episodes and retrieves them via a timeindexed agentic reader, and MemOpt, a reinforcement learning framework that optimizes the memory writer to produce faithful, informative, and retrievable memories.

MemLife System for Memory Writing and Reading

MemLife separates question-independent memory construction from question-dependent memory access. The writer generates a textual description for each memory segment using two core principles: first, it anchors memories in time and grounds entities across modalities, aligning spoken references with people and places; second, it narrates in the first person, matching how users phrase questions about their own lives. To exploit these memories, MemLife’s reader combines agentic semantic search with time-scoped memory fetching and presents retrieved episodes in chronological order for reasoning over long histories. The reader operates by selecting actions from an action space that includes Rewrite, SearchMemory, FetchMemory, and FetchVideo.

MemOpt Framework for Optimizing Memory Writer

MemOpt is a learning framework that optimizes the memory writer parameters through the FIRM reward system, which rewards Faithful, Informative, and Retrievable Memories. The theoretical foundation decomposes the uncertainty about an answer into two gaps: memory writing loss (informativeness gap) and memory recall loss (retrievability gap). The framework defines three specific rewards:

  1. Faithfulness: Penalizes hallucinated memories unsupported by the source video, assessed by checking if every claim is supported by its source segment.

  2. Informativeness: Encourages the writer to preserve details critical to answering future questions, measured by testing whether a candidate memory entails a source-grounded key fact.

  3. Retrievability: Favors concise descriptions that enable easy and precise retrieval, measured by checking whether the relevant memory is returned during QA time.

Experimental Results and Generalizability

MemLife improves over the strongest training-free baseline by 4.6–12.0% across four long-horizon benchmarks without query-time video access. When MemOpt is applied to improve the writer, it consistently improves MemLife by 2.7–5.0%. The combined system, MemLife + MemOpt, outperforms the strongest prior baseline by 4.0–17.0%, while reducing memory size by up to 31× (Appendix E). The trained writer demonstrates strong transferability: it consistently improves accuracy when plugged into other memory systems (e.g., EgoRAG) and backbones, and on out-of-domain videos and question types.

Ablation Studies and Component Importance

Ablation studies confirm the importance of specific components in both the writer and reader designs. For the writer, entity grounding generally improves accuracy, while first-person narration further improves accuracy. For the reader, time anchoring in the search tool provides its clearest benefit on optimized EgoLifeQA memories, and chronological ordering further improves accuracy despite small or mixed recall changes. The analysis of reward dynamics during MemOpt training shows that retrievability and informativeness improve for longer, indicating that the writer continues to preserve more answer-relevant evidence without a sustained trade-off in retrievability.

Conclusion

The paper demonstrates that learning what to remember is an effective and generalizable approach to long-term video question answering. MemLife provides the structure for constructing persistent episodic memory, while MemOpt provides the mechanism to optimize that construction through multi-granular feedback on faithfulness, informativeness, and retrievability. These results show that optimizing the memory writer alone yields significant gains across different systems and distributions. The final results indicate that learning what to remember is an effective and generalizable approach to long-term video question answering. This work establishes a scalable alternative to raw video reprocessing for personalized AI assistants.

The gist: MemLife introduces an agentic memory system that constructs time- and entity-anchored, first-person episodes from egocentric videos, while MemOpt optimizes the memory writer using a FIRM reward system to ensure memories are faithful, informative, and retrievable.

How it works

MemLife constructs persistent episodic memory by generating textual descriptions for each video segment based on three designs: multimodal fusion (jointly interpreting frames and transcript), entity grounding (aligning speech with the memory segment using named entities), and first-person narration (adopting a first-person perspective). The reader is an agentic system that takes a question and timestamp as input and operates over multiple rounds by selecting actions from an action space, including SearchMemory, FetchMemory, and FetchVideo.

MemOpt Framework for Optimizing Memory Writer

MemOpt updates the writer parameters through supervision using the FIRM reward model.

Improvements for AI systems

Here are the specific improvements and capabilities derived from the MemLife/MemOpt framework:

  1. The system gains a robust, long-term episodic memory by converting raw egocentric video streams into structured, time-anchored text episodes that are grounded in both visual and audio modalities.

  2. The system can answer complex questions about past life experiences (e.g., Where did I leave my ring last month?) by utilizing an agentic reader that performs a flexible search and retrieval strategy over this memory.

  3. The memory system is optimized using the MemOpt framework, which employs a Reinforcement Learning (RL) approach focused solely on the writer to produce memories that are:

  4. Faithful: Directly supported by the source video evidence, eliminating hallucinations.

  5. Informative: Preserve all details critical for answering future questions.

  6. Retrievable: Concise enough to be found precisely when needed during a query, ensuring low retrieval competition in growing search spaces.

Abstract

Long-term egocentric video enables personalized AI assistants to reason about daily life. However, as video histories grow to hundreds of hours spanning months or years, reprocessing raw clips for every query becomes computationally prohibitive. Memory systems offer a scalable alternative by compacting videos into text representations, but often fail on practical benchmarks: either the memory does not preserve key evidence, or the retriever fails to locate relevant entries due to retrieval competition in growing search spaces. To address these challenges, we introduce MemLife, a multimodal memory system that constructs entity-grounded, first-person text episodes and retrieves them via a time-indexed agentic reader. Without training or query-time video access, MemLife improves over the strongest training-free baseline by 4.6--12.0% across four long-horizon benchmarks. To further improve memory quality, we propose MemOpt, a reinforcement learning framework that optimizes the memory writer to produce faithful, informative, and retrievable memories. MemOpt consistently improves MemLife by 2.7--5.0% across different video and question distributions, with gains that generalize across writer and reader backbones and memory systems.

Sources

Related papers