UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory

arXiv:2608.30508 · cs.CL · Submitted 2026-08-31 · Read on arXiv

cs.CL

Submitted: 2026-08-31

Updated: 2026-08-31

Code: https://github.com/peijunallin/UtilMem

Terminology

• Benchmark — 45× in this paper · explained in 2 episodes
• Robustness — 18× in this paper · explained in 5 episodes
• Hallucination — 14× in this paper · explained in 4 episodes
• Consistency — 9× in this paper · explained in 3 episodes
• Benchmarking — 5× in this paper · explained in 4 episodes
• RAG — 4× in this paper · explained in 3 episodes
• Accuracy — 3× in this paper · explained in 1 episode
• Agent Memory — 3× in this paper · explained in 1 episode
• Core — 3× in this paper · explained in 2 episodes
• Degradation Rate — 3× in this paper
• LLM judge — 3× in this paper · explained in 2 episodes
• Aggregation — 2× in this paper · explained in 1 episode
• Fidelity — 2× in this paper · explained in 2 episodes
• Grounding — 2× in this paper · explained in 3 episodes
• Normalized Robustness — 2× in this paper
• Verification — 2× in this paper · explained in 2 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Abstention — 1× in this paper · explained in 1 episode
• Abstraction — 1× in this paper · explained in 1 episode
• Adaptation — 1× in this paper · explained in 1 episode
• Agentic Memory System — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Calibration — 1× in this paper · explained in 7 episodes
• Clustering — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• In International Conference on Learning Representations — 1× in this paper
• Knowledge Graph — 1× in this paper · explained in 7 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Multi-hop Reasoning — 1× in this paper · explained in 3 episodes
• Pairwise Comparisons — 1× in this paper · explained in 2 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Resource Constraints — 1× in this paper · explained in 3 episodes
• Robustness Score — 1× in this paper
• Semantic Similarity — 1× in this paper · explained in 2 episodes
• Temporal Reasoning — 1× in this paper · explained in 2 episodes
• Transformer — 1× in this paper · explained in 2 episodes

Sources

• RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction
• HaluMem: Evaluating Hallucinations in Memory Systems of Agents
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
• Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding
• Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization
• EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning
• Memory OS of AI Agent
• DialSim: A Dialogue Simulator for Evaluating Long-Term Multi-Party Dialogue Understanding of Conversational Agents
• MemOS: A Memory OS for AI System
• Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems
• LightMem: Lightweight and Efficient Memory-Augmented Generation
• LightRAG: Simple and Fast Retrieval-Augmented Generation
• The StudyChat Dataset: Analyzing Student Dialogues With ChatGPT in an Artificial Intelligence Course
• A Survey of Context Engineering for Large Language Models
• Nomic Embed: Training a Reproducible Long Context Text Embedder
• MemGPT: Towards LLMs as Operating Systems
• Convomem Benchmark: Why Your First 150 Conversations Don't Need RAG
• Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents
• Zep: A Temporal Knowledge Graph Architecture for Agent Memory

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering