SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue
cs.CL, cs.AI, cs.IR, cs.LG
Submitted: 2026-09-22
Updated: 2026-09-23
Comments: Project Page: https://2022hpsk.github.io/SpeakerMemR1 , Code: https://github.com/2022hpsk/SpeakerMemR1
Code: https://github.com/2022hpsk/SpeakerMemR1
Project page: https://2022hpsk.github.io/SpeakerMemR1
License: http://creativecommons.org/licenses/by/4.0/
The gist: Long-term conversational memory in multi-party settings requires more than retrieving relevant content from long-term conversations: it must distinguish who said what, whom each statement concerns,
Terminology
Abstract
Long-term conversational memory in multi-party settings requires more than retrieving relevant content from long-term conversations: it must distinguish who said what, whom each statement concerns, how individuals perceive one another, what information is shared by the group, and how states change over time. Recent studies on multi-party dialogue benchmarks show that existing general-purpose LLM memory systems tend to lose person and group relations or struggle to integrate clues distributed across members, groups, and time. Together, these issues reveal two core bottlenecks: message attribution and relational understanding in multi-party dialogue, and state reconstruction from interleaved histories. To address both, we propose SpeakerMem-R1: its dual-track memory stores speaker-labeled verbatim messages and derived states organized into person-level and group-level views, then combines evidence from both tracks by entity, event, and time at query time. To reduce attribution and update errors during structured memory construction while enabling local deployment, we train Writer-R1 with SpeakerLevenshtein and speaker-conditioned GRPO. On GroupMemBench, SocialMemBench, and EverMemBench, SpeakerMem-R1 achieves binary accuracies of 47.9%, 69.2%, and 61.9%, respectively. On the publicly reported EverMemBench leaderboard from EverMind-AI, we achieves 62.33%, the best reported result among the latest state-of-the-art frameworks. It also achieves 70.85% on all 1,986 LoCoMo questions, which we use as a two-person long-term conversation boundary test. In a controlled evaluation of 305 questions, RL raises the SFT Writer's mean accuracy from 57.38% to 68.20%. We report both binary accuracy and token-F1, and ablations show that the verbatim and structured tracks, as well as person-level and group-level views, are complementary under the standardized evaluation interface.
Sources
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- LightRAG: Simple and Fast Retrieval-Augmented Generation
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
- RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory
- REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
- MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models
- MemOS: A Memory OS for AI System
- Joint Optimization of Multi-agent Memory System
- SocialMemBench: Are AI Memory Systems Ready for Social Group Settings?
- MemGPT: Towards LLMs as Operating Systems
- Zep: A Temporal Knowledge Graph Architecture for Agent Memory
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turns
- R^2-Mem: Reflective Experience for Memory Search
- MIRIX: Multi-Agent Memory System for LLM-Based Agents
- Mem-{\alpha}: Learning Memory Construction via Reinforcement Learning
- Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents
- GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations
- DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Agent Memory
- DeltaMem: Towards Agentic Memory Management via Reinforcement Learning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering