Auditing Long-Term Memory Evaluation: Repeated Judging, Reader Variation, and Negative Controls

arXiv:2609.38021 · cs.CL, cs.AI, cs.IR · Submitted 2026-09-29 · Read on arXiv

cs.CL, cs.AI, cs.IR

Submitted: 2026-09-29

Updated: 2026-10-02

Code: https://github.com/JordanMcCann/agentmemory

Terminology

• Benchmark — 21× in this paper · explained in 2 episodes
• Abstention — 15× in this paper · explained in 1 episode
• Verifier — 8× in this paper · explained in 2 episodes
• Consistency — 4× in this paper · explained in 3 episodes
• Self-consistency — 4× in this paper · explained in 2 episodes
• Provenance — 2× in this paper · explained in 4 episodes
• Verification — 2× in this paper · explained in 2 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Accuracy — 1× in this paper · explained in 1 episode
• Agent Memory — 1× in this paper · explained in 1 episode
• Agentic Memory System — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Core — 1× in this paper · explained in 2 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Knowledge Graph — 1× in this paper · explained in 7 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Trace — 1× in this paper · explained in 2 episodes

Sources

• Chronos: Temporal-Aware Conversational Agents with Structured Event Retrieval for Long-Term Memory
• Agent Zero Memory: Provenance-Aware Long-Term Memory for LLM Agents
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
• MemGPT: Towards LLMs as Operating Systems
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
• Zep: A Temporal Knowledge Graph Architecture for Agent Memory
• Evaluating Very Long-Term Conversational Memory of LLM Agents
• Passage Re-ranking with BERT
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering