CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video
cs.AI, cs.CV
Submitted: 2026-09-15
Updated: 2026-09-15
Comments: EMNLP 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Wearable assistants require episodic memory over egocentric video, yet current vision-language models face bounded frame budgets, growing visual-token costs, and long-context retrieval failures.
Terminology
Abstract
Wearable assistants require episodic memory over egocentric video, yet current vision-language models face bounded frame budgets, growing visual-token costs, and long-context retrieval failures. Under these practical constraints, we study whether textual captions can serve as reusable episodic memory. We define the Episodic Memory Video Caption QA task and introduce CapMem, a human-annotated benchmark with 75 videos totaling 33.7 hours, and 1,000 multiple-choice questions across 16 scenarios. On long videos (>20 min), full-coverage CaptionQA with 30s and 60s caption windows outperforms direct VideoQA for 10/12 and 8/12 models, respectively. On the same video subset, a matched-frame control across six Qwen models retains mean accuracy gains of 3.22 and 2.55 points, respectively. Our caption-guided retrieve-and-verify harness further improves accuracy by up to 5.3 points. These results support the effectiveness of caption memory for episodic reasoning over long egocentric video.
Sources
- Qwen3-VL Technical Report
- LabOS: The AI-XR Co-Scientist That Sees and Works With Humans
- Project Aria: A New Tool for Egocentric Multi-Modal AI Research
- Small Vision-Language Models are Smart Compressors for Long Video Understanding
- VideoVista: A Versatile Benchmark for Video Understanding and Reasoning
- World Model on Million-Length Video And Language With Blockwise RingAttention
- EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding
- CaptionQA: Is Your Caption as Useful as the Image Itself?
- Watch Before You Answer: Learning from Visually Grounded Post-Training
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection