SIMLIFE: Pattern Understanding for Long-Horizon Human-Agent Partnership
cs.AI
Submitted: 2026-09-17
Updated: 2026-09-17
Comments: COLM 2026 Learning from Situated and Embodied Interaction Workshop
Code: https://github.com/gkamradt/LLMTest_NeedleInAHaystack
Project page: https://roihn.github.io/SimLife
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Understanding humans over long horizons requires agents to infer not only what people need in the moment, but also how routines form, why they repeat, and when they change.
Terminology
Abstract
Understanding humans over long horizons requires agents to infer not only what people need in the moment, but also how routines form, why they repeat, and when they change. We introduce SimLife, a scalable platform for simulating long-term household life with rich visual observations, ground-truth action logs, and synthetic dialogues with audio. Built on SimLife, SimLife-BP evaluates long-context pattern understanding: the ability to infer latent behavioral rules from weeks or months of everyday observations. The benchmark contains 106 episodes averaging 15.49 hours and 38.57 in-game days, and 1,439 question-answer pairs. Each task probes direct, counterfactual, noisy, and inverse reasoning under different levels of rule hints. Evaluating frontier models and architectures, we find that current models often achieve surface-level prediction without comprehensive rule understanding, rely on frequency-based heuristics rather than if-then reasoning over evidence, and struggle to adapt when behavioral patterns change. These findings suggest that long-context pattern understanding remains a major bottleneck for future embodied agents, while SimLife opens a broader space for studying memory, personalization, adaptation, and long-horizon planning in everyday human-AI interaction.
Sources
- Qwen3-VL Technical Report
- Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents
- Longformer: The Long-Document Transformer
- Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline
- On the Measure of Intelligence
- ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
- Fast Spatial Memory with Elastic Test-Time Training
- Communication and Verification in LLM Agents towards Collaboration under Information Asymmetry
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds
- Gemma 3 Technical Report
- Qwen3-Omni Technical Report
- TeleEgo: Benchmarking Egocentric AI Assistants in the Wild
- MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?
- EgoLife: Towards Egocentric Life Assistant
- Test-Time Training Done Right
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection