Social Caption: Evaluating Social Understanding in Multimodal Models

arXiv:2601.14569 · cs.CL, cs.LG · Submitted 2026-01-21 · Read on arXiv

cs.CL, cs.LG

Submitted: 2026-01-21

Updated: 2026-08-31

Code: https://github.com/abwilf/Social-IQ-2

Terminology

• Accuracy — 16× in this paper · explained in 1 episode
• Alignment — 16× in this paper · explained in 1 episode
• Social Inference — 12× in this paper
• Directed Social Analysis — 9× in this paper
• Holistic Social Analysis — 9× in this paper
• Benchmark — 7× in this paper · explained in 2 episodes
• Actor — 5× in this paper · explained in 3 episodes
• Annotator Agreement — 5× in this paper · explained in 1 episode
• F1 Score — 5× in this paper · explained in 3 episodes
• Adam — 3× in this paper · explained in 2 episodes
• Consistency — 3× in this paper · explained in 3 episodes
• Core — 3× in this paper · explained in 2 episodes
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• Foundation Model — 3× in this paper · explained in 3 episodes
• Foundation Models — 3× in this paper · explained in 7 episodes
• Benchmarking — 2× in this paper · explained in 3 episodes
• Conference on Computer Vision and Pattern Recognition — 2× in this paper
• Instruction Tuning — 2× in this paper · explained in 2 episodes
• Prompt Adherence — 2× in this paper · explained in 2 episodes
• Reasoning Traces — 2× in this paper · explained in 3 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Context Length — 1× in this paper
• Embeddings — 1× in this paper · explained in 2 episodes
• Evaluation Dimensions — 1× in this paper · explained in 3 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Multimodal Reasoning — 1× in this paper · explained in 5 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes

Sources

• Qwen2.5-VL Technical Report
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
• GPT-4o System Card
• SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning
• Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution
• Social Genome: Grounded Social Reasoning Abilities of Multimodal Models
• MR. Judge: Multimodal Reasoner as a Judge
• Learning Transferable Visual Models From Natural Language Supervision
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• EgoNormia: Benchmarking Physical Social Norm Understanding
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling
• Qwen2.5-Omni Technical Report
• VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understanding
• Qwen2 Technical Report
• MiniCPM-V: A GPT-4V Level MLLM on Your Phone
• Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models
• Sigmoid Loss for Language Image Pre-Training

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering