OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

arXiv:2610.12458 · cs.CV · Submitted 2026-10-08 · Read on arXiv

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Code: https://github.com/Jingkang50/FunQA

Project page: https://01yzzyu.github.io/OmniCapBench

Terminology

• Benchmark — 48× in this paper · explained in 2 episodes
• Hallucination — 36× in this paper · explained in 3 episodes
• Core — 20× in this paper · explained in 2 episodes
• Grounding — 20× in this paper · explained in 3 episodes
• Alignment — 17× in this paper · explained in 1 episode
• Consistency — 14× in this paper · explained in 3 episodes
• Verification — 14× in this paper · explained in 2 episodes
• Benchmarking — 9× in this paper · explained in 3 episodes
• LLM judge — 9× in this paper · explained in 2 episodes
• Failure Modes — 7× in this paper · explained in 3 episodes
• Fidelity — 7× in this paper · explained in 2 episodes
• Reinforcement Learning — 5× in this paper · explained in 4 episodes
• Structural Integrity — 5× in this paper · explained in 3 episodes
• Trace — 5× in this paper · explained in 2 episodes
• Transformer — 5× in this paper · explained in 2 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Cross-Shot Coreference Consistency — 4× in this paper
• Decoupling — 3× in this paper · explained in 2 episodes
• F1 Score — 3× in this paper · explained in 3 episodes
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• Computer Vision and Pattern Recognition — 2× in this paper
• Foundation Model — 2× in this paper · explained in 3 episodes
• Foundation Models — 2× in this paper · explained in 7 episodes
• Multi-agent Collaboration — 2× in this paper · explained in 2 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Structural Constraints — 2× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Actor — 1× in this paper · explained in 3 episodes
• Analyzer — 1× in this paper · explained in 1 episode
• Failure Taxonomy — 1× in this paper · explained in 2 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Granularity Mismatch — 1× in this paper · explained in 2 episodes
• Graph Construction — 1× in this paper · explained in 3 episodes
• Introspection — 1× in this paper · explained in 2 episodes
• Iterative Refinement — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Multi-Agent System — 1× in this paper · explained in 6 episodes
• Multimodal Reasoning — 1× in this paper · explained in 5 episodes
• Structural Reasoning — 1× in this paper · explained in 2 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
• Temporal consistency — 1× in this paper · explained in 2 episodes

Sources

• Qwen2.5-VL Technical Report
• Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity
• TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
• AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
• AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration
• DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Models
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
• VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
• Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction
• VGI-Bench: Probing Visual Intelligence in Video Generation Models
• WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs
• Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
• Baichuan-Omni Technical Report
• Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
• Ola: Pushing the Frontiers of Omni-Modal Language Model
• Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models