OmniFysics-Captioner Technical Report: Grounding Omni-Modal Understanding in the Physical World for Better Captioning

arXiv:2609.31714 · cs.CV · Submitted 2026-09-21 · Read on arXiv

cs.CV

Submitted: 2026-09-21

Updated: 2026-09-21

Code: https://github.com/Fysics-AI/OmniFysics-Captioner

Terminology

• Benchmark — 78× in this paper · explained in 2 episodes
• Accuracy — 14× in this paper · explained in 1 episode
• Alignment — 13× in this paper · explained in 1 episode
• Grounding — 8× in this paper · explained in 3 episodes
• Ablation — 7× in this paper · explained in 1 episode
• Verification — 7× in this paper · explained in 2 episodes
• Aggregation — 6× in this paper · explained in 1 episode
• Commentary — 6× in this paper · explained in 2 episodes
• Consistency — 5× in this paper · explained in 3 episodes
• F1 Score — 5× in this paper · explained in 3 episodes
• Physical Perception Model — 5× in this paper
• Fine-Tuning — 4× in this paper · explained in 5 episodes
• Benchmarking — 3× in this paper · explained in 3 episodes
• Hallucination — 3× in this paper · explained in 3 episodes
• Unified Framework — 3× in this paper · explained in 5 episodes
• World Model — 3× in this paper · explained in 7 episodes
• Category-Aware Temporal Anchor Aggregation — 2× in this paper
• Instruction Tuning — 2× in this paper · explained in 2 episodes
• Active Perception — 1× in this paper · explained in 2 episodes
• Affective Computing — 1× in this paper · explained in 1 episode
• Causal Inference — 1× in this paper · explained in 4 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Momentum transfer — 1× in this paper · explained in 2 episodes
• Provenance — 1× in this paper · explained in 3 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Temporal Reasoning — 1× in this paper · explained in 2 episodes

Sources

• OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimization
• ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
• video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models
• AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration
• Active Perception Agent for Omnimodal Audio-Video Understanding
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
• Qwen2.5-Omni Technical Report
• Qwen3-Omni Technical Report
• Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding
• UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarks
• AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning
• Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities
• A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
• MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Models
• PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos
• PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models
• VideoMind: An Omni-Modal Video Dataset with Intent Grounding for Deep-Cognitive Video Understanding
• WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation
• Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models