A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation

arXiv:2609.01315 · cs.AI · Submitted 2026-09-01 · Read on arXiv

cs.AI

Submitted: 2026-09-01

Updated: 2026-09-09

Comments: 12 pages, 3 figures. Code: https://github.com/naver-ai/omni-evaluator

Code: https://github.com/naver-ai/omni-evaluator

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 72× in this paper · explained in 2 episodes
• Verifier — 52× in this paper · explained in 2 episodes
• Foundation Model — 7× in this paper · explained in 3 episodes
• Accuracy — 6× in this paper · explained in 1 episode
• Adam — 6× in this paper · explained in 2 episodes
• Verification — 5× in this paper · explained in 2 episodes
• Provenance — 4× in this paper · explained in 4 episodes
• Trace — 3× in this paper · explained in 2 episodes
• Conference on Computer Vision and Pattern Recognition — 2× in this paper
• Foundation Models — 2× in this paper · explained in 7 episodes
• LLM judge — 2× in this paper · explained in 2 episodes
• Reasoning Traces — 2× in this paper · explained in 3 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 4 episodes
• Decoupling — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• Hallucination — 1× in this paper · explained in 4 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Modular Architecture — 1× in this paper · explained in 4 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes
• Verifiable Rewards — 1× in this paper · explained in 2 episodes
• Visual Grounding — 1× in this paper · explained in 2 episodes

Sources

• Lessons from the Trenches on Reproducible Evaluation of Language Models
• xVerify: Efficient Answer Verifier for Reasoning Model Evaluations
• MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
• Gemini: A Family of Highly Capable Multimodal Models
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• ANLS* -- A Universal Document Processing Metric for Generative Large Language Models
• HyperCLOVA X 8B Omni
• GPT-4o System Card
• Qwen3.5-Omni Technical Report
• Qwen2.5-Omni Technical Report
• Qwen3-Omni Technical Report

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection