Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing

arXiv:2608.24112 · cs.AI · Submitted 2026-08-25 · Read on arXiv

cs.AI

Submitted: 2026-08-25

Updated: 2026-08-25

Code: https://github.com/blackforest-labs/flux

Terminology

• Benchmark — 18× in this paper · explained in 2 episodes
• Aggregation — 8× in this paper · explained in 1 episode
• Transformer — 4× in this paper · explained in 2 episodes
• Alignment — 3× in this paper · explained in 1 episode
• Core — 3× in this paper · explained in 2 episodes
• Foundation Model — 3× in this paper · explained in 3 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Verification — 2× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Diffusion Models — 1× in this paper · explained in 6 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Rectified Flow — 1× in this paper · explained in 3 episodes
• Semantic Alignment — 1× in this paper · explained in 3 episodes

Sources

• Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist
• HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer
• HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer
• Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models
• FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark
• X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again
• EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation
• ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
• OctoT2I: A Self-Evolving Agentic Text-to-Image Router
• GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation
• GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation
• Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation
• Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?
• Cost-Aware Routing for Efficient Text-To-Image Generation
• Evaluating Text-to-Visual Generation with Image-to-Text Generation
• ERNIE-Image Technical Report
• LongCat-Image Technical Report
• PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models
• Image-POSER: Reflective RL for Multi-Expert Image Generation and Editing
• WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection