Quantifying the Gap between Understanding and Generation within Unified Multimodal Models

A bidirectional benchmark, GAPEVAL, was introduced to quantify and measure the gap between understanding and generation capabilities within Unified Multimodal Models (UMMs), revealing that current models achieve only surface-level unification rather than deep cognitive convergence.

GAPEVAL Benchmark Design

GAPEVAL is a bidirectional benchmark where each question can be answered through either text or image, establishing a fair and symmetric testbed for UMMs. Each item consists of an image (or None) and a corresponding text instruction pair, denoted as texti = und texti, gen texti. The ground truth varies across task categories: for understanding tasks, the ground-truth yu is a reference textual answer; for generation tasks, the answer yg consists of a reference image accompanied by descriptive text. This design allows for the evaluation of model performance on identical underlying knowledge and quantify the misalignment between its understanding and generation capabilities.

Taxonomy of Tasks

The benchmark encompasses four categories: Instruction Following, Numerical Perception, World Knowledge, and Reasoning. Each task is designed to test specific aspects of UMMs:

1. Instruction Following evaluates whether UMMs can follow instructions by incorporating both explicit edits (requiring articulation of changes) and implicit edits (examining the ability to follow underlying rules), assessing cross-modal consistency of instruction-grounded editing.

2. Numerical Perception focuses on interpreting and manipulating quantitative information, requiring models to accurately perceive numerical structure and consistently express the corresponding quantitative modification across modalities.

3. World Knowledge tests a model’s ability to recognize, reason, and apply broad world knowledge when grounded in visual inputs, demanding that models map visual evidence to the appropriate realworld entities.

4. Reasoning assesses comprehension of textual and visual reasoning problems through subcategories such as Image Selection, Knowledge Selection (rendering knowledge into images), Real-world Reasoning (establishing physical contexts), and Logical Reasoning (performing visual symbolic reasoning).

Gap Quantification Metric

To quantify the modality gap, a specific metric called the Gap Score is proposed, grounded in Multidimensional Item Response Theory (MIRT). This two-stage evaluation methodology first measures capability correctness using GPT-5-mini as a judge model to assign binary labels. The second stage uses MIRT to introduce item difficulty and model ability as latent variables, allowing the framework to continuously and sensitively reflect the nuanced performance variations between different model levels and varying task difficulties. The final capability gap is normalized between 0 and 100.

Knowledge Manipulation Analysis

Empirical studies from the perspective of knowledge manipulation investigate why UMMs lack genuine cross-modal consistency. These experiments involve injecting or editing knowledge entities, defined as a tuple (subject, relation, object), where both subject and object can be text or image. Results reveal that knowledge within UMMs often remains disjoint, with findings indicating that Training on one side fails to generalize to the other. This suggests that knowledge representations are non-unified, and unbalanced training introduces modality-specific knowledge drift, where updates in one space fail to propagate coherently to the other.

Synergy and Conclusion

The paper argues that true synergy relies on alignment, stating that alignment is the structural prerequisite for Synergy. A strong negative correlation was observed between Gap Score and performance on Synergy Benchmarks (GIR-Bench), confirming that narrowing the gap is a prerequisite for real-world applications. The findings conclude that current models achieve only surface-level unification, necessitating a next-level unification to achieve deep cognitive convergence. Furthermore, the study shows that higher performance does not always imply stronger unification, revealing a decoupling between state-of-the-art performance and the degree of unification.

Data Construction Details

The data collection is systematic, integrating human curation with automated generation across four knowledge manipulation scenarios: Knowledge Injection (introducing novel concepts), Knowledge Editing (altering existing conceptual associations), and various task subsets. For instance, in Knowledge Injection, entities are selected based on their low public prominence to test model novelty. In Knowledge Editing, a counter-factual label assignment is used to force the model to overwrite its internal alignment between the visual and textual modalities. The datasets are constructed bidirectionally for both VQA and T2I components to ensure robust analysis of knowledge manipulation within UMMs.

Metric Implementation

The detailed metric implementation employs a Bayesian maximum a posteriori (MAP) framework, optimizing the joint log-likelihood L(Θ, β) to estimate latent ability vectors θi and difficulty parameters β. The capability gap is then computed as Gabs(∆θi) = ∆θi / (1 + ∆θi), which represents the normalized absolute capability gap. A post-hoc reward–penalty on the gap uses observed co-occurrence statistics to adjust the logit space, encouraging consistency when capable and penalizing simultaneous failures.

Based on the scientific paper Quantifying the Gap between Understanding and Generation within Unified Multimodal Models, here are specific, actionable improvements for AI systems:

1. Improve Cross-Modal Consistency through Bidirectional Benchmarking (GAPEVAL):

2. Implement Knowledge Manipulation for Deep Cognitive Integration:

3. Develop a Metric-Driven Gap Quantification System (Gap Score/MIRT):

4. Enhance Model Training via Asymmetric Knowledge Alignment:

5. Improve Cross-Modal Consistency through Bidirectional Benchmarking (GAPEVAL):

Incorporate the GAPEVAL framework into the training and fine-tuning pipeline of Unified Multimodal Models (UMMs). This moves beyond single-direction benchmarks by requiring models to consistently perform under both text and image modalities for every query.

The improved system can:

• Identify models that achieve surface-level unification versus those demonstrating deep cognitive convergence.

• Provide a quantitative, symmetric assessment of a model's bidirectional inference capability (e.g., testing if understanding an image leads to the correct generation, and vice versa).

6. Implement Knowledge Manipulation for Deep Cognitive Integration:

Leverage the findings from Section 4 (Empirical Analysis on UMMs Gap) by systematically injecting or editing knowledge entities within UMMs via single-sided manipulation (both text and image modalities).

The improved system can:

• Diagnose the underlying mechanism of knowledge decoupling—determining whether knowledge representations are truly shared or disjoint across modalities.

• Identify where modality-specific learning dynamics cause knowledge drift, allowing for targeted architectural modifications to enforce cross-modal knowledge propagation during training.

7. Develop a Metric-Driven Gap Quantification System (Gap Score/MIRT):

Replace simple accuracy scores with the proposed Gap Score, grounded in Multidimensional Item Response Theory (MIRT), which models both model ability and item difficulty as latent variables.

The improved system can:

• Provide a continuous, sensitive measure of the understanding-generation gap normalized between 0 and 100.

• Automatically distinguish failures caused by limited model capability versus those caused by high item difficulty (overcoming biases from simple averaging).

• Use Bayesian MAP optimization to jointly fit model ability and item difficulty, leading to a more robust and nuanced performance evaluation than current methods.

8. Enhance Model Training via Asymmetric Knowledge Alignment:

Utilize the findings from Section 5 (Training Expense) regarding the asymmetric convergence dynamics where understanding improves faster than generation during fine-tuning. Implement training strategies that explicitly address this lag, such as:

• For models like Bagel, use joint training when single-modality fine-tuning causes catastrophic loss of capability in the other modality.

• For knowledge editing, employ counterfactual label assignment to force the model to overwrite its internal alignment between visual and textual modalities simultaneously.

The improved system can:

• Achieve more balanced performance across both understanding and generation tasks by optimizing for mutual enhancement rather than functional coupling.

• Reduce the training budget required for effective knowledge adaptation by recognizing that generative pathways require a substantially larger optimization effort to internalize new information compared to comprehension tasks.

Recent advances in unified multimodal models (UMM) have demonstrated remarkable progress in both understanding and generation tasks. However, whether these two capabilities are genuinely aligned and integrated within a single model remains unclear. To investigate this question, we introduce GapEval, a bidirectional benchmark designed to quantify the gap between understanding and generation capabilities, and quantitatively measure the cognitive coherence of the two "unified" directions. Each question can be answered in both modalities (image and text), enabling a symmetric evaluation of a model's bidirectional inference capability and cross-modal consistency. Experiments reveal a persistent gap between the two directions across a wide range of UMMs with different architectures, suggesting that current models achieve only surface-level unification rather than deep cognitive convergence of the two. To further explore the underlying mechanism, we conduct an empirical study from the perspective of knowledge manipulation to illustrate the underlying limitations. Our findings indicate that knowledge within UMMs often remains disjoint. The capability emergence and knowledge across modalities are unsynchronized, paving the way for further exploration.