Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models

As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge. Current evaluation protocols predominantly treat generative and discriminative capabilities as separate tasks, leaving a gap in system-level evaluation for unified multimodal models (UMMs). The paper states: "Existing evaluations commonly assess generation and understanding using separate metrics... These component-wise evaluations are important for diagnosing specific strengths and weaknesses. However, they do not directly provide a unified assessment of how well a UMM performs when understanding and generation are both involved within the same model."

The authors note that "two UMMs may exhibit opposite capability profiles, with one stronger in generation and the other stronger in understanding. Separate scores can reveal these differences, but they do not directly indicate whether the model can function well when both abilities must interact within a unified process."

The paper proposes Self-Generative-Understanding (SGU), described as a novel, annotation-free evaluation framework that probes the integrated capabilities of unified models through a semantic closed-loop challenge. The framework operates as follows:

1. Visual description stage: The understanding function MU perceives the original visual input v and generates a textual description tg.

2. Visual generation stage: Based solely on the self-generated description tg, the generation function MG synthesizes a proxy visual context v̂.

3. Self understanding stage: The model then uses MU to resolve the original textual task on the synthesized image v̂ and produces the final response t̂.

The process is formalized as: tg = MU(v), v̂ = MG(tg), t̂ = MU(v̂, t)

The SGU score is calculated as: summ = E(v,q,a)∼D [I(Match(MU(MG(MU(v))), q), a)]

The paper also introduces a relative SGU score normalized by the base understanding score: summ,r = summ/sbase, where sbase measures VQA performance on the original images and serves as a model-specific upper-bound reference.

The paper highlights three main advantages of SGU:

1. Integrated capability evaluation: "The closed-loop process activates multiple capabilities of UMMs, including image understanding, textual representation, visual generation, and downstream reasoning, without relying on external judge models or additional generation systems."

2. System-level sensitivity: The final score reflects the combined effect of these stages: information loss in understanding, intermediate representation, generation, or self-reasoning will reduce the final performance.

3. Annotation-free scalability: SGU can be instantiated on existing image-grounded reasoning benchmarks, such as MMBench, by reusing their original questions and ground-truth answers, requiring no new annotations.

The paper evaluates SGU on four benchmarks: MMStar, MMBench, MathVista, and OCR-VQA, using six representative UMMs: Janus-Pro-7B, BAGEL-7B, UniWorld-V1, Show-o2-7B, OmniGen2, and Ovis-U1-3B.

All stages are executed in a stateless manner: "the same unified model is reused across stages, but each stage runs in an isolated session with no memory or intermediate features preserved across steps. Throughout the evaluation, no external judge models or human annotators are introduced."

Q1: System-level view beyond isolated evaluations. The results show a consistent drop from direct VQA accuracy on the original image (sbase) to the SGU score (summ). Importantly, "models with similar sbase can exhibit very different SGU scores. For example, on OCR-VQA, UniWorld-V1 achieves sbase=81.33 but drops to summ=28.67, while OmniGen2 has a comparable sbase=79.39 but retains a much higher summ=56.59."

Q2: Joint effects of understanding and generation. Stage-wise replacement experiments (replacing understanding with Qwen3-VL-8B and generation with Qwen-Image-2512) show that "both interventions change SGU performance, confirming that the score is affected by both understanding and generation. Replacing the generation stage yields larger positive gains across models, suggesting that visual reconstruction is a major bottleneck in the current SGU loop."

Q3: Prompt sensitivity. Testing alternative prompts for understanding and generation stages showed results stay close to the default score, suggesting that SGU is insensitive primarily to reasonable prompt choices within the loop.

Q4: Shortcut exploitation. An image-swap test pairing models with similar direct VQA accuracy found Cross remains broadly comparable to Self across pairs, with only minor and non-systematic differences, suggesting no consistent shortcut signals that materially influence SGU performance.

The paper includes a text-bottleneck analysis showing that increasing the caption limit from 256 to 512 tokens on OCR-VQA with Janus-Pro-7B changes caption-only QA accuracy only marginally, from 49.50 to 49.53, suggesting degradation is not simply due to token limits.

A caption-only QA comparison with OmniGen2 on OCR-VQA shows: direct VQA on the original image achieves 79.39, caption-only QA obtains 60.05, and the full SGU loop obtains 56.59, demonstrating that both understanding-side information extraction and generation contribute to the final score.

The paper concludes: "SGU evaluates whether a model can operate through its own understanding-and-generation process and complete tasks grounded in the original images, providing a complementary system-level view rather than replacing component-wise evaluations. The authors note that even high-performing UMMs often struggle to reason over their own generated contexts, revealing limitations that are not captured by separate evaluations of understanding or generation alone."

The paper acknowledges: "SGU is a system-level evaluation signal, not a standalone theoretical criterion for UMM alignment. Its score naturally aggregates effects from multiple stages, so fine-grained failure attribution should be performed with stage-wise analyses and component-level metrics. The current implementation also uses text as an intermediate representation, which may introduce information bottlenecks for detail-intensive tasks and fine-grained visual evidence."

Based on the SGU framework, here are specific improvements I can implement in AI systems:

Improvement: Add a closed-loop self-check mechanism where the model generates a description of an input, reconstructs the visual from that description, and verifies whether its original understanding is preserved through the cycle.

Capability: The AI can detect when its own understanding is lossy or incomplete by measuring the semantic gap between original input and reconstructed output. This enables automatic flagging of uncertain or degraded reasoning, improving reliability in high-stakes applications like medical imaging or autonomous navigation.

Improvement: Train the model to minimize information loss when compressing visual input into textual descriptions, specifically optimizing for reconstructability rather than just descriptive accuracy.

Improvement: Incorporate a training objective where the understanding module is fine-tuned to reason effectively over images that were generated from its own descriptions, not just real images.

Improvement: Implement an adaptive intermediate representation selector that dynamically chooses between text, structured scene graphs, or latent embeddings based on task complexity and detail requirements.

Improvement: Build a diagnostic layer that decomposes SGU-style failures into understanding loss, generation loss, and reasoning loss, providing stage-specific error signals during inference.

Improvement: Use SGU-style scores as a training reward signal, specifically penalizing models that achieve high direct accuracy but low closed-loop consistency.