Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling

arXiv:2605.13062 · cs.CV · Submitted 2026-05-13 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Edit-Compass & EditReward-Compass".

Tom: As a fastidious and diligent researcher, I have meticulously analyzed the provided excerpts from what appears to be a technical paper concerning image editing systems and reward modeling benchmarks.

Jane: First, who's behind it and why it matters.

Paper summary: Lu: Looking at the whole scope of "Edit-Compass and EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling," it seems they’ve created a tool that systematically addresses the dual challenges of evaluating image editing systems and their associated reward models.

Tom: The authors are focusing on creating this unified suite to provide a comprehensive framework that moves beyond the limitations of older, narrower benchmarks by covering six progressively challenging task categories in Edit-Compass.

Jane: In simple terms, they’re saying that we need better tools to see if an AI can actually edit images the way a human would want it edited, which involves checking both the final output and the underlying guidance mechanism.

Meng: From an engineering standpoint, I see this as a necessary step for anyone trying to build reliable systems because you can’t trust your optimization process if you don't have a solid way to test the reward component separately.

Lalam: My take is that this framework helps culture because it establishes a standardized vocabulary for what constitutes high-quality, reasoned image manipulation in AI, which could guide development across different teams.

Lu: And I think the implications are huge because by linking editing performance to reward model preference data using methods inspired by FlowGRPO and stochastic differential equations, they give us a multidimensional analysis capability that was previously missing.

Tom: The title itself, "Edit-Compass and EditReward-Compass," suggests they are providing two distinct but related ways to navigate the complex landscape of image editing and reward modeling evaluation.

Jane: They are essentially giving researchers a standardized way to compare different models by seeing not just the final pixels, but also how well their internal decision-making process aligns with desired outcomes.

Meng: It seems like this work has direct implications for practical AI development because it provides a more reliable metric than relying on simple automated metrics alone for assessing model progress in this area.

Lalam: The real impact could be in developing more robust AI systems where the editing capabilities are tightly coupled with accurate, human-aligned reward signals, leading to more intentional and useful image generation.

Conclusion: Tom: So we've been diving deep into how Edit-Compass and EditReward-Compass work, and now we're wrapping up by talking about what these tools actually mean for the field of image editing.

Jane: It’s a really important paper, Tom, because it brings together two previously separate evaluation systems into one comprehensive benchmark called Edit-Compass and EditReward-Compass.

Lu: I find the unification aspect fascinating; it suggests that we can finally evaluate the entire pipeline from raw manipulation all the way up to how a reward model guides that manipulation in a single framework.

Meng: From an engineering standpoint, having two distinct but linked benchmarks makes sense because you can test the model's output quality separately from its reinforcement learning strategy evaluation.

Lalam: I think this work has massive implications for culture because it sets a rigorous standard for what we consider high-quality, reasoned image manipulation in AI systems moving forward.

Tom: Exactly, Lalam, and that brings us to the title itself: "Edit-Compass and EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling." It really tells you exactly what this paper is about—creating a single place to measure both the editing performance and the reward modeling accuracy.

Jane: That title explains the core concept perfectly; they’re providing one consolidated way to test both the image generation aspect and how well those models are rewarded for doing things right.

Lu: It's creative because it moves beyond just looking at static images; they are building a dynamic system that evaluates complex, multi-step reasoning through these structured tasks.

Meng: I’m thinking about the practical impact: if we can reliably benchmark both the editing and the reward component, it helps us pinpoint exactly where an AI is failing—is it bad at following instructions or is its internal reward signal flawed?

Lalam: And that's where the real cultural shift happens; when we have this kind of unified tool, developers won't just be chasing pretty pictures anymore; they’ll be optimizing for verifiable reasoning and consistent performance across multiple dimensions.

Tom: So, it boils down to giving us a holistic view of these complex AI systems rather than just looking at isolated metrics.

Jane: Precisely, Tom; it gives us the context needed to truly understand the capabilities and weaknesses of these advanced image editing models.

Lu: The way they constructed EditReward-Compass using FlowGRPO inspired strategies really shows how deep they went into simulating realistic RL scenarios for this evaluation suite.

Meng: It’s impressive that they managed to bridge the gap between the visual outputs and the underlying reward signals so systematically within one benchmark structure.

Lalam: This work opens up a pathway where we can build image editing AI that isn't just creative, but genuinely follows complex, multi-faceted human intent in a consistent way.

Tom: And that sets us up perfectly for our next topic: we’re going to look at some of the specific results they found when comparing different types of models on this new benchmark.

Xuehai Bai, Yang Shi, Yi-Fan Zhang, Xuanyu Zhu, Yuran Wang, Yifan Dai, Xinyu Liu, Yiyan Ji

HDU Institute of Technology Development University of Petroleum and Mining University Kling Team CASIA

cs.CV

Submitted: 2026-05-13

Updated: 2026-10-05

Code: https://github.com/bxhsort/Edit-Compass-and-EditReward-Compass

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: As a fastidious and diligent researcher, I have meticulously analyzed the provided excerpts from what appears to be a technical paper concerning image editing systems and reward modeling benchmarks.

Key concepts

Edit-Compass
This is the core benchmark for image editing systems. It contains 2,388 annotated tasks covering general manipulations, dynamic actions, world knowledge reasoning (like math and cause/effect), and multi-image tasks. It uses five metrics—Instruction Following, World Knowledge Awareness, Unedited Region Consistency, Identity Consistency, and Visual Quality—to score system performance.
EditReward-Compass
This benchmark evaluates reward models used in Reinforcement Learning for image editing. It contains 2,251 preference pairs simulating real RL scenarios. It is constructed using Edit-Compass data to simulate sampling via FlowGRPO and stochastic differential equations, allowing researchers to analyze reward model performance across various dimensions.
Instruction Following (IF)
This metric assesses how accurately the image editing system executes the specific instructions given in a prompt. High IF means the model successfully performs exactly what was asked, which is crucial for practical application and task completion in complex editing scenarios.
World Knowledge Awareness (WA)
This measures a model's understanding of real-world facts and relationships relevant to the image editing task. It tests advanced cognitive skills such as temporal, causal, math, and chemical reasoning, ensuring the system can reason beyond simple visual manipulation.

Terminology

Summary

As a fastidious and diligent researcher, I have meticulously analyzed the provided excerpts from what appears to be a technical paper concerning image editing systems and reward modeling benchmarks. The goal is to synthesize these fragmented pieces into a comprehensive, detailed summary of the work presented in "Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling."

Here is the combined, detailed summary:

This research introduces a unified benchmark suite comprising Edit-Compass and EditReward-Compass, designed to provide a comprehensive, human-aligned framework for rigorously evaluating frontier image editing systems and their associated reward models. The work addresses fundamental limitations in current state-of-the-art models by focusing on deep reasoning capabilities, world knowledge understanding, and complex multi-image manipulation.

Edit-Compass serves as the core evaluation suite for image editing systems. It is characterized by its extensive coverage and fine granularity:

  • Task Coverage: Edit-Compass contains 2,388 carefully annotated instances spanning six progressively challenging task categories. These tasks are designed to probe a wide spectrum of capabilities, including world knowledge reasoning, visual reasoning, and multi-image editing.

  • Task Taxonomy: The evaluation is structured across diverse categories:

  • General Tasks: Covering fundamental manipulations such as Subject Addition, Subject Removal, Subject Replacement, Subject Extraction, Change Color & Size (material), and Change Material.

  • Dynamic Manipulation Tasks: Focusing on object-level actions like Object Movement, Object Swap, and Object Interaction.

  • World Knowledge Reasoning Tasks: Testing advanced cognitive skills through Temporal Reasoning, Causal Reasoning, Math Reasoning, and Chemical Reasoning.

  • Algorithmic Visual Reasoning Tasks: Including Longest Word Discovery and Knapsack Selection.

  • Multi-Image Tasks: Assessing capabilities such as Multi-Image Awareness and Virtual Try-On.

  • Complex Tasks: Addressing high-level instructions like Complex Instruction Following and Complex Paint operations.

  • Evaluation Framework: Beyond broad coverage, Edit-Compass employs a fine-grained multidimensional evaluation framework. This framework is built upon structured reasoning and carefully designed scoring rubrics, utilizing five key metrics:

  1. Instruction Following (IF)

  2. World Knowledge Awareness (WA)

  3. Unedited Region Consistency (URC)

  4. Identity Consistency (IC)

  5. Visual Quality (VQ)

In parallel with the image editing benchmark, EditReward-Compass is designed to systematically evaluate reward models used during Reinforcement Learning (RL) optimization for these editing tasks.

  • Preference Data: It contains 2,251 preference pairs, which are constructed to simulate realistic reward modeling scenarios encountered during RL optimization. Each pair consists of an editing instruction and two candidate edited images.

  • Simulation Fidelity: To ensure the simulation is realistic, EditReward-Compass is constructed using Edit-Compass as its source data. The construction process simulates the sampling process via a FlowGRPO-inspired strategy [22] and introduces necessary stochasticity through stochastic differential equations [36]. This methodology allows for multidimensional analysis of reward models across crucial dimensions: instruction following, visual consistency, perceptual quality, and reasoning-aware editing preference.

  • Annotation Rigor: The annotation process is highly rigorous. It involves eight human experts in image editing. The process is staged: an initial stage where three annotators independently review sampled outputs to create candidate pairs (flagging and resolving ambiguities through discussion), followed by a second stage where five annotators conduct fine-grained verification of the selected pairs, ensuring both task validity and preference correctness. This protocol achieves a high correlation with expert ratings, validating its reliability.

The combined evaluation suite yields several critical insights into the current landscape of image editing technology:

  • Performance Gap: The results reveal a substantial performance gap between proprietary (closed-source) and open-source systems. Specifically, Nano Banana Pro [7] was noted as performing best among closed-source models, showing consistent advantages across task categories. Among open-source models, QwenImage-Edit [48] achieved the best overall performance under both English and Chinese instructions, leveraging a 20B diffusion transformer integrated with a 7B Qwen-VL model.

  • Persistent Weaknesses: Fine-grained analysis exposes persistent weaknesses across the board in multi-image understanding, world knowledge awareness, and visual reasoning, even for frontier models.

Improvements for AI systems

As a fastidious researcher, I have analyzed the core contributions of "Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling." The paper establishes a rigorous, human-aligned framework for evaluating frontier image editing models and their associated reward models.

Here are the specific improvements that can be made to AI systems by leveraging this benchmark suite:

The primary improvement is the transition from coarse-grained or unrealistic evaluation to a comprehensive, multidimensional assessment grounded in human judgment. This allows for targeted, high-impact improvements across four distinct capability axes.

The improved AI system can achieve the following specific capabilities:

  2. Enhanced Reasoning and Knowledge Integration (World Knowledge & Complex Reasoning):

  4. High-Fidelity Visual Manipulation (Algorithmic & Dynamic Manipulation):

  6. Precise Multimodal Composition and Cross-Image Transfer:

Detailed breakdown of improvements based on Edit-Compass capabilities:

  1. A system can perform complex edits that require deep, real-world understanding, such as predicting future states (What changes will occur after two hours?), inferring past states from current appearance (Temporal Reasoning), or reasoning under external conditions (Causal Reasoning).

  2. The system can execute intricate, multi-step instructions that integrate various concepts—for example, combining a Style Transfer with a Subject Replacement based on specific world knowledge (e.g., replacing a statue with jade)—achieving high scores in the Complex Instruction category where atomic task decomposition is strictly enforced.

  3. The system can perform highly precise object manipulation, such as executing actions (Action), swapping objects while maintaining spatial relationships (Object Swap), or modeling physical interactions between multiple subjects (Object Interaction) with high fidelity, moving beyond simple attribute changes to dynamic scene understanding.

  4. The system can execute tasks requiring deep visual reasoning over structured data, such as identifying the optimal path in a grid (Optimal Path Identification), finding the maximum value submatrix in a grid (Maximum Submatrix Sum Identification), or performing complex geometric reconstructions like finding the convex hull of a point set.

  5. The system can seamlessly integrate information from multiple input images to perform sophisticated composition, such as accurately composing scenes by transferring specific attributes (like pose or garment) from one image onto another (Multi-Image Awareness and Virtual Try-On).

  6. The system can execute highly controlled, multimodal text modifications, such as precisely editing Chinese or English text within an image while maintaining the original font style and color (Visual Text Editing), ensuring high adherence to complex visual guidance markers (Complex Paint).

The improvement in Reward Models stems from using EditReward-Compass:

A reward model can be trained in a realistic RL environment where it must compare candidate edits generated by the same model under the same instruction, thereby learning to prioritize not just surface-level changes, but those that satisfy complex criteria like Instruction Adherence, Visual Consistency (URC/IC), and Visual Quality (VQ).

This allows the AI system to be optimized for human-aligned outcomes rather than simply maximizing a generic loss function.

Abstract

Recent image editing models have achieved remarkable progress in instruction following, multimodal understanding, and complex visual editing. However, existing benchmarks often fail to faithfully reflect human judgment, especially for strong frontier models, due to limited task difficulty and coarse-grained evaluation protocols. In parallel, reward models have become increasingly important for RL-based image editing optimization, yet existing reward model benchmarks still rely on unrealistic evaluation settings that deviate from practical RL scenarios. These limitations hinder reliable assessment of both image editing models and reward models. To address these challenges, we introduce Edit-Compass and EditReward-Compass, a unified evaluation suite for image editing and reward modeling. Edit-Compass contains 2,388 carefully annotated instances spanning six progressively challenging task categories, covering capabilities such as world knowledge reasoning, visual reasoning, and multi-image editing. Beyond broad task coverage, Edit-Compass adopts a fine-grained multidimensional evaluation framework based on structured reasoning and carefully designed scoring rubrics. In parallel, EditReward-Compass contains 2,251 preference pairs that simulate realistic reward modeling scenarios during RL optimization.

Sources

Related papers