CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation

arXiv:2606.15867 · cs.CV · Submitted 2026-06-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation".

Tom: CogCanvas introduces a comprehensive benchmark designed to systematically evaluate multi-subject reference-based image generation by simultaneously testing identity preservation, object/fashion binding, and background scene consistency.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Hey Jane, I'm really excited about this paper we're looking at today, it's called "CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation," and it seems to tackle a problem that existing benchmarks completely miss.

Jane: I agree Tom, the title itself tells you exactly what’s important here—it’s setting up a proper test for when an AI needs to generate pictures with several people all interacting in a specific setting. It sounds like they are trying to get past the old way of testing things that only handle one person at a time.

Lu: What strikes me immediately is how CogCanvas sets up its data, pairing one thousand nine hundred fifty-two reference images with prompts involving two to five different people, objects, and backgrounds all at once. That complexity is where most current models struggle when you try to do it in one go.

Meng: So the core idea seems to be creating a rigorous test that checks identity preservation, object binding, and background consistency simultaneously, rather than testing them separately on different small datasets. That sounds like a much harder challenge for any model to meet.

Lalam: From my side, this benchmark structure is really interesting because it forces the AI to handle the compounding difficulties of spatial arrangement and attribute fidelity across multiple subjects in a single generation attempt. It’s like moving from solving individual puzzles to solving a whole complex scene puzzle at once.

Tom: Exactly, and that complexity is what makes this paper so significant because it shows how models fail when they have to juggle all those things at the same time, not just one by one. So, CogCanvas introduces a way to systematically measure these failures across different group sizes.

Jane: That’s right; the paper explains that previous benchmarks only looked at these axes in isolation, which means they didn't capture how much harder it actually gets when you move from two subjects to five subjects. It really highlights a bottleneck in how we’ve been evaluating multi-subject generation.

Lu: The structure they use for the prompts, formalizing it as a tuple with background reference and interaction graphs, gives them a very precise way to define what success looks like in this complex space. It’s not just asking for an image; it’s asking for a specific spatial layout with certain objects tied to certain people within a defined background.

Title and authors: Meng: I wonder how the evaluation metrics they introduce, like BG-Sim and Attr-VQA, actually help us diagnose *why* something went wrong when the generation failed, beyond just looking at the final picture quality.

Lalam: Those new metrics are really powerful because they move beyond just looking at how pretty an image is and start checking if specific attributes and interactions were actually bound correctly through structured questions, which is a much more concrete way to measure fidelity.

Tom: It sounds like the authors are showing us not just what's wrong, but giving us the tools to pinpoint exactly where the AI breaks down when things get complicated. So, they’re essentially building a map of failure modes for multi-subject generation.

Jane: That’s a huge step because it moves the conversation from general performance scores to understanding specific failure modes like attribute leakage or background ignoring. It makes it much easier to understand the underlying mechanics of these models.

Lu: They also introduce a training-free baseline called Sequential Reference Inpainting, which is designed specifically to suppress that cross-subject reference leakage by inpainting one subject at a time. That suggests they have found a way to manage the difficulty of multiple references sequentially.

Meng: As an engineer, I’m interested in that training-free approach; if it can suppress leakage by separating the generation into passes, that has huge implications for how we might structure our generation pipeline right now.

Lalam: If we think about the broader impact, this paper helps us see that simply making models bigger or better at one thing doesn't automatically solve the problem of managing multiple references coherently. It forces a more thoughtful approach to scaling these visual generation systems.

Tom: So, when we look at the results, it’s quite sobering because they show that as you increase the number of subjects from two to five, almost every single axis degrades sharply. That scaling collapse is a key finding.

Jane: And that scaling collapse is what really shows us the limits of current methods; for instance, object and fashion binding starts collapsing beyond three subjects in these evaluations. It’s a clear signal that models aren't robust yet at higher group sizes.

Title and authors: Lu: The results quantify this collapse quite clearly, showing that for DINO-Sim, four out of the five prior methods score below two point seven at four subjects and drop below zero point five when you go up to five subjects. That level of quantitative evidence is really helpful for the community.

Meng: Quantifying that specific collapse means we can start targeting those weak points in our own model architectures, instead of just guessing where the problem lies. It moves us from general failure to specific fix targets.

Lalam: And the trade-off they observed between identity preservation and prompt layout adherence is also important; sometimes improving one part actually makes another part worse, which adds another layer of complexity. It shows that optimization isn't always straightforward.

Tom: So, to wrap up the main findings, CogCanvas demonstrates that current methods consistently ignore the background reference and suffer from attribute-person binding collapse as the group size grows. It’s a clear demonstration of where we need to focus our research efforts next.

Jane: This paper really provides a high-fidelity testbed, but the authors are also upfront about its limitations; they mention risks like identity overlap with pretraining corpora and their reliance on a single multimodal LLM family for evaluation, which means human-in-the-loop auditing is still necessary to avoid bias.

Lu: It’s a fair limitation; relying on one specific type of LLM for verification does introduce potential bias in how we measure the results, which is something we have to keep in mind as we look at future evaluation frameworks.

Meng: That means any future work needs to be careful about the tools used for evaluation, not just the generation model itself; it's a holistic problem. It’s not just about making the generator better; it’s about making the entire verification process more reliable.

Lalam: I think this paper really pushes us toward a future where we have comprehensive diagnostic tools, like those structured VQA queries, to verify interaction graphs and spatial layouts, which is a massive step for how we validate AI outputs. That kind of verification is what will make these systems trustworthy in real applications.

Tom: So, in summary, CogCanvas gives us a rigorous way to test multi-subject reference-based image generation by checking identity, object binding, and background consistency together. It highlights that current models struggle significantly as the number of subjects increases beyond three.

Title and authors: Jane: Exactly; the implications are that we need a new way to benchmark these complex scenes because the old methods are failing to capture how difficult it is to maintain all those details at once. It gives us a clear roadmap for where the next generation of models needs to focus their training efforts.

Lu: From a creative standpoint, I see the potential for this framework to inspire entirely new ways of composing complex scenes from reference inputs, moving beyond simple concatenation into truly structured multi-agent visual synthesis. It opens up a lot of conceptual territory.

Meng: On the practical side, if we can use this benchmark to systematically find out that attribute leakage collapses at five subjects, we can design specific regularization techniques to target that exact failure point in our next iteration. That's actionable engineering.

Lalam: For culture and application, this research suggests that the goal should be to build systems where the AI isn't just making pretty pictures, but where it reliably understands and executes complex instructions involving multiple entities in a specific context. That level of reliability is what makes these tools useful for real-world personalization.

Tom: So, to wrap up our discussion on "CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation," we see a very clear picture of the challenges involved in scaling these kinds of generation tasks. It’s a valuable tool for anyone working on complex visual composition.

Jane: It is certainly a valuable tool because it forces us to look at the whole system—the people, the objects, and the scene—as an integrated unit that needs to be managed correctly. We'll be watching how these results shape what we try next in AI research.

Lu: I think this benchmark really pushes the boundaries of what we consider a complete visual generation task; it demands coherence across multiple complex constraints simultaneously. The potential for using this structure to guide future generative architectures is huge.

Meng: I’ll be thinking about how to implement a system that can use these evaluation metrics dynamically during the training process, rather than just testing at the end, because that seems like the next logical step for practical deployment.

Lalam: Ultimately, this work shows us that achieving reliable multi-subject generation means mastering not just individual components but the entire interaction and context of the scene. It sets a high bar for what we need from generative AI in complex visual tasks.

The paper's summary: Tom: So, CogCanvas is basically setting up a super detailed testing ground to see how well image generation AI handles creating scenes with multiple people, objects, and backgrounds all at once.

Jane: That's right; it’s building this comprehensive benchmark that checks identity keeping, object binding, and scene consistency all in one go. It really addresses the problem where previous tests only looked at these things separately.

Lu: What I find fascinating is how they structure the prompts using interaction graphs and specific references for each subject, which gives them a very rigorous way to define success in this complex visual space. It moves beyond just asking for a picture to demanding a specific spatial layout with certain people interacting in a defined background.

Meng: From my side, I'm thinking about how this structured approach helps us pinpoint where the AI is failing when things get complicated, which is something we need for practical deployment.

Lalam: The most impactful vision I see from this paper is that it provides the diagnostic toolkit needed to move past just looking at visual quality and actually verify if specific attributes and interactions are correctly bound through structured questions. That kind of verification can really help improve how we design these systems in a way that makes them more reliable for real-world use.

Tom: Exactly, Lalam, that structured verification is what separates this work from other image generation benchmarks because it’s not just about how good the final image looks; it’s about proving the underlying logic of the generation.

Jane: It means we can start understanding exactly why models struggle when you go from two subjects to five, and they show a clear pattern where attribute binding collapses as the group size increases.

Lu: That scaling collapse is what really tells us that current methods aren't robust enough to handle this level of complexity consistently across different numbers of people.

Meng: I’m also looking at the baseline they introduce, Sequential Reference Inpainting, which sounds like a training-free way to try and stop those references from leaking across subjects by handling them one at a time.

Lalam: That baseline suggests there might be an architectural strategy—a sequential approach—that can help manage the difficulty of multiple references without needing massive retraining efforts for every new configuration.

Tom: So, CogCanvas isn't just showing us where models fail; it’s giving us a structured way to diagnose those failures and suggesting potential paths forward, like using sequential generation techniques to suppress leakage.

Jane: It sounds like the real value here is moving the conversation from just "is this picture pretty?" to "did the AI correctly handle all these complex constraints simultaneously?"

Lu: And that shift in focus toward structured fidelity, using metrics like BG-Sim and Attr-VQA, opens up a whole new area for research into how we can measure reasoning capabilities in visual synthesis.

Meng: If we can use this benchmark to systematically find out exactly where attribute leakage collapses at four or five subjects, then we can design specific regularization techniques to target those weaknesses in our own model architectures.

Lalam: I think the cultural impact of this kind of reliability is huge because it suggests that future AI systems could be trusted more when they're dealing with complex scenarios involving multiple entities and specific contexts.

Tom: It really does; CogCanvas gives us a solid foundation for figuring out how to build generative systems that aren't just making cool pictures but are actually capable of reliable scene construction.

Jane: We’ll be watching closely how the community uses these structured evaluation metrics, because they seem like the best way to move toward building more robust and trustworthy visual AI tools.

The paper's improvements: Tom: So, CogCanvas isn't just stopping at measuring problems; it’s actually laying out some concrete ways to fix those issues in future research.

Jane: That’s right; the authors are suggesting practical improvements, like using a training-free method called Sequential Reference Inpainting to manage those pesky cross-subject references by inpainting one person at a time.

Lu: I think that sequential approach is really clever because it tackles the leakage problem directly rather than trying to force one massive model to handle five subjects perfectly from the start. It’s a way of breaking down the hard problem into easier, manageable steps.

Meng: If that sequential decomposition actually works in practice and helps suppress leakage, then it gives us a really tangible strategy for deploying multi-subject generation tools without having to wait for massive, slow retraining cycles every time we want to add another person.

Lalam: The implication here is that we might shift from building one monolithic model trying to do everything at once toward a pipeline of specialized generation steps, which could make the whole process more stable and predictable.

Tom: That sounds like a major shift in how we think about training these models; moving away from one huge black box to a series of coordinated, simpler operations seems like a very sensible direction.

Jane: It means that for applications where you need high fidelity across many subjects, this idea of sequential refinement could be the next logical step in our development process.

Lu: And I think the authors’ focus on structured prompts with ground-truth interaction graphs is another key improvement; it shows that specifying *how* people interact is as important as just knowing who they are and what they're wearing.

Meng: That grounding in interaction data, paired with those new verification metrics like Attr-VQA, means the system isn't just guessing; it’s being checked against a known spatial relationship, which is much more reliable for engineering specifications.

Lalam: From a cultural perspective, if AI can reliably generate scenes where people are positioned and interacting in a specific way—like demonstrating complex social dynamics—it opens up possibilities for creating highly nuanced virtual environments that feel much more realistic.

Tom: So, the paper is suggesting we focus on two things: using techniques like sequential inpainting to control reference leakage, and demanding better structural grounding through interaction data during prompt creation.

Jane: Exactly; it’s a dual approach—improving the generation process itself while also improving how we guide that generation with structured information.

Lu: It really pushes the idea that success in multi-subject synthesis depends less on just brute-force model size and more on how coherently the model can handle complex, multi-layered constraints.

Meng: I’m interested in seeing how the authors actually implement this sequential pass; we need to know if it adds significant overhead or if it's a worthwhile trade-off for better control.

Lalam: The potential impact is that this framework could lead to AI systems that are not only visually convincing but also logically coherent in their composition, which is essential for serious applications in digital media and simulation.

Conclusion: Tom: So, to wrap up our talk on "CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation," we see that this research provides a really solid way to test AI when it has to manage multiple subjects and complex scene details simultaneously.

Jane: It really boils down to showing us that previous testing methods simply weren't equipped to handle the compounding difficulties of generating scenes with several distinct individuals interacting in a specific environment.

Lu: I think the core contribution is setting up this rigorous framework, which forces the AI to prove its capability across identity, object binding, and background consistency all at once. It’s about demanding coherence under pressure.

Meng: From my side, it’s clear that this benchmark gives us specific data points to target in our own model architecture when we’re trying to improve reliability for more complex tasks involving multiple agents.

Lalam: The most significant implication I see is that this work provides the necessary diagnostic tools—like those structured verification queries—to move beyond just judging a picture's aesthetics and actually confirming if the AI has understood the underlying spatial relationships.

Tom: Right, Lalam, that’s exactly what we need; verifiable accuracy over mere pretty visuals. The results show that as the number of subjects increases, things break down systematically, which is super valuable information for engineers.

Jane: It shows us a clear scaling challenge: current methods struggle to maintain attribute binding when you go beyond three subjects, and that's a concrete limitation we can work against.

Lu: And the paper’s suggestion of using techniques like sequential inpainting to suppress reference leakage is a really creative way to think about solving that scalability issue without needing an entirely new kind of model.

Meng: I’m keen to see if implementing that sequential approach actually provides a practical speedup or if the overhead makes it too slow for real-world use cases.

Lalam: Ultimately, this research points toward a future where AI systems in complex scenarios can be trusted more because we have metrics that verify their understanding of context and interaction, which could improve how people interact with these tools daily.

Tom: So, CogCanvas gives us a clear roadmap for what the next generation of multi-subject generation models needs to focus on: structured grounding and robust handling of scaling complexities.

Jane: It’s a fantastic piece of work because it forces the community to look at multi-subject generation as an integrated challenge rather than just a collection of smaller, independent tasks.

Lu: We're really excited about how this benchmark opens up new avenues for creative composition and spatial reasoning in generative AI.

Meng: I’m looking forward to seeing how other teams use these metrics to drive their iterative improvements on the generation pipeline.

Long-Bao Nguyen, Quang-Khai Le, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le

University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · University of Dayton, Ohio

cs.CV

Submitted: 2026-06-14

Updated: 2026-09-30

Code: https://github.com/black-forest-labs/flux

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 92/100

The gist: CogCanvas introduces a comprehensive benchmark designed to systematically evaluate multi-subject reference-based image generation by simultaneously testing identity preservation, object/fashion

Key concepts

Multi-subject Reference-based Generation: This is the task being tested: creating a single image that accurately depicts 2 to 5 different people, each matching a specific reference photo provided by the user. The goal is to ensure all subjects look correct and interact logically within one scene.
Identity Preservation (ID-Sim): This metric measures how well the model keeps the faces and core identities of each person consistent across multiple generations. A high score means that if you ask for a specific person, they look like that same person in every generated image, preventing identity leakage.
Attribute & Interaction Fidelity (Attr-VQA): This uses a large language model to check the details of the scene. It verifies if the attributes of each subject (like clothing or accessories) are correct and if their spatial relationships and interactions with objects are accurately represented in the final image.

Terminology

Benchmark — 67× in this paper · explained in 2 episodes
Fidelity — 26× in this paper · explained in 2 episodes
Verification — 20× in this paper · explained in 2 episodes
Consistency — 14× in this paper · explained in 3 episodes
Alignment — 13× in this paper · explained in 1 episode
Accuracy — 11× in this paper · explained in 1 episode
Grounding — 11× in this paper · explained in 3 episodes
Benchmarking — 9× in this paper · explained in 3 episodes
Sequential Reference Inpainting — 9× in this paper
Human-in-the-Loop — 7× in this paper · explained in 3 episodes
Cosine Similarity — 6× in this paper · explained in 3 episodes
Diffusion Model — 6× in this paper · explained in 8 episodes
Diffusion Models — 6× in this paper · explained in 6 episodes
Failure Modes — 6× in this paper · explained in 3 episodes
Personalization — 6× in this paper · explained in 2 episodes
Core — 4× in this paper · explained in 2 episodes
Embeddings — 3× in this paper · explained in 2 episodes
Generalization — 3× in this paper · explained in 13 episodes
In International Conference on Learning Representations — 3× in this paper
Semantic Alignment — 3× in this paper · explained in 3 episodes
Conference on Computer Vision and Pattern Recognition — 2× in this paper
Diagnostic Tools — 2× in this paper · explained in 2 episodes
Generative AI — 2× in this paper · explained in 2 episodes
Latent Diffusion Model — 2× in this paper · explained in 2 episodes
Latent Diffusion Models — 2× in this paper · explained in 2 episodes
Regularization — 2× in this paper · explained in 3 episodes
Transformer — 2× in this paper · explained in 2 episodes
Affordance — 1× in this paper · explained in 1 episode
Decoupling — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Flow Matching — 1× in this paper · explained in 7 episodes
Formal Verification — 1× in this paper · explained in 2 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 2 episodes
Latent Space — 1× in this paper · explained in 7 episodes
Rectified Flow — 1× in this paper · explained in 3 episodes
Semantic Similarity — 1× in this paper · explained in 2 episodes
Trace — 1× in this paper · explained in 2 episodes
Unified Framework — 1× in this paper · explained in 5 episodes

Summary

CogCanvas introduces a comprehensive benchmark designed to systematically evaluate multi-subject reference-based image generation by simultaneously testing identity preservation, object/fashion binding, and background scene consistency. This work is significant because existing benchmarks evaluate these axes in isolation, failing to capture the compounding difficulties models face when generating scenes with 2–5 distinct individuals while maintaining complex spatial arrangements and attribute fidelity. CogCanvas addresses this bottleneck by pairing a curated set of reference images with structured compositional prompts that include ground-truth interaction and position graphs, providing a rigorous framework for assessing the performance of state-of-the-art diffusion models in this challenging domain.

Benchmark Composition and Structure

CogCanvas is built around three organizing principles: diversity, multireference complexity, and logic-focused evaluation. The dataset comprises 1,952 curated reference images spanning 100 celebrity identities (Person), 80 unique objects (Object), 35 fashion items (Fashion), and 29 distinctive background scenes (Background). Compositional prompts involve sampling a set of N ∈ N = 2–5 identities, each paired with specific object/fashion references, grounded in a specified background scene. This structure is formalized as the tuple (B, [Hi] N i=1, [Oj] M j=1, [Rj] M j=1), where B is the background reference and Rj specifies human–object interaction.

Evaluation Metrics and Ground Truth

The benchmark scores generation across six complementary dimensions: Identity preservation (ID-Sim), Object & fashion attribute binding (DINO-Sim), Text-image semantic alignment (CLIP-T), Background-scene consistency (BG-Sim), Perceptual aesthetic quality (Aesth.), and Attribute & interaction fidelity. Crucially, CogCanvas introduces two new metrics: BG-Sim, which measures background fidelity on foregroundmasked regions via SAM 3 masks and DINOv3 features; and Attr-VQA, which uses a multimodal LLM (Gemma-4) to verify per-subject attribute binding and interaction graphs through structured visual question-answering queries. These metrics allow for fine-grained verification of interaction fidelity and spatial layout accuracy via structured VQA queries, rather than relying solely on perceptual quality.

Baseline and Comparison with State-of-the-Art Methods

The paper introduces Sequential Reference Inpainting (SRI), a training-free baseline designed to suppress cross-subject reference leakage by inpainting one subject per pass. The experimental setup compares SRI against five state-of-the-art methods: OmniGen2, UNO, DreamO, XVerse, and MOSAIC. Results show that across all evaluated methods, every axis degrades sharply from N = 2 to 5 subjects, with object/fashion binding collapsing beyond three subjects. SRI leads on the two axes that isolate reference grounding (ID-Sim and Attr-VQA) at every group size, showing a 2.5× margin in attribute binding against the best prior method at N=2.

Key Findings on Scaling Challenges

The experimental results systematically quantify failure modes as group size increases. For DINO-Sim, four of the five prior methods fall below 2.7 at N = 4 and below 0.5 at N = 5, indicating a failure to bind specific object and fashion attributes when subjects exceed three. Furthermore, Attr-VQA scores drop to near-zero for most methods at N=5, confirming that attribute leakage becomes catastrophic at scale. A recurring trade-off observed is between identity preservation (ID-Sim) and prompt layout adherence (CLIP-T), where methods achieving higher ID-Sim scores tend to sacrifice CLIP-T alignment.

Dataset Curation Details

The reference database curation pipeline ensures high fidelity across all categories. Person identities are selected from CelebA-HQ and audited for demographic balance. Object and fashion references are curated via web crawling, deduplicated using DINOv2 feature similarity, filtered by the LAION aesthetic predictor v2.5, and manually reviewed for distinctiveness. Background scenes include real-world landmarks to ensure world-knowledge background consistency. Prompt construction is managed by a multimodal LLM (MLLM) that extracts structured metadata (age, gender, ethnicity, size category) to condition the sampler and verify prompt plausibility through three stages of Gemma 4 verification queries.

Conclusion and Limitations

CogCanvas demonstrates that current methods fail to address the integrated challenges of multi-subject generation. The benchmark exposes systematic weaknesses in prior models by showing that they consistently ignore the background reference and suffer from attribute–person binding collapse as N grows. While CogCanvas is a high-fidelity testbed for real-world applications, it notes risks such as identity overlap with pretraining corpora and reliance on a single MLLM family for evaluation, which requires human-in-the-loop auditing to mitigate bias.

Improvements for AI systems

Here are the specific improvements an AI system can make by adopting or integrating the methodologies described in CogCanvas:

Improved Identity-Specific Compositional Generation: The system will be capable of generating scenes with 2 to 5 distinct, identity-specific people that maintain unique visual identities (face features, demographics) across all subjects simultaneously.
Accurate Attribute and Object/Fashion Binding: The system can correctly bind specific outfitting (e.g., Person A wears a red jacket) and objects (e.g., Person B holds the iPhone 15 Pro Max) to their assigned individuals, rather than mixing attributes between subjects or failing to bind them entirely when the group size increases.
Precise Spatial Reasoning and Interaction Modeling: The system can accurately realize complex spatial arrangements (e.g., A stands to the left of B) and fine-grained physical interactions (e.g., A and B shake hands), verified by ground-truth interaction graphs derived from the benchmark structure.
Robust Background Consistency: The system will ensure that the generated scene strictly adheres to a specified real-world background environment (e.g., a castle courtyard or a specific Vietnamese landmark like Ha Long Bay), preventing generic studio backdrops from replacing requested scenes.
Systematic Failure Mode Identification and Mitigation: By using the CogCanvas framework, researchers can systematically expose and quantify the exact failure modes of current state-of-the-art models (e.g., attribute leakage, scaling collapse at N=4/5 subjects). This allows for targeted architectural improvements to address these specific weaknesses.
Training-Free Leakage Suppression: Integration of the Sequential Reference Inpainting (SRI) baseline demonstrates a method to suppress cross-subject reference leakage by decomposing generation into sequential, single-subject inpainting passes, which can be used as a robust training strategy for complex multi-subject tasks.
Enhanced Diagnostic Capabilities: The inclusion of structured evaluation metrics like BG-Sim (background fidelity via SAM/DINOv3) and Attr-VQA (MLLM verification of binding/interaction) provides a comprehensive diagnostic toolkit to measure not just perceptual quality, but also the fidelity of identity grounding and semantic correctness.
Improved Reference Retrieval Performance: The system can be optimized for a reference retrieval subtask, allowing it to select the highest-fidelity reference image for each slot given a prompt, which improves grounding accuracy before final generation.

Abstract

Multi-subject reference-based image generation requires jointly preserving multiple human identities, binding per-person objects and fashion items, and respecting a specified background scene, a regime where current diffusion models remain brittle. Existing benchmarks evaluate only one axis at a time and none jointly captures multi-identity composition with human-object interaction, background grounding, and spatial plausibility. We introduce CogCanvas, a benchmark of 1,952 curated reference images spanning 100 celebrity identities, 115 distinctive objects and fashion items, and 29 real-world background scenes including landmarks, from which we construct 1,361 compositional prompts covering 2-5 person group sizes. The curation pipeline combines DINOv2-based deduplication, two-stage aesthetic filtering, and automated derivation of structured interaction and position graphs that serve as ground-truth supervision. CogCanvas supports three tasks, reference-based multi-human-object generation (primary), text-to-image compositional generation, and reference retrieval, under a unified six-axis evaluation protocol. We introduce two metrics tailored to the multi-reference setting: BG-Sim, which scores background fidelity on SAM 3-masked regions via DINOv3 feature similarity, and Attr-VQA, which uses a multimodal LLM to verify per-subject attribute binding and inter-person interactions against the structured graphs. Benchmarking five SOTA methods reveals that every model degrades substantially as group size grows from 2 to 5, with near-complete failure on object/fashion binding beyond three subjects.