Auditing Metrics for Synthetic Data Utility: A Cross-Domain Study of Appearance and Geometry

summary

Video file (mp4)

The gist

The gist The SADGE metric provides a zero-shot method for estimating synthetic-data usefulness by jointly modeling appearance and geometry similarity to predict downstream performance before training

In short

SADGE is a quantitative metric designed to predict how useful synthetic image datasets will be for real-world computer vision tasks before any model training occurs. The study found that neither appearance nor geometry alone is sufficient; the best performance comes from combining them using a specific bilinear formula. This fused metric strongly correlates with downstream task success across various benchmarks.

Key concepts

Appearance Similarity Score
This measures how semantically similar a real image is to a synthetic image by comparing their high-level features, often derived from deep learning embeddings like DINOv3. It assesses whether the synthetic image looks like it belongs in the target domain.
Geometric Similarity Score
This score quantifies structural fidelity by counting how many points align between a real image and a synthetic one using geometric matching methods (like MASt3R). It checks if the spatial relationships and shapes of objects are preserved correctly.
SADGE Metric
SADGE is the final, fused metric that combines appearance and geometry scores. It uses a constrained bilinear interaction model to capture the non-linear way these two factors must work together to predict how well a dataset will help train a model for tasks like object detection.

Terminology used across episodes

This episode discusses

The paper

Auditing Metrics for Synthetic Data Utility: A Cross-Domain Study of Appearance and Geometry · Read on arXiv

Adam Mickiewicz University · KAUST · Kiel University

We propose SADGE, a quantitative similarity metric that predicts the performance of synthetic image datasets for common computer vision tasks without downstream model training. Estimating whether a synthetic dataset will lead to a model that performs well on real-world data remains a bottleneck in model development. Existing evaluation metrics (e.g., PSNR, FID, CLIP) primarily measure semantic alignment between real and synthetic images (Appearance Similarity Score). Less commonly, structural similarity between images is considered to assess the domain gap (Geometric Similarity Score). However, to the best of our knowledge there exists no studies that evaluate which similarity metric is the best downstream predictor for a given synthetic dataset. In this paper, we show over a wide variety of different synthetic datasets and downstream tasks that neither appearance nor geometry alone can reliably predict downstream performance; rather, it is their non-linear interplay that dictates synthetic data utility. Specifically, we measure how commonly used Appearance and Geometric Similarity metrics computed between synthetic and real images correlate with downstream performance in object detection, semantic segmentation, and pose estimation. Across five public synthetic-to-real benchmark families and 15 dataset-level variants (79k image pairs), SADGE achieves the strongest association with downstream transfer performance under both linear and rank-based criteria, reaching Pearson r=0.88 and Spearman rho=0.77. We compute for each combination of geometry-based methods and appearance-based approaches SADGE scores across all benchmark families. The best configuration is obtained by fusing DINOv3 appearance similarity with MASt3R geometric consistency through a constrained bilinear interaction, outperforming both the strongest geometry-only baseline and the strongest appearance-only baseline.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Auditing Metrics for Synthetic Data Utility".

Jane: The gist The SADGE metric provides a zero-shot method for estimating synthetic-data usefulness by jointly modeling appearance and geometry similarity to predict downstream performance before training How it works SADGE…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re looking at this paper now called "Auditing Metrics for Synthetic Data Utility: A Cross-Domain Study of Appearance and Geometry". Jane, what’s the main takeaway from just reading that title?

Jane: It sounds like they are trying to figure out which metrics we should actually trust when we look at synthetic data. They’re not just looking at one thing, but how appearance and geometry interact.

Tom: Exactly. They suggest that existing methods, like just looking at appearance scores or just looking at geometry scores on their own, don't tell us much about whether a synthetic dataset is going to actually make a good model later on >

Lu: What’s interesting is that they are explicitly testing which similarity metric—appearance versus geometry—is the best predictor for downstream performance across many different datasets and tasks >

Meng: So, they’re essentially trying to create a single score that tells you how useful synthetic data is before you spend any time training. That sounds like it could save us a lot of time if it works on real traffic >

Lalam: It seems the paper is setting up a framework where we can rank candidate synthetic datasets based on this joint appearance and geometry model instead of just picking one metric in isolation >

The paper's summary: Jane: So, how does SADGE actually work? If we break it down for someone who isn't deep into computer vision, what’s the core idea they are pushing here?

Tom: The core idea is that you need to check two things simultaneously for every real image you have against a set of synthetic candidates. First, how similar do they look—that’s the appearance score. Then, how well do their shapes and structures match up—that’s the geometry score >

Lu: They define appearance similarity using cosine similarity between image embeddings from encoders like DINOv2 or CLIP, which measures if they are in the same visual space >

Jane: And for geometry, they use something called inlier count. It counts how many geometrically valid matches there are between a real image and a synthetic one using tools like MASt3R or LoFTR to see if the structure is preserved >

Meng: So, they’re getting these two scores—appearance and geometry—and then they fuse them together using this constrained bilinear interaction model called SADGE >

Tom: That fusion step is key because it’s not just adding the two scores together; it's a specific mathematical way of combining them to capture how they complement each other >

Lalam: It’s interesting that they standardized both those raw appearance and geometry signals using z-score normalization based on statistics from the training set variants before putting them into that final bilinear formula >

The paper's improvements: Tom: Looking at their results, the paper shows that neither appearance-only nor geometry-only metrics work reliably on their own when predicting how well a model will transfer.

Jane: That’s a strong finding because it validates the idea that we need both signals. They found SADGE achieves a Pearson correlation of zero point eight seven nine with downstream performance across several tasks like pose estimation and semantic segmentation >

Lu: What really stands out is the leave-one-dataset-out evaluations, where they show that SADGE keeps its strong correlation in every single one of those tests, even when excluding datasets like DIMO or RarePlanes >

Meng: That suggests that the utility of synthetic data isn't determined by one factor alone; it’s really about this joint structure–appearance relationship. If you only focus on geometry, your score drops significantly when you exclude DIMO, for example >

Lalam: They show that SADGE is best in every single leave-one-dataset-out split, with Pearson correlation values ranging from zero point six three seven to zero point nine zero seven depending on the exclusion >

Conclusion: Jane: So, to wrap up on "Auditing Metrics for Synthetic Data Utility: A Cross-Domain Study of Appearance and Geometry", what is the big picture for someone who just wants to use this paper?

Tom: The main point is that SADGE gives us an efficient ranking metric. It lets us prioritize candidate synthetic datasets before we spend time and money training a model on them >

Lu: They are essentially proposing a way to audit those rendering pipelines or domain randomization schedules before committing resources, because they show how the joint structure–appearance relationship dictates performance >

Meng: If I’m deploying models, this gives me a principled way to compare different synthetic data sources beyond just looking at the visual quality or the geometric fidelity alone >

Lalam: They also mention a limitation: their reliability might decrease if the target domain you are aiming for is very different from the data they used to train their encoders >

Tom: That’s fair. So, SADGE is a strong proxy ranking metric, not a guarantee that its coefficients will be perfect for every future domain. It’s a useful tool for comparison under public protocols >

Jane: Indeed, it gives us a better way to compare different synthetic data sources by capturing that non-linear interplay between appearance and geometry. That’s what this paper is about, "Auditing Metrics for Synthetic Data Utility: A Cross-Domain Study of Appearance and Geometry".

More episodes

← Home