SR-Prominence: A Crowdsourced Protocol and Dataset Suite for Perceptually-Weighted Super-Resolution Artifact Evaluation

summary

The gist

The gist: The authors introduce artifact prominence as an evaluative target, defined as the fraction of viewers who judge a highlighted region to contain a noticeable artifact, and construct

In short

The authors introduced 'artifact prominence,' measuring how often viewers notice an image artifact, moving beyond simple presence/absence. They built SR-Prominence, a dataset of 3,935 masks annotated with this metric from crowdsourced evaluations across multiple datasets. This allows researchers to evaluate super-resolution methods based on actual perceptual impact rather than just binary defect detection.

Key concepts

Artifact Prominence: This is an evaluative target defined as the fraction of viewers who judge a specific highlighted region in an image to contain a noticeable artifact. It shifts evaluation from whether an artifact exists to how much it bothers the human observer, quantifying perceptual impact.
SR-Prominence Dataset: A comprehensive suite containing 3,935 artifact masks derived from various sources like DeSRA and Urban100. These masks are annotated with prominence scores based on crowdsourced viewing sessions, allowing for robust benchmarking of different SR methods against perceptual quality metrics.
Full-Reference Metrics: Metrics like SSIM and DISTS that use the original high-resolution image as a ground truth to assess structural similarity. The paper found these provide surprisingly strong signals for viewer noticeability of artifacts, outperforming some no-reference methods in this context.
Crowdsourced Annotation Protocol: A structured method where 30 participants view paired 'Original' and 'Upscaled' images with a highlighted artifact region and vote on whether it is noticeable. This protocol ensures data integrity by discarding responses from participants who make multiple errors, yielding reliable prominence scores.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes

This episode discusses

The paper

SR-Prominence: A Crowdsourced Protocol and Dataset Suite for Perceptually-Weighted Super-Resolution Artifact Evaluation · Read on arXiv

AI Center, Lomonosov Moscow State University

Modern image super-resolution methods generate detailed, visually appealing results, but they often introduce visual artifacts: unnatural patterns and texture distortions that degrade perceived quality. These defects vary widely in perceptual impact--some are barely noticeable, while others are highly disturbing--yet existing detection methods treat them equally. We propose artifact prominence as an evaluative target, defined as the fraction of viewers who judge a highlighted region to contain a noticeable artifact. We design a crowdsourced annotation protocol and construct SR-Prominence, a dataset suite containing 3,935 artifact masks from DeSRA, Open Images, Urban100, and a realistic no-ground-truth Urban100-HR setting, annotated with prominence. Re-annotating DeSRA reveals that 48.2% of its in-lab binary artifacts are not noticed by a majority of viewers. Across the suite, we audit SR artifact detectors, image-quality metrics, and SR methods. We find that classical full-reference metrics, especially SSIM and DISTS, provide surprisingly strong localized prominence signals, whereas no-reference IQA methods and specialized artifact detectors often fail to generalize across datasets and reference settings. SR-Prominence is released with an objective scoring protocol that allows new metrics to be benchmarked on our suite without further crowdsourcing. Together, the data and protocols enable SR artifact evaluation to move from binary defect presence toward perceptual impact. SR-Prominence is available at https://huggingface.co/datasets/imolodetskikh/sr-artifact-prominence.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "SR-Prominence: A Crowdsourced Protocol and Dataset Suite for Perceptually-Weighted Super-Resolution Artifact Evaluation".

Jane: The gist: The authors introduce artifact prominence as an evaluative target, defined as the fraction of viewers who judge a highlighted region to contain a noticeable artifact,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So wrapping up on "SR-Prominence: A Crowdsourced Protocol and Dataset Suite for Perceptually-Weighted Super-Resolution Artifact Evaluation," the authors are presenting this dataset and protocol as a way to make super-resolution artifact evaluation more grounded in human perception.

Tom: They’ve created this whole dataset of three thousand nine hundred thirty-five masks annotated with prominence scores across various image types, and they’ve shown that binary labels alone don't tell us much about the actual visual impact.

Lu: The authors found that classical metrics like SSIM and DISTS provide strong localized prominence signals, but no-reference IQA methods often fail to generalize across different datasets and reference settings.

Conclusion: Tom: So we’re wrapping up on SR-Prominence... basically, they’ve created this whole dataset and protocol to stop just counting errors in super-resolution images.

Tom: Exactly. They call this artifact prominence... which is the fraction of viewers who judge a highlighted area to have a noticeable artifact in it.

Jane: So what this means for us is we might need to change how we grade super-resolution results. We have to start looking at what the human eye actually notices when an AI messes up.

Tom: It shifts the evaluation goal from just finding any pixel mismatch to understanding perceptual impact, which is a much more grounded way to look at SR failure.

Lu: And that opens up a new way for researchers and builders to test if their models are actually producing results that look good, not just technically "correct."

Meng: For me, it means when we ship models to production, we can use this kind of data to quickly see if our errors are in the right places—the stuff that actually breaks the user experience.

Jane: It really grounds the entire evaluation process in what matters most: how a real person perceives a super-resolution image.

Tom: Next up, we’re going to look at some of the specific metrics they used to prove this point.

More episodes

← Home