APEX: Assumption-free Projection-based Embedding eXamination Metric for Image Quality Assessment

The gist The APEX framework introduces an assumption-free projection-based embedding examination metric for image quality assessment by leveraging the Sliced Wasserstein Distance with CLIP and DINOv2 embeddings to overcome limitations in existing metrics.

How it works

1. APEX measures the Sliced Wasserstein Distance (SWD) between image embeddings extracted by state-of-the-art foundation models, specifically CLIP and DINOv2 <ref:2605.07786#pg15> The Sliced Wasserstein Distance [Rabin et al., 2011] computes the average of the 2-Wasserstein distance on one-dimensional projections of two probability distributions µ and ν <ref:2605.07786#pg4> In practice, the SWD can be approximated by using a simple Monte Carlo scheme that uniformly draws L sample directions θl on S d−1, and replaces the integration with a finite-sample average <ref:2605.07786#pg5>.

2. The framework is embedding-agnostic and uses two open-vocabulary foundation models, CLIP and DINOv2, as feature extractors <ref:2605.07786#pg6> APEX inherits effective scalability to high-dimensional spaces, as we prove with theoretical and empirical evidences <ref:2605.07786#pg2>. The SWD in APEX computes the distance between two distributions of image embeddings, specifically extracted either from CLIP or DINOv2 <ref:2605.07786#pg4>.

Key Contributions

** We audit failure modes and limitations of existing image quality assessment metrics, analysing how their behaviour is affected by backbone choices, distributional assumptions, kernel configurations, finite-sample effects, and cross-domain shifts <ref:2605.07786#pg2>. The work introduces APEX as a projection-based evaluation framework for comparing image distributions in foundation model embedding spaces <ref:2605.07786#pg2>. 3. We investigate the stability of evaluations by analysing the number of random projections required for a reliable SWD estimation, providing both theoretical and empirical ablations, identifying a trade-off between estimation stability and computational cost <ref:2605.07786#pg2>. 4. We benchmark APEX against established metrics via an extensive evaluation protocol spanning heterogeneous visual domains, various degradations, analysing finite-sample stability, cross-dataset consistency, and sensitivity to progressive generation and refinement <ref:2605.07786#pg2>. 5. A dedicated human perceptual study further shows that APEX is competitive with the strongest baselines in human alignment, while offering more stable behaviour across domains <ref:2605.07786#pg2>. The remainder of the paper is organized as follows <ref:2605.07786#pg2>. 3. The APEX metrics propose a feature-distribution metric that measures the SWD between image embedding extracted by state-of-the-art foundation models, specifically CLIP and DINOv2 <ref:2605.07786#pg4>. The APEX-CLIP leverages OpenAI’s CLIP ViT-L/14@336px yielding 768-dimensional [CLS] embeddings <ref:2605.07786#pg5>. Conversely, APEX-DINO captures a richer architectural hierarchy by concatenating the 1024-dimensional [CLS] tokens from layers 6, 12, and 23 of DINOv2 ViT-L/14 into a single 3,072-dimensional representation <ref:2605.07786#pg5>. These metrics can be used in synergy: APEX-CLIP provides a robust assessment of global semantic fidelity, while APEX-DINO offers diagnostic utility by isolating which level of visual abstraction is affected by specific distribution shifts <ref:2605.07786#pg5>. 3.1 Analysis on the number of projections investigates the minimal number of projections L required for a stable and robust Monte Carlo approximation of the (squared) Sliced Wasserstein Distance <ref:2605.07786#pg6>. Theorem 1 provides a bound on the estimation error, showing that if L ≥ 2D4τ 2k log 8CD 2τ − log δ squared then the estimation error SW 2(µ, ν) − SW[2] L ≤ τ with probability at least 1 − δ <ref:2605.07786#pg2>. The number of projections L needed to maintain the estimation error within a certain threshold scales linearly with k, i.e. the intrinsic dimension of the manifold where embedding distributions are supported <ref:2605.07786#pg2>. CLIP and DINOv2 have a l2-normalization layer by design, making the boundedness assumption consistent with our experimental setting <ref:2605.07786#pg2>. The number of projections L needed to maintain the estimation error within a certain threshold scales linearly with k, i.e. the intrinsic dimension of the manifold where embedding distributions are supported <ref:2605.07786#pg2>. 3.2 Evaluation Protocol defines a multi-axis evaluation protocol going beyond aggregate benchmark scores <ref:2605.07786#pg2>. All experiments were conducted on a workstation equipped with an Intel Core Ultra 9 285K CPU with 24 physical cores, an NVIDIA GeForce RTX 5080 GPU with 16 GB of VRAM and CUDA 13.0, and a 2 TB SSD <ref:2605.07786#pg2>. The protocol evaluates responsiveness to diverse pixel- and latent-level distortions (Section 4.3.1), intra-dataset stability, efficiency under finite sample sizes, invariance of the degradation signal across different domains (Section 4.3.2), capability to capture subtle details during the generative refinement phase (Section 4.3.3), and overall alignment with human perceptual judgments (Section 4.3.4) <ref:2605.07786#pg2>. The degradation signal is defined as ∆k(τ, s) = dϕ(τs(Dk)), ϕ(Dk) <ref:2605.07786#pg2>. A well-behaved metric should satisfy monotonicity relative to the degradation signal as severity increases: ∆k(τ, s1) < ∆k(τ, s2) whenever s1. 4. The experiments are conducted across five heterogeneous evaluation domains—natural images, faces, dermoscopy, radiography, and remote sensing <ref:2605.07786#pg2>. Datasets include COCO-30k (natural scenes), HAM10000 (dermoscopic images), CelebA-HQ (high-resolution facial images), NIH ChestX-ray14 (medical radiographs), and NWPU-RESISC45 (remote sensing) <ref:2605.07786#pg2>. 4.2 Image Quality Assessment Metrics compares APEX with generative evaluation standards and recent foundation-model-based baselines <ref:2605.07786#pg2>. Table 1 summarizes the technical specifications of the evaluated metrics, categorizing them into AF (Assumption-Free feature distributions), FE (Foundation model Embeddings), and OT (Optimal Transport distance, no kernel/moment matching) <ref:2605.07786#pg2>. APEX-CLIP leverages OpenAI’s CLIP ViT-L/14@336px yielding 768-dimensional [CLS] embeddings <ref:2605.07786#pg5>. Conversely, APEX-DINO captures a richer architectural hierarchy by concatenating the 1024-dimensional [CLS] tokens from layers 6, 12, and 23 of DINOv2 ViT-L/14 into a single 3,072-dimensional representation <ref:2605.07786#pg5>. APEX is formulated as a feature-distribution metric that measures the SWD between image embedding extracted by state-of-the-art foundation models, specifically CLIP and DINOv2 <ref:2605.07786#pg4>. 4.3 Evaluation Protocol assesses metrics based on five fundamental criteria: responsiveness to diverse pixel- and latent-level distortions (Section 4.3.1), intra-dataset stability, efficiency under finite sample sizes, invariance of the degradation signal across different domains (Section 4.3.2), capability to capture subtle details during the generative refinement phase (Section 4.3.3), and overall alignment with human perceptual judgments (Section 4.3.4) <ref:2605.07786#pg2>. In Section 4, we present the results of the evaluation protocol above <ref:2605.07786#pg2>. 4.3.1 Degradation Sensitivity extends the protocol of [Jayasumana et al., 2024] with pixel- and latent-space perturbations across domains <ref:2605.07786#pg2>.

1. Bold header: Assumption-free similarity measurement for image quality assessment

APEX provides a mathematically grounded, assumption-free similarity measure by leveraging the Sliced Wasserstein Distance (SWD), which not relying on rigid parametric hypotheses or kernel configurations. This allows AI systems to evaluate synthetic images using modern foundation models like CLIP and DINOv2 features without being constrained by outdated feature spaces or rigid Gaussian assumptions inherent in FID.

2. Bold header: Foundation model agnostic evaluation framework

APEX is designed to be embedding-agnostic and uses two open-vocabulary foundation models, CLIP and DINOv2, as feature extractors, meaning the system can be adapted to natively supports any modern feature space. This enables AI systems to maintain high evaluation standards across diverse generative architectures without being bottlenecked by a single pre-trained backbone like Inception-v3.

3. Bold header: Robustness against visual degradations and domain shifts

The framework demonstrates superior robustness to visual degradations and exhibits strong intra- and cross-dataset robustness, as evidenced by the low cross-dataset consistency metric (Λ) for APEX variants compared to baselines. This capability allows AI systems to reliably assess the quality of generated content even when subjected to various pixel-level corruptions, latent distortions, or shifts between domains like medical scans and satellite imagery.

4. Bold header: Sample efficiency and computational scalability

APEX metrics show rapid sample convergence and provide stable evaluations with significantly fewer samples compared to MMD-based baselines. Furthermore, the computation for APEX scales as O(N log N), bypassing the prohibitive quadratic overhead of MMD variants, making it highly efficient for large-scale generative model training pipelines.

5. Bold header: Diagnostic utility via multi-scale feature analysis

The use of APEX-DINO allows for a diagnostic utility by isolating which level of visual abstraction is affected by specific distribution shifts, as shown by the per-layer analysis on APEX-DINO focusing on the response of intermediate layers to progressive degradation. This enables AI researchers to pinpoint whether failures in image quality stem from low-level texture corruption or high-level semantic inconsistencies.

As generative models achieve unprecedented visual quality, the gold standard for image evaluation remains traditional feature-distribution metrics (e.g., FID). However, these metrics are provably hindered by the closed-vocabulary bottleneck of outdated features and the assumptive bias of rigid parametric formulations. Recent alternatives exploit modern backbones to solve the feature bottleneck, yet continue to suffer from parametric limitations. To close this gap, we introduce APEX (Assumption-free Projection-based Embedding eXamination), a novel evaluation framework leveraging the Sliced Wasserstein Distance as a mathematically grounded, assumption-free similarity measure. APEX inherits effective scalability to high-dimensional spaces, as we prove with theoretical and empirical evidences. Moreover, APEX is embedding-agnostic and uses two open-vocabulary foundation models, CLIP and DINOv2, as feature extractors. Benchmarking APEX against established baselines reveals superior robustness to visual degradations. Additionally, we show that APEX metrics exhibit intra- and cross-dataset stability, ensuring highly stable evaluations on out-of-domain datasets.