WARP: A Unified Benchmark for Invisible Image Watermarking -- Robustness and Protection Against Attacks

Digital image watermarking is increasingly critical in media contexts, as emerging regulations and industry practices require marking AI-generated content and ensuring traceable sources to prevent manipulation or misuse. The gist: WARP introduces a unified framework and benchmark for evaluating the robustness of invisible watermarks by incorporating 32 recent classical, deep, and generative methods against 34 different erasing techniques to provide standardized protocols for evaluating perceptual quality, watermark readability, and attack resilience.

The Goal of the Benchmark

The primary objective is to develop a comprehensive, reproducible benchmark that standardizes the assessment of both perceptual quality and robustness across modern techniques. This framework aims to cover the full spectrum of existing approaches by assembling a representative set of methods that collectively cover the full spectrum of existing approaches to both watermarking and corresponding attack strategies. Specifically, it addresses three core research questions: RQ1 Can any watermarks survive both standard perturbations, erasing attacks, and cross-domain adversarial purification and re-generation techniques? RQ2 What systematic failure modes exist within specific method classes when exposed to state-of-the-art removal attacks? RQ3 To what extent can an existing watermark be overwritten via competitive re-embedding without compromising the carrier image?

The Benchmark Composition

WARP incorporates a large collection of watermarking and attack methods. The benchmark includes 32 classical, deep, and generative watermarking methods, spanning zero-bit to multi-bit scenarios (e.g., 1024 bits in ChunkySeal). It also incorporates 34 different erasing techniques ranging from traditional distortions (like JPEG compression, noise) to sophisticated adversarial attacks such as purification and re-embedding attacks. The evaluation utilizes subsets of COCO and DiffusionDB datasets, with more than 2.8M processed images and >8k GPU-hours across experiments.

Evaluation Metrics and Protocols

The framework employs a unified protocol for consistent comparison. For zero-bit watermarking, robustness is assessed with binary classification metrics like the p-value or True Positive Rate at a specific False Positive Rate (TPR@x%FPR). For multi-bit watermarking, robustness is typically assessed by comparing the extracted message against the original using Bit Error Rate (BER) and Word Error Rate (WER). Image quality evaluation utilizes full-reference metrics such as PSNR, SSIM, LPIPS, and CLIP-IQA for watermarks, and FID for generative methods.

Key Findings on Robustness Trade-offs

The results reveal that No method is uniformly robust (RQ1), indicating that robustness reflects threat-model alignment rather than intrinsic superiority. Learned and generative approaches improve over baselines but remain vulnerable to geometric, purification, and regeneration attacks. A crucial observation is the separation between post-hoc methods and generative watermarks: post-hoc methods are suitable when the main risk is conventional distortion or non-destructive image processing, whereas generative watermarks are preferable when the adversary can perform strong image reconstruction or synthesis.

Systematic Failure Modes Identified

The study identifies several systematic failure modes (RQ2):

1. Geometric alignment remains a crucial weakness; rotation-aware design is still a meaningful architectural advantage, as rotation through a non-right angle (30°) is the single most damaging geometric transformation.

2. Regeneration-based attacks—such as Flux Regeneration and Flux Rinsing—consistently produce the most severe watermark degradation, pushing BER toward 0.5 and TPR toward zero for the vast majority of post-hoc methods, irrespective of their architecture family.

3. Post-hoc methods exhibit a notable robustness gap compared to generative approaches, suggesting that the choice between classes should be guided by the expected threat model.

Re-embedding Behavior Analysis

Regarding competitive re-embedding (RQ3), most watermarks exhibit limited resilience to self-re-embedding, as a second application of the same method effectively removes the original watermark. However, methods that rely on a key can survive re-embedding provided a different key is used. Notably, only the DCT CAISS method demonstrates strong resistance to self-overlays. Built-in watermarks prove to be practically immune to posthoc overlays.

Conclusion and Practical Insights

The work concludes that stronger training-time augmentation, and explicit robustness-oriented objectives employed within these methods can substantially improve the quality–robustness balance. The results suggest that for practical deployment, post-hoc methods with explicit synchronization are suitable when the expected threat is conventional processing. Conversely, in-generation watermarks are preferable whenever the adversary can perform strong image reconstruction or synthesis. The framework is designed as a modular and extensible framework, allowing future updates under consistent evaluation protocols.

The gist

WARP introduces a unified framework and benchmark for evaluating the robustness of invisible watermarks by incorporating 32 recent classical, deep, and generative methods against 34 different erasing techniques to provide standardized protocols for evaluating perceptual quality, watermark readability, and attack resilience.

How it works

Here are specific improvements for AI systems based on the WARP benchmark, categorized by capability:

) AI System Improvement 1: Adversarially Robust Watermarking Architectures (For Content Provenance)

Based on the findings in Section 6 and Figure 4b, which show that generative watermarks (Gaussian Shading, MaxSive) are fundamentally superior to post-hoc methods under regeneration attacks (Flux Regeneration), the system should prioritize or integrate these architectures.

• The AI system can be improved by adopting a Generative Embedding Module that integrates directly into the diffusion process (similar to Stable Signature). This module would embed the watermark signal during the initial noise generation phase rather than post-hoc transformation.

• This improved AI system will be able to guarantee watermark persistence even when subjected to high-fidelity image reconstruction or synthesis attacks (like Flux Regeneration), which are currently devastating to post-hoc watermarks like InvisMark and StegaStamp.

) AI System Improvement 2: Geometry-Aware Robustness Modules (For Geometric Transformations)

Section 5.2 highlights that geometric alignment is a major failure mode, with non-right angle rotations (30°) being particularly destructive, and that methods with explicit synchronization mechanisms (DWSF, DFT Circle) show higher resistance.

• The AI system can be improved by incorporating a Geometric Synchronization Layer into the embedding pipeline. This layer would utilize spatial awareness to ensure the watermark signal is anchored relative to key image features or structural elements, making it robust against non-rigid transformations like rotation and cropping.

• This improved AI system will maintain high detection accuracy (TPR) even when the input image undergoes significant geometric warping or scaling, a scenario where most current methods fail.

) AI System Improvement 3: Capacity-Aware Watermarking Strategy (For High-Bandwidth Provenance)

The results in Section 5.1 and Figure 8 show a non-monotonic relationship between payload capacity and robustness; increasing bits beyond an optimal point (e.g., ChunkySeal vs. PixelSeal) does not yield better robustness, suggesting the bottleneck shifts from embedding to detection efficiency at very high capacities.

• The AI system can be improved by implementing a dynamic Capacity Allocation Engine. This engine would automatically select the optimal watermark payload size based on the anticipated threat model (e.g., low capacity for high-distortion scenarios, higher capacity for low-distortion scenarios) and the current computational budget (CPU vs. GPU).

• This improved AI system will provide an optimized balance between message density and resilience, ensuring that high-capacity watermarks do not suffer disproportionate robustness degradation when deployed in real-world systems.

) AI System Improvement 4: Metric-Adaptive Detection Thresholding (For Low False Positive Rate Deployment)

Section C.1 and Figure 9 clearly demonstrate that low-bit methods (e.g., HiDDeN with 30 bits) suffer a dramatic drop in TPR when the False Positive Rate (FPR) target is pushed to very low levels (down to 10−8).

• The AI system can be improved by integrating a Confidence-Aware Detection Module. This module would dynamically adjust the extraction threshold based on the required FPR for the specific application. If high confidence is needed, it defaults to high-capacity methods; if low FPR is paramount, it might temporarily switch to a method with higher inherent statistical slack (like ChunkySeal) or apply sophisticated post-extraction filtering.

• This improved AI system will allow for deployment in security-critical environments where the risk of false positives must be minimized, ensuring reliable detection performance across diverse operational requirements.

) AI System Improvement 5: Cross-Domain Quality Assessment (For Universal Evaluation)

Section C.5 reveals that no single quality metric (PSNR, SSIM, CLIP-IQA) fully characterizes visibility; full-reference metrics capture fidelity while no-reference metrics capture perceptual plausibility, and attacks can cause high LPIPS without affecting semantic content.

• The AI system can be improved by implementing a Multimodal Quality Validator. This validator would simultaneously utilize both full-reference (for fidelity checks) and no-reference (for aesthetic plausibility checks) metrics during the embedding phase.

• This improved AI system will provide a more holistic assessment of the embedded content, ensuring that the watermark is both perceptually indistinguishable from the source (high PSNR/SSIM) and semantically consistent with the surrounding image context (low LPIPS), thus reducing misclassification risks in complex media streams.

Digital image watermarking is increasingly critical in media contexts, as emerging regulations and industry practices require marking AI-generated content and ensuring traceable sources to prevent manipulation or misuse. Recent advances in invisible watermarking methods highlight the need to update existing benchmarking practices to reflect current techniques and evaluation criteria. We address this by introducing WARP -- a unified framework and benchmark for evaluating the robustness of invisible watermarks. WARP incorporates 32 recent classical, deep, and generative watermarking methods, as well as 34 different erasing techniques, ranging from traditional distortions to more sophisticated adversarial, purification, and re-embedding attacks. It provides standardized, reproducible, and easily scalable protocols for evaluating perceptual quality, watermark readability, and attack resilience. Using WARP, we extensively evaluate current invisible watermarking techniques, collecting the largest robustness benchmark in the field. Results identify the most robust approaches under both distortion and adversarial conditions, and reveal consistent relationships between watermarking methods and the attack strategies most effective against them. Our experiments also highlight that some of the watermarking methods considered are highly vulnerable to reembedding, even if they are robust to standard distortions. The code is made available at https://github.com/ispras/wibe.