IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed

The gist: IS-Diff proposes a training-free approach to improve diffusion-based image inpainting by using initial seeds sampled from unmasked areas and incorporating a dynamic selective refinement mechanism to adjust initialization strength based on distributional cross-entropy.

How it works

IS-Diff addresses the issue of mismatched semantic information in masked regions by employing initial seeds sampled from unmasked areas to imitate the masked data distribution, thereby setting a promising direction for the diffusion procedure. This strategy helps prevent the generation from irrational inpaintings of previous works, such as filling masked regions with a uniform color. The initial seed is constructed by using the unmasked region to estimate the overall image distribution, which might closely match the distribution of the masked area. This process involves fitting a Gaussian Mixture Model (GMM) to fit the distribution of the unmasked image as defined in Eq. (4). The approximate distribution is then defined as xmasked ∼ X K k=1 πˆkN(xiµˆk, Σˆk), where πˆk, µˆk and Σˆk represent the mixing proportion, mean and variance of the k-th component in the GMM, respectively. The primary initialization xini is then obtained by concatenating the random seed xmasked sampled from the approximate distribution with the areas outside the mask xunmasked to obtain the primary initialization xini by xini = xmasked · m + xunmasked · (1 − m), where m is a binary metrics used to represent the inpainting mask.

Dynamic Selective Refinement

A dynamic selective refinement mechanism is proposed to detect severe unharmonious inpaintings in intermediate latent and adjust the strength of our initialization prior dynamically. This mechanism operates by iteratively evaluating intermediate inpainting results at an appropriate “checkpoint” timestep to determine whether the masked and unmasked regions are distributionally harmonious. The harmonization quality is assessed using the Cross-Entropy between the histograms of intensity/texture Distributions (DCE) in the masked and unmasked regions as a metric, defined as DCE = −Hmasked(x0tc) log Hunmasked(x0tc), where H denotes the histogram of inpainting result, and x0tc denotes the estimated x0t at a specific timestep tc during the diffusion process. If this DCE metric is above a pre-defined threshold ϵ, it indicates severe misalignment between the masked and unmasked regions, leading to further refinement by increasing the ratio of initialization to noise. The strength of our proposed primary seed initialization is adjusted dynamically on the basis of DCE metrics by reversing primary initialization to the timestep tˆ by xtˆ = p αtˆxˆ0 + p 1 − αtˆε, where tˆ = T at the very beginning.

Validation and Performance

The method is validated on both standard and large-mask inpainting tasks using the CelebA-HQ, ImageNet, and Places2 datasets. Quantitative experiments compare IS-Diff with state-of-the-art training-free diffusionbased methods, including DDNM [8], RePaint [7], and Stable Inpainting 2.0 [12]. Results demonstrate that IS-Diff consistently enhances performance across various baseline frameworks, validating its universal value as a plug-and-play module. Qualitatively, the method shows significantly better performance on challenging cases such as side faces and extensively masked images compared to previous methods.

Key Contributions

The contributions of IS-Diff can be summarized as follows:

We reveal a critical oversight in the field of image inpainting by clarifying that a distributional compatible initialization is crucial to satisfactory inpainting

We propose IS-Diff, where a semantically meaningful initial seed is constructed to drive the diffusion process, leading to coherent and consistent inpainting

We introduce an dynamic elective refinement mechanism to adjust the initialization strength and mitigate the inherent conflict between initialization and unmasked semantic

Limitations

The authors acknowledge that while IS-Diff improves robustness and realism, it is significantly slower than GAN-based and Autoregressive-based methods. Additionally, the method can be abused for image manipulation potentially.

Social Impact

Our proposed IS-Diff, as an image inpainting method, can be used for image editing, object removal, and other typical applications. Moreover, IS-Diff can be easily integrated into any diffusion-based methods to enhance their inpainting capabilities.

Ablation Study Insights

Ablation studies confirm the effectiveness of the components; for instance, when evaluating Primary Seed Initialization without dynamic iterative refinement on the ImageNet dataset and comparing with the DDNM baseline, IS-Diff consistently improves the baseline on all settings. Furthermore, in testing under extended time consumption for extreme large mask conditions like “expand”, IS-Diff is observed to achieve better performance even with only 10% the time consumption of Repaint (T = 250). The optimal quantitative outcomes for dynamic adjustment are noted to occur when ∆t = 100.

Conclusion

In this paper, we propose the Initial Seed refined Diffusion Model (IS-Diff) to set a promising direction for the diffusion process with a refined initial seed. Estimated data distribution from the unmasked images is extracted as a primary initial seed. Furthermore, a interative refinement mechanism is introduced to seek finer initialization by adjusting the initializationto-noise ratio daynamically. Extensive experimental results demonstrate that our method can generate more harmonious and realistic results in a training-free, efficient, and flexible manner.

References

[1] S. Wang, C. Saharia, C. Montgomery, J. Pont-Tuset, S. Noy, S. Pellegrini, Y. Onoe, S. Laszlo, D. Fleet, R. Soricut, J.

1. Bold initialization for diffusion process: we propose to sample the seed from unmasked image areas to imitate the masked data distribution, thereby setting a promising direction for the diffusion procedure. This addresses mismatched semantic information in masked regions by using initial seeds that are distributionally harmonious, which is shown to lead to harmonious results (as indicated by Fig. 1(b)).

2. Dynamic selective refinement mechanism: we propose a dynamic selective refinement mechanism to detect severe unharmonious inpaintings in intermediate latent and adjust the strength of our initialization prior dynamically. This mechanism uses the distributional cross-entropy (DCE) to evaluate diffusion intermediate latent, and when it exceeds a threshold, it increases the ratio of initialization to noise, enabling better initialization selection and harmonious inpainting.

3. Improved robustness on challenging scenarios: The model can generate more realistic peacock feathers and is capable of inpainting objects that have been truncated in the middle because the method demonstrates stronger consistency with the original image’s texture and semantics (Fig. 7).

Diffusion models have shown promising results in free-form inpainting. Recent studies based on refined diffusion samplers or novel architectural designs led to realistic results and high data consistency. However, random initialization seed (noise) adopted in vanilla diffusion process may introduce mismatched semantic information in masked regions, leading to biased inpainting results, e.g., low consistency and low coherence with the other unmasked area. To address this issue, we propose the Initial Seed refined Diffusion Model (IS-Diff), a completely training-free approach incorporating distributional harmonious seeds to produce harmonious results. Specifically, IS-Diff employs initial seeds sampled from unmasked areas to imitate the masked data distribution, thereby setting a promising direction for the diffusion procedure. Moreover, a dynamic selective refinement mechanism is proposed to detect severe unharmonious inpaintings in intermediate latent and adjust the strength of our initialization prior dynamically. We validate our method on both standard and large-mask inpainting tasks using the CelebA-HQ, ImageNet, and Places2 datasets, demonstrating its effectiveness across all metrics compared to state-of-the-art inpainting methods.