Instance-Specific Classifier-Free Guidance for Text-to-Image Diffusion/Flow-Based Editing

In recent years, a variety of prompt-based image editing methods have been developed based on text-to-image (T2I) diffusion/flow models. Image editing introduces an inherent trade-off between adhering to the original source image and realizing the change specified by the target prompt, which are typically measured by the LPIPS and CLIP-T metrics, respectively. Intuitively, this trade-off can be controlled by the classifier-free guidance (CFG) scale. Most existing editing methods, though, use a fixed CFG scale, ignoring that different edits require varying levels of modification. In this paper, we first explore the potential benefits of using an instance-specific CFG by exhaustively re-running prominent editing methods with a wide range of CFG scale values on benchmark datasets. We show that instance-specific setting can significantly improve LPIPS without compromising CLIP-T. Yet, tuning based on repeatedly executing an editing pipeline is inefficient. To prevent slow inference, we train a light-weight neural network that predicts the metrics of the underlying editing method for a given edit instance and CFG scale. At inference time, these predictions enable fast selection of an instance-specific CFG scale. Experiments across multiple editing methods and benchmark settings show the performance advantage of our approach over the baseline methods.