StawGAN: Structural-Aware Generative Adversarial Networks for Infrared Image Translation

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "StawGAN: Structural-Aware Generative Adversarial Networks for Infrared Image Translation".

Jane: The gist The proposed StawGAN introduces a novel structural-aware generative adversarial network to enhance the quality of translating night-time thermal infrared images into daytime color images by focusing on better-shaped…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So to recap, this paper introduces StawGAN, a model that focuses on improving the quality of translating night-time thermal infrared images into daytime color images by emphasizing better-shaped and high-definition objects. They test this approach using aerial imagery from the DroneVeichle dataset where they have paired RGB and infrared images.

Jane: The core idea is to move beyond simple colorization, which means they are building something that respects the structure of what it’s generating, not just the colors. This structural awareness is what they call StawGAN.

Lu: They describe a shared architecture between two different flows inside the generator and target decoder, both interconnected by shared layers to handle this translation process from source domain to target domain, which is something they show in their architecture diagram on page two of this work.

Meng: The model takes three things as input for the generator G: the image in the source domain s, a segmented image rs, and a one-hot vector representing the target domain t. Its output is both the translated image xt and its segmentation rt.

Lalam: They use several loss functions to guide this generation process, including an adversarial loss for training both generators and discriminators, plus a modality classification loss on the generator G to force it into the correct target modality.

Tom: Then there’s that shape consistency loss called Lshape-r, which tries to control slight differences in structural shape by optimizing a Structural Dissimilarity Index Measure or DSSIM loss encouraging spatial smoothness in the generated images.

Jane: That DSSIM loss is defined as Lssim equals one minus SSIM of xpaired minus xt, where xpaired is the original image but in the same domain of xt, which is instead the translated sample.

Lu: They also use a reconstruction loss Lrec-z aimed at preserving the original characteristics of source images, employing it with that DSSIM loss as suggested in reference fifteen.

Meng: And they have a crossing loss Lcross to force the generator to pay attention to the target area when generating the translated image. Plus an L1 loss LL1 is applied, taking the fake image with applied factors and the paired image serving in reducing production of more gray-like and more colorful images on page three.

Lalam: The model architecture itself has a convolutional encoder-decoder couple in the generator where one part uses convolution, instance normalization, and Leaky ReLU for convolution, while the upsampling part involves the same operation but with an upsampling done by interpolation at the beginning.

Tom: And on page three they mention that after each convolutional layer of the discriminators, they adopted a layer of spectral normalization to further stabilize training.

Conclusion: Jane: So, wrapping up this overview of StawGAN: it introduces a neural approach that strengthens the edges in the translation process, which makes it useful for various sub-tasks like object detection or classification.

Tom: The authors suggest this method can be used with both paired or unpaired datasets, showing that it’s generalizable to different scenarios when you look at the overall capability of StawGAN: Structural-Aware Generative Adversarial Networks for Infrared Image Translation.

Lu: The main contribution they highlight is enforcing a better generation of targets using different flows in the same model for both the image and the segmented target, with these flows being interconnected by a shared architecture on page two.

Meng: This method sits in the middle between paired and unpaired image translation models because it can be generalizable to unpaired problems but you can also use it in paired ones taking advantage of ad hoc losses.

Lalam: The DSSIM loss they employ is used to improve the estimation of both the luminance and chrominance pixels of infrared images, which is what leads to those better-shaped objects.

Tom: They show that this structural awareness produces better-shaped objects, and they compare it against pix2pixHD and TarGAN, showing that StawGAN surpasses pix2pixHD in most metrics except for the FID value being higher than pix2pixHD's.

Jane: For translation from infrared to RGB, both color and shape of the StawGAN image are more appealing compared to pix2pixHD, while for RGB to IR, both color and shape of the StawGAN image are also more appealing with respect to pix2pixHD.

Lu: TarGAN loses information on the targets because its images are darker, which makes it difficult to distinguish the shape and type of vehicle on page three.

Meng: And in terms of segmentation performance, the comparison shows that our model gains better performance with respect to TarGAN in segmentation metrics too, like Dice similarity coefficient DSC and Segmentation score S-Score.

Lalam: The paper concludes by saying the proposed model produces high-definition samples with realistic colorization and well-shaped and defined targets for those translation tasks.