MiniGPT-Reverse-Designing: Predicting Image Adjustments Utilizing MiniGPT-4

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "MiniGPT-Reverse-Designing: Predicting Image Adjustments Utilizing MiniGPT-4".

Jane: The paper was written by Vahid Azizi and Fatemeh Koochaki from.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the show, everyone. Today we’re digging into a paper called “MiniGPT-Reverse-Designing: Predicting Image Adjustments Utilizing MiniGPT-four.” Jane, I have to say, just the title alone got me curious—reverse designing sounds like something out of a puzzle book.

Jane: It really does, Tom. And the idea is actually pretty clever. Normally, when we edit an image, we know what we did—we brightened it, we cropped it, we changed the colors. But reverse designing flips that. You’re given the original image, the edited version, and maybe a vague text description, and you have to figure out exactly what edits were made and with what values.

Tom: So it’s like looking at a before-and-after photo and trying to guess the recipe of changes. That’s not just a simple task—that’s asking the model to understand the relationship between two images and text all at once.

Jane: Exactly. And that’s why the authors chose MiniGPT-four which is a vision-language model that can process both images and text together. They wanted to see if an off-the-shelf model like that could be extended to handle this more complex, multi-image task.

Tom: And the authors are Vahid Azizi and Fatemeh Koochaki. They’ve made the code available, which is great for anyone who wants to try it out themselves. But Jane, what’s the real-world hook here? Why should we care about reverse designing?

Jane: Think about image editing software. If you could automatically figure out what edits were made to a photo, you could build better revision control for images, or even teach people how to achieve certain effects by showing them the steps. It’s like having a chef reverse-engineer a dish just by tasting it.

Tom: That’s a tasty analogy. And I love that they’re not just using the model as-is—they’re fine-tuning it specifically for this task. That’s where the real work happens, and I’m excited to see how they did it.

Jane: Me too. Let’s get into the details of how they set up the whole thing.

Summary: Tom: So Jane, we’ve got the big picture—reverse designing is about predicting edits from before-and-after images. But how did they actually make MiniGPT-four do this? What’s the setup?

Jane: Great question. They took MiniGPT-four which already aligns a frozen vision encoder with a frozen language model using a simple linear projection layer. The key is that they only train that projection layer, keeping the rest of the model frozen. So they’re not retraining the whole thing—they’re just teaching the model how to connect the two images with the text.

Tom: That’s smart because it saves a ton of compute. But how do they feed two images into a model that was originally designed for one?

Jane: They process both images through the same vision encoder, so each image becomes a set of tokens. Then those tokens are combined with the text tokens using a template prompt, and everything goes into the language model, which is LLAMA-two in this case. The model then outputs the operations and their parameters as a sentence.

Tom: And the dataset—what are they training on? I remember they used something called I-MAD-Dense.

Jane: Right. The I-MAD dataset, which stands for Image Multi-Adjustment Dataset. The dense version has about twenty-two thousand triplets, each with a source image, an edited image, and a high-level creative edit description. The descriptions are intentionally vague, so the model has to rely on the images themselves to figure out the specific edits.

Tom: Vague descriptions sound like a challenge. But they also converted the ground truth operations into sentences, so the model learns to output things like “increase brightness by zero point three” or “apply saturation adjustment.”

Jane: Exactly. And they designed two sets of prompts—one that includes the edit description and one that doesn’t. That way they could test how much the text helps. They also made sure that half the training data had the description and half didn’t, to keep things balanced.

Tom: So the model is learning to predict edits both with and without textual hints. That’s a nice way to test its true understanding of the images.

Jane: And the results showed that including the text description made a huge difference. With the command, accuracy jumped to over ninety-four percent, but without it, accuracy dropped to around fifty percent. That tells us the text is doing a lot of heavy lifting.

Tom: Wow, that’s a big gap. So the model is really leaning on the language to guide it. But what happens when the text isn’t there? That’s where the model struggles, and that’s probably where the improvements come in.

Jane: Exactly. Let’s talk about what they tried to improve the performance.

Improvements: Tom: So Jane, we saw that the model performs well with text but struggles without it. What did the authors try to fix that?

Jane: They ran a few different experiments. The first was just fine-tuning the original MiniGPT-four without any changes. That gave them a baseline. Then they added an auxiliary loss—specifically, mean squared error between the predicted operation parameters and the ground truth. That’s like adding a second teacher that focuses only on getting the numbers right.

Tom: And did that help?

Jane: It did, actually. The accuracy went up slightly, and the error went down. The MSE dropped from zero point two nine to zero point two three, which is a solid improvement. So adding that extra signal helped the model nail down the values of the operations.

Tom: But they didn’t stop there. They also tried a heuristic auxiliary loss to deal with the problem of the model predicting too many or too few operations. How did that go?

Jane: That one backfired, unfortunately. The heuristic penalized the model when the number of predicted operations didn’t match the ground truth, and also when the intersection of predicted and actual operations was empty. But the overall accuracy dropped to seventy point six nine percent, which is worse than the baseline. So sometimes adding more constraints doesn’t help—it just confuses the model.

Tom: Interesting. So the simpler MSE loss worked better than a more complex heuristic. What about adding special tokens? I saw something about a ⟨break⟩ token.

Jane: Right. They tried adding a special token to indicate transitions between modalities, like between the two images and the text. But that also hurt performance. The accuracy dropped to sixty-eight point nine eight percent. And when they tried adding even more special tokens, the performance fell so much they didn’t even report the numbers.

Tom: So the takeaway is that the model works best when you keep it simple and just add a focused loss on the parameters. But Jane, what does this mean for the future? Is this the end of the road for reverse designing?

Jane: Not at all. The authors suggest that using a higher-resolution image encoder could help capture finer details, and they also mention fine-tuning on a smaller, human-generated dataset called I-MAD-Pro, which might be more accurate. There’s also the possibility of using human feedback to train the model further.

Tom: So there’s room to grow. And honestly, even with the current results, being able to predict edits with over ninety-four percent accuracy when text is available is pretty impressive. Let’s wrap up with our final thoughts.

Conclusion: Tom: We’ve been talking about “MiniGPT-Reverse-Designing: Predicting Image Adjustments Utilizing MiniGPT-four” and I think we can all agree this is a fascinating step forward. Jane, what’s the big takeaway for our listeners?

Jane: The big takeaway is that vision-language models like MiniGPT-four can be extended to handle complex tasks like reverse designing, but they still rely heavily on textual cues. The model does great when it has a description to work with, but it struggles when it has to rely purely on the images. That tells us there’s still a gap in visual understanding.

Tom: And that gap is exactly where future work will focus. The authors have laid out clear next steps—better image encoders, more accurate datasets, and maybe even human feedback. It’s not a finished story, but it’s a promising start.

Jane: Absolutely. And the fact that they made the code available means other researchers can build on this work. That’s how progress happens.

Tom: Well said. We’ll be keeping an eye on this line of research. Thanks for joining us, and we’ll see you next time with another paper.

Jane: Take care, everyone.