POET: Preference Optimization for Enhanced Text-to-Image Generation

arXiv:2510.12041 · cs.CL · Submitted 2025-10-14 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "POET: Preference Optimization for Enhanced Text-to-Image Generation".

Jane: This work introduces an input-side inference-time scaling framework that enhances text-to-image generation through LLM-based prompt rewriting, demonstrating that optimizing only the input text can significantly improve image quality, alignment,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So Jane, we're diving into this paper now called "POET: Preference Optimization for Enhanced Text-to-Image Generation." Essentially, the big idea here is that instead of retraining the massive text-to-image models themselves, we can just focus on improving what users type. They claim that by using a large language model to rewrite the initial user prompt before it hits the image generator, we can significantly boost image quality and make things look better.

Jane: That makes sense for us to think about because it avoids all that heavy retraining work on the T2I backbones, which is a huge practical win for anyone trying to use these tools right now. The paper suggests this input-side strategy is model-agnostic, meaning it should work across different types of image generators without needing custom training for each one.

Lu: I find the idea of treating the T2I model as a black box really fascinating; it opens up possibilities because we don't need to understand the internal workings of every diffusion process to get better outputs. It shifts our focus entirely to how language interacts with visual concepts, which is where the real creativity lies for future AI systems.

Meng: From an engineering standpoint, that model-agnostic part is what interests me; if it works across different backbones without retraining them, that drastically lowers the barrier to entry for deployment and fine-tuning new models. But I wonder how robust those reward systems are when you switch from one T2I architecture to another.

Lalam: If we look at this from a cultural perspective, this means we can democratize high-quality visual generation; if the prompt rewriting is effective, anyone can get better results without needing deep machine learning expertise to tune the generator itself. This could make incredible visual content accessible everywhere.

Tom: Exactly, Lalam; it's about making powerful tools more usable for everyone. Jane, you mentioned avoiding retraining—what specifically does this framework claim it does for the user input? What's the core thesis of "POET: Preference Optimization for Enhanced Text-to-Image Generation"?

Jane: Well, the paper proposes a prompt rewriting framework that uses LLMs to refine user inputs before they are fed into T2I backbones. The central claim is that optimizing only the input text can substantially improve image quality, alignment with the text, and overall aesthetics across different T2I models. They achieve this by training an RL-trained prompt rewriter using iterative Direct Preference Optimization, which means it learns what makes a prompt better through pairwise comparisons of generated images.

Paper summary: Lu: The reward design they use is quite sophisticated; they don't just look at one thing like fidelity or style, but they break quality down into four dimensions: image quality, general alignment, physical alignment between the image and text, and aesthetics. This multi-faceted approach shows a deep understanding of what makes an image successful visually.

Meng: That seems comprehensive for evaluation; having those separate metrics allows them to see exactly where the prompt is failing—maybe it’s just not matching the subject matter, or maybe it’s missing some artistic flair entirely. It gives us diagnostic power before we even generate the final image.

Lalam: I think that emphasis on aesthetics is really important because in a creative field like visual art, just getting the subject right isn't enough; there's a whole layer of visual appeal that these reward functions seem to capture well. It speaks to how nuanced human perception works when looking at an image.

Tom: That’s a great point about nuance, Lalam; and this leads us into the training part. They bypass supervised fine-tuning entirely and use iterative Direct Preference Optimization, which is pretty clever because it doesn't require massive datasets of paired inputs and outputs to get started. How does that RL training actually happen without standard supervison?

Jane: The methodology involves generating several candidate rewrites for a single user prompt, synthesizing those candidates with a frozen T2I model, and then having a multimodal LLM judge these images to give pairwise preferences. Those chosen and rejected prompt pairs are then used to update the rewriter policy through the DPO loss defined in Equation one.

Lu: That iterative loop of generate, synthesize, judge, and update is what gives the system its learning capability; it’s a self-correcting mechanism driven by preferences rather than explicit instruction sets. It’s less about teaching the model rules and more about showing it preferred outcomes through comparison.

Meng: So, the core mechanism is this cycle of candidate generation and preference learning; that's a solid engineering pipeline structure we can definitely work with for system development. But I still have to ask about scalability; if we train this rewriter on one backbone, does it perform well when tested on an entirely different T2I model?

Paper summary: Lalam: The paper specifically addresses transferability, showing that the rewriter trained on one T2I backbone can generate improvements across different backbones during testing. That suggests the learned refinements aren't tied too closely to the specific architecture of the original image generator.

Tom: That transferability is a huge piece of evidence for its practicality; it means we don't have to build a custom prompt rewriter for every single new T2I model that comes out. We can train one robust tool and it works everywhere, which is incredibly efficient for the industry.

Jane: Indeed, Tom; and when we look at the results, they show that this prompt rewriting framework consistently improves image quality, aesthetics, and text–image alignment compared to existing strong baselines. It’s not just incremental improvement; it's a measurable jump in how well the generated image matches the user's intent.

Lu: The ablation study they conducted really hammers home that every single component, including those four reward dimensions, contributes something specific to the final performance metrics. Removing any one of them visibly drops the corresponding quality score, which validates the necessity of that composite reward structure.

Meng: That validation is crucial for us; it tells us exactly where we need to focus our optimization efforts if we decide to adapt this for a specific application, like generating product mockups versus artistic illustrations. Knowing which reward term matters most helps us prioritize development work.

Lalam: And the trade-off they found between aesthetics and alignment is something that really resonates; you can get a pronounced improvement in aesthetics by adding the rewards, but you see that it simultaneously reduces alignment, which is a necessary caution for any creative application. It shows there’s always a balance to strike.

Tom: So we've covered the summary, the training concept, and how they measure success with these four reward dimensions. Now we move toward what this all means for the broader application of prompt engineering in AI. Jane, how do you see the title "POET: Preference Optimization for Enhanced Text-to-Image Generation" fitting into the practical reality of using these tools today?

Jane: The title itself highlights that their focus is on optimization via preference learning rather than just simple instruction following. It frames this as a method to systematically enhance the generation process by optimizing what users prefer, which is a very different way to approach prompt design.

Paper summary: Lu: I see this as moving beyond simply describing an image; it’s about teaching the AI system what constitutes a *good* image according to complex criteria that humans use. It elevates the prompt from a description to a kind of subjective instruction set.

Meng: From a practical impact view, this means we can start building more sophisticated user interfaces where users don't have to guess the right keywords; the system itself helps guide them toward better language, which is a huge step toward making AI tools truly intuitive.

Lalam: For me, I see it empowering creators immensely; it takes away some of the guesswork from achieving a specific visual look, allowing artists and designers to focus on their vision rather than obsessing over technical prompting minutiae. It’s about unlocking more creative potential.

Tom: So to wrap up this discussion on "POET: Preference Optimization for Enhanced Text-to-Image Generation," we've seen that it's a framework that uses LLMs and DPO to refine inputs based on a composite reward system focusing on quality, alignment, and aesthetics across models. Jane, do you have one final thought on the long-term implications of this input-side scaling approach?

Jane: I think the most important implication is establishing input-side optimization as a robust strategy that doesn't rely on costly model retraining cycles. It provides a scalable way to enhance T2I systems by focusing our effort where it matters most: refining the language that drives the generation process.

Lu: The future potential lies in applying this same reward structure concept to other generative AI domains, not just images, because the idea of defining quality through multiple preference dimensions is fundamentally transferable across modalities.

Meng: I'm interested in seeing how much these gains translate when we move from text-to-image to other complex generative tasks, like code or three dee asset creation; the principle of refining the input remains a strong foundation.

Lalam: If this method becomes standard, it means that high-quality visual output won't be dependent on having access to massive proprietary training sets for every single generation task; it becomes more accessible through smart language manipulation.

Tom: That’s a great summary of the impact we’re hearing today. We've seen how "POET: Preference Optimization for Enhanced Text-to-Image Generation" uses iterative DPO and multi-dimensional rewards to improve T2I outputs without touching the model weights, showing a very practical way forward for enhancing generation quality.

Conclusion: Tom: So we've seen how POET uses LLMs to rewrite prompts to improve images, now let's talk about what that title actually means for us as listeners and creators.

Jane: It sounds like the authors are really focusing on preference optimization, which suggests they aren't just trying to make things look pretty but are teaching the AI what a "good" image looks like through comparison.

Lu: I see it as them moving away from rigid instructions toward a system that learns aesthetic and alignment preferences directly from how humans judge outputs. That’s a really creative way to think about guiding generation.

Meng: From an engineering standpoint, focusing on preference optimization implies they're building a feedback loop where the model learns through what we like, which is much more practical than just feeding it more text examples.

Lalam: For me, this title points toward a future where AI systems don't just follow commands but actively refine their creative output based on nuanced human judgment across quality and style.

Tom: Exactly! The authors are showing us that the way we talk to the AI—the prompt—is a trainable part of the generation process itself, not just an input for a static machine.

Jane: It simplifies things by suggesting that we don't need to perfectly engineer every single pixel or texture; instead, we can optimize the language driving those pixels.

Lu: This feels like shifting the focus from brute-force technical specification to a more intuitive form of creative guidance, which is really exciting for AI development.

Meng: I wonder how this preference-based learning translates when we apply it to different kinds of complex outputs beyond just images, like generating functional code or three dee assets.

Lalam: That potential extends far beyond visuals; if this approach works on images, the underlying principle could be applied anywhere there’s a subjective quality standard to achieve.

Tom: It really opens up new avenues for how we design user interfaces because the system learns what users prefer, making it more intuitive for everyone to get great results.

Jane: That accessibility is huge; if this works across different AI backbones, it means high-quality generation won't be locked behind just one specific model architecture anymore.

Lu: It suggests a path where we can develop more flexible generative tools that adapt their creative language based on the desired outcome.

Meng: I think the real impact is in making the creation of complex visual content significantly less dependent on deep, specialized machine learning expertise from the end user.

Lalam: That empowerment of creators is a huge cultural shift; it means high-end creative capabilities become more distributed and accessible to a wider audience.

Tom: So we're looking at a system that learns what's good through comparison, which means we can expect even more refined and aesthetically pleasing results from AI tools soon. What aspect should we look into next?

Ruibo Chen, Jiacheng Pan, Heng Huang, Zhenheng Yang

TikTok · University of Maryland

cs.CL

Submitted: 2025-10-14

Updated: 2026-09-28

Code: https://github.com/black-forest-labs/flux

Importance score: 83/100

The gist: This work introduces an input-side inference-time scaling framework that enhances text-to-image generation through LLM-based prompt rewriting, demonstrating that optimizing only the input text can

Key concepts

Prompt Rewriting Framework
This is the core idea where a large language model (LLM) acts as an editor to refine user text prompts. Instead of changing the image generation model itself, this framework focuses solely on improving the input text. The LLM learns to suggest better descriptions based on desired visual outcomes, making it model-agnostic.
Composite Reward Function
The system uses four different metrics—image quality, general alignment, physical alignment, and aesthetics—to guide the prompt rewriter's training. A 'general rewriter' balances quality and alignment, while an 'aesthetics rewriter' focuses heavily on visual appeal. This multi-faceted approach ensures the rewritten prompts optimize for several desired qualities simultaneously.
Iterative DPO Without SFT
The training method uses Direct Preference Optimization (DPO) to teach the rewriter how to generate better prompts. Instead of needing expensive supervised fine-tuning data, the system generates several prompt candidates, and a multimodal LLM judge ranks them. This feedback loop updates the prompt rewriter policy iteratively, improving performance without requiring traditional SFT pairs.
Model-Agnostic Transferability
The research shows that prompts rewritten for one text-to-image model often work well on completely different models. This means the improvements learned by the rewriter generalize across various T2I backbones. The rewriter learns universal principles of good prompting rather than memorizing specific model quirks, making it highly practical.

Terminology

Summary

This work introduces an input-side inference-time scaling framework that enhances text-to-image generation through LLM-based prompt rewriting, demonstrating that optimizing only the input text can significantly improve image quality, alignment, and aesthetics across diverse T2I backbones without requiring model retraining.

How it works

The core of the approach is a prompt rewriting framework that leverages large language models (LLMs) to refine user inputs before they are fed into a frozen T2I backbone. This strategy treats the T2I model as a black box and optimizes the input text only, which is both model-agnostic and training-free for the T2I models. The process involves training an RL-trained prompt rewriter using iterative Direct Preference Optimization (DPO), starting from a short user-provided prompt.

Reward Design

The rewriter is trained using a composite reward function that integrates multiple dimensions of quality. The paper defines four reward dimensions: (1) image quality, denoted as rQuality; (2) general image–text alignment, denoted as rGeneral-Alignment; (3) physical image–text alignment, denoted as rPhysical-Alignment; and (4) aesthetics, denoted as rAesthetics. Two distinct rewriters are trained: a general rewriter, which prioritizes semantic faithfulness and overall image quality by combining the first three rewards, and an aesthetics rewriter, which emphasizes visual appeal by including the aesthetics reward.

Learning Algorithm: Iterative DPO Without SFT

The training methodology bypasses Supervised Fine-Tuning (SFT) entirely, opting instead for iterative Direct Preference Optimization (DPO). At each training round, the policy generates n candidate rewrites for a user prompt. These candidates are then synthesized by a frozen T2I model and evaluated by a multimodal LLM judge that provides pairwise preferences. The chosen and rejected prompts form a triplet used to update the policy via the DPO loss defined in Equation 1, which utilizes pairwise image comparisons to reduce variance compared to scalar rewards.

Scalability and Transferability

The study systematically investigates scalability by evaluating how performance gains scale with the capacity of the large LLM used as the rewriter. It also examines transferability by training multiple rewriters on different T2I backbones and testing them on a fixed target backbone. Results show that rewritten prompts consistently improve over the original prompts, irrespective of whether the training and testing backbones match, suggesting that the rewriter acquires refinements that generalize across T2I backbones without needing per-model adaptation.

Key Findings

Empirical evaluations across diverse T2I models and benchmarks demonstrate that the method consistently improves image quality, aesthetics, and text–image alignment, outperforming strong baselines. The research also shows a controllable trade-off: incorporating the aesthetics reward yields a pronounced improvement in aesthetics but simultaneously reduces alignment. Furthermore, the performance scales with LLM size; results show that larger backbones generally yield higher averaged GPT-4o win rates, with Llama-3-70B-Instruct achieving the highest average win rate. The paper concludes that prompt rewriting is an effective, scalable, and practical model-agnostic strategy for improving T2I systems.

Ablation Study

The ablation study confirms the effectiveness of each reward term: removing the quality reward substantially decreases the image–quality win rate, while excluding either general or physical alignment rewards reduces alignment. Incorporating the aesthetics reward yields a pronounced improvement in aesthetics, increasing from 0.476 to 0.818, but simultaneously reduces alignment from 0.561 to 0.424, highlighting this inherent trade-off between visual appeal and faithfulness/alignment. The study also found that LoRA training is a strong and practical choice for prompt-rewriter training compared to full-model finetuning.

Conclusion

The proposed framework establishes input-side optimization as a practical strategy for advancing T2I systems by developing an RL-trained prompt rewriter that optimizes only the input text, improving both diffusion and autoregressive T2I backbones without SFT pairs or parameter updates. This method achieves state-of-the-art performance while avoiding the substantial costs associated with SFT data collection and curation.

The gist

A prompt rewriting framework trained with iterative DPO and composite rewards consistently improves image quality, alignment, and aesthetics across diverse T2I backbones without modifying or retraining them.

**(Self-Correction Note: The provided text is from a paper titled IMPROVING TEXT-TO-IMAGE GENERATION WITH INPUT-SIDE INFERENCE-TIME SCALING, not POET: Preference Optimization for Enhanced Text-to-Image Generation. The summary above strictly adheres to the content of the provided text, as requested.

Improvements for AI systems

Based on the scientific paper IMPROVING TEXT-TO-IMAGE GENERATION WITH INPUT-SIDE INFERENCE-TIME SCALING, here are specific, actionable improvements for AI systems and what those improved systems could achieve:

) 1. System Improvement: Training a Model-Agnostic Prompt Rewriter (The Core Improvement)

The system can be upgraded to utilize the proposed Reinforcement Learning with Iterative Direct Preference Optimization (DPO) framework to create a standalone, training-free prompt rewriter.

  • Specific Implementation: Train a Large Language Model (LLM) specifically as a prompt rewriter using DPO, leveraging multimodal LLM judges (like Qwen2.5-VL-72B-Instruct) to generate pairwise preferences across four reward dimensions: Image Quality, General Alignment, Physical Alignment, and Aesthetics.

  • What the Improved System Can Do: This system can take a simple or vague user prompt (e.g., city skyline) and iteratively refine it into highly detailed, contextually rich instructions (e.g., "A futuristic cityscape at dusk, with sleek skyscrapers and neon lights illuminating the horizon, serving as the backdrop for a dramatic scene in a Japanese-style visual novel, complete with dialogue bubbles and character sprites."). This eliminates the need for expensive Supervised Fine-Tuning (SFT) data collection or model-specific annotations.

) 2. System Improvement: Introducing Dual Rewriter Strategies

The system can be enhanced to offer two specialized rewriter policies based on user intent, allowing for fine-grained control over the output trade-off.

  • Specific Implementation: Implement two distinct DPO training pipelines:

Spinner A (General Rewriter): Optimized using the reward function: (rQuality + rGeneral-Alignment + rPhysical-Alignment). This variant prioritizes semantic faithfulness and overall image quality.

Spinner B (Aesthetics Rewriter): Optimized using the reward function: (rQuality + rGeneral-Alignment + rPhysical-Alignment + rAesthetics). This variant maximizes visual appeal, allowing the user to explicitly trade off strict adherence to prompt details for artistic merit.

  • What the Improved System Can Do: Users can select which rewriter they want to employ based on their goal. For example, a designer focused purely on photorealism can use Spinner A, while an artist seeking highly stylized or vibrant imagery can use Spinner B.

) 3. System Improvement: Enhancing Transferability and Scalability

The system's architecture should be designed to maximize cross-model performance without retraining.

  • Specific Implementation: The DPO training process should be structured to learn prompt refinements that generalize across different Text-to-Image (T2I) backbones (e.g., training on a rewriter paired with FLUX.1 and testing it on Stable Diffusion 3.5). Furthermore, systematically scale the LLM backbone capacity (e.g., comparing performance gains from Llama-3 8B vs. Llama-3 70B).

  • What the Improved System Can Do: The resulting prompt rewriter becomes a versatile tool that works across any T2I model without requiring per-model adaptation or retraining. It can be scaled by simply swapping the underlying LLM backbone to leverage larger models for more complex prompt generation.

) 4. System Improvement: Automated Quality and Alignment Benchmarking

The system should include built-in evaluation mechanisms to quantify its performance benefits objectively.

  • Specific Implementation: Integrate the reward dimensions (rQuality, rGeneral-Alignment, rPhysical-Alignment, rAesthetics) directly into a real-time scoring mechanism during the iterative DPO process. This allows for continuous monitoring of the trade-offs being made between these factors.

  • What the Improved System Can Do: The system can provide immediate feedback on which aspect of image generation (e.g., This prompt has high alignment but low aesthetic score) is being prioritized in a given iteration, enabling dynamic user steering toward their desired outcome.

) 5. System Improvement: Improving Prompt Length and Complexity

The training process should be tuned to encourage the rewriter to generate more informative and complex prompts over time.

  • Specific Implementation: Monitor the average token length of generated rewritten prompts across DPO rounds (as shown in Figure 5). The training objective should implicitly reward longer, more specific outputs.

  • What the Improved System Can Do: The system will evolve from simple expansions to generate highly complex, detailed prompts that specify numerous attributes, spatial relations (on/under, left/right), and lighting conditions, leading directly to higher fidelity and better image results.

) Summary of Overall Capabilities for the Improved AI System:

The improved system transforms a T2I pipeline from a static input-output mapping into an active, intelligent refinement loop. It enables users to achieve state-of-the-art image generation quality, aesthetics, and text–image alignment using only simple user inputs. Crucially, it achieves this without the prohibitive costs of supervised fine-tuning data collection and allows for robust deployment across diverse T2I models by ensuring prompt refinements are model-agnostic.

Sources

Related papers