RAC: Rectified Flow Auto Coder

arXiv:2603.05925 · cs.CV, cs.AI · Submitted 2026-03-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "RAC: Rectified Flow Auto Coder".

Jane: Inconsistent generation and reconstruction results in traditional Variational Autoencoders (VAEs) are addressed by proposing a Rectified Flow Auto Coder (RAC),

Tom: First, who's behind it and why it matters.

Paper summary: Tom: To summarize what we've seen so far, RAC proposes replacing the standard VAE decoder with this continuous flow mechanism to achieve multi-step decoding and inherent bidirectional inference <ref:2603.05925#pg0>. Essentially, they're using a velocity field to integrate a state tensor from an initial latent variable all the way to the target image state over time <ref:2603.05925#pg1>. Jane Right, so instead of one shot reconstruction, you get a path where you can correct things along the way, which is what they call a "straight and correctable" decoding path <ref:2603.05925#pg0>. Lu The paper highlights that this shared model structure allows the decoder to also act as an encoder through time reversal, which is what gives it that bidirectional capability <ref:2603.05925#pg1>. Meng That bidirectional nature is key because it reduces the parameter count by nearly forty-one percent compared to having a separate encoder and decoder pair <ref:2603.05925#pg0>.

Lalam: I see how that shared design leads to better parameter efficiency, which is always important when we're dealing with large generative models <ref:2603.05925#pg1>. Tom And the training objectives they use are quite detailed, focusing on reconstruction loss, path consistency loss, latent alignment loss for encoding, and a round-trip consistency loss <ref:2603.05925#pg0>. Jane That latent alignment part is interesting because it tries to ensure the latent representations themselves are better structured when they're being used for both encoding and decoding functions <ref:2603.05925#pg1>.

Lu: They even introduce an optional mean-velocity regularizer inspired by rectified flow, which adds another layer of control over the velocity field during training <ref:2603.05925#pg0>. Meng From a practical standpoint, having these multiple loss functions working together suggests they are trying to tightly constrain the model's behavior across all these different tasks simultaneously. Lalam It seems like this entire framework aims to address the generation-reconstruction gap by giving the model more control over how it moves through the latent space during image synthesis.

Conclusion: Tom: So, wrapping up our discussion on RAC: Rectified Flow Auto Coder, we’re talking about how this work tackles those known issues in VAEs by using a continuous flow to provide a multi-step decoding process <ref:2603.05925#pg0>. Jane It really seems like the authors are pushing the idea that integrating the decoder into the generation framework, as they did here, helps improve generation quality because it allows for that necessary calibration along the path <ref:2603.05925#pg1>. Lu The implication here is significant because it shows how flow-based methods can be adapted to solve problems in representation learning and generative modeling, moving beyond just standard diffusion techniques <ref:2603.05925#pg2>. Meng If this approach scales well, the parameter efficiency gains mentioned from the bidirectional inference could mean we can deploy much more complex generative models on less computational hardware than before. Lalam And for AI culture, if these models become inherently better at handling latent structure through these flow mechanisms, it could lead to much more coherent and reliable creative outputs across various applications.

Tom: The title itself, "RAC: Rectified Flow Auto Coder," tells us exactly what the method is: it's using rectified flow concepts to build an auto coder that goes beyond the simple VAE structure <ref:2603.05925#pg0>. Jane And the authors are doing a lot of heavy lifting by showing how this unified training objective manages to keep both encoding and decoding paths consistent through time reversal, which is what makes it work <ref:2603.05925#pg1>. Lu It points toward a future where generative models naturally have an inherent ability to correct their own internal representations during the generation process <ref:2603.05925#pg1>.

Meng: Practically, this means that for engineers building real-world systems, we can expect models that are more robust when they have to perform iterative refinement tasks instead of just generating a final image in one go <ref:2603.05925#pg0>. Lalam I think the biggest impact will be on how we define quality in generative AI; if generation and reconstruction become tightly coupled and correctable, it fundamentally changes what we consider a successful output <ref:2603.05925#pg1>.

Rutgers University · Nanyang Technological University · University of Wisconsin-Madison

cs.CV, cs.AI

Submitted: 2026-03-06

Updated: 2026-10-02

Comments: 12 Figures, 8 Tables. Project Page at https://world-snapshot.github.io/RAC/

Code: https://github.com/black-forest-labs/flux

Project page: https://world-snapshot.github.io/RAC

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: Inconsistent generation and reconstruction results in traditional Variational Autoencoders (VAEs) are addressed by proposing a Rectified Flow Auto Coder (RAC), which replaces standard VAE decoding

Key concepts

Rectified Flow Auto Coder (RAC)
RAC replaces the standard VAE decoder with a continuous flow model. This flow integrates a state tensor from an initial latent point to the final target image state over time. It allows for multi-step decoding and enables bidirectional inference by reversing the flow, creating a unified encoder and decoder.
Time-Conditioned Velocity Field
The core mechanism is modeling the decoder as a function describing how a state evolves over time: ds(t)/dt = vθ(s(t), t). This velocity field predicts the direction and speed of movement from one state to another, allowing the system to smoothly integrate states from $t=0$ (initial latent) to $t=1$ (target image).
Bidirectional Consistency
RAC achieves bidirectional inference by using a single shared model. Encoding is performed by reversing the flow direction, and decoding is performed by integrating the flow forward in time. This shared design avoids duplicating backbones, leading to significant parameter efficiency gains and cleaner latent representations.
Path Consistency Loss
This training objective ensures that the generated sequence of intermediate states follows a uniform, correctable path between the start and end points. It penalizes deviations from a linear progression defined by the target state difference, enforcing smooth transitions during decoding.

Terminology

Summary

Inconsistent generation and reconstruction results in traditional Variational Autoencoders (VAEs) are addressed by proposing a Rectified Flow Auto Coder (RAC), which replaces standard VAE decoding with a continuous-time velocity field integration to create a multi-step, correctable decoding path. This method inherently supports bidirectional inference by using the decoder as an encoder through time reversal, achieving parameter sharing and significantly improving generation quality while reducing computational cost.

How it works

The core innovation of RAC is replacing the single-step VAE decoder with a time-conditioned velocity field, denoted as a continuous flow:

“We model the decoder as a time-conditioned velocity field: d s(t)/dt = vθ(s(t), t), t∈[0,1].”

This flow integrates a state tensor from a latent-derived initialization to the target image state. The network predicting this velocity field is lightweight, utilizing a VAE backbone and incorporating an explicit time channel and (optionally) relative positional encoding. Decoding is performed by integrating this flow from t=0 to t=1:

s 1 = s 0 + ∫01 vθ(s(t), t) dt.

The same model yields an encoder by reversing the flow in time:

s 0 = s 1 - ∫01 vθ(s(t), t) dt.

Training Objectives and Mechanism

The training objective is designed to enforce three critical properties: accurate reconstruction, rectified paths, and latent consistency. The total loss function is defined as:

  1. A standard reconstruction loss: Lrecon = s K − s∗22.

  2. A path consistency loss to encourage a uniform, correctable path: L path dec = (1/(K-1))∑k=1 K-1 s k - (s0 + k/K(s∗ − s0))22.

  3. Latent alignment loss for encoding: Llatent = ẑ − zT22.

  4. Pixel reconstruction loss using the teacher decoder: Lpixel = DecT(ẑ) − x22.

  5. Round-trip consistency loss to ensure encoding and decoding return to the same state: L rt = Flowθ(Flow−1θ(s∗)) - s∗22.

  6. An optional mean-velocity regularizer inspired by rectified flow: L mv = vθ(s(t), t) - (v − t∂t v)22.

Bidirectional Consistency and Efficiency

RAC achieves bidirectional inference through a shared model operating in the full-resolution state space. Unlike conventional VAE designs that use separate encoder and decoder backbones, RAC uses a single shared model where encoding and decoding are realized by reversing the flow direction. This design avoids duplicating the backbone, leading to substantial parameter efficiency:

This shared design avoids the need for a dedicated encoder-decoder pair and therefore provides a substantially more parameter-efficient bidirectional formulation.

The paper reports that this mechanism reduces parameter count by nearly 41%. Furthermore, the latent representations induced by RAC exhibit significantly cleaner and more coherent structures compared to the standard SD-VAE, suggesting that the objectives jointly regularize the state space.

Experimental Results

Experiments demonstrate that RAC surpasses State-of-the-Art (SOTA) VAEs in both reconstruction and generation quality while maintaining approximately 70% lower computational cost.

RAC consistently outperforms existing SOTA VAEs in terms of both reconstruction and generation performance, but with approximately 70% lower computational cost.

The method shows improvements across various backbones (SD-VAE, IN-VAE, VA-VAE) and model scales (SiT-B, SiT-L, SiT-XL), consistently achieving the best performance. For instance, RAC reduces gFID from 24.1 to 14.8 for SD-VAE on ImageNet-1K while improving IS from 75.0 to 78.3 under the +RAC configuration compared to REPA-E baseline results in Table 1(a). Additionally, the generative decoder functions as an iterative refinement mechanism, where multi-step decoding progressively refines image quality even under ultra-short training (e.g., 1k steps).

Generation–Reconstruction Gap Analysis

The paper confirms the hypothesis that generation and reconstruction are not perfectly aligned in traditional VAEs. The line graph analysis shows that the generation ability is weaker than the reconstruction, directly confirming the assumption that RAC successfully unifies these two processes with a joint training objective.

Improvements for AI systems

Here are the specific improvements that can be made to AI systems based on the Rectified Flow Auto Coder (RAC) paper, and what those improved systems will be capable of:

  1. The AI system can achieve superior performance in both image reconstruction and novel content generation compared to traditional VAEs, while simultaneously reducing computational costs by approximately 40-70%.

  2. By replacing the single-step VAE decoder with a time-conditioned velocity field integration (Rectified Flow), the system gains a correctable multi-step decoding path. This allows the AI to refine latent variables along this path, leading to significantly higher generation quality and better alignment between generated outputs and ground truth data compared to models that commit to a single projection.

  3. The model inherently supports bidirectional inference by using the same velocity field for both encoding (time reversal) and decoding (forward time flow). This shared architecture reduces the parameter count by nearly 41%, leading to more compact and efficient AI systems capable of both synthesizing images from latent codes and compressing images into latent representations.

  4. The joint training objective—enforcing path consistency loss, latent alignment loss, and reconstruction constraints—ensures that the learned latent space is structurally regularized. This results in cleaner, more organized latent representations that are less prone to artifacts (like high-frequency noise or coarse blocks) when compared to standard VAEs.

  5. The system can perform iterative refinement during inference. By using multiple Euler steps with optional Gaussian noise per step, the AI can leverage multi-step decoding to progressively enhance image quality, recovering finer textures and sharper edges even under limited training budgets (e.g., ultra-short training).

  6. The improved latent space regularity allows the system to generalize better across different pre-trained VAE backbones (like SD-VAE or REPA-E), acting as a general refinement mechanism rather than being tailored to a specific architecture, making it highly adaptable for plug-in enhancement.

Sources

Related papers