Auto-Regressive Models Need Structural Registers: Semantic Foresight Improves Coherent Driving Video Continuation

arXiv:2412.03758 · cs.CV · Submitted 2024-12-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Auto-Regressive Models Need Structural Registers".

Tom: ARCON introduces a scheme that alternates between generating semantic and RGB tokens to explicitly learn high-level structural video information,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So we’re looking at the title and authors for this paper, "Auto-Regressive Models Need Structural Registers: Semantic Foresight Improves Coherent Driving Video Continuation." The authors are Ruibo Ming, Jingwei Wu, Zhewei Huang, Zhuoxuan Ju, Jianming Hu, and Lihui Peng.

Jane: Those names sound very academic and focused on deep learning research. The title itself points to the core idea: that standard auto-regressive models need some kind of structural scaffolding to do a better job when generating video sequences for driving.

Lu: The authors are from a mix of institutions, which often leads to really interesting cross-disciplinary approaches in these kinds of vision papers, and their focus on autonomous driving scenarios gives the work a very tangible goal.

Meng: I wonder if the team behind this research has considered how these structural registers integrate with real-time processing constraints that autonomous vehicles face.

Lalam: I think the collaboration across different university settings suggests they’re aiming for a comprehensive solution, which is important because we need models that can handle diverse and complex real-world inputs effectively.

The paper's summary: Tom: Now let’s get into what this paper actually does. They introduce ARCON, which alternates between generating semantic tokens and RGB tokens to help the Large Vision Model learn structural video information explicitly during continuation tasks.

Jane: That means instead of just predicting the next pixel or frame blindly, the model is forced to pay attention to both what things look like (RGB) and where things are in space (semantic maps).

Lu: The summary points out that this interleaving process helps break down the video continuation task into two parts: continuing the auxiliary modality and translating between modalities, which they argue simplifies the task for the Large Vision Model.

Meng: So, they’re saying that by separating structure from texture generation using these tokens, you can handle generating new scenes or objects more reliably than just looking at raw pixels.

Lalam: This structural understanding is a huge step because it moves beyond just pixel matching; it implies the model is learning the underlying scene layout rather than just surface appearance, which has implications for how AI understands environments.

The paper's improvements: Tom: The paper highlights several key improvements. First, they use an interleaving generation scheme during training and inference to tackle the degeneration phenomenon when generating very long videos.

Jane: They also employ a simple method inspired by reference-based super resolution, using optical flow to stitch texture information from high-resolution input frames onto the low-resolution generated results.

Lu: The authors show that placing semantic tokens before RGB tokens in an auto-regressive model helps the model focus more on video structure, which they claim improves long-term generation capabilities by mitigating degradation, evidenced by better average optical flow vector magnitude between neighboring frames.

Meng: So they’re using a flow-based technique for texture enhancement during decoding to fix quality loss, which is a necessary step when you’re trying to generate high-definition results from discrete tokens.

Lalam: That combination of explicit structural learning and texture stitching seems like it addresses two major hurdles at once: maintaining long-term coherence and ensuring visual fidelity in the final output.

Conclusion: Tom: To wrap things up, the authors conclude that this scheme successfully separates structure from texture by interleaving semantic and RGB token generation, showing a clear effect of semantic tokens on RGB token generation.

Jane: They’ve demonstrated that this approach helps preserve video structural information and leads to better consistency between modalities in autonomous driving video continuation tasks.

Lu: The implications here are substantial for building more robust world models because they show how explicit structural registers can guide the generative process, moving beyond purely implicit learning within the transformer architecture.

Meng: From a practical standpoint, this suggests that future autonomous systems won't just react to immediate visual input but will have a stronger internal model of the physical scene structure over extended periods.

Lalam: Ultimately, ARCON’s work on "Auto-Regressive Models Need Structural Registers: Semantic Foresight Improves Coherent Driving Video Continuation" shows how integrating structural awareness can lead to more physically reasonable and temporally stable video generation across different modalities.

Tsinghua University · Megvii Technology 3 · University of the Chinese Academy of Sciences 1

cs.CV

Submitted: 2024-12-04

Updated: 2026-10-06

Importance score: 83/100

The gist: ARCON introduces a scheme that alternates between generating semantic and RGB tokens to explicitly learn high-level structural video information, which significantly improves temporal consistency and

Key concepts

Interleaving Generation
ARCON alternates between generating semantic tokens, which represent the structural layout of a scene, and RGB tokens, which capture visual details. This process breaks the video continuation task into two parts: continuing the auxiliary modality (structure) and translating between modalities (appearance). This helps the model simplify its job of creating long videos.
Semantic Tokens
These are discrete tokens that encode high-level structural information about a video frame, such as object layouts and scene organization. By generating these first, the model pays more attention to the underlying structure of the video, which is crucial for maintaining physical coherence over long sequences. They help capture structural details explicitly.
Token Decoder
Since discrete tokens lack high-definition visual detail, ARCON uses a simple method inspired by reference-based super resolution. This technique borrows texture information from high-resolution input frames to decode the low-resolution tokens into realistic images, ensuring that the generated visuals have good quality and texture.
Semantic Token First
Placing semantic tokens before RGB tokens in the auto-regressive process forces the model to prioritize understanding video structure. This ordering helps capture structural details explicitly and mitigates degradation during long-term generation, resulting in better preservation of video structure and improved consistency between modalities.

Terminology

Summary

ARCON introduces a scheme that alternates between generating semantic and RGB tokens to explicitly learn high-level structural video information, which significantly improves temporal consistency and physical reasonableness when autoregressively generating long videos in autonomous driving scenarios.

The gist

ARCON is a large auto-regressive transformer trained on tokenized video frames that utilizes an interleaving generation scheme of semantic tokens and RGB tokens to enhance the model's ability to create longer videos with better temporal consistency, showing high degree of correspondence between generated semantic maps and RGB images.

How it works

The ARCON model comprises three decoupling components: (1) an image tokenizer encoding images and semantic maps into discrete tokens, (2) a Large Vision Model (LVM) trained with the next token prediction task to perform the video continuation task in an auto-regressive manner, and (3) an image decoder that can decode discrete tokens to images. The model is built upon a large auto-regressive transformer with up to 20B parameters trained on large-scale tokenized video frames.

Interleaving generation

The core innovation lies in the interleaving of semantic tokens and RGB tokens during the LVM's training and inference phases. This approach is designed to break down the video continuation task into two sub-tasks: continuing the auxiliary modality and translating between modalities. By incorporating the task of continuing the auxiliary modal sequences alongside RGB tokens, ARCON aims to capture structural details with fewer tokens while enabling the generation of new objects and scenes that were not present in the input sequence. This interleaving is specifically designed to help the LVM simplify the video continuation task and mitigate the degeneration phenomenon.

Token decoder

To address quality loss inherent in representing high-definition images with few discrete tokens, ARCON employs a simple method inspired by reference-based super resolution to borrow texture information from high-resolution input frames. This involves training a flow-based feature warping method on the BDD100K dataset using the open-MAGVIT-v2 scheme. During inference, this method uses reference tokens from high-resolution input frames with the shape of 224 × 224 × 3 to guide the decoder, which is fine-tuned on the MAGVIT-v2 decoder.

Semantic token first

The paper demonstrates that placing semantic tokens before RGB tokens in an auto-regressive model helps pay more attention to the structure of the videos. This ordering essentially breaks down the video continuation task into a continuation and translation subtasks, allowing the model to capture structural details explicitly. The results show that incorporating a semantic token generation step before image token generation can improve long-term generation capabilities through mitigating degradation, as evidenced by improved performance quantified by metrics like the average optical flow vector magnitude between neighboring frames. This strategy is shown to help preserve video structural information and leads to better consistency between modalities.

Experiments and results

Experiments on the BDD100K dataset show that semantic tokens are highly effective; for instance, semantic tokens occupy only 1% of the codebook space to encompass 50% of the training data, in contrast to RGB tokens which utilize 23%. Furthermore, ARCON's results demonstrate strong generalization ability, as it outperforms other baselines on Frechet Video Distance (FVD) scores even without fine-tuning on the nuScenes dataset. Qualitative results show that the model exhibits autonomous learning processes, acquiring fundamental traffic knowledge, and can generate completely new scenes while better preserving the structural information of objects compared to methods like DMVFN or CogVideoX. The use of optical flow-based texture stitching further enhances visual quality, yielding results that exceed expectations.

Ablation study

The study investigates the efficacy of the simple flow-based feature warping method in the decoding process across different frame rates. Table 3 shows that adding feature warping significantly reduces FVD scores on both BDD100K and K700 datasets, indicating benefits in both in-distribution and out-of-distribution scenarios. Additionally, Table 4 analyzes the impact of interleaved generation and inference temperature, suggesting that the interleaving approach is beneficial for long-term generation capabilities. The model's capability to generate a multitude of diverse future outcomes from the same input frames further highlights its creative flexibility.

Conclusion

ARCON successfully develops a scheme that separates structure from texture by interleaving semantic and RGB token generation, demonstrating the effect of semantic tokens on RGB token generation. Future research is suggested in exploring novel encoding strategies for efficiency and investigating the integration of flow-based techniques with generative methods at the decoding stage, alongside further exploration into physical consistency for autonomous driving applications.

Improvements for AI systems

Here are specific improvements to AI systems based on the ARCON model described in this paper:

  1. Improve autonomous driving perception and decision-making by enabling minute-level video continuation, allowing vehicles to generate coherent, long-term driving scenarios that respect physical laws (e.g., consistent lane changes, proper traffic light adherence).

  2. Enhance generative capabilities for autonomous systems by allowing the model to generate novel, diverse future scenes based on initial driving conditions (e.g., generating multiple plausible outcomes for a complex intersection or turn scenario), significantly increasing robustness against unforeseen situations compared to diffusion-based models alone.

  3. Improve temporal consistency and structural understanding in video generation tasks by explicitly learning high-level structural video information via the interleaving of semantic tokens and RGB tokens, leading to more temporally stable and physically realistic generated videos over extended sequences.

  4. Develop a multimodal foundation model capable of unifying diverse visual modalities (RGB, depth maps, segmentation maps) into a single discrete token space, allowing language models (LLMs) to perform cross-modal reasoning and control generation using auxiliary structural information efficiently.

  5. Create high-fidelity video reconstruction/inpainting systems by leveraging an optical flow-based texture stitching method during decoding; this allows the model to borrow fine-grained texture details from high-resolution input frames into low-resolution generated results, significantly reducing visual artifacts and blurring.

  6. Increase model efficiency for long-sequence generation by exploring novel encoding strategies for the image tokenizer, which currently requires hundreds of tokens per image and a large parameter count in the LVM; this could lead to faster inference speeds without sacrificing quality.

  7. Develop controllable generative models for autonomous driving by integrating explicit semantic control (via semantic tokens) into the autoregressive generation loop, enabling the model to focus on structural planning while maintaining texture generation fidelity, which is crucial for applications requiring precise scene understanding.

Sources

Related papers