Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

summary

The gist

The gist: This work presents a systematic study of long-context continued pre-training for vision-language models, introducing MMProLong, which improves long-document VQA performance by 7.1% and

In short

This work systematically studies long-context continued pre-training for vision-language models by introducing MMProLong. The method extends a 7B model to 128K context using long-document VQA as primary data. Results show MMProLong boosts performance by 7.1% at longer contexts and maintains strong results beyond 128K without further training, proving a transferable recipe for large models.

Key concepts

LongPT: Long-context continued pre-training is a technique used to train vision-language models on very long sequences of text and images. It involves continuing the model's training process beyond standard short context limits to enable it to understand and reason over massive documents.
MMProLong: MMProLong is the specific LongPT recipe developed for this study. It uses a combination of synthesizing data from long-document VQA tasks and natural sampling from a large document pool, specifically optimized for achieving high performance on long contexts up to 128K tokens.
Long-Document VQA: This task involves asking questions about entire documents rather than single images or short passages. The training data is synthesized by taking small, coherent page segments and generating QA pairs from them, placing those questions within the context of the full document for the model to answer.
Generalization Beyond 128K: This refers to the ability of a trained model to perform well on contexts significantly longer than what it was explicitly trained on. MMProLong demonstrates that this training method allows models to maintain strong performance at extended lengths like 512K tokens without needing additional, task-specific fine-tuning.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Retrieval-Augmented Generation (RAG) — explained in 20 episodes
Sample Complexity — explained in 20 episodes
Quantum Fisher Information (QFI) — explained in 19 episodes
Flow Matching — explained in 18 episodes
Federated Learning — explained in 17 episodes

This episode discusses

The paper

Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context · Read on arXiv

Zhaowei Wang, Lishu Luo, Haodong Duan, Weiwei Liu, Sijin Wu, Ji Luo, Shen Yan, Shuai Peng, Sihang Yuan, Chaoyi Huang

Department of Computer Science, Hong Kong University of Science and Technology Department of ByteDance Seed

Long-context modeling is becoming a core capability of modern large vision-language models (LVLMs), enabling sustained context management across long-document understanding, video analysis, and multi-turn tool use in long-horizon agents. Yet practical training recipes remain insufficiently explored, particularly for designing and balancing long-context data mixtures. In this work, we present a systematic study of long-context continued pre-training (LongPT) for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data. We first show that long-document VQA is substantially more effective than OCR transcription. Building on this observation, our ablations further yield three key findings: i) for sequence-length distribution, balanced data outperforms target-length-focused data (e.g., 128K), suggesting that long-context ability requires generalizable key-information retrieval across various lengths and positions; ii) retrieval remains the primary bottleneck, favoring retrieval-heavy mixtures with modest reasoning data for task diversity; iii) pure long-document VQA largely preserves short-context capabilities, suggesting that instruction-formatted long data reduces the need for short-data mixing. Based on these findings, we introduce MMProLong, obtained by long-context continued pre-training from Qwen2.5-VL-7B with only a 5B-token budget. MMProLong improves long-document VQA scores by 7.11% and maintains strong performance at 256K and 512K contexts beyond its 128K training window, without additional training. It further generalizes to webpage-based multimodal needle retrieval, long-context vision-text compression, and long-video understanding without task-specific supervision. Overall, our study establishes a practical LongPT recipe and an empirical foundation for advancing long-context vision-language models.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context".

Jane: Here's the gist of the paper. The gist: This work presents a systematic study of long-context continued pre-training for vision-language models, introducing MMProLong, which improves long-document VQA performance by 7.1% and maintains strong performance at contexts beyond 128K without additional training.

Tom: Systematic Study of LongPT The paper presents a systematic study of long-context continued pre-training (LongPT) for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data.

Jane: The authors first show that "long-document VQA is substantially more effective than OCR transcription".

Tom: Building on this observation, the ablations yield three key findings regarding long-context ability.

Title and authors: Jane: Now, let's talk about the title and who wrote this: "Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context <ref:2605.13831#pg1,Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context>." Who are the authors, Tom?

Tom: The team is a group of researchers from places like HKUST and ByteDance Seed. They are focused on how to make these models work well across much longer sequences than what we usually test them on.

Lu: It’s interesting because they aren't just looking at one small context length; they are testing the generalization ability beyond 128K tokens <ref:2605.13831#pg1>.

Jane: So, what does that mean for us right now? It suggests that a training recipe isn't fixed; it needs to be systematic enough to push these models past those standard benchmarks.

Meng: If you can train something on 128K and it still performs well at 256K or even 512K without retraining, that makes scaling much more predictable for production systems.

Lalam: The authors are trying to fill a gap in the literature where practical recipes for developing long-context vision-language models are still quite unexplored.

The paper's summary: Tom: So, the summary of this work is that they conduct a systematic study of LongPT, extending a 7B model from 32K to 128K context using long-document data and doing extensive ablations to find out what works <ref:2605.13831#pg1,extending a 7B model from 32K to 128K context>.

Jane: What’s the key finding they point out right away? They show that long-document VQA is substantially more effective than just using OCR transcription for training purposes.

Lu: That comparison is important because it suggests that having a high-quality, instruction-formatted supervision signal from the document itself is much better for learning long context.

Meng: I need to know what they are actually doing with the data synthesis part then, beyond just picking documents. Are they just feeding raw text in?

Tom: No, they construct five specific training tasks using a large pool of over one point five million PDF-formatted documents, selecting documents that are between thirty-two and fifty pages long.

Lalam: For the long-document VQA part, they use a segment-level pipeline to generate QA pairs from short, coherent page segments and then place that back into the full document context for the model to learn from.

The paper's improvements: Jane: So, what are the specific improvements or findings they highlight from their ablations? What did they discover about how to structure this training?

Tom: They found three key things. First, for the sequence length distribution, balanced data outperforms data that is strictly focused on a target length like 128K <ref:2605.13831#pg1>.

Lu: That’s a crucial point because it means we don't need to force every single training sample to be exactly 128K long; mixing different lengths seems better <ref:2605.13831#pg1>.

Meng: How does that translate to deployment? Does it mean our pipeline needs to generate samples of varying lengths, or is there a simpler way for us to curate our data?

Tom: They also found an eight:two mixture between extraction tasks and reasoning tasks yields the best overall scores, which they say preserves task diversity while maximizing performance <ref:2605.13831#pg3>.

Jane: And what about how far this works? The most impressive part is that MMProLong improves long-document VQA scores by seven point one percent at both the 64K and 128K contexts.

Conclusion: Tom: So, to wrap up, the core takeaway from "Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context" is that we have a practical LongPT recipe called MMProLong <ref:2605.13831#pg1,Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context>.

Jane: This recipe takes Qwen2 point 5-VL-7B and extends its context window to 128K, achieving a seven point one percent lift in long-document VQA performance compared to previous setups without needing extra training for contexts beyond that point <ref:2605.13831#pg1>.

Lu: The implication here is that we have an empirical foundation for advancing long-context vision-language models with data-efficient and transferable recipes based on the LongPT study.

Meng: From a deployment standpoint, this suggests a very scalable path toward training these models without needing massive amounts of task-specific supervision for every new long context scenario.

Lalam: The study also shows that this recipe generalizes to broader multimodal tasks, like webpage-based needle-in-a-haystack evaluation and long-context vision-text compression, even without task supervision.

Tom: We’ve covered the title, the summary, the specific recipe improvements like MMProLong, and what this means for pushing context lengths beyond 128K <ref:2605.13831#pg1>. That’s what we had on this paper.

More episodes

← Home