RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs

This paper, authored by Qiyanhui Lu, Han Wu, Rongjian Xu, Tingzhang Luo, Cheng Fan, Xinghao Chen, Minjing Dong, Jufeng Yang, and Jianyuan Guo (corresponding author), presents RoRA, a training-free framework for visual token pruning in multimodal large language models (MLLMs). The work was published under the Association for the Advancement of Artificial Intelligence (AAAI) copyright and is available with code at https://github.com/LukieLuu/RoRA.

The paper addresses the inference cost problem in MLLMs caused by long visual token sequences: "Multimodal large language models (MLLMs) represent images as long sequences of visual tokens. High-resolution images, multi-image inputs, and videos can easily yield hundreds to thousands of tokens, accounting for most of the prefilling cost, KV-cache size, and latency in subsequent LLM layers. The authors note that raw text-conditioned attention is often positionally biased and concentrates around image borders or other systematic hotspots, causing methods like FastV to select tokens around biased regions while overlooking the true semantic evidence."

The paper identifies a common limitation in existing training-free pruning methods: tokens are selected by importance, diversity, or spatial coverage, but not by the role they should play under a fixed token budget. The authors argue that answering a multimodal query requires three distinct types of visual evidence: (1) core—a small set of high-confidence tokens must lock onto the queried object; (2) context—additional tokens are more useful as complementary scene or secondary-object cues outside the support; and (3) detail—a few fine-grained tokens are still needed for text, boundaries, and small parts, which are easy to miss under coarse importance ranking or crop-level quotas.

RoRA is formally described as follows: "RoRA, a training-free framework that casts visual token pruning as role-oriented regional evidence allocation. Given a fixed budget, RoRA partitions tokens into a protected semantic core, complementary context, and fine-grained detail."

Framework structure. RoRA is applied at an early LLM layer and retains a total budget K of visual tokens, decomposed as S = P ∪̇ C ∪̇ D, where P is the protected semantic-core set, C is the complementary-context set, D is the detail-token set, and Kp + Kc + Kd = K.

Object-calibrated attention and core protection. Following D2 Pruner's debiasing principle, RoRA calibrates instance-level attention with an offline positional-attention prior, and further incorporates a soft, prompt-calibrated object prior. The priors are obtained by sampling 1,000 unlabeled images from GQA: the positional prior uses the prompt Please describe the image., while the object-aware prior uses List all visible objects in the image. The calibrated attention score is defined as r i = Norm(a i/(b i + ε)) and p i = Norm(r i + λ obj o i), where a i is text-conditioned attention, b i is the positional-bias prior, and o i is the object-aware prior. The Kp highest-scoring tokens form the protected semantic core: P = TopK Kp p i.

Attention-Anchored Regions (AARs). RoRA selects Ma region anchors and expands their local neighborhoods on the 2D token grid: A = TopK Ma p i and R AAR = ∪ N ra(a). The paper states: An AAR is not an exact object mask; it is a lightweight spatial proxy of object-related support already covered by the core.

Complementary context allocation. Context scores are defined as c i = Norm(p i + λ ctx I[i ∉ R AAR]), giving a small exploration boost to tokens outside AARs. Redundancy is handled by a lightweight similarity filter: candidates with cosine similarity exceeding threshold τ sem against retained tokens are skipped. The context set is C = Select τ sem,Kc c i i ∉ P.

AAR-guided detail repair. Detail scores combine attention response, feature magnitude, and local contrast: d i = ω a â i + ω m m̂ i + ω l l̂ i + λ det η i, where η i = I[i ∈ R AAR] provides a bonus for detail tokens inside AARs. The detail set is D = TopK Kd d i i ∉ P ∪ C.

The paper lists four main contributions: (1) Role-aware budget allocation—organizing retained tokens into three complementary roles with different selection criteria under a shared budget; (2) Object region prior—a prompt-calibrated signal identifying object-centric regions; (3) Attention-anchored regions for structured pruning—enabling outside-AAR context exploration and inside-AAR detail preservation without requiring dense global similarity graphs; (4) Extensive experiments across several MLLMs verifying accuracy–efficiency gains.

Models and baselines. RoRA was evaluated on LLaVA-1.5-7B, LLaVA-NeXT-7B, Qwen2.5-VL-7B-Instruct, and Qwen3-VL-8B-Instruct, compared against 15 baselines including ToMe, FastV, MustDrop, LLaVA-PruMerge, PDrop, VisionZip, SparseVLM, DART, HoloV, and D2 Pruner.

Results on LLaVA models. On LLaVA-1.5-7B with budgets K = 192, 128, and 64, RoRA retains 99.8%, 99.1%, and 96.5% of the unpruned model's normalized average performance, respectively, achieving the best result among all compared methods at every matched budget. The authors note that the relative advantage of RoRA increases as pruning becomes more aggressive. On LLaVA-NeXT-7B at K = 320 (88.9% pruning), RoRA achieves a normalized average of 95.5%, outperforming all baselines.

Results on Qwen models. On Qwen2.5-VL-7B, RoRA achieves normalized averages of 106.9% and 96.7% at 75% and 90% pruning, respectively, outperforming all compared baselines. On Qwen3-VL-8B, it reaches 98.0% and 87.9%, surpassing D2 Pruner by 5.4 and 5.3 points, respectively, demonstrating consistent generalization to newer dynamic-resolution MLLMs.

Ablation studies. The semantic core was validated on RefCOCO-family localization splits: at K = 144, the normalized average increases from 35.44% for FastV to 74.44% after introducing the protected semantic core; at K = 58, it increases from 9.57% to 17.36%. Adding AAR-guided context further improves averages from 74.44% to 75.03% at K = 144 and from 17.36% to 17.47% at K = 58. For detail repair on TextVQA and VizWiz at K = 128, the average normalized score increases monotonically from 101.61% (without detail tokens) to 101.80% (standard configuration) and 101.94% (18 detail tokens).

Efficiency analysis. On LLaVA-1.5-7B with full POPE and one NVIDIA H800 GPU at K = 192, "RoRA completes the full evaluation in 5min43s and achieves an average per-sample latency of 0.038 s... Its selector overhead is only 0.704 ms, while it retains 98.4% of the unpruned model's POPE performance. Compared with the unpruned runtime of 7 min 35 s, this corresponds to a 24.6% reduction in end-to-end inference time. On LLaVA-NeXT-7B with an RTX PRO 6000 GPU, RoRA achieves 1.25× speedup at 66.7% pruning and 2.03× at 90% pruning. RoRA has complexity O(N + N t2), which is much lighter in practice than D2 Pruner's O(N2) full-graph construction and comparable to HoloV's crop-wise O(N2/C)."

The paper concludes that "by organizing visual tokens into a protected semantic core, complementary context, and fine-grained detail evidence, RoRA explicitly improves coverage of retained tokens through lightweight Attention-Anchored Regions (AARs). This design enables asymmetric budget allocation, expanding complementary evidence outside attended regions while recovering local details inside them, without constructing costly dense pairwise redundancy graphs. Extensive evaluations across LLaVA and Qwen-VL families show a superior accuracy–efficiency profile, consistently outperforming existing training-free baselines across different models even at aggressive pruning ratios."

Improvements:

• Implement role-oriented visual token allocation instead of single-criterion importance ranking. Split any fixed token budget into three roles: protected semantic core (high-confidence object evidence), complementary context (diverse scene support outside attended regions), and fine-grained detail (text, boundaries, small parts).

• Calibrate attention scores with a positional-bias prior and a prompt-calibrated object prior, preventing attention from concentrating on image borders or systematic hotspots.

• Use object-calibrated attention to identify a small “semantic core” that locks onto the queried object even under extreme pruning.

• Build lightweight Attention-Anchored Regions (AARs)—local 2D neighborhoods around top attention tokens—to guide structured pruning: tokens outside AARs get a small exploration boost for context coverage, while tokens inside AARs get a bonus for detail recovery. This avoids dense global pairwise similarity graphs.

• Apply asymmetric budget allocation across the three roles, allowing context to expand outside attended regions and detail to be preserved inside them.

• Add a lightweight similarity filter to skip redundant context tokens, keeping O(N + N t2) complexity rather than O(N2).

• Use mixed detail scoring (attention response + feature magnitude + local contrast + AAR bonus) to retain fine-grained tokens that coarse importance ranking misses.

What the improved AI system can do:

• Prune 75–90% of visual tokens in MLLMs with near-lossless accuracy—for example, retaining 99.8%, 99.1%, and 96.5% of unpruned performance on LLaVA-1.5-7B at budgets of 192, 128, and 64 tokens, and outperforming all training-free baselines on LLaVA-NeXT, Qwen2.5-VL, and Qwen3-VL.

• Run faster with lower memory overhead: achieve 1.25× speedup at 66.7% pruning and 2.03× at 90% pruning, with selector overhead under 1 ms and no costly full-graph similarity construction.

• Maintain semantic grounding and object localization at extreme budgets—improving normalized RefCOCO performance from 35.44% (FastV) to 74.44% at K=144 by adding the protected semantic core.

• Better handle high-resolution, multi-image, and video inputs by reducing KV-cache size and prefilling latency, making long-token multimodal inference practical on limited hardware.

• Retain text, boundaries, and small parts that coarse pruning typically loses—demonstrated by monotonic gains on TextVQA and VizWiz when detail tokens are allocated.

• Generalize to newer dynamic-resolution MLLMs (e.g., Qwen2.5-VL, Qwen3-VL) without retraining or model-specific adaptation, simply by plugging in the training-free pruning framework.

Multimodal large language models (MLLMs) encode images as long visual token sequences, making prefilling and KV-cache storage expensive. Existing training-free pruning methods select tokens by importance, diversity, or spatial coverage, but treat retained tokens as interchangeable and do not explicitly track which object-related regions are already covered. We present RoRA, a training-free framework that casts visual token pruning as role-oriented regional evidence allocation. Given a fixed budget, RoRA partitions tokens into a protected semantic core, complementary context, and fine-grained detail. It first calibrates text-conditioned attention with a positional prior and a prompt-calibrated object prior, then builds Attention-Anchored Regions (AARs) from high-confidence anchors as lightweight proxies for covered object support. Context is explored mainly outside AARs, while a small AAR-guided budget restores local detail; pairwise similarity is used only for context-stage redundancy filtering. Under matched budgets, RoRA consistently outperforms strong training-free baselines across LLaVA and Qwen-VL families, retaining most of the unpruned accuracy even at aggressive pruning ratios, e.g., 96.5% of full performance at 88.9% pruning on LLaVA-1.5, and improving over D2Pruner by about 5% on Qwen3-VL at 75-90% pruning. At a 66.7% pruning ratio, RoRA requires only 0.7 ms for token selection and reduces end-to-end inference time by 24.6%, corresponding to a 1.33x speedup over unpruned inference on an NVIDIA H800.