RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

arXiv:2609.06712 · cs.CV · Submitted 2026-09-06 · Read on arXiv

cs.CV

Submitted: 2026-09-06

Updated: 2026-09-21

Terminology

• Sparsity — 40× in this paper · explained in 4 episodes
• Transformer — 18× in this paper · explained in 2 episodes
• Fidelity — 15× in this paper · explained in 2 episodes
• Sparse Attention — 15× in this paper · explained in 5 episodes
• Aggregation — 13× in this paper · explained in 1 episode
• Fine-Tuning — 10× in this paper · explained in 5 episodes
• Ablation — 9× in this paper · explained in 1 episode
• Adaptation — 9× in this paper · explained in 1 episode
• Benchmark — 6× in this paper · explained in 2 episodes
• Consistency — 6× in this paper · explained in 3 episodes
• Alignment — 4× in this paper · explained in 1 episode
• Embeddings — 4× in this paper · explained in 2 episodes
• Cosine Similarity — 3× in this paper · explained in 3 episodes
• Diffusion Model — 3× in this paper · explained in 8 episodes
• Diffusion Models — 3× in this paper · explained in 6 episodes
• Computational overhead — 2× in this paper · explained in 2 episodes
• Generative Model — 2× in this paper · explained in 2 episodes
• Generative Models — 2× in this paper · explained in 2 episodes
• Ablation study — 1× in this paper · explained in 1 episode
• Activation Function — 1× in this paper · explained in 1 episode
• Complex Landscape — 1× in this paper
• Dynamic Spatial Relationship — 1× in this paper
• Flow Matching — 1× in this paper · explained in 7 episodes
• Instance Preservation — 1× in this paper
• Motion Order Understanding — 1× in this paper
• Motion Rationality — 1× in this paper
• Theoretical Guarantees — 1× in this paper · explained in 2 episodes
• Two-Stage Optimization — 1× in this paper · explained in 2 episodes

Sources

• SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation
• Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile
• SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer
• LTX-Video: Realtime Video Latent Diffusion
• Flow Matching for Generative Modeling
• Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers
• RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers
• OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation
• Linearized Relative Positional Encoding
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• SLA2: Sparse-Linear Attention with Learnable Routing and QAT
• VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness
• SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models