Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
cs.CV, cs.AI, cs.LG
Submitted: 2026-09-28
Updated: 2026-09-28
Terminology
Sources
- Distilling the Knowledge in a Neural Network
- ViT-AdaLA: Adapting Vision Transformers with Linear Attention
- DiD It in 87 Minutes: A Label-Free Softmax-to-Linear Adaptation of Vision Transformers for Object Detection
- Attention Transfer Is Not Universally Effective for Vision Transformers
- Mimetic Initialization of MLPs
- Mimetic Initialization Helps State Space Models Learn to Recall
- Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models