Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement

arXiv:2609.19964 · cs.CV · Submitted 2026-09-17 · Read on arXiv

cs.CV

Submitted: 2026-09-17

Updated: 2026-09-17

Code: https://github.com/xingyitong1/TPRD

Terminology

• Distillation — 131× in this paper · explained in 2 episodes
• Knowledge Distillation — 31× in this paper · explained in 11 episodes
• Transformer — 15× in this paper · explained in 2 episodes
• Robustness — 5× in this paper · explained in 5 episodes
• Alignment — 4× in this paper · explained in 1 episode
• Consistency — 4× in this paper · explained in 3 episodes
• Negative Prediction Suppression — 4× in this paper
• Positive Prediction Correction — 4× in this paper
• Teacher Prediction Refinement Distillation — 4× in this paper
• Accuracy — 3× in this paper · explained in 1 episode
• Maximum Dark Knowledge Preservation — 3× in this paper
• Ablation — 2× in this paper · explained in 1 episode
• Benchmark — 2× in this paper · explained in 2 episodes
• Computational overhead — 2× in this paper · explained in 2 episodes
• Edge Devices — 2× in this paper · explained in 2 episodes
• Ablation study — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Decoupled Knowledge Distillation — 1× in this paper
• Feature Extraction — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Sparsity — 1× in this paper · explained in 4 episodes

Sources

• Distilling the Knowledge in a Neural Network
• Knowledge Distillation via Query Selection for Detection Transformer
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR
• DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
• Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity
• D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement
• FitNets: Hints for Thin Deep Nets
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Contrastive Representation Distillation
• Function-Consistent Feature Distillation
• NormKD: Normalized Logits for Knowledge Distillation
• Masked Distillation with Receptive Tokens
• ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillation
• CLoCKDistill: Consistent Location-and-Context-aware Knowledge Distillation for DETRs
• OD-DETR: Online Distillation for Stabilizing Training of Detection Transformer
• detrex: Benchmarking Detection Transformers

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models