DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
cs.LG, cs.CL
Submitted: 2026-09-28
Updated: 2026-09-28
Code: https://github.com/JIBSIL/dualgoose
Terminology
Sources
- Zoology: Measuring and Improving Recall in Efficient Language Models
- Simple linear attention language models balance the recall-throughput tradeoff
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
- Structured Denoising Diffusion Models in Discrete State-Spaces
- Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It
- RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale
- Qwen2.5-Coder Technical Report
- Pointer Sentinel Mixture Models
- Revisiting associative recall in modern recurrent models
- RWKV-7 "Goose" with Expressive Dynamic State Evolution
- Simple and Effective Masked Diffusion Language Models
- DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone
- Qwen3 Technical Report
- Gated Delta Networks: Improving Mamba2 with Delta Rule
- Dream 7B: Diffusion Large Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks