Lizard: An Efficient Linearization Framework for Large Language Models

arXiv:2507.09025 · cs.CL, cs.LG · Submitted 2025-07-11 · Read on arXiv

cs.CL, cs.LG

Submitted: 2025-07-11

Updated: 2026-04-18

Code: https://github.com/EleutherAI/lm-evaluation-harness

Terminology

• Benchmark — 25× in this paper · explained in 2 episodes
• Transformer — 21× in this paper · explained in 2 episodes
• Core — 14× in this paper · explained in 2 episodes
• Gated Linear Attention — 10× in this paper
• Attention Mechanism — 9× in this paper · explained in 2 episodes
• Fine-Tuning — 8× in this paper · explained in 5 episodes
• Generalization — 8× in this paper · explained in 13 episodes
• Accuracy — 7× in this paper · explained in 1 episode
• Ablation — 6× in this paper · explained in 1 episode
• Sliding Window Attention — 4× in this paper
• Adaptation — 3× in this paper · explained in 1 episode
• Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
• Ablation study — 2× in this paper · explained in 1 episode
• Distillation — 2× in this paper · explained in 2 episodes
• Fidelity — 2× in this paper · explained in 2 episodes
• Transformer Architecture — 2× in this paper · explained in 2 episodes
• Adam — 1× in this paper · explained in 2 episodes
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Downstream Tasks — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• Hybrid Architectures — 1× in this paper · explained in 3 episodes
• Hybrid Models — 1× in this paper · explained in 2 episodes
• In-Context Learning — 1× in this paper · explained in 4 episodes
• KV cache — 1× in this paper · explained in 3 episodes
• Low-Rank Adaptation (LoRA) — 1× in this paper · explained in 9 episodes
• Office of the Director of National Intelligence — 1× in this paper
• Parameter-Efficient Fine-Tuning — 1× in this paper · explained in 2 episodes
• Transformer Models — 1× in this paper · explained in 2 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
• Zamba: A Compact 7B SSM Hybrid Model
• RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale
• The Llama 3 Herd of Models
• GPT-4 Technical Report
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Measuring Massive Multitask Language Understanding
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• Mistral 7B
• LLaMA: Open and Efficient Foundation Language Models
• Language Models are Few-Shot Learners
• Linformer: Self-Attention with Linear Complexity
• RWKV: Reinventing RNNs for the Transformer Era
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering