The Extender: A Log-Structured Transformer
cs.LG, cs.AI
Submitted: 2026-09-26
Updated: 2026-09-26
Code: https://github.com/karpathy/nanochat
Terminology
Sources
- Relational inductive biases, deep learning, and graph networks
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Gemma 2: Improving Open Language Models at a Practical Size
- GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Distilling the Knowledge in a Neural Network
- Training Compute-Optimal Large Language Models
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- Fast Transformer Decoding: One Write-Head is All You Need
- GLU Variants Improve Transformer
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Retentive Network: A Successor to Transformer for Large Language Models
- LLaMA: Open and Efficient Foundation Language Models
- MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks