Do Value Vectors in Deep Layers Need Context from the Residual Stream?
Muyu He, Yuchen Liu, Qingya Huang, Li Zhang
cs.CL
Submitted: 2026-08-23
Updated: 2026-08-25
Comments: 13 pages, 5 figures. Code: https://github.com/RiddleHe/nanochat
Code: https://github.com/RiddleHe/nanochat
License: http://creativecommons.org/licenses/by/4.0/
The gist: The success of the transformer architecture as the backbone of modern LLMs is in large part due to its use of attention layers.
Terminology
Abstract
The success of the transformer architecture as the backbone of modern LLMs is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn only a context-free value vector to preserve the original token information, without drawing on any context from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component provides little additional benefit for aggregate benchmark performance. Such context-free value vectors can be stored as sparse model parameters, eliminating the need to recompute or persistently cache these values. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. Across 135M and 780M models, BoV improves validation loss over standard attention and, at 780M, the average score across 21 benchmarks, matching the previous best method that adds token information to the value vector with less compute and memory.
Sources
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
- MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers
- Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals
- DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
- Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers
- The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference
- Attention Residuals
- MoVE: Mixture of Value Embeddings -- A New Axis for Scaling Parametric Memory in Autoregressive Models
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
- Fast Transformer Decoding: One Write-Head is All You Need
- Value Residual Learning
- LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- Mixture-of-Depths Attention
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering