LoGo: Token-Level Dynamic Local-Global Attention
cs.CL, cs.LG
Submitted: 2026-08-30
Updated: 2026-08-30
Code: https://github.com/ByteDance-Seed/seed-oss
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- gpt-oss-120b & gpt-oss-20b Model Card
- Longformer: The Long-Document Transformer
- Conditional Computation in Neural Networks for faster models
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths
- Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning
- Zamba: A Compact 7B SSM Hybrid Model
- The Llama 3 Herd of Models
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Mistral 7B
- MiniMax-01: Scaling Foundation Models with Lightning Attention
- Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
- Jamba: A Hybrid Transformer-Mamba Language Model
- LoLA: Low-Rank Linear Attention With Sparse Caching
- Stem: Rethinking Causal Information Flow in Sparse Attention
- Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Retentive Network: A Successor to Transformer for Large Language Models
- Gemma 2: Improving Open Language Models at a Practical Size
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering