A.X K2 Technical Report
cs.AI, cs.CL
Submitted: 2026-08-31
Updated: 2026-08-31
Comments: https://huggingface.co/skt/A.X-K2
Code: https://github.com/deepseek-ai/DeepGEMM
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce A.X K2, a 688B-parameter Mixture-of-Experts (MoE) language model trained from scratch as a high-performance foundation for agentic applications.
Terminology
Abstract
We introduce A.X K2, a 688B-parameter Mixture-of-Experts (MoE) language model trained from scratch as a high-performance foundation for agentic applications. Trained on approximately 8.5T tokens---fewer than its predecessor, A.X K1---on a smaller but higher-quality mixture with substantially expanded agentic and software-engineering data, it nonetheless improves over A.X K1 across the board, by over 30 percentage points on some benchmarks, reflecting large gains in token efficiency. To support long contexts efficiently, we introduce Sparse Gated Attention (SGA), which combines sparse attention with gated attention, and adopt Gated Norm (GN) to stabilize large-scale training. SGA is trained natively at 128K through a sparse indexer warmup that optimizes the indexer against its own sparse top- k selection rather than the dense attention distribution, making adaptation markedly cheaper: each query reads only 2,048 positions, yet long-context quality is unchanged and A.X K2 scores 94.6 on RULER out to 256K. The outlier suppression of GN in turn keeps 4-bit NVFP4 serving within one point of FP8 accuracy. A simple yet effective Think-Fusion recipe further lets users switch between thinking and non-thinking modes within a single unified model. Extensive evaluations show that A.X K2 performs competitively against strong open-weight baselines, matching or exceeding them on math and Korean-language benchmarks.
Sources
- EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Scaling Vision Transformers to 22 Billion Parameters
- Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
- The Llama 3 Herd of Models
- Gemma 3 Technical Report
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- ThinkDial: An Open Recipe for Controlling Reasoning Effort in Large Language Models
- From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Kimi K2: Open Agentic Intelligence
- Reducing Activation Recomputation in Large Transformer Models
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
- DeepSeek-V3 Technical Report
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection