Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories
cs.LG, cs.AI, cs.CL
Submitted: 2026-09-29
Updated: 2026-09-29
Terminology
Sources
- Program Synthesis with Large Language Models
- WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models
- Evaluating Large Language Models Trained on Code
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- DiLoCo: Distributed Low-Communication Training of Language Models
- Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models
- Measuring Massive Multitask Language Understanding
- Measuring Mathematical Problem Solving With the MATH Dataset
- Training Compute-Optimal Large Language Models
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
- Scaling Laws for Neural Language Models
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
- GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
- CCPM: A Chinese Classical Poetry Matching Dataset
- Model Merging in Pre-training of Large Language Models
- MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks