SPT: Skills as Pre-Training Data for Agentic Language Models

arXiv:2608.26563 · cs.CL · Submitted 2026-08-27 · Read on arXiv

cs.CL

Submitted: 2026-08-27

Updated: 2026-08-27

Code: https://github.com/huggingface/trl

License: http://creativecommons.org/licenses/by/4.0/

The gist: Agentic (tool-using) language models are mainly trained on tool-call traces and agent trajectories during post-training.

Terminology

Abstract

Agentic (tool-using) language models are mainly trained on tool-call traces and agent trajectories during post-training. These data provide direct behavioral supervision, but producing them requires task environments, execution, and verification, making broad tool and task coverage expensive. Publicly available skills offer another source of training data: they encode reusable tool semantics and workflows but are typically used only as inference-time context. We introduce Skill Pre-Training (SPT), a mid-training method that applies causal language modeling to SkillCorpus, a collection of public multi-file skill packages, optionally mixed with general data. To preserve relations among files within each package, we also introduce Reference Insert, a reference-aware assembly strategy that places supporting files near their mentions in the primary instruction. Experiments across multiple model scales and post-training recipes show that SPT consistently improves agentic performance over mid-training on general or trajectory data, while largely preserving general performance. Data mixture experiments show additional benefits from combining skill data with general annealing corpora. These results indicate that skill packages are a valuable data source for pre-training agentic language models.

Sources

Related papers