Agentic Instruction Data Selection: Let DataMaster Interpret Your Intent

The paper introduces DataMaster, an agentic instruction data selection system that translates natural-language user requirements into task-specific selection strategies for Supervised Fine-Tuning (SFT) of Large Language Models (LLMs).

Motivation and Problem: Existing instruction data selection methods use static, single-metric or hard-coded multi-metric strategies that fail to generalize across diverse real-world datasets and dynamic user training needs. The paper states: these methods typically evaluate data using a static selection strategy, whereas the desired subset depends jointly on the heterogeneous composition of the data pool and the user’s dynamic training needs. Even within the same evaluation dimension, different user intentions lead to distinct selection configurations; for example, "for the user instructions 'Select mathematical reasoning data' and 'Select medical question answering data', even if both are filtered based on difficulty levels, the former, a reasoning task, may favor samples with moderate difficulty relative to the target model, whereas the latter, a knowledge-based task, may favor harder samples with long-tail knowledge."

Proposed Method: DataMaster's "central novelty lies in dynamic, intent-conditioned strategy configuration: rather than applying a fixed metric or a hard-coded combination of metrics, DataMaster jointly interprets the user’s instruction and inspects the candidate data pool to determine how selection should be performed for the current task. The system is instantiated through four cascaded agents operating over domain, characteristic, informativeness, and quality," with each agent configuring its selection behavior according to the inferred intent and the data encountered at its stage. The selection function is formalized as D∗ = Fqual ◦ Finfo ◦ Fchar ◦ Fdom (I, D, Φ), where I is the natural language instruction, D is the data pool, and Φ is the shared backbone LLM.

• Stage 1 (Domain Agent): Interprets user requirements, embeds samples, forms semantic clusters (using HDBSCAN or KMeans depending on size), labels clusters via LLM, and selects clusters matching the user's target domain.

• Stage 2 (Characteristic Agent): Uses tool-assisted data inspection to analyze data characteristics and dynamically generates filtering rules (e.g., keyword blacklists, length constraints, linguistic requirements) tailored to both the data and user intent.

• Stage 3 (Informativeness Agent): Assesses learning value relative to the target model by dynamically generating a task-specific weight vector over four metrics: NLL, token-level entropy, semantic drift, and MeanDiff (adapted from ResoFilter). The weights are allocated based on user intent; for example, NLL is prioritized for knowledge-intensive tasks to detect informational gaps but receives lower weights in reasoning tasks.

• Stage 4 (Quality Agent): Uses LLM-as-a-Judge with a dynamically formulated scoring rubric and weights, prioritizing criteria like factual correctness for knowledge-intensive tasks and logical rigor for reasoning tasks.

Experiments and Results: DataMaster was evaluated on six single-domain training pools spanning math, medical, and code, plus two multi-domain pools, using three 7B/8B-scale models (Qwen2.5-7B, Llama-3.1-8B, Olmo-3-7B-Instruct) with DeepSeek-V4-Flash as the backbone LLM.

• Single-domain results: DataMaster outperforms the static baselines in 16 of the 18 test cases and ranks first in 13 of 18 settings. It outperforms Random in all 18 settings with gains from 3.47 to 9.94 points. Against the dynamic baseline Claude Code, DataMaster nevertheless achieves a higher average score in 14 of 18 settings. Training on DataMaster's 10K-sample subsets outperforms training on the corresponding full, uncurated pool in 12 of the 18 settings, despite Full Pool using 2×–41× more examples.

• Multi-domain results: DataMaster outperforms all four static baselines in every test case, with gains ranging from 0.33 to 2.84 points over the strongest one. It also surpasses Claude Code... by 0.04 to 1.34 points across all trials.

• Ablation studies: Stage 1 ablation shows that using DataMaster's domain selection as a preprocessing module for SuperFiltering yields consistent improvements, but still lags behind the full pipeline. Stages 2-4 ablation shows the full cascade peaks on 16 of 18 benchmark rows, supporting complementary contributions of successive stages.

• Cost: Hosted-LLM call costs average less than US 5 per data-selection task (approximately US 4 for single-domain and US5 for multi-domain settings).

Conclusion: The paper concludes that DataMaster "achieves strong overall performance against static and agentic baselines in both single- and multi-domain settings while selecting compact subsets that can surpass full-pool training. By reducing reliance on manually designed heuristics, DataMaster offers a practical path to scalable, task-adaptive data curation for diverse model development." Future work includes examining larger models, broadening comparisons across curator models and inference budgets, and adapting stage activation and ordering.

Improvements to AI systems:

1. Intent-Conditioned Dynamic Data Curation: Replace static, one-size-fits-all data selection pipelines with a multi-agent cascade that interprets natural-language user instructions (e.g., select math reasoning data vs. select medical QA data) and dynamically configures filtering criteria, metric weights, and scoring rubrics per task. This enables SFT data selection to adapt to heterogeneous data pools and evolving training goals without manual re-engineering.

2. Adaptive Metric Weighting for Learning Value: Implement a stage that computes a task-specific weight vector over multiple informativeness metrics (NLL, token entropy, semantic drift, MeanDiff) based on the inferred intent—prioritizing NLL for knowledge-intensive tasks and logical rigor metrics for reasoning tasks—rather than using fixed or equal weights. This improves sample selection for target-model learning gaps.

3. Tool-Assisted, Context-Aware Filtering Rules: Integrate an agent that inspects the actual data pool (via embeddings, keyword analysis, length distributions) to generate bespoke filtering rules (e.g., keyword blacklists, length constraints) that are conditionally applied only when relevant to the user’s intent, reducing noise from irrelevant heuristics.

4. Dynamic LLM-as-a-Judge Rubric Generation: Replace static quality scoring with an agent that formulates a weighted rubric on the fly (e.g., factual correctness for knowledge tasks, logical coherence for reasoning tasks) and applies it via LLM-as-a-Judge, aligning quality assessment with the downstream task objective.

5. Cascaded Domain-to-Quality Selection Pipeline: Structure data selection as four sequential, complementary stages (domain → characteristic → informativeness → quality) where each stage narrows the pool and conditions the next stage’s behavior, leading to compact, high-yield subsets that can outperform full-pool training with 2×–41× fewer examples.

6. Cost-Efficient Agentic Orchestration: Enable the system to run on a hosted LLM backbone with a total cost under 5 per selection task, making adaptive, agent-driven data curation practical for routine model development rather than a research-only luxury.

What the improved AI system can do:

• Accept a natural-language instruction (e.g., select diverse code snippets for debugging skill) and automatically produce a tailored, high-quality SFT dataset from a mixed or single-domain pool, without human heuristic tuning.

• Dynamically re-weight selection criteria based on the task type (reasoning vs. knowledge) and the target model’s current weaknesses, leading to better downstream performance on domain-specific benchmarks (e.g., +3.47 to +9.94 points over random selection; outperforming static baselines in 16/18 single-domain and all multi-domain tests).

• Generate compact training subsets (e.g., 10K samples) that match or exceed the performance of training on the full uncurated pool (12/18 settings), reducing compute and data costs.

• Adapt to multi-domain pools by automatically partitioning and selecting relevant sub-clusters, outperforming static baselines and even a strong agentic baseline (Claude Code) across all trials.

• Provide interpretable, stage-wise data curation (domain clusters, filtering rules, informativeness weights, quality scores) that can be inspected and modified by users, improving transparency and trust in automated data engineering.

Although existing instruction data selection methods have introduced various metrics, the inherent complexity of real-world datasets makes it impractical for any single metric to generalize across all scenarios. Developers are thus often forced to manually inspect data and craft heuristic rules for each new application---a tedious and error-prone process. In this paper, we propose a paradigm shift from manual configuration to automated orchestration via the Instruction Data Selection Agent (DataMaster), which interprets user intent and autonomously composes optimal selection strategies. By allowing users to specify data needs through natural language descriptions, DataMaster simplifies data curation and removes the burden of manual strategy design. Extensive experiments across the math, medical, and code domains show that DataMaster outperforms static baselines in most settings and surpasses full-pool training in a substantial number of cases. The implementation of DataMaster and the scripts needed to reproduce the reported pipeline are publicly available at https://github.com/nju-websoft/DataMaster.