SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement
Arther Tian, Alex Ding, Simon Wu, Aaron Chan
cs.LG, cs.AI, cs.CR
Submitted: 2026-07-21
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Procuring supervised fine-tuning (SFT) data forces a buyer to decide, before any downstream training, whether a candidate corpus is worth acquiring.
Terminology
Abstract
Procuring supervised fine-tuning (SFT) data forces a buyer to decide, before any downstream training, whether a candidate corpus is worth acquiring. We present, a statistics-first gating architecture that treats procurement as a cost-aware routing problem over three intrinsic quality axes -- diversity, utility, and redundancy. Cheap blind measurements are summarised into per-axis estimates with confidence intervals; a gate accepts a decision only when intervals are tight, sample sizes are adequate, and the axes agree, otherwise it escalates the case to an adjudicative debate between a buy-advocate and a reject-advocate judge, resolved by a presiding verdict. On a controlled benchmark of 12 datasets (2 times 3 times 2 grid over the three axes) with 5 seeds, the gate reaches 0.90 accuracy and 0.83 F 1 at 0.017 per unit, sitting between an always-verify baseline (0.75) and an oracle upper bound (0.98) while spending less than always-escalate (0.020). We further report honest negative diagnostics of the debate path: a con-side win rate of 0.80 (p about3 times 10-6) and a 52% position-flip rate under advocate swapping expose negativity and positional biases that a naive LLM-judge would hide. We frame the injected-knob evaluation explicitly as a controlled synthetic benchmark for measurement fidelity and routing calibration, and delimit external validity as future work.
Sources
- SemDeDup: Data-efficient learning at web-scale through semantic deduplication
- A Survey on Data Selection for Language Models
- Instruction Mining: Instruction Data Selection for Tuning Large Language Models
- AlpaGasus: Training A Better Alpaca with Fewer Data
- Improving Factuality and Reasoning in Language Models through Multiagent Debate
- Data Shapley: Equitable Valuation of Data for Machine Learning
- On Calibration of Modern Neural Networks
- AI safety via debate
- Deduplicating Training Data Makes Language Models Better
- From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning
- Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models
- Training language models to follow instructions with human feedback
- Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- Large Language Models are not Fair Evaluators
- Self-Instruct: Aligning Language Models with Self-Generated Instructions
- Finetuned Language Models Are Zero-Shot Learners
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks