PruneShift: A Framework for Evaluating Decision Reliability in Structured Pruning

PruneShift: A Framework for Evaluating Decision Reliability in Structured Pruning

The paper introduces PruneShift, an evaluation framework designed to address a critical gap in structured pruning evaluations where direct task evaluation over every feasible mask is too expensive. Current methods often report average surrogate error or rank correlation on broadly sampled masks, but these summaries fail to account for the fact that the selector, however, searches for masks with unusually favorable surrogate values. This creates an evaluation gap: the surrogate is checked on a broad sample, while the selection mechanism targets specific regions. The paper argues that an error that is rare under broad sampling can be common among the masks that compete for selection, meaning average error and rank correlation can still look strong. The selected mask can nevertheless be poor.

PruneShift addresses this by defining a decision reliability evaluation framework that separates three distinct domains:

1. Broad predictive fidelity: Fidelity across generic feasible masks (the broad validation distribution Q U).

2. Fidelity near selector outputs: Accuracy in the neighborhoods reached by selection (Q N).

3. Quality of the selected pruning decision: The performance against declared alternatives on independent data (the comparison domain C S).

The paper makes four primary contributions:

1. Definition of Decision Reliability: It defines a decision reliability evaluation for pruning surrogates, separating broad prediction, selector-neighborhood prediction, and finite comparison regret, while also separating estimator error from selector error.

2. Theoretical Guarantees: The authors prove that Spearman and Kendall agreement can approach one while normalized selection regret remains maximal. They further derive sufficient conditions based on key metrics including uniform error, selector suboptimality, decision margin, density ratio, and comparison mass, and also yield a finite pool certificate with an explicit excess cost bound.

3. ** Operational Routes to Stronger Evidence:** The paper provides two operational routes for achieving stronger evidence: a fixed-look finite-pool certificate (which provides a guaranteed cost within a fixed set) and independent confirmation (which directly tests a prespecified comparison).

4. ** Empirical Testing:** The framework is tested through four distinct studies: external transport, finite-pool selection, controlled coverage intervention, and public OSSCAR reconstruction.

The empirical results demonstrate the necessity of this separation:

• External TextbookQA confirmation: This study found that 7 of 20 simultaneous intervals favor the surrogate-selected mask, 6 favor its fixed comparator, and 7 cross zero, indicating heterogeneity in transfer.

• Natural Questions finite pool: On a fixed pool, strict improvement holds in one of four settings. However, the certificate is scoped to this pool and does not compare the pool winner with a different pruning strategy.

• Controlled QQP experiment: This study showed that the proposed coverage mechanism in all 16 prespecified endpoints, although the derived sufficient bounds were noted to be conservative.

• OSSCAR reconstruction study: A restricted analysis on OPT-125M shows better local than broad fidelity in 68 of 75 primary endpoints.

In conclusion, the paper asserts that these findings show why predictive fit, decision reliability, and pruning method quality require separate evidence, providing a clearer standard for evaluating the decisions made by structured pruning surrogates.

The following improvements represent a fundamental shift from evaluating surrogate performance to evaluating decision reliability.

1. Tripartite Evaluation Framework Implementation:

We replace traditional single-metric evaluation (e.g., average surrogate error or rank correlation) with the PruneShift framework, integrating three distinct validation domains:

• Broad Validation (Q U): A generic, uniform sampling of the feasible mask space to measure global predictive fidelity (Spearman/Kendall agreement). This serves as a baseline.

• Selector Neighborhood (Q N: A targeted distribution focusing on regions surrounding the masks most likely to be selected by the pruning algorithm. This explicitly measures local error exposure that is often missed by broad sampling.

• Decision Comparison Set (C S): A fixed, finite pool containing the selected mask and several declared alternatives. Evaluation here measures actual task loss (regret) on this known set, providing a ground truth for decision quality independent of the surrogate's performance.

2. Deployment of Theoretical Certification Conditions:

We move beyond heuristic checks by integrating specific mathematical guarantees derived from PruneShift theory:

• Uniform Error (epsilon): Quantifying the maximum deviation between the surrogate and actual loss within a finite set C.

• Decision Margin (C): Calculating the separation between a selected mask and its alternatives. If C > 2 epsilon, we can mathematically certify that the chosen mask is locally optimal compared to the alternatives in C.

• Comparison Mass (q): Assessing how much evaluation weight is assigned to critical decision points, ensuring that low-mass, high-regret candidates are not overlooked.

• Coverage Transfer Mechanism: Explicitly measuring the ratio kappa = dQ/dP (density ratio) between the broad validation and the selector neighborhood. This determines if a single tail error can be amplified by N-fold when transitioning from general validation to targeted selection, allowing us to quantify where predictive fit is unreliable.

3. Refining Surrogate Optimization Goals:

We modify the objective function optimization process (the selector) to explicitly account for the observed local fidelity (Q N) and the guaranteed comparison quality (C S), rather than simply chasing maximum surrogate reduction.

The improved system provides a comprehensive, quantifiable assessment of a pruning decision, moving from correlation to certification:

• Determine Reliability vs. Quality: It can distinguish between a model that looks good based on average surrogate scores (high Q U fidelity) and a model whose actual choice is reliable when compared against alternatives (low C S regret).

• Quantify Risk: It provides an explicit upper bound on the potential excess cost of the selected mask relative to its competitors, based on measured epsilon, C, and q (e.g., With 95% confidence, the selected mask's regret is no more than X relative to the best candidate in this pool).

• Diagnose Failure Modes: It identifies why a pruning strategy fails: Is it because the surrogate is inaccurate globally (low Q U), or because it fails specifically in the critical regions explored by the selection process (high amplification factor kappa from Q U to Q N)?

• ** Certify Local Optimality:** It can mathematically prove that a selected mask is locally optimal within a specified, finite set of alternatives, even if it does not achieve global optimality across the entire feasible space.