Invertible Logits Transformation for Accuracy-Preserving Post-Hoc Uncertainty Calibration

The paper introduces Invertible Logits Transformation (InvLT), a post-hoc calibration method that applies a learned scalar MLP f: R to R element-wise to pre-softmax logits. The method is designed to correct nonlinear miscalibration while preserving the original classifier's predictions (accuracy preservation), scaling gracefully to large label spaces, and avoiding the computational overhead of prior monotone calibrators.

Post-hoc calibration aligns a classifier's predicted confidences with its empirical accuracy without retraining. The paper identifies that existing methods violate at least one of three desirable properties: correcting nonlinear miscalibration, scaling to large label spaces, and preserving original predictions. Specifically:

• Temperature scaling (TS) divides all logits by a single learned scalar T > 0, preserving the argmax but having only one degree of freedom—it cannot correct miscalibration that varies across confidence levels.

• More flexible parametric alternatives introduce parameters that grow with the number of classes C.

• Other expressive methods do not preserve the rank ordering of class scores and may alter the predicted class.

InvLT applies a shared scalar function element-wise to logits:

[

phi f(z) = (f(z 1), f(z 2),, f(z C))

]

where f: R to R is invertible and shared across all classes and samples. The calibrated probabilities are cal = softmax(phi f(z)). The key property: strict increase of f is sufficient to preserve accuracy because the ordering of all logits is unchanged.

Rather than enforcing monotonicity through architectural constraints (as UMNN does via numerical integration), InvLT uses a reconstruction regularizer based on the classical result that a continuous injective function on an interval is strictly monotone (Proposition 1, proved via the Intermediate Value Theorem in Appendix A).

The method trains f theta jointly with an auxiliary inverse network g psi, minimizing:

[

L rec = 1 over K sum k=1 K g psi(f theta(u k)) - u k squared

]

over reference points u k k=1 K drawn from the operating range of calibration logits. This drives f toward injectivity, and hence toward monotonicity, without imposing any architectural restriction.

The complete objective combines a calibration loss (NLL or Brier) with the reconstruction regularizer:

[

L = L Brier/NLL + lambda times L rec

]

where lambda > 0 controls regularization strength. A warm-up technique activates the regularization after certain epochs, allowing the calibrator to first reduce calibration error before monotonicity is enforced. Both networks are optimized jointly with Adam.

1. Training avoids numerical integration: each step adds only K scalar evaluations of g f on a precomputed grid.

2. Inference reduces to a small MLP applied element-wise, comparable in cost to temperature scaling.

3. ** f can be any standard MLP** with any activation function, simplifying implementation and tuning.

4. **Parameter count is independent of C **: sharing f across all logit dimensions makes this scale gracefully to ImageNet (C = 1, 000).

5. Introduction of InvLT, a post-hoc calibrator applying a shared scalar MLP element-wise to logits with parameter count independent of C.

6. A reconstruction regularizer via an inverse network that softly enforces monotonicity, avoiding numerical integration required by prior monotone calibration methods.

7. Consistent outperformance of all baselines across CIFAR-10, CIFAR-100, and ImageNet with seven architectures, while being substantially faster to train than UMNN.

• CIFAR-10 and CIFAR-100: ResNet-50 as primary backbone, evaluated across five random seeds.

• CIFAR-100 architecture generalization: VGG-16/19, DenseNet-121, Wide ResNet, and ViT-B/16.

• ImageNet: pretrained ResNet-50 and ResNet-152 checkpoints.

• Calibration set sizes: N = 500, N = 5, 000, N = 25, 000 held-out samples.

Compared against No Calibration, Temperature Scaling, Vector/Matrix Scaling, Ensemble Temperature Scaling (ETS), Parameterized Temperature Scaling (PTS), Histogram Binning, Isotonic Regression, Spline Calibration, Dirichlet Calibration, and UMNN Calibration. Methods are separated into those that may alter the predicted class and those that preserve the original classification decision by construction.

Expected Calibration Error (ECE, 15 equal-width bins), Adaptive ECE (AdaECE), KDE-ECE, and Negative Log-Likelihood (NLL).

• CIFAR-10: InvLT achieves ECE = 0.60% and NLL = 0.15, both best among all methods. KDE-ECE = 0.98%, outperforming Histogram Binning at 1.09%.

• CIFAR-100: InvLT achieves ECE = 1.67%, outperforming the best accuracy-preserving baseline UMNN at 2.48%, and matches the lowest NLL (0.79).

• ImageNet: InvLT achieves ECE = 0.39%, ahead of UMNN (0.56%) and far below Spline Calibration (1.10%), while also obtaining the best NLL.

InvLT consistently achieves the best ECE and NLL at all three sizes, with its largest lead over UMNN in the low-data val 500 regime. Notably, class-dependent methods degrade severely at val 500: Vector Scaling reaches ECE = 43.44% and Isotonic Regression 33.46%, while Matrix Scaling collapses to the identity map.

InvLT ranks first on every backbone with an average ECE of 2.41% vs. 3.08% for the next-best method (PTS). Temperature scaling is consistently 2–7× worse in ECE.

InvLT matches or outperforms UMNN on ECE and NLL in every setting while being 3.5× faster to fit (22.7 s vs. 80.2 s on CIFAR-100) and 5× faster at inference (74.8 ms vs. 372.7 ms). The gap stems from UMNN's numerical quadrature at every forward pass versus InvLT's lightweight reconstruction penalty discarded at test time.

Figure 1 shows learned transformations from val 1,000 and val 5,000 are nearly indistinguishable on every dataset, demonstrating stability even from limited calibration data. All learned maps are strictly monotone. The learned maps deviate substantially from identity in a dataset-dependent manner: on CIFAR-10, f theta compresses high logits; on CIFAR-100 and ImageNet, it amplifies positive logits relative to negative ones.

Figure 2 shows InvLT achieves the closest alignment to the perfect-calibration diagonal with uniformly small gaps across confidence bins, while TS and ETS show systematic overconfidence in the mid-confidence range.

Table 5 compares InvLT with and without the reconstruction loss on CIFAR-10 (ResNet-50, val 5,000):

• Without regularization: classification error increases (4.52% → 4.69%), confirming an unregularized f theta can become non-monotone and alter the argmax.

• With regularization: classification error returns to the original model's accuracy (4.52%), and both ECE and NLL improve, suggesting the invertibility constraint acts as a beneficial inductive bias.

Additional ablations (Table 13) show the number of reconstruction samples K controls a trade-off: K at least 25 ensures accuracy preservation while maintaining competitive calibration. Warmup delay has negligible effect.

The paper acknowledges two limitations in Section 5:

1. Element-wise design cannot capture cross-class dependencies in miscalibration structure. A natural extension would introduce lightweight cross-class interactions, though this risks overfitting on small calibration sets.

2. Soft monotonicity enforcement does not formally guarantee argmax preservation; in extremely safety-critical settings, a post-hoc argmax check may be advisable.

InvLT offers a simple, efficient, and flexible approach to post-hoc calibration: a shared scalar network applied element-wise to logits, with monotonicity softly enforced via a paired inverse network that is discarded at test time. Across CIFAR-10, CIFAR-100, and ImageNet with seven architectures, InvLT consistently achieves the best ECE and NLL among all compared methods, preserves accuracy, and is substantially faster to train than UMNN. Future work includes exploring hard monotonicity constraints via non-negative weights, incorporating cross-class interactions, and extending evaluation beyond image classification.

Improvements to AI systems based on this paper:

1. Add a lightweight, accuracy-preserving calibration layer to any classifier – The Invertible Logits Transformation (InvLT) can be appended post-hoc to any pre-trained model (e.g., vision, NLP, or speech) to correct confidence scores without retraining or altering predictions. The improved system will output calibrated probabilities that match empirical accuracy across all confidence levels, reducing overconfidence in mid-range predictions.

2. Enable scalable calibration for large label spaces – Since InvLT’s parameter count is independent of the number of classes (shared scalar MLP), it can calibrate models with thousands of classes (e.g., ImageNet, fine-grained classification, or multi-label tasks) where prior methods like matrix scaling become computationally prohibitive. The improved system will maintain calibration quality even with 1,000+ classes.

3. Provide fast, low-overhead calibration for real-time deployment – InvLT’s inference cost is comparable to temperature scaling (a single small MLP applied element-wise), making it suitable for edge devices, real-time inference, or high-throughput APIs. The improved system will correct miscalibration without adding meaningful latency or memory overhead.

4. Improve reliability in safety-critical decision-making – By preserving the original argmax (via monotonicity) and aligning confidence with accuracy, the improved AI system will enable safer thresholding for autonomous systems (e.g., medical diagnosis, self-driving cars, fraud detection) where overconfident wrong predictions are costly. The system can flag low-confidence cases for human review more accurately.

5. Enable robust calibration with small calibration datasets – InvLT remains stable and accurate even with as few as 500 calibration samples (as shown on ImageNet), unlike class-dependent methods that collapse. The improved system will be deployable in low-data regimes, such as specialized domains with limited labeled examples, without sacrificing calibration quality.

6. Provide a plug-and-play module for existing ML pipelines – InvLT can be integrated as a final post-processing step after any classifier’s logits, requiring only a small held-out calibration set and a few minutes of training. The improved system will automatically correct miscalibration in production models without architectural changes or retraining, reducing deployment friction.

7. Support flexible calibration loss functions – The method works with both NLL and Brier score, allowing the improved system to optimize for either probabilistic accuracy or squared-error calibration depending on the application’s needs (e.g., Brier for ranking, NLL for generative tasks).

8. Enable interpretable calibration behavior – The learned scalar function f can be visualized to reveal dataset-specific miscalibration patterns (e.g., compression of high logits on CIFAR-10 vs. amplification of positive logits on ImageNet). The improved system can provide insights into model bias, helping developers understand and debug confidence errors systematically.

9. Reduce training time for calibration – InvLT is 3.5× faster to fit than prior monotone calibrators (UMNN) and 5× faster at inference, enabling rapid iteration on calibration for multiple models or datasets. The improved system can be recalibrated frequently (e.g., after each model update) without significant compute cost.

10. Serve as a baseline for future calibration research – The improved system can incorporate InvLT as a strong, simple baseline in any calibration benchmark, allowing researchers to compare novel methods against a method that is both accurate and efficient, accelerating progress in the field.

Post-hoc calibration aligns a classifier's predicted confidences with its empirical accuracy without retraining. An ideal calibrator should correct nonlinear miscalibration, scale gracefully to large label spaces, and preserve the original predictions; existing methods typically violate at least one of these properties---temperature scaling lacks expressivity, more flexible parametric alternatives introduce parameters that grow with the number of classes C, and other expressive methods do not preserve the rank ordering of class scores and may alter the predicted class. We propose Invertible Logits Transformation (InvLT), which applies a learned scalar MLP f: R to R element-wise to the pre-softmax logits. Sharing f across all logit dimensions makes the parameter count independent of C. Monotonicity of f ---and hence preservation of the argmax prediction---is softly encouraged via a paired inverse network rather than enforced through the numerical integration required by prior monotone calibrators; this avoids their computational overhead while empirically preserving the original classification accuracy in every setting we evaluate. Across standard image classification benchmarks and a range of architectures, InvLT consistently outperforms a broad set of post-hoc baselines on standard calibration metrics.