Post-Calibration Reliability Reranking of Relevance Decisions via Label-wise Monotone Projection

Web search, product search, and question-answering retrieval systems often assign a relevance label and confidence score to each query-candidate pair. The relevance label describes how well a page, product, or passage matches the query, while the confidence often guides downstream use or fallback decisions. Post-hoc calibration is therefore needed because misaligned confidence can make systems over-trust wrong predictions or unnecessarily defer correct ones. However, calibration mainly aligns confidence with average correctness, and does not remove predicted-label-dependent reliability differences that remain within the same calibrated confidence level. We address this gap with Label-wise Monotone Reliability Projection (MRP), which learns label-wise monotone functions that map calibrated confidence to correctness reliability while preserving the original predicted labels and class probabilities. The resulting reliability score reranks fixed predictions according to residual risk. Across six information access relevance datasets and multiple post-hoc calibrators, MRP improves reliability reranking and average fallback utility while preserving full-coverage accuracy and ECE. Structural ablations show that the main gains come from label-wise residual reliability rather than from global confidence remapping. We further analyze when MRP reliability scores can be embedded back into top-label probability geometry, showing that this projection is useful as a compatibility analysis but is distinct from the main reliability-reranking objective. The implementation will be made publicly available.

Improvements to AI Systems:

1. Add a post-hoc reliability-reranking layer to any retrieval or QA system that outputs confidence scores. After calibration, train label-wise monotone functions (MRP) on validation data to map calibrated confidence to a separate reliability score per predicted label. This layer reorders the fixed top-k predictions by residual risk, so the system can surface the most trustworthy candidate first without retraining the base model.

2. Enable label-aware fallback decisions in production systems. Instead of using a single global confidence threshold for deferral (e.g., to human review or a more expensive model), use the MRP reliability score per predicted label. This allows the system to defer only when the reliability for that specific label is low, reducing unnecessary fallbacks for high-reliability labels and catching more errors for low-reliability labels.

3. Improve ensemble or multi-stage retrieval pipelines by using MRP scores as a secondary ranking signal. After the first-stage retriever produces candidates and a cross-encoder assigns calibrated probabilities, apply MRP to re-rank within each predicted-label group. This yields better final ordering for downstream tasks like answer generation or product recommendation, especially when the top-1 is wrong but a lower-ranked candidate is correct.

4. Provide a compatibility check for probability-based explanations. Use the MRP projection back into top-label probability space as a diagnostic tool: if the reliability score can be embedded into the probability geometry, the system can report a reliability-adjusted probability for interpretability. This helps users understand when a high-confidence prediction is still risky.

What the Improved AI System Can Do:

• Reduce over-trust in wrong high-confidence predictions by re-ranking them below correct lower-confidence predictions, without changing the model’s original labels or probabilities.

• Maintain full-coverage accuracy and calibration error (ECE) while improving average fallback utility—so it can safely defer fewer correct cases and catch more incorrect ones.

• Adapt reliability per label (e.g., relevant vs. not relevant in search, or entailed vs. not entailed in QA), avoiding the pitfall of treating all labels with the same confidence-reliability curve.

• Operate as a drop-in post-processing module on top of any existing calibrated retriever or classifier, requiring only a small validation set to fit the monotone functions, with no changes to inference latency beyond a simple lookup or monotone interpolation.