Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)

arXiv:2609.24052 · cs.CL · Submitted 2026-09-21 · Read on arXiv

cs.CL

Submitted: 2026-09-21

Updated: 2026-09-21

Code: https://github.com/pozapas/jev-calibrated-narra

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Calibration — 141× in this paper · explained in 7 episodes
• Accuracy — 32× in this paper · explained in 1 episode
• Generative Model — 14× in this paper · explained in 2 episodes
• Generative Models — 10× in this paper · explained in 2 episodes
• Abstention — 6× in this paper · explained in 1 episode
• Steering — 5× in this paper · explained in 2 episodes
• Fidelity — 4× in this paper · explained in 2 episodes
• Transformer — 4× in this paper · explained in 2 episodes
• Verification — 3× in this paper · explained in 2 episodes
• Aggregation — 2× in this paper · explained in 1 episode
• Benchmark — 2× in this paper · explained in 2 episodes
• Consistency — 2× in this paper · explained in 3 episodes
• Crash Records Information System — 2× in this paper
• Domain Shift — 2× in this paper · explained in 4 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Agent Harness — 1× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 4 episodes
• Calibration Failure — 1× in this paper · explained in 2 episodes
• EMMA — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Human-in-the-Loop — 1× in this paper · explained in 3 episodes
• International Conference on Machine Learning and Applications — 1× in this paper
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Systematic Bias — 1× in this paper · explained in 3 episodes
• Theoretical Guarantees — 1× in this paper · explained in 2 episodes

Sources

• Conformal Risk Control
• Benchmarking Frontier Large Language Models Against Official Crash Database Coding Using Police Crash Narratives
• Reliability, Sufficiency, and the Decomposition of Proper Scores
• From Narratives to Probabilistic Reasoning: Predicting and Interpreting Drivers' Hazardous Actions in Crashes Using Large Language Model
• Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
• Calibration of Pre-trained Transformers
• Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited
• SelectiveNet: A Deep Neural Network with an Integrated Reject Option
• Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination
• Analysis of Railway Accidents' Narratives Using Deep Learning
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Language Models (Mostly) Know What They Know
• Active Testing: Sample-Efficient Model Evaluation
• Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
• Verified Uncertainty Calibration
• Teaching Models to Express Their Uncertainty in Words
• An Agentic Workflow for Detecting Personally Identifiable Information in Crash Narratives
• Large Language Models in Analyzing Crash Narratives -- A Comparative Study of ChatGPT, BARD and GPT-4

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering