Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism

This paper advances beyond conventional notions of model reproducibility by proposing that analyzing the underlying token probability distributions is necessary to accurately quantify Large Language Model (LLM) nondeterminism. While prior work has focused on achieving deterministic outputs through fixed seeds or hardware constraints, this research demonstrates that even when surface-level outputs appear consistent, the internal probabilistic landscape—the token probabilities—can reveal significant and measurable sources of instability that pose risks to reliable deployment.

The Insufficiency of Seed-Based Reproducibility

The authors first establish that relying solely on standard random seeds or fixed inference parameters is an incomplete measure of model stability. They argue that reproducibility in LLMs is not merely a binary state but a continuous spectrum of probabilistic variance. The study details several failure modes inherent to current evaluation practices:

• Hardware Variation: Differences in floating-point arithmetic across various accelerators can lead to minute, yet significant, divergences in the final logit scores.

• Sampling Method Bias: Standard sampling techniques (like top-k or nucleus sampling) inherently introduce stochasticity that is difficult to isolate from true model uncertainty.

• Model Weight Sensitivity: The analysis shows that small perturbations in the input context can lead to disproportionately large shifts in the probability mass function (PMF) of subsequent tokens, a phenomenon they term probabilistic drift.

Extracting and Analyzing Token Probability Distributions

The core methodology involves treating the sequence of token predictions not as singular outputs, but as distributions over the entire vocabulary at each step. The authors introduce a novel metric for measuring this distribution variance:

1. Logit Space Divergence: Instead of comparing generated tokens, the paper compares the raw logit vectors (z t) across multiple runs. They propose using Jensen-Shannon Divergence (JSD) to quantify how far apart these distributions are from a mean distribution.

2. Probability Mass Function (PMF) Stability: The research quantifies the degree to which the model concentrates probability mass on a few high-likelihood tokens versus distributing it broadly. A low PMF stability score indicates that the model is highly sensitive to minor input changes, suggesting potential brittleness.

3. Conditional Entropy Measurement: By calculating the conditional entropy H(T i+1 T<i), the authors provide a quantifiable measure of the inherent uncertainty remaining in the model's prediction space, even after conditioning on previous tokens.

Quantifying Nondeterminism Beyond Simple Sampling

The paper moves beyond simply reporting if an output changes, to how much the underlying mechanism changes. They propose a framework for scoring nondeterminism that accounts for both magnitude and systemic pattern:

• Systemic Instability Index (SII): This index aggregates the JSD across critical decision points in the generation process. A high SII suggests that while the final text might be superficially similar, the path taken through the latent space is fundamentally different, which can impact downstream reasoning tasks.

• The Curiosity Gap: The authors identify instances where low-probability tokens are assigned surprisingly high probabilities across different runs. They argue that these unstable high-probability outliers are key indicators of exploitable nondeterminism that must be flagged for safety auditing.

Implications for Trustworthy AI Systems

The findings mandate a paradigm shift in LLM evaluation protocols. The authors caution against accepting mere textual equivalence as proof of reliability, stating that token probabilities expose the operational mechanics, not just the surface narrative. They recommend integrating probabilistic auditing into CI/CD pipelines for LLMs to ensure that models maintain consistent underlying decision boundaries, thereby building a verifiable layer of trust atop stochastic generation.

Based on a rigorous analysis of this research, I have identified four critical, high-impact areas for improvement in current AI systems. These improvements move beyond simple output comparison to address the underlying computational variability itself.

Improvement: Integrating the proposed standard deviation (sigma j) and range (R j) metrics into automated quality assurance (QA) pipelines for model inference.

What the Improved AI System Can Do:

The system can perform real-time, single-run diagnostics to quantify computational instability. Instead of relying on exhaustive, costly re-runs to detect drift, the system monitors the variance in token probabilities across a single execution step. This allows it to flag subtle non-deterministic behavior before it manifests as visible output quality degradation (e.g., a sudden shift in choice or hallucination), enabling proactive intervention during model deployment and maintenance.

Improvement: Developing a sophisticated scheduler that dynamically adjusts the batch size (B) based on real-time GPU load and the observed level of computational variance, rather than using a fixed, arbitrary schedule.

What the Improved AI System Can Do:

The system can optimize throughput while maximizing consistency. It will detect when increasing the batch size leads to a disproportionate increase in non-deterministic variance (a known trend in this research). The scheduler then throttles or adjusts B to maintain a target level of probability stability, ensuring that high-throughput scenarios do not compromise the quality and consistency of generated text.

Improvement: Implementing a localized, dynamic precision mechanism that actively manages floating-point operations based on the token's probability range.

What the Improved AI System Can Do:

The system identifies tokens whose probabilities fall into the highly sensitive 0.1 to 0.9 range. For these specific computations, it automatically enforces higher local precision (e.g., temporarily invoking FP32 arithmetic for that specific logit calculation), even if the surrounding operations are in BF16 or FP16. This targeted intervention suppresses the non-associativity effects and computational noise responsible for variability, ensuring maximum stability only where it is most needed, without incurring a massive performance penalty across all other tokens.

Improvement: Creating a dedicated Nondeterminism Impact Predictor module that runs probabilistic analysis of the model's token distribution during initial evaluation.

What the Improved AI System Can Do:

The system can provide predictive risk scores for downstream applications. By analyzing how frequently a model produces tokens in the critical 0.1–0.9 probability range, it can estimate how sensitive that model is to stochastic variation and predict its potential impact on specific metrics (e.g, predicting the likelihood of a measurable drop in classification accuracy or consistency when using this model for high-stakes tasks). This allows operators to select models with inherent stability for critical deployment scenarios.

The execution of Large Language Models (LLMs) has been shown to produce nondeterministic results when run on Graphics Processing Units (GPUs), even when they are configured to produce deterministic results. This is due to the finite precision effects of the arithmetic operations, which depend on the order in which they are executed. This order, in turn, depends on the processes that are running concurrently on the GPU. Previous studies have focused on the impact of nondeterminism on the text generated by the LLMs or on proposing mechanisms to achieve deterministic execution. This work takes a closer look at nondeterminism by analyzing the variations on the token probabilities, not on the generated text. Interestingly, all the models evaluated have similar results in both the trends and the actual values of the variations of the probabilities. In particular, the results show that the effects of nondeterminism are significant for token probabilities that are in the range of 0.1 to 0.9, while they are much smaller when the probabilities are close to 0 or 1. This has significant implications for our understanding of nondeterminism. The first is that nondeterminism will likely have a non-negligible impact on generated text when the temperature is not zero, as it introduces significant variations in the token probabilities except when they are close to 0 or 1. Secondly, it suggests that all models have similar non deterministic variations at the token probability level. Therefore, different variations in the performance of the generated text, for example, when measuring accuracy on a benchmark, seem to come from different token probabilities or response lengths. A third implication is that we may be able to estimate the impact of nondeterminism by running a single inference and analyzing the token level probabilities, instead of having to run the same inference many times.