Coverage is not enough: Frequentist tests of simulation-based inference for primordial non-Gaussianity

arXiv:2605.00980 · astro-ph.CO, astro-ph.IM, stat.ME · Submitted 2026-05-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Astrophysics Radio. Generated commentary on the latest astrophysics papers.

Vera: I'm Vera, and with me are Jocelyn and Subrahmanyan, guest researcher.

Jocelyn: Today's paper: "Coverage is not enough".

Vera: Simulation-based inference (SBI) methods are being developed to extract cosmological parameters from complex, non-linear data where analytical likelihoods are unavailable, but their reliability remains questionable.

Jocelyn: First, who's behind it and why it matters.

Paper summary: Vera: Well team, I'm really looking forward to discussing this paper today. It’s titled "Coverage is not enough: Frequentist tests of simulation-based inference for primordial non-Gaussianity," and it seems like it tackles a really tricky area in cosmology where we can't just rely on simple analytical likelihoods.

Jocelyn: I agree, Vera; the abstract makes it sound like they are poking holes in how we validate these simulation-based inference methods when trying to constrain primordial non-Gaussianity using dark matter halo simulations. What’s the core idea they are pushing?

Subrahmanyan: Basically, the paper is investigating whether the standard validation strategies used right now actually give us reliable uncertainty quantification for constraints on f NL derived from these simulations. The authors are questioning if current methods truly capture what we need for practical work.

Vera: Exactly; they focus on how standard coverage diagnostics might be misleading because they only check calibration in an averaged way rather than looking at the posterior shape at a specific parameter value, which is where we actually need to be precise.

Jocelyn: So, if I'm understanding correctly, their main point is that these diagnostic tests don't fully constrain how the posterior behaves when you fix a parameter like f NL to its true value, which is crucial for our real-world inference.

Subrahmanyan: That’s right; they are suggesting that diagnostics sensitive to the posterior shape and behavior at a fixed parameter value are absolutely necessary if we want robust uncertainty quantification in precision cosmology. This work is looking at limitations in extracting f NL using simulations of the dark matter halo density field.

Vera: It seems like they are comparing two main ways of doing inference: Likelihood-Based Inference and Simulation-Based Inference, specifically using Contrastive Neural Ratio Estimation for the latter, focusing on different summary statistics.

Jocelyn: And they aren't just looking at one statistic; they’re comparing the power spectrum, bispectrum, and Wavelet Scattering Transform coefficients to see how these methods perform across different observables.

Subrahmanyan: That comparison is important because it shows how the choice of summary statistics can affect the results when trying to constrain primordial non-Gaussianity. They use simulations of dark matter halo density fields as their testbed for this, which gives them a realistic scenario for testing their inference frameworks.

Paper summary: Vera: They found some interesting discrepancies between these two approaches; while they agree on things like posterior means and skewness, the variance shows weaker consistency in SBI when compared to LBI, particularly when looking at the combined power spectrum and bispectrum.

Jocelyn: That is a key finding because it suggests that even when combining different data vectors, the SBI posteriors can be systematically broader than those from traditional likelihood-based inference.

Subrahmanyan: And they pointed out that the kurtosis shows larger differences, which indicates distinct behaviors in the posterior tails between the two methods. This highlights how different frameworks respond differently to sampling noise and how they encode that posterior structure.

Vera: The paper also found that higher-order statistics are quite powerful; specifically, the Wavelet Scattering Transform coefficients provide substantially stronger constraints on f NL than using just the combination of the power spectrum and bispectrum alone, even when we only look at large scales.

Jocelyn: So, if we're looking for better constraints on primordial non-Gaussianity, the WST coefficients seem to be a pretty strong tool here based on their findings in "Coverage is not enough: Frequentist tests of simulation-based inference for primordial non-Gaussianity."

Subrahmanyan: That leads us to the conclusion that diagnostics sensitive to posterior shape and behavior at fixed parameter value will be necessary for robust uncertainty quantification in precision cosmology, as current coverage tests are insufficient to guarantee reliable posterior shapes. This work really underscores the need for more stringent validation methods in this area.

Vera: It feels like a call to action for how we validate these complex inference pipelines before we start deploying them widely in cosmological analyses. It’s about making sure those uncertainty estimates actually reflect the true behavior of the posteriors under realistic conditions.

Jocelyn: I think it means that relying solely on standard coverage checks isn't enough when you're dealing with things like primordial non-Gaussianity where the likelihoods are inherently hard to nail down analytically. We need tests that look deeper into the structure of the posterior itself.

Subrahmanyan: From a theoretical standpoint, this reinforces our understanding that simply checking if a credible region contains the true value doesn't tell us much about how well our model captures the full uncertainty in complex parameter spaces like those governing f NL. The paper makes it very clear that we need to test the posterior shape directly.

Paper summary: Vera: So, to wrap up this part, "Coverage is not enough: Frequentist tests of simulation-based inference for primordial non-Gaussianity" seems to be urging us away from just checking coverage and towards more detailed diagnostics that look at how the posterior actually behaves at specific points.

Jocelyn: Indeed, it’s a paper that shows us where the current validation procedures fall short when we're trying to get reliable constraints on things like f NL from simulation data. It’s a significant piece of work for anyone working on these complex inference pipelines.

Subrahmanyan: I think the implication here is that if we want to use SBI reliably for cosmological parameters, we can't just rely on the usual tests; we need methods that probe calibration in a way that captures the full structure of the posterior, not just an average frequency.

Vera: It's encouraging to see this level of scrutiny applied to how these tools are validated, especially when it comes to something as subtle as primordial non-Gaussianity. We need these kinds of detailed tests if we want our cosmological constraints to be solid.

Jocelyn: I think the real impact is showing that we need more sophisticated tools for assessing inference reliability in this domain, moving beyond simple coverage checks to methods that are sensitive to the posterior's behavior at fixed parameter values.

Subrahmanyan: This paper sets a clear direction for future work in developing better validation protocols for simulation-based inference techniques applied to cosmological problems where analytical likelihoods are unavailable. It gives us a roadmap for how to build more trustworthy tools.

Vera: It’s certainly an important piece of reading, and I think it really highlights the challenges we face when trying to extract subtle signals from noisy, complex data like that from dark matter halos.

Jocelyn: I'm excited to see how this discussion influences the next generation of SBI pipelines as researchers try to move forward with constraining primordial non-Gaussianity.

Subrahmanyan: I share that excitement, and I think this work provides a solid foundation for developing more rigorous statistical frameworks for cosmological inference in these non-linear regimes.

Conclusion: Vera: So, we've been diving into the technical details of how these simulation-based inference methods work to constrain primordial non-Gaussianity, and now we need to talk about what this specific paper is actually saying in plain English.

Jocelyn: It’s titled "Coverage is not enough: Frequentist tests of simulation-based inference for primordial non-Gaussianity," and the authors are basically telling us that just checking if our results fall within a certain region isn't enough to trust the uncertainty we get.

Subrahmanyan: That's right; they are arguing that standard coverage diagnostics, while common, only check calibration in an averaged sense and don't constrain how the posterior actually behaves at a fixed parameter value, which is where we really need precision.

Vera: Exactly; they are pushing for a more rigorous statistical approach to validation because the current methods seem insufficient for robust uncertainty quantification in precision cosmology.

Jocelyn: I mean, if we can't trust the shape of the posterior, then any constraints we derive on something as subtle as primordial non-Gaussianity become suspect.

Subrahmanyan: Precisely; they found that higher-order statistics, like those from Wavelet Scattering Transforms, give substantially stronger constraints on f NL than combining just the power spectrum and bispectrum.

Vera: That’s a huge implication for us observing the sky; it tells us that we need to look at more complex data structures beyond the basic two- and three-point functions to really probe these non-Gaussian effects.

Jocelyn: It suggests that moving forward, we should prioritize using those richer statistics when trying to extract signals from dark matter halo simulations.

Subrahmanyan: I think it means that the theoretical picture of how these signals manifest in the data is more complex than we previously thought, and our inferential tools need to catch that complexity.

Vera: This paper really highlights a crucial step in making sure our cosmological measurements are solid before we start relying on these simulation-based techniques for final parameter estimates.

Jocelyn: It’s a strong call to action for the community to develop new validation protocols that look at posterior shape directly, not just coverage frequency.

Subrahmanyan: And I think this work sets a clear direction for future theoretical and computational efforts aimed at building more trustworthy tools in this area.

Vera: So, the main message is that we need better tests than what's currently available to ensure our constraints on primordial non-Gaussianity are reliable.

Jocelyn: It’s about moving past simple checks and demanding a deeper understanding of how our inference methods truly handle uncertainty.

Argelander-Institut für Astronomie, Universität Bonn

astro-ph.CO, astro-ph.IM, stat.ME

Submitted: 2026-05-01

Updated: 2026-10-07

Comments: 13+4 pages, 5+7 figures, 2 tables. Updated to match the published version: A&A, 714, A66 (2026)

Journal ref: Astron. Astrophys. 714, A66 (2026)

DOI: 10.1051/0004-6361/202660668

Code: https://github.com/changhoonhahn/pySpectrum

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 73/100

The gist: Simulation-based inference (SBI) methods are being developed to extract cosmological parameters from complex, non-linear data where analytical likelihoods are unavailable, but their reliability

Key concepts

Simulation-Based Inference (SBI)
A method that uses simulations to infer cosmological parameters when analytical likelihoods are unavailable. This approach is implemented here using Contrastive Neural Ratio Estimation (CNRE) to learn the likelihood-to-evidence ratio from complex data like dark matter halo simulations.
Likelihood-Based Inference (LBI)
An alternative inference framework that uses parametric surrogate models based on perturbation theory to approximate the likelihood function for summary statistics. It serves as a comparison point against the more simulation-driven SBI method.
Coverage Diagnostics
Statistical tests, such as Simulation-Based Calibration (SBC) and TARP, used to check if credible regions correctly contain the true parameter value with expected frequency. The paper argues these tests fail because they only probe calibration in an averaged sense and miss crucial information about posterior shape.
Higher-Order Statistics
Summary statistics beyond the standard power spectrum (like the bispectrum and Wavelet Scattering Transform coefficients) are analyzed. These higher-order statistics provide significantly stronger constraints on primordial non-Gaussianity (PNG) than using only the power spectrum alone.

Terminology

Summary

Simulation-based inference (SBI) methods are being developed to extract cosmological parameters from complex, non-linear data where analytical likelihoods are unavailable, but their reliability remains questionable. This work investigates the limitations of current validation strategies by testing whether standard coverage diagnostics ensure reliable uncertainty quantification for primordial non-Gaussianity (PNG) constraints derived from dark matter halo simulations.

The gist

Diagnostic tests sensitive to posterior shape and behavior at fixed parameter value will be necessary for robust uncertainty quantification in precision cosmology.

How it works: Inference Frameworks and Statistics

The study compares two primary inference frameworks: Likelihood-Based Inference (LBI) and Simulation-Based Inference (SBI). SBI is implemented using Contrastive Neural Ratio Estimation (CNRE), which learns the likelihood-to-evidence ratio, while LBI uses parametric surrogate models based on perturbation theory to approximate the likelihood for summary statistics. The comparison focuses on three complementary summary statistics: the power spectrum, bispectrum, and Wavelet Scattering Transform (WST) coefficients.

How it works: Validation Strategies

The paper critically examines existing validation methods. Standard diagnostics include coverage-based tests like Simulation-Based Calibration (SBC) and Tests of Accuracy with Random Points (TARP), which assess whether credible regions contain the true parameter value with the expected frequency under the prior predictive distribution. The authors argue that these tests probe calibration only in an averaged sense and do not constrain posterior behavior at a fixed parameter value, which is relevant for practical inference.

How it works: Comparison of Results and Discrepancies

The results reveal systematic differences between SBI and LBI posteriors across various data vectors. While both methods agree well on posterior means and skewness, they show weaker realization-by-realization consistency in the variance, particularly for the combined power spectrum and bispectrum. The kurtosis shows larger discrepancies, indicating different behaviors in the posterior tails. Specifically, for the combined P + B data vector, SBI posteriors are systematically broader than LBI posteriors and can yield weaker constraints than either statistic individually.

How it works: Higher-Order Statistics and Conclusion

The analysis demonstrates that higher-order statistics provide substantial additional constraining power on PNG beyond traditional two- and three-point functions. The WST coefficients, in particular, provide substantially stronger constraints on fNL than the combination P+B alone, even when restricted to large scales. The paper concludes that diagnostics sensitive to posterior shape and behavior at fixed parameter value are necessary for robust uncertainty quantification in precision cosmology, as current coverage tests are insufficient to guarantee reliable posterior shapes.

Key Findings Summary

  1. Only the power spectrum (P) consistently passes all multivariate normality tests, justifying the Gaussian-likelihood assumption underlying LBI for this statistic.

  2. The bispectrum, WST coefficients, and their combinations all show measurable departures from multivariate Gaussianity, indicating that a Gaussian likelihood may not accurately capture the full structure of these distributions.

  3. The combined P + B data vector exhibits systematic underconfidence in SBI posteriors relative to LBI, failing to recover expected gains in constraining power when combining datasets.

  4. The WST coefficients provide substantially stronger constraints on fNL than the combination P+B alone, suggesting that additional summary statistics are crucial for probing PNG.

  5. The observed discrepancies suggest that systematic differences arise from how LBI and SBI respond to sampling noise and encode posterior structure, rather than solely from the non-Gaussianity of the data-generating process.

Key phrases quoted:

Coverage is not enough: Frequentist tests of simulation-based inference for primordial non-Gaussianity

diagnostics sensitive to posterior shape and behavior at fixed parameter value will be necessary for robust uncertainty quantification in precision cosmology.

SBI posteriors are systematically underconfident in this regime.

The WST coefficients provide substantially stronger constraints on fNL than the combination P+B alone.

**"Diagnostics sensitive to posterior shape and behavior at fixed parameter value will be necessary for robust uncertainty quantification in precision cosmology.

Improvements for AI systems

As a fastidious researcher, I have analyzed this manuscript, Coverage is not enough: Frequentist tests of simulation-based inference for primordial non-Gaussianity. The core scientific finding is that standard coverage diagnostics (like SBC and TARP) are insufficient to guarantee the reliability of Bayesian posterior estimates in Simulation-Based Inference (SBI), especially when comparing them to Likelihood-Based Inference (LBI) at fixed parameter values.

Here are the specific, high-impact improvements for AI systems based on these findings:

  1. Implement Tail-Sensitive Calibration Diagnostics

  2. Develop a Fixed Parameter Value Validator Module

  3. Improve Uncertainty Quantification for Non-Gaussian Data (P+B)

  4. Enhance Summary Statistic Selection Strategy

  5. The improved AI system can perform the following specific tasks:

5.1. Tail-Sensitive Calibration Diagnostics: The system will incorporate a new validation layer that explicitly tests posterior behavior at a fixed truth parameter value (i.e., frequentist coverage). Unlike current systems that only check if the true value falls within a credible region with expected frequency (prior-predictive coverage), this module will use diagnostics like the Tests of Accuracy with Random Points (TARP) or similar methods to assess calibration against regions defined relative to a fixed reference point.

5.2. Fixed Parameter Value Validator: The system will move beyond marginal validation tests by integrating a fixed truth testing regime. This involves drawing data from the prior predictive distribution and then evaluating the coverage of the posterior for specific, fixed values of parameters (e.g., setting fNL to a specific value like 0). This allows the system to explicitly detect systematic biases or mis-estimations that only appear when conditioning on a single realization, which current coverage tests miss.

5.3. Uncertainty Quantification for Non-Gaussian Data (P+B): When dealing with joint summary statistics (like the power spectrum and bispectrum, P+B), the system will be trained to identify underconfidence regimes where SBI posteriors are systematically broader than LBI posteriors, as evidenced by the observed discrepancies in Figure 4. The system can then automatically apply necessary rescaling factors (like the Dodelson–Schneider correction or Percival’s probability-matching prior approximation) when inferring constraints from these combined vectors to ensure frequentist consistency.

5.4. Summary Statistic Selection Strategy: The system will be equipped with a learned strategy to select the most informative subset of summary statistics for inference. Based on the study's findings, it can prioritize statistics that are known to be more Gaussian (like the Power Spectrum, P), while flagging or weighting higher-order statistics (Bispectrum, WST) based on their demonstrated ability to provide stronger constraints on specific parameters like fNL. This prevents the system from relying solely on potentially misleading high-dimensional combinations (like P+B) when a lower-dimensional statistic might be sufficient and more reliably calibrated.

Sources

Related papers