Distribird: Literature-Informed Prior Distribution Design for Bayesian Model Calibration

The paper presents Distribird, an agentic web application that automates the construction of informative prior distributions for Bayesian model calibration from scientific literature. Despite decades of methodological work, researchers almost always fall back on uniform priors because building informative priors from scientific literature is slow and needs both domain and statistical expertise. Distribird deploys "a multi-agent LangGraph pipeline that searches Semantic Scholar and OpenAlex in parallel, reads the retrieved papers in full, extracts reported numerical values, weights them by how well each source’s study context matches the target domain, and fits the best-matching probability distribution via AIC model selection from a predefined set of distributions. When no literature is available, the system falls back to sensible uninformative alternatives (a uniform prior or a wide Normal), and clearly reports both the evidence behind and the confidence level of every prior it produces. The tool is designed for physically interpretable parameters in process-based models, where domain knowledge exists in the published literature. Evaluated on 24 parameters across ten scientific domains using three open-weight models (Qwen3.6 27B, Gemma 4 31B, Mistral Small 4 119B) run entirely on local hardware, the full pipeline matches this baseline rather than beating it on prior quality. Its contribution is different: Every prior is traced to the specific papers and values from which it was constructed; a built-in validity layer declines to produce priors for out-of-scope requests, flagging all five out-of-scope test parameters on every model, whereas the single-prompt baseline returns confident but unfounded priors for them in 11 of 30 model–parameter cases; and every language-model call runs on local open weights, so no parameter description or unpublished modelling detail is transmitted to a third-party LLM provider (only generated search terms reach the public literature databases). The authors argue these properties matter more than a marginal improvement in point-estimate accuracy" for scientific use.

Process-based models "describe complex natural systems (eg. crop growth, water movement through soil, biogeochemical cycles, disease spread, etc.) using mechanistic equations with parameters that represent physical, chemical, or biological quantities. Because many parameters cannot be measured directly, they must be inferred from observations by adjusting parameter values until the model output matches available data, a process called model calibration, or inverse modeling. Bayesian calibration is arguably the soundest approach to this problem, on both statistical and philosophical grounds as it combines prior knowledge about the parameters with the information in the observed data, and returns a posterior distribution that captures not only the most likely parameter values but also the uncertainty around them."

The Bayesian approach requires the researcher to specify a prior distribution for each parameter before seeing the data. The mathematically convenient answer is to use a uniform prior, which is almost universally adopted in practice but almost always inadequate. Pericchi and Walley [1991] showed no single uninformative prior works in every case. Uniform priors are not invariant under reparameterisation, and with a uniform prior the posterior mode coincides exactly with the maximum-likelihood estimate. The information lost "has real consequences: the sampler converges more slowly, credible intervals are wider than they need to be, and when data are scarce, posteriors are poorly constrained where an informative prior would have stabilised them. Informative priors improve all of these properties. The knowledge needed exists. It is in the scientific literature. The problem is that gathering and combining it is expensive, taking days of reading for a model with twenty parameters. Researchers default to uniform priors not because they believe them appropriate, but because the alternative is too costly."

Recent LLMs that can search databases on their own have created a new option. Directly prompting an LLM for a prior has in fact been shown to yield well-placed, informative distributions quickly and cheaply. However, "LLMs return priors with no traceable basis: as we demonstrate, a model will readily produce a confident, informative prior for a parameter that has no empirical referent, a typographic error, or a software-internal tuning constant. Also important is the question of attribution: when a language model synthesises findings from the literature without citing its sources, the researchers who produced that empirical work receive no credit. The objective is not to extract a more accurate point estimate from a language model, but to make literature-based prior construction trustworthy, meaning: every value is traceable to a cited source; out-of-scope requests are declined rather than confabulated; and the entire process runs on open-weight models on the researcher’s own hardware. Distribird is not a general-purpose prior elicitation tool but is designed specifically for the class of problems where literature-based prior construction is most appropriate and most impactful: process-based models with physically interpretable parameters, active publishing communities, and genuine prior knowledge encoded in the scientific record. The explicit trade-off: on our benchmark the full pipeline does not produce more accurate priors than a single well-prompted LLM call, and it is considerably more expensive to run. What it provides is provenance, robustness against fabricated evidence, and fully local operation."

Distribird is implemented as a Python 3.10+ package distributed via PyPI and exposing two user-facing interfaces: a REST API built on FastAPI and an interactive Streamlit web application. Both call the same core pipeline. The user provides a ParameterInput object specifying the parameter name, a plain-language physical description, the measurement unit, a domain context string, and optional physical constraints. The system returns a PipelineResult containing "the fitted prior distribution, its confidence level, the number of contributing sources, all search queries attempted, the full list of retrieved papers with extracted values, an optional enrichment context, and, when multi-agent deliberation is enabled, the moderator’s rationale and any excluded papers."

The core is a LangGraph StateGraph that orchestrates a multi-agent pipeline with three feedback loops and a conditional forward path. The pipeline allows cycles: when the initial evidence is too thin, the graph routes back to earlier nodes for another attempt before moving on to synthesis. All nodes read from and write to a shared PipelineState typed dictionary that follows the blackboard pattern.

The pipeline comprises "eleven primary processing nodes: Enrich (parameter semantics expansion), SearchStrategy (search-scope planning), QueryGen (search query generation), Search (parallel API queries), RelevanceJudge (paper scoring), CrossEnrich (citation snowballing), FetchFulltext (PDF retrieval and parsing), Extract (numerical value extraction), QualityGate (routing logic), Synthesize (distribution fitting), and ValidityCheck (out-of-scope classification). There are three feedback loops and one conditional forward path":

• Loop A, Search refinement: When the quality gate finds zero extracted values but papers were retrieved, the pipeline routes to a RefineSearch node, which generates refined search queries targeting more specific experimental or calibration studies. This loop may execute up to two iterations.

• Loop B, Domain broadening: When the quality gate finds too few domain-relevant values (fewer than domain broadening min relevant, default 2) and search breadth can be widened, the pipeline routes to a BroadenSearch node, which advances the current breadth one tier toward broad and returns control to QueryGen. This loop may execute up to domain broadening max times (default 2).

• Conditional path, Cross-enrichment: if the maximum iteration number permits and at least two high-relevance papers have been identified, the pipeline routes through CrossEnrich (citation snowballing) before fetching full texts. This is executed at most once per pipeline invocation.

• Loop C, Extraction refinement: When the quality gate finds extracted values but none are high-confidence and the coefficient of variation exceeds a predefined threshold, the pipeline routes to a RefineExtraction node that uses web-assisted search to locate additional confirming or disconfirming evidence. This loop may execute once.

An IterationBudget object tracks the number of iterations consumed by each loop and enforces a global cap on total LLM calls (default: 30), guaranteeing termination.

A scope-planning node (SearchStrategy) classifies the request’s domain-specificity as high, medium, or low and maps it to a starting search breadth (strict, mixed, or broad, respectively). The domain-broadening loop widens the breadth after the fact when the first pass returns too little on-target evidence. Papers found at each tier accumulate rather than replacing one another, so broadening only ever adds evidence. Broadening escalates at most domain broadening max times (default 2), so the pipeline starts as narrow as the request allows and widens only as far as it must.

Every request is classified as Valid, Suspicious, Likely Invalid, or Unknown. The early-skip runs right after enrichment: if the LLM does not recognise the parameter name, the pipeline goes straight to the validity node and skips search, extraction, and synthesis, saving 80–95% in our benchmarks of runtime. The terminal classifier combines the enrichment self-reports, the number of refined queries, papers, and extracted values, and the prior’s confidence into a verdict. If the heuristics return Suspicious, a single second-opinion LLM call is made that can sharpen the verdict to Likely Invalid or confirm it as Suspicious, but never overrides a Valid result.

The search subsystem queries two academic APIs in parallel: Semantic Scholar and OpenAlex, both restricted to open-access papers. Two more source agents are optional: a deep-research agent hands the task to an LLM with web-search capabilities and a web search agent runs a similar checked LLM search. Both agents' results are checked against Semantic Scholar before they are included, to guard against made-up references.

The FetchFulltext node retrieves and reads the complete text of each selected paper using a tiered fetch fallbacks cascade: URL-derived resolver, primary PDF, open-access mirror, last-resort resolvers (DOI→PMC, arXiv preprint), and an opt-in stealth browser (Camoufox) for JavaScript bot walls. PDFs are read as Markdown via PyMuPDF4LLM rather than flattened plain text, preserving tables, headings, and page boundaries. Extraction reads the entire paper, not just its opening pages, splitting long papers into overlapping pages sized to the model's context window.

The fitting strategy has tiers set by how many values survive extraction and quality-gate filtering:

• ≥5 values: AIC selection across admissible families (Normal, Truncated Normal, Gamma, Log-Normal, Beta), marked high confidence.

• 2–4 values: Moment matching with σ widened ×1.5 and a min-width floor against a Truncated Normal, marked medium confidence.

• 1 value: Centred on the reported value; σ from paper’s uncertainty, else half the reported value, marked low confidence.

• 0 values: Centred at midpoint of bounds with σ = (u − l)/4 (or effectively unbounded with µ = 0, σ = 1000), marked none / uninformative.

Relevance-aware synthesis is enabled by default: each value-bearing paper receives a dedicated LLM judgment of how well its study context matches the searched domain, yielding a label of high, medium, or low. This changes synthesis in three ways: selection (fit from the strongest subset), weighting (relevance factors 1.0, 0.6, 0.25), and confidence ceiling (capped by relevance). Values are weighted by the square root of its sample size when reported.

Distribird exports priors in three formats: JSON (a structured record containing the parameter name, distribution family, fitted parameters, confidence level, informative/uninformative flag, fitting rationale, number of contributing sources, and a citation list), Python (an executable scipy.stats script), and R (an executable R script using base distribution functions). All formats carry the full provenance chain.

An optional integer seed is forwarded to the LLM API’s seed field, and sampling temperature is set per task class. An opt-in debug-trace framework captures a complete structured record of a run, every LLM prompt and response, each search request, each PDF fetch outcome, the extracted values, and the AIC candidates considered during fitting.

All runs use open-weight models served locally through llama.cpp as Unsloth Dynamic (UD) GGUF quantisations: Qwen3.6 27B and Gemma 4 31B at Q8 K XL, and Mistral Small 4 119B at Q4 K XL, all on a single NVIDIA H100 GPU (80 GB). The baseline is a naive single-prompt elicitation with no literature search at all, run on the same three local models and for reference, on three cloud frontier models (Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5). The benchmark covers 24 parameters across 12 use cases spanning ten scientific domains (climate, pharmacokinetics, hydrology, structural engineering, epidemiology, astrophysics, three ecology systems, geophysics, robotics, economics).

Running the full pipeline locally is computationally expensive: a single parameter consumes on the order of a million LLM tokens over roughly 80 model calls and 20–40 minutes of time on local hardware. Full-text reading and extraction dominate. Per-parameter figures span from 0.4 M tokens and 12 minutes to 4.7 M tokens and 96 minutes for the Hubble constant.

Prior accuracy is scored as accuracy = 1 − prior mode − data optimum / range. The two approaches place priors about equally well: Across all 24 parameters their mean accuracy is within a few points on every model. Seven parameters are benchmark artifacts where the synthetic data force the true value close to the top of the allowed range, higher than any published value. On the remaining 17 fair parameters, Distribird is ahead by about seven points on Mistral (83% vs. 76%) and behind by less than one point on Gemma 4 and Qwen. A two-sample Kolmogorov–Smirnov test over all 72 parameter–model pairs finds D is small and the p-value is far above the usual 0.05 threshold, so we cannot reject the hypothesis that the accuracy scores come from one common distribution. The pipeline shows no measurable difference in prior placement from a single well-written prompt.

Against flagship cloud models, on the fair set all three flagship baselines score 84%, which is essentially the same as Distribird on local open-weight models (83–84%). The local, auditable pipeline places priors about as well as a direct prompt to the strongest cloud models.

Effective Sample Size (ESS) from four NUTS chains shows Distribird’s priors give a higher ESS on 46–54% of the parameters, and the naive priors on about 62% compared to a flat prior.

Distribird attaches "a complete, machine-readable provenance record: the search queries it issued, the papers it found, the numerical values it extracted with their in-text context, the per-paper domain-relevance judgments, the AIC family and score, and a citation list with DOIs. For example, the COVID-19 incubation-period prior built by Qwen3.6 27B is an 'AIC-selected lognormal from 85 values (85 high- and medium-relevance values kept, 12 low-relevance dropped)' drawn from 43 source papers. On saturated hydraulic conductivity, the naive single-prompt model had no value it trusted and fell back to an uninformative prior (accuracy 0.50), whereas Distribird on Gemma 4 read 69 papers, kept the 9 values its relevance judge rated high or medium, dropped 6 low-relevance ones, and fit a high-confidence truncated Normal that placed the prior near the data optimum (accuracy 0.91)."

The test set is "six requests: two fabricated names with no scientific meaning (mumblesnort factor, fake quantum correction xyz), three real but non-empirical model-internal quantities, and one real empirical control (specific leaf area). Distribird's validity layer flagged all 15 fabricated or non-empirical requests (five per model) as non-valid, likely invalid or suspicious, never valid and recovered the real control on all three (3/3). On the 24 in-scope benchmark parameters, it never once hard-rejected a real parameter and marked 71% (51/72) valid. The naive baseline returned a confident informative prior for the fabricated parameters in 11 of 30 model–parameter cases and never once refused. Fabricated names are caught by the early-skip route in 12 s to 3.6 min using only 2–6 k tokens, while requests reaching the full pipeline cost 40 k–1.2 M tokens and 7–90 minutes."

Prior elicitation has a large methodological literature falling into expert elicitation protocols and empirical Bayes methods. Distribird fills a different niche: automated literature-based elicitation, where the prior knowledge comes from the published scientific record rather than from a human expert or a separate dataset. The authors state: "We are not aware of another system that automates literature-based prior construction end to end, combining multi-agent literature search, per-source relevance weighting, AIC-based distribution fitting, and explicit confidence and out-of-scope reporting."

The prior problem in Bayesian calibration is not a problem of missing knowledge; it is a problem of the cost of getting to it. Distribird makes that knowledge available automatically, turning a task that used to take days of expert effort into one that takes minutes. The evaluation shows "the full Distribird pipeline does not produce more accurate priors than a single, well-prompted call to the same language model, and it is substantially more expensive to run. Its contribution is trustworthiness rather than accuracy. The tool is openly available, pip-installable, and deployable via Docker. For informal use, a single LLM call is a cheaper and comparably accurate choice. For scientific use, where a prior must be defensible, auditable, and robust against silently fabricated evidence, and where transmitting requests and unpublished model details to a commercial LLM provider may be unacceptable, these are precisely the properties Distribird provides."

Distribird produces its best results under: Physically interpretable parameters with a physical or biological interpretation that is discussed in the scientific literature; Literature-active domain with an active publishing community; Univariate distributions describable by a standard univariate probability distribution (Normal, truncated Normal, Beta, Gamma, Log-Normal); and Known physical bounds. When conditions are not fully met, Distribird degrades gracefully: it states the evidence behind its output clearly, flags low-confidence results, and falls back to sensible uninformative alternatives. It is not designed for parameters without physical interpretation (e.g. neural network weights), purely empirical tuning coefficients with no literature base, or parameters whose behaviour is fundamentally multivariate.

Based on the paper, here are specific improvements I can implement in AI systems:

• Improvement: Add a mandatory citation-traceability layer to any AI system that generates scientific claims, parameter values, or data summaries. Every output must include a machine-readable chain linking each value to its source paper, DOI, and extraction context.

• What the improved system can do: Generate priors, parameter estimates, or literature syntheses where every number is auditable—a reviewer can click through from the final distribution to the exact sentence in each cited paper. It refuses to emit unverifiable claims.

• Improvement: Implement a two-gate validity classifier (early-skip + terminal check) that detects fabricated, typo-ridden, or non-empirical inputs before expensive processing. The system must decline to answer with a confident prior when the parameter has no empirical referent.

• What the improved system can do: For nonsense inputs like mumblesnort factor or fake quantum correction xyz, it returns likely invalid in seconds, saving 80–95% of compute. For real but non-empirical quantities (e.g., Kalman filter noise covariance), it flags them as suspicious rather than inventing a distribution.

• Improvement: Replace single-pass pipelines with a cyclic LangGraph architecture that has three feedback loops (search refinement, domain broadening, extraction refinement) and a global iteration budget to guarantee termination.

• What the improved system can do: When initial evidence is thin, it automatically widens search scope or refines queries, accumulating papers across tiers rather than discarding them. The budget cap (default 30 LLM calls) prevents runaway compute on pathological inputs.

• Improvement: Add a per-source relevance judgment layer that labels each extracted value as high/medium/low relevance to the target domain, then weight values accordingly (1.0/0.6/0.25) during distribution fitting, with a confidence ceiling tied to relevance.

• What the improved system can do: A porosity measurement from carbonate rock is down-weighted when calibrating a sandstone model, preventing off-domain values from dominating the prior. High confidence is only achievable when genuinely high-relevance evidence exists.

• Improvement: Implement a four-tier fitting strategy based on the number of surviving values: AIC model selection (≥5 values, high confidence), moment matching with widened σ (2–4 values, medium), single-value centering (1 value, low), and uninformative fallback (0 values, none).

• What the improved system can do: Automatically select between Normal, Truncated Normal, Gamma, Log-Normal, and Beta via AIC, while explicitly labeling the confidence level so downstream users never mistake an uninformative fallback for evidence-backed guidance.

• Improvement: Configure all LLM calls to run on local open-weight models (e.g., Qwen3.6 27B, Gemma 4 31B) via quantized GGUF files, with no third-party API calls for inference.

• What the improved system can do: Process unpublished modelling details, proprietary data, or sensitive parameter descriptions entirely on the researcher's hardware. Only generated search terms reach public literature databases (Semantic Scholar, OpenAlex), never the raw input.

• Improvement: Use Markdown-based PDF extraction (PyMuPDF4LLM) that preserves tables as pipe-tables, headings, and page boundaries, rather than flattening to plain text.

• What the improved system can do: Extract numerical values with their row labels, column headers, and unit associations intact, dramatically reducing extraction errors when values appear in calibration tables or appendices rather than prose.

• Improvement: Expose a fixed seed parameter and per-task-class temperature settings (low for precise tasks like extraction, higher for creative tasks like enrichment), plus an optional debug-trace framework that records every prompt, response, and intermediate result.

• What the improved system can do: Repeated runs on the same input converge to identical priors, enabling scientific reproducibility. The trace can be rendered as an HTML viewer, allowing any prior to be traced back to the exact model calls and evidence that produced it.

• Improvement: Add a conditional forward path that, when at least two high-relevance papers are found, follows their citations to surface additional key papers before full-text retrieval.

• What the improved system can do: Discover seminal works that direct keyword search might miss, expanding the evidence base by 20–30% on well-studied parameters (e.g., the incubation period example surfaced 18 additional papers).

• Improvement: When zero values are extracted, return a wide Truncated Normal centered at the midpoint of bounds with σ = (u−l)/4, and set is informative=false with confidence none.

• What the improved system can do: Never silently produce a false-confidence prior. The user sees a clear warning that the prior is uninformative, preventing downstream misinterpretation in MCMC sampling.

What the fully improved system can do: Given a parameter name, description, and domain context, it autonomously searches and reads dozens of papers, extracts and relevance-weights numerical values, fits the best distribution via AIC, and returns a fully traceable prior with explicit confidence—all while refusing fabricated inputs, running entirely on local hardware, and producing reproducible results that a reviewer can audit end-to-end.

Bayesian calibration of process-based models requires a prior distribution for each model parameter. Despite decades of methodological work, researchers almost always fall back on uniform priors. The main reason is that building informative priors from scientific literature is slow and needs both domain and statistical expertise. We present Distribird, an agentic web application that automates this process. Given a parameter name, physical description, and domain context, Distribird deploys a multi-agent pipeline that searches the literature, extracts and weights reported values by domain relevance, and fits a probability distribution via AIC model selection. When no literature is available, the system falls back to sensible uninformative alternatives, and clearly reports both the evidence behind and the confidence level of every prior it produces. It is designed for the problems where the models have physically interpretable parameters, where domain knowledge exists in the published literature. We evaluate the tool on 24 parameters across 10 scientific domains comparing three open-weight models (Qwen3.6 27B, Gemma 4 31B, Mistral Small 4 119B) with a single-prompt LLM baseline. On prior quality the full pipeline matches this baseline. Every prior is traced to the specific papers and values from which it was constructed; a built-in validity layer declines to produce priors for out-of-scope requests, whereas the single-prompt baseline returns confident but unfounded priors for them in 11 of 30 model--parameter cases; and every language-model call runs locally, so no parameter description or unpublished modelling detail is transmitted to a third-party LLM provider (only generated search terms reach the public literature databases). For scientific use, we argue these properties matter more than a marginal improvement in point-estimate accuracy.