quivers.diagnostics.calibration¶
Simulation-based calibration
checks an inference procedure against the program it targets (Talts et al., 2018). Each simulation
draws parameters from the prior and data from the program, fits the simulated data, and ranks each
true quantity among its \(L\) posterior draws. An exact procedure gives ranks uniform on
{0, ..., L}. Ties are broken at random so that discrete quantities qualify. Uniformity is tested
with the simultaneous ECDF statistic (Säilynoja et al., 2022) calibrated under the exact discrete
null, and coverage of the central rank band is judged against its exact null mass with
Clopper-Pearson intervals (Clopper and Pearson, 1934). Test quantities that read the data can expose
miscalibration the parameters alone miss (Modrák et al., 2025).
calibration
¶
Simulation-based calibration (SBC) of Bayesian inference procedures.
SBC (Talts, Betancourt, Simpson, Vehtari, and Gelman, 2018,
doi:10.48550/arXiv.1804.06788 <https://doi.org/10.48550/arXiv.1804.06788>_)
checks an inference procedure against the model it targets. Each
simulation draws parameters from the prior and data from the model
given those parameters, fits the simulated data, and records the rank
of each true quantity among the posterior draws of that quantity. If
the procedure samples the exact posterior, every rank is uniform on
{0, ..., L} for L posterior draws, whatever the model; a
posterior that is too narrow piles ranks at the ends, one too wide
piles them in the middle, and a biased one tilts them.
sbcruns the whole procedure for a program and a caller-supplied fit (MCMC, a guide, or any sampler returning chain-by-draw tensors) and returns anSBCResult.sbc_ranksis the rank primitive, with ties between the true value and posterior draws broken uniformly at random, so that discrete and mixed quantities have uniform ranks too.sbc_uniformitytests the ranks against the discrete uniform distribution with the simultaneous ECDF statistic of Säilynoja, Bürkner, and Vehtari (2022,doi:10.1007/s11222-022-10090-6 <https://doi.org/10.1007/s11222-022-10090-6>_), calibrated by Monte Carlo under the exact discrete null.sbc_coveragereports how often the central rank band holds the true value, against that band's exact null mass, with Clopper-Pearson intervals adjusted for the number of statistics.
The module needs only torch; quivers.diagnostics.to_datatree_from_sbc
converts an SBCResult for ArviZ's PIT-ECDF plots.
SBCStatistic
module-attribute
¶
A test quantity: statistic(parameters, data). parameters
maps each latent site to a batch of draws with a leading draw axis,
shape (n, *constrained_shape); data is the simulated
observations together with the host data, as handed to the fit. The
result has shape (n,) for one quantity or (n, *shape) for an
array of them, one row per draw.
SBCPosterior
module-attribute
¶
SBCPosterior = MCMCResult | Guide | Mapping[str, torch.Tensor]
What a fit returns: an MCMCResult, a fitted Guide (sampled
num_guide_draws times), or a mapping of site name to draws laid
out (num_chains, num_draws, *constrained_shape).
SBCThinning
module-attribute
¶
How posterior draws are thinned before ranking: "ess" keeps
about one draw per effective sample, an integer keeps every k-th draw
of each chain.
SBCResult
didactic-model
¶
SBCResult(*, ranks: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), num_draws: int, statistic_names: tuple[str, ...], truths: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), thinning: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), retained: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), seed: int)
Bases: ValueModel
The rank statistics of a simulation-based calibration run.
| ATTRIBUTE | DESCRIPTION |
|---|---|
ranks |
Int64, shape
TYPE:
|
num_draws |
TYPE:
|
statistic_names |
The name of each column of
TYPE:
|
truths |
Float64, shape
TYPE:
|
thinning |
Int64, shape
TYPE:
|
retained |
Int64, shape
TYPE:
|
seed |
The seed that reproduces the run.
TYPE:
|
SBCCoverage
didactic-model
¶
SBCCoverage(*, mass: float, null_mass: float, lower_rank: int, upper_rank: int, family_error: float, coverage: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), lower: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), upper: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), consistent: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY))
Bases: ValueModel
Coverage of the central rank band, per statistic.
A rank in the band [lower_rank, upper_rank] means the true
value lies inside the central posterior interval spanned by the
corresponding order statistics. Under calibration the probability
of that event is exactly null_mass, which differs from the
requested mass because ranks are discrete.
| ATTRIBUTE | DESCRIPTION |
|---|---|
mass |
The requested central mass.
TYPE:
|
null_mass |
The band's exact probability under uniform ranks,
TYPE:
|
lower_rank |
The smallest rank in the band.
TYPE:
|
upper_rank |
The largest rank in the band.
TYPE:
|
family_error |
The error rate shared by all statistics' intervals; each
interval has confidence
TYPE:
|
coverage |
Float64, shape
TYPE:
|
lower |
Float64, shape
TYPE:
|
upper |
Float64, shape
TYPE:
|
consistent |
Bool, shape
TYPE:
|
sbc_ranks
¶
Rank true values among posterior draws, breaking ties at random.
The rank of a true value t among draws d_1, ..., d_L is
#{d_l < t} + U, with U uniform on {0, ..., #{d_l = t}}.
With continuous draws ties have probability zero and U = 0;
with discrete or mixed quantities the random tie-break is what
makes the rank uniform on {0, ..., L} under calibration.
| PARAMETER | DESCRIPTION |
|---|---|
truth
|
True values, any shape
TYPE:
|
draws
|
Posterior draws, shape
TYPE:
|
generator
|
Source of the tie-breaking draws.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Tensor
|
Int64 ranks, shape |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If |
Source code in src/quivers/diagnostics/calibration.py
sbc_uniformity
¶
sbc_uniformity(ranks: Tensor, num_draws: int, *, num_reference: int = 10000, seed: int = 0) -> Tensor
Monte Carlo p-values for the uniformity of SBC ranks.
The test statistic is the smallest two-sided binomial tail
probability of the rank ECDF over every threshold, the quantity
behind the simultaneous ECDF bands of Säilynoja, Bürkner, and
Vehtari (2022). Its null distribution is simulated from
num_reference sets of ranks drawn exactly from the discrete
uniform distribution on {0, ..., L}, so the test has its
nominal size at any L and any number of simulations, where a
continuous Kolmogorov-Smirnov test or an unequal-bin histogram
would not. The p-value is (1 + #{reference <= observed}) /
(num_reference + 1), a valid p-value for a Monte Carlo test.
Each column is tested separately; with several statistics, adjust
for multiplicity (for instance by comparing the smallest p-value
with alpha / num_statistics).
| PARAMETER | DESCRIPTION |
|---|---|
ranks
|
Integer ranks, shape
TYPE:
|
num_draws
|
TYPE:
|
num_reference
|
Monte Carlo draws of the null distribution.
TYPE:
|
seed
|
Seed of the reference draws, so the p-values are reproducible.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Tensor
|
Float64 p-values, shape |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If |
Source code in src/quivers/diagnostics/calibration.py
sbc_coverage
¶
sbc_coverage(ranks: Tensor, num_draws: int, *, mass: float = 0.9, family_error: float = 0.05) -> SBCCoverage
Coverage of the central rank band against its exact null mass.
The band holds the ranks lower_rank, ..., L - lower_rank with
lower_rank = floor((1 - mass) / 2 * (L + 1)), the narrowest
symmetric band whose null mass is at least mass. With L
finite, that exact mass, null_mass, is the reference the
observed coverage is judged against, not mass itself. Each
statistic's coverage gets a Clopper-Pearson interval (Clopper and
Pearson, 1934, doi:10.1093/biomet/26.4.404
<https://doi.org/10.1093/biomet/26.4.404>_) at confidence
1 - family_error / num_statistics (a Bonferroni adjustment),
so that under calibration every interval holds null_mass
with probability at least 1 - family_error.
| PARAMETER | DESCRIPTION |
|---|---|
ranks
|
Integer ranks, shape
TYPE:
|
num_draws
|
TYPE:
|
mass
|
Requested central mass, in
TYPE:
|
family_error
|
Error rate shared by all statistics' intervals, in
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
SBCCoverage
|
The band, its null mass, and each statistic's coverage and interval. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If |
Source code in src/quivers/diagnostics/calibration.py
438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 | |
sbc
¶
sbc(model: MonadicProgram, x: Tensor, fit: Callable[[dict[str, Tensor], int], SBCPosterior], *, observed_names: Iterable[str], num_simulations: int, seed: int, data: Mapping[str, Tensor] | None = None, statistics: Mapping[str, SBCStatistic] | None = None, thin: SBCThinning = 'ess', num_guide_draws: int = 1000) -> SBCResult
Run simulation-based calibration of an inference procedure.
Each of num_simulations simulations
- runs
modelforward at the one-row inputxand the hostdata, drawing every latent from its prior and every site inobserved_namesfrom the model given those latents; - calls
fit(observations, fit_seed)with the simulated observations merged withdata, and reads its posterior draws (seeSBCPosterior); - evaluates the test quantities at the prior draw and at every
posterior draw, thins the posterior draws (
thin), and keeps them.
The posterior draws of all simulations are then cut to the
smallest retained count L (taking evenly spaced draws), and
each true quantity is ranked among its L draws with
sbc_ranks. Following Talts et al. (2018), thin="ess" keeps
about one draw per effective sample, since autocorrelated draws
make the ranks of even an exact sampler non-uniform.
The procedure is reproducible from seed: it derives three
seeds per simulation, for the prior simulation, the fit, and the
tie-breaks; the prior simulation and the fit run with Torch's
CPU and available accelerator generators forked and initialized from
locally seeded generators. Neither scope depends on or changes the
caller's random state, and a fit that draws from global generators is
reproducible too. States are restored even if the fit raises.
| PARAMETER | DESCRIPTION |
|---|---|
model
|
The program whose inference is checked.
TYPE:
|
x
|
A one-row program input: one run, as MCMC and the guides take.
TYPE:
|
fit
|
The inference procedure:
TYPE:
|
observed_names
|
The sites treated as data.
TYPE:
|
num_simulations
|
Number of simulations.
TYPE:
|
seed
|
Seed of the whole procedure.
TYPE:
|
data
|
Host data the program reads by name (covariates), held fixed.
TYPE:
|
statistics
|
Named test quantities; None ranks every coordinate of every
latent site. Quantities that read the data, such as a
log-likelihood, can detect miscalibration that the parameters
alone miss (Modrák et al., 2025,
TYPE:
|
thin
|
TYPE:
|
num_guide_draws
|
Draws taken from a fit that returns a
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
SBCResult
|
The ranks and the bookkeeping that reproduces them. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If |
TypeError
|
If |
Source code in src/quivers/diagnostics/calibration.py
754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 | |
Cited sources¶
Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating Bayesian Inference Algorithms with Simulation-Based Calibration. arXiv preprint arXiv:1804.06788, 2018. doi:10.48550/arXiv.1804.06788.
Teemu Säilynoja, Paul-Christian Bürkner, and Aki Vehtari. Graphical Test for Discrete Uniformity and Its Applications in Goodness-of-Fit Evaluation and Multiple Sample Comparison. Statistics and Computing, 32(2):32, 2022. URL: https://doi.org/10.1007/s11222-022-10090-6, doi:10.1007/s11222-022-10090-6.
C. J. Clopper and E. S. Pearson. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika, 26(4):404–413, 1934. URL: https://doi.org/10.1093/biomet/26.4.404, doi:10.1093/biomet/26.4.404.
Martin Modrák, Angie H. Moon, Shinyoung Kim, Paul-Christian Bürkner, Niko Huurre, Kateřina Faltejsková, Andrew Gelman, and Aki Vehtari. Simulation-Based Calibration Checking for Bayesian Computation: The Choice of Test Quantities Shapes Sensitivity. Bayesian Analysis, 20(2):461–488, 2025. URL: https://doi.org/10.1214/23-BA1404, doi:10.1214/23-BA1404.