Skip to content

quivers.diagnostics.calibration

Simulation-based calibration checks an inference procedure against the program it targets (Talts et al., 2018). Each simulation draws parameters from the prior and data from the program, fits the simulated data, and ranks each true quantity among its \(L\) posterior draws. An exact procedure gives ranks uniform on {0, ..., L}. Ties are broken at random so that discrete quantities qualify. Uniformity is tested with the simultaneous ECDF statistic (Säilynoja et al., 2022) calibrated under the exact discrete null, and coverage of the central rank band is judged against its exact null mass with Clopper-Pearson intervals (Clopper and Pearson, 1934). Test quantities that read the data can expose miscalibration the parameters alone miss (Modrák et al., 2025).

calibration

Simulation-based calibration (SBC) of Bayesian inference procedures.

SBC (Talts, Betancourt, Simpson, Vehtari, and Gelman, 2018, doi:10.48550/arXiv.1804.06788 <https://doi.org/10.48550/arXiv.1804.06788>_) checks an inference procedure against the model it targets. Each simulation draws parameters from the prior and data from the model given those parameters, fits the simulated data, and records the rank of each true quantity among the posterior draws of that quantity. If the procedure samples the exact posterior, every rank is uniform on {0, ..., L} for L posterior draws, whatever the model; a posterior that is too narrow piles ranks at the ends, one too wide piles them in the middle, and a biased one tilts them.

  • sbc runs the whole procedure for a program and a caller-supplied fit (MCMC, a guide, or any sampler returning chain-by-draw tensors) and returns an SBCResult.
  • sbc_ranks is the rank primitive, with ties between the true value and posterior draws broken uniformly at random, so that discrete and mixed quantities have uniform ranks too.
  • sbc_uniformity tests the ranks against the discrete uniform distribution with the simultaneous ECDF statistic of Säilynoja, Bürkner, and Vehtari (2022, doi:10.1007/s11222-022-10090-6 <https://doi.org/10.1007/s11222-022-10090-6>_), calibrated by Monte Carlo under the exact discrete null.
  • sbc_coverage reports how often the central rank band holds the true value, against that band's exact null mass, with Clopper-Pearson intervals adjusted for the number of statistics.

The module needs only torch; quivers.diagnostics.to_datatree_from_sbc converts an SBCResult for ArviZ's PIT-ECDF plots.

SBCStatistic module-attribute

SBCStatistic = Callable[[Mapping[str, torch.Tensor], Mapping[str, torch.Tensor]], torch.Tensor]

A test quantity: statistic(parameters, data). parameters maps each latent site to a batch of draws with a leading draw axis, shape (n, *constrained_shape); data is the simulated observations together with the host data, as handed to the fit. The result has shape (n,) for one quantity or (n, *shape) for an array of them, one row per draw.

SBCPosterior module-attribute

SBCPosterior = MCMCResult | Guide | Mapping[str, torch.Tensor]

What a fit returns: an MCMCResult, a fitted Guide (sampled num_guide_draws times), or a mapping of site name to draws laid out (num_chains, num_draws, *constrained_shape).

SBCThinning module-attribute

SBCThinning = Literal['ess'] | int

How posterior draws are thinned before ranking: "ess" keeps about one draw per effective sample, an integer keeps every k-th draw of each chain.

SBCResult didactic-model

SBCResult(*, ranks: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), num_draws: int, statistic_names: tuple[str, ...], truths: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), thinning: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), retained: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), seed: int)

Bases: ValueModel

The rank statistics of a simulation-based calibration run.

ATTRIBUTE DESCRIPTION
ranks

Int64, shape (num_simulations, num_statistics): the rank of each true quantity among num_draws posterior draws, in {0, ..., num_draws}, with ties broken uniformly at random.

TYPE: Tensor

num_draws

L, the posterior draws each rank is taken against, after thinning and equalizing across simulations.

TYPE: int

statistic_names

The name of each column of ranks: "site" for a scalar site, "site[i,j]" for one coordinate of an array site, and likewise for a named test quantity.

TYPE: tuple[str, ...]

truths

Float64, shape (num_simulations, num_statistics): the value of each quantity at the parameters drawn from the prior.

TYPE: Tensor

thinning

Int64, shape (num_simulations,): the interval each simulation's chains were thinned by.

TYPE: Tensor

retained

Int64, shape (num_simulations,): the draws each simulation kept after thinning, before all are cut to num_draws.

TYPE: Tensor

seed

The seed that reproduces the run.

TYPE: int

SBCCoverage didactic-model

SBCCoverage(*, mass: float, null_mass: float, lower_rank: int, upper_rank: int, family_error: float, coverage: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), lower: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), upper: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), consistent: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY))

Bases: ValueModel

Coverage of the central rank band, per statistic.

A rank in the band [lower_rank, upper_rank] means the true value lies inside the central posterior interval spanned by the corresponding order statistics. Under calibration the probability of that event is exactly null_mass, which differs from the requested mass because ranks are discrete.

ATTRIBUTE DESCRIPTION
mass

The requested central mass.

TYPE: float

null_mass

The band's exact probability under uniform ranks, (upper_rank - lower_rank + 1) / (num_draws + 1).

TYPE: float

lower_rank

The smallest rank in the band.

TYPE: int

upper_rank

The largest rank in the band.

TYPE: int

family_error

The error rate shared by all statistics' intervals; each interval has confidence 1 - family_error / num_statistics.

TYPE: float

coverage

Float64, shape (num_statistics,): the fraction of simulations whose rank lies in the band.

TYPE: Tensor

lower

Float64, shape (num_statistics,): the Clopper-Pearson lower bound of each coverage.

TYPE: Tensor

upper

Float64, shape (num_statistics,): the Clopper-Pearson upper bound of each coverage.

TYPE: Tensor

consistent

Bool, shape (num_statistics,): whether null_mass lies inside each interval.

TYPE: Tensor

sbc_ranks

sbc_ranks(truth: Tensor, draws: Tensor, *, generator: Generator) -> Tensor

Rank true values among posterior draws, breaking ties at random.

The rank of a true value t among draws d_1, ..., d_L is #{d_l < t} + U, with U uniform on {0, ..., #{d_l = t}}. With continuous draws ties have probability zero and U = 0; with discrete or mixed quantities the random tie-break is what makes the rank uniform on {0, ..., L} under calibration.

PARAMETER DESCRIPTION
truth

True values, any shape S.

TYPE: Tensor

draws

Posterior draws, shape (L, *S).

TYPE: Tensor

generator

Source of the tie-breaking draws.

TYPE: Generator

RETURNS DESCRIPTION
Tensor

Int64 ranks, shape S, in {0, ..., L}.

RAISES DESCRIPTION
ValueError

If draws is not shaped (L, *truth.shape) with L >= 1, or either tensor holds a non-finite value.

Source code in src/quivers/diagnostics/calibration.py
def sbc_ranks(
    truth: torch.Tensor,
    draws: torch.Tensor,
    *,
    generator: torch.Generator,
) -> torch.Tensor:
    """Rank true values among posterior draws, breaking ties at random.

    The rank of a true value ``t`` among draws ``d_1, ..., d_L`` is
    ``#{d_l < t} + U``, with ``U`` uniform on ``{0, ..., #{d_l = t}}``.
    With continuous draws ties have probability zero and ``U = 0``;
    with discrete or mixed quantities the random tie-break is what
    makes the rank uniform on ``{0, ..., L}`` under calibration.

    Parameters
    ----------
    truth : torch.Tensor
        True values, any shape ``S``.
    draws : torch.Tensor
        Posterior draws, shape ``(L, *S)``.
    generator : torch.Generator
        Source of the tie-breaking draws.

    Returns
    -------
    torch.Tensor
        Int64 ranks, shape ``S``, in ``{0, ..., L}``.

    Raises
    ------
    ValueError
        If ``draws`` is not shaped ``(L, *truth.shape)`` with ``L >= 1``,
        or either tensor holds a non-finite value.
    """
    if draws.dim() != truth.dim() + 1 or tuple(draws.shape[1:]) != tuple(truth.shape):
        raise ValueError(
            f"sbc_ranks: draws must have shape (L, *truth.shape) = "
            f"(L, *{tuple(truth.shape)}); got {tuple(draws.shape)}"
        )
    if draws.shape[0] < 1:
        raise ValueError("sbc_ranks: need at least one posterior draw")
    if not bool(torch.isfinite(truth).all()) or not bool(torch.isfinite(draws).all()):
        raise ValueError("sbc_ranks: truth and draws must be finite")
    below = (draws < truth).sum(dim=0)
    ties = (draws == truth).sum(dim=0)
    uniform = torch.rand(
        truth.shape, generator=generator, dtype=torch.float64, device="cpu"
    ).to(device=truth.device)
    tie_share = torch.floor(uniform * (ties + 1).to(dtype=torch.float64))
    return (below + tie_share.to(dtype=below.dtype)).to(dtype=torch.int64)

sbc_uniformity

sbc_uniformity(ranks: Tensor, num_draws: int, *, num_reference: int = 10000, seed: int = 0) -> Tensor

Monte Carlo p-values for the uniformity of SBC ranks.

The test statistic is the smallest two-sided binomial tail probability of the rank ECDF over every threshold, the quantity behind the simultaneous ECDF bands of Säilynoja, Bürkner, and Vehtari (2022). Its null distribution is simulated from num_reference sets of ranks drawn exactly from the discrete uniform distribution on {0, ..., L}, so the test has its nominal size at any L and any number of simulations, where a continuous Kolmogorov-Smirnov test or an unequal-bin histogram would not. The p-value is (1 + #{reference <= observed}) / (num_reference + 1), a valid p-value for a Monte Carlo test.

Each column is tested separately; with several statistics, adjust for multiplicity (for instance by comparing the smallest p-value with alpha / num_statistics).

PARAMETER DESCRIPTION
ranks

Integer ranks, shape (num_simulations,) or (num_simulations, num_statistics), in {0, ..., num_draws}.

TYPE: Tensor

num_draws

L, the number of posterior draws each rank is taken against.

TYPE: int

num_reference

Monte Carlo draws of the null distribution.

TYPE: int DEFAULT: 10000

seed

Seed of the reference draws, so the p-values are reproducible.

TYPE: int DEFAULT: 0

RETURNS DESCRIPTION
Tensor

Float64 p-values, shape () for one-dimensional ranks and (num_statistics,) otherwise.

RAISES DESCRIPTION
ValueError

If ranks is not an integer tensor of the stated layout with values in {0, ..., num_draws}, or num_reference < 1.

Source code in src/quivers/diagnostics/calibration.py
def sbc_uniformity(
    ranks: torch.Tensor,
    num_draws: int,
    *,
    num_reference: int = 10_000,
    seed: int = 0,
) -> torch.Tensor:
    """Monte Carlo p-values for the uniformity of SBC ranks.

    The test statistic is the smallest two-sided binomial tail
    probability of the rank ECDF over every threshold, the quantity
    behind the simultaneous ECDF bands of Säilynoja, Bürkner, and
    Vehtari (2022). Its null distribution is simulated from
    ``num_reference`` sets of ranks drawn exactly from the discrete
    uniform distribution on ``{0, ..., L}``, so the test has its
    nominal size at any ``L`` and any number of simulations, where a
    continuous Kolmogorov-Smirnov test or an unequal-bin histogram
    would not. The p-value is ``(1 + #{reference <= observed}) /
    (num_reference + 1)``, a valid p-value for a Monte Carlo test.

    Each column is tested separately; with several statistics, adjust
    for multiplicity (for instance by comparing the smallest p-value
    with ``alpha / num_statistics``).

    Parameters
    ----------
    ranks : torch.Tensor
        Integer ranks, shape ``(num_simulations,)`` or
        ``(num_simulations, num_statistics)``, in ``{0, ..., num_draws}``.
    num_draws : int
        ``L``, the number of posterior draws each rank is taken
        against.
    num_reference : int
        Monte Carlo draws of the null distribution.
    seed : int
        Seed of the reference draws, so the p-values are reproducible.

    Returns
    -------
    torch.Tensor
        Float64 p-values, shape ``()`` for one-dimensional ``ranks``
        and ``(num_statistics,)`` otherwise.

    Raises
    ------
    ValueError
        If ``ranks`` is not an integer tensor of the stated layout with
        values in ``{0, ..., num_draws}``, or ``num_reference < 1``.
    """
    table = _check_ranks(ranks, num_draws, "sbc_uniformity")
    if num_reference < 1:
        raise ValueError(
            f"sbc_uniformity: num_reference must be >= 1, got {num_reference}"
        )
    num_simulations = table.shape[0]
    lower, upper = _binomial_tails(num_simulations, num_draws)
    observed = _ecdf_statistic(table, num_draws, lower, upper)
    generator = torch.Generator().manual_seed(seed)
    reference_parts: list[torch.Tensor] = []
    remaining = num_reference
    while remaining > 0:
        size = min(remaining, _REFERENCE_CHUNK)
        reference_ranks = torch.randint(
            0, num_draws + 1, (num_simulations, size), generator=generator
        )
        reference_parts.append(
            _ecdf_statistic(reference_ranks, num_draws, lower, upper)
        )
        remaining -= size
    reference = torch.cat(reference_parts)
    at_most = (reference.unsqueeze(0) <= observed.unsqueeze(1)).sum(dim=1)
    p_values = (1.0 + at_most.to(dtype=torch.float64)) / (num_reference + 1.0)
    return p_values[0] if ranks.dim() == 1 else p_values

sbc_coverage

sbc_coverage(ranks: Tensor, num_draws: int, *, mass: float = 0.9, family_error: float = 0.05) -> SBCCoverage

Coverage of the central rank band against its exact null mass.

The band holds the ranks lower_rank, ..., L - lower_rank with lower_rank = floor((1 - mass) / 2 * (L + 1)), the narrowest symmetric band whose null mass is at least mass. With L finite, that exact mass, null_mass, is the reference the observed coverage is judged against, not mass itself. Each statistic's coverage gets a Clopper-Pearson interval (Clopper and Pearson, 1934, doi:10.1093/biomet/26.4.404 <https://doi.org/10.1093/biomet/26.4.404>_) at confidence 1 - family_error / num_statistics (a Bonferroni adjustment), so that under calibration every interval holds null_mass with probability at least 1 - family_error.

PARAMETER DESCRIPTION
ranks

Integer ranks, shape (num_simulations,) or (num_simulations, num_statistics), in {0, ..., num_draws}.

TYPE: Tensor

num_draws

L.

TYPE: int

mass

Requested central mass, in (0, 1).

TYPE: float DEFAULT: 0.9

family_error

Error rate shared by all statistics' intervals, in (0, 1).

TYPE: float DEFAULT: 0.05

RETURNS DESCRIPTION
SBCCoverage

The band, its null mass, and each statistic's coverage and interval.

RAISES DESCRIPTION
ValueError

If ranks is malformed, or mass or family_error lies outside (0, 1).

Source code in src/quivers/diagnostics/calibration.py
def sbc_coverage(
    ranks: torch.Tensor,
    num_draws: int,
    *,
    mass: float = 0.9,
    family_error: float = 0.05,
) -> SBCCoverage:
    """Coverage of the central rank band against its exact null mass.

    The band holds the ranks ``lower_rank, ..., L - lower_rank`` with
    ``lower_rank = floor((1 - mass) / 2 * (L + 1))``, the narrowest
    symmetric band whose null mass is at least ``mass``. With ``L``
    finite, that exact mass, ``null_mass``, is the reference the
    observed coverage is judged against, not ``mass`` itself. Each
    statistic's coverage gets a Clopper-Pearson interval (Clopper and
    Pearson, 1934, `doi:10.1093/biomet/26.4.404
    <https://doi.org/10.1093/biomet/26.4.404>`_) at confidence
    ``1 - family_error / num_statistics`` (a Bonferroni adjustment),
    so that under calibration every interval holds ``null_mass``
    with probability at least ``1 - family_error``.

    Parameters
    ----------
    ranks : torch.Tensor
        Integer ranks, shape ``(num_simulations,)`` or
        ``(num_simulations, num_statistics)``, in ``{0, ..., num_draws}``.
    num_draws : int
        ``L``.
    mass : float
        Requested central mass, in ``(0, 1)``.
    family_error : float
        Error rate shared by all statistics' intervals, in ``(0, 1)``.

    Returns
    -------
    SBCCoverage
        The band, its null mass, and each statistic's coverage and
        interval.

    Raises
    ------
    ValueError
        If ``ranks`` is malformed, or ``mass`` or ``family_error`` lies
        outside ``(0, 1)``.
    """
    table = _check_ranks(ranks, num_draws, "sbc_coverage")
    if not 0.0 < mass < 1.0:
        raise ValueError(f"sbc_coverage: mass must lie in (0, 1), got {mass}")
    if not 0.0 < family_error < 1.0:
        raise ValueError(
            f"sbc_coverage: family_error must lie in (0, 1), got {family_error}"
        )
    # The band edge is computed in exact rational arithmetic on the
    # shortest decimal spelling of ``mass``, so that a mass such as 0.8
    # with L = 9 yields the band 1..8 rather than losing a rank to
    # binary rounding.
    tail_mass = (1 - Fraction(repr(float(mass)))) / 2
    lower_rank = math.floor(tail_mass * (num_draws + 1))
    upper_rank = num_draws - lower_rank
    null_mass = (upper_rank - lower_rank + 1) / (num_draws + 1)
    trials, num_statistics = table.shape
    inside = ((table >= lower_rank) & (table <= upper_rank)).sum(dim=0).tolist()
    confidence = 1.0 - family_error / num_statistics
    bounds = [_clopper_pearson(int(k), trials, confidence) for k in inside]
    low = torch.tensor([b[0] for b in bounds], dtype=torch.float64)
    high = torch.tensor([b[1] for b in bounds], dtype=torch.float64)
    return SBCCoverage(
        mass=float(mass),
        null_mass=null_mass,
        lower_rank=lower_rank,
        upper_rank=upper_rank,
        family_error=float(family_error),
        coverage=torch.tensor(inside, dtype=torch.float64) / trials,
        lower=low,
        upper=high,
        consistent=(low <= null_mass) & (null_mass <= high),
    )

sbc

sbc(model: MonadicProgram, x: Tensor, fit: Callable[[dict[str, Tensor], int], SBCPosterior], *, observed_names: Iterable[str], num_simulations: int, seed: int, data: Mapping[str, Tensor] | None = None, statistics: Mapping[str, SBCStatistic] | None = None, thin: SBCThinning = 'ess', num_guide_draws: int = 1000) -> SBCResult

Run simulation-based calibration of an inference procedure.

Each of num_simulations simulations

  1. runs model forward at the one-row input x and the host data, drawing every latent from its prior and every site in observed_names from the model given those latents;
  2. calls fit(observations, fit_seed) with the simulated observations merged with data, and reads its posterior draws (see SBCPosterior);
  3. evaluates the test quantities at the prior draw and at every posterior draw, thins the posterior draws (thin), and keeps them.

The posterior draws of all simulations are then cut to the smallest retained count L (taking evenly spaced draws), and each true quantity is ranked among its L draws with sbc_ranks. Following Talts et al. (2018), thin="ess" keeps about one draw per effective sample, since autocorrelated draws make the ranks of even an exact sampler non-uniform.

The procedure is reproducible from seed: it derives three seeds per simulation, for the prior simulation, the fit, and the tie-breaks; the prior simulation and the fit run with Torch's CPU and available accelerator generators forked and initialized from locally seeded generators. Neither scope depends on or changes the caller's random state, and a fit that draws from global generators is reproducible too. States are restored even if the fit raises.

PARAMETER DESCRIPTION
model

The program whose inference is checked.

TYPE: MonadicProgram

x

A one-row program input: one run, as MCMC and the guides take.

TYPE: Tensor

fit

The inference procedure: fit(observations, seed), where observations holds the simulated observed sites and the host data, and seed is an integer the procedure may pass to a seeded sampler (for instance MCMC(..., seed=seed)).

TYPE: Callable[[dict[str, Tensor], int], SBCPosterior]

observed_names

The sites treated as data.

TYPE: Iterable[str]

num_simulations

Number of simulations.

TYPE: int

seed

Seed of the whole procedure.

TYPE: int

data

Host data the program reads by name (covariates), held fixed.

TYPE: Mapping[str, Tensor] or None DEFAULT: None

statistics

Named test quantities; None ranks every coordinate of every latent site. Quantities that read the data, such as a log-likelihood, can detect miscalibration that the parameters alone miss (Modrák et al., 2025, doi:10.1214/23-BA1404 <https://doi.org/10.1214/23-BA1404>_).

TYPE: Mapping[str, SBCStatistic] or None DEFAULT: None

thin

"ess" (the default) or a fixed interval; use 1 for a fit whose draws are independent.

TYPE: SBCThinning DEFAULT: 'ess'

num_guide_draws

Draws taken from a fit that returns a Guide.

TYPE: int DEFAULT: 1000

RETURNS DESCRIPTION
SBCResult

The ranks and the bookkeeping that reproduces them.

RAISES DESCRIPTION
ValueError

If x has more than one row, num_simulations or num_guide_draws is less than one, an observed name is not a site of the model, thin is invalid, a simulation's latent sites or statistics differ from the first simulation's, or a true value or posterior draw is non-finite.

TypeError

If fit returns none of the accepted posterior forms.

Source code in src/quivers/diagnostics/calibration.py
def sbc(
    model: MonadicProgram,
    x: torch.Tensor,
    fit: Callable[[dict[str, torch.Tensor], int], SBCPosterior],
    *,
    observed_names: Iterable[str],
    num_simulations: int,
    seed: int,
    data: Mapping[str, torch.Tensor] | None = None,
    statistics: Mapping[str, SBCStatistic] | None = None,
    thin: SBCThinning = "ess",
    num_guide_draws: int = 1000,
) -> SBCResult:
    """Run simulation-based calibration of an inference procedure.

    Each of ``num_simulations`` simulations

    1. runs ``model`` forward at the one-row input ``x`` and the host
       ``data``, drawing every latent from its prior and every site in
       ``observed_names`` from the model given those latents;
    2. calls ``fit(observations, fit_seed)`` with the simulated
       observations merged with ``data``, and reads its posterior
       draws (see `SBCPosterior`);
    3. evaluates the test quantities at the prior draw and at every
       posterior draw, thins the posterior draws (``thin``), and keeps
       them.

    The posterior draws of all simulations are then cut to the
    smallest retained count ``L`` (taking evenly spaced draws), and
    each true quantity is ranked among its ``L`` draws with
    `sbc_ranks`. Following Talts et al. (2018), ``thin="ess"`` keeps
    about one draw per effective sample, since autocorrelated draws
    make the ranks of even an exact sampler non-uniform.

    The procedure is reproducible from ``seed``: it derives three
    seeds per simulation, for the prior simulation, the fit, and the
    tie-breaks; the prior simulation and the fit run with Torch's
    CPU and available accelerator generators forked and initialized from
    locally seeded generators. Neither scope depends on or changes the
    caller's random state, and a fit that draws from global generators is
    reproducible too. States are restored even if the fit raises.

    Parameters
    ----------
    model : MonadicProgram
        The program whose inference is checked.
    x : torch.Tensor
        A one-row program input: one run, as MCMC and the guides take.
    fit : Callable[[dict[str, torch.Tensor], int], SBCPosterior]
        The inference procedure: ``fit(observations, seed)``, where
        ``observations`` holds the simulated observed sites and the
        host data, and ``seed`` is an integer the procedure may pass
        to a seeded sampler (for instance ``MCMC(..., seed=seed)``).
    observed_names : Iterable[str]
        The sites treated as data.
    num_simulations : int
        Number of simulations.
    seed : int
        Seed of the whole procedure.
    data : Mapping[str, torch.Tensor] or None
        Host data the program reads by name (covariates), held fixed.
    statistics : Mapping[str, SBCStatistic] or None
        Named test quantities; None ranks every coordinate of every
        latent site. Quantities that read the data, such as a
        log-likelihood, can detect miscalibration that the parameters
        alone miss (Modrák et al., 2025,
        `doi:10.1214/23-BA1404 <https://doi.org/10.1214/23-BA1404>`_).
    thin : SBCThinning
        ``"ess"`` (the default) or a fixed interval; use ``1`` for a
        fit whose draws are independent.
    num_guide_draws : int
        Draws taken from a fit that returns a `Guide`.

    Returns
    -------
    SBCResult
        The ranks and the bookkeeping that reproduces them.

    Raises
    ------
    ValueError
        If ``x`` has more than one row, ``num_simulations`` or
        ``num_guide_draws`` is less than one, an observed name is not a
        site of the model, ``thin`` is invalid, a simulation's latent
        sites or statistics differ from the first simulation's, or a
        true value or posterior draw is non-finite.
    TypeError
        If ``fit`` returns none of the accepted posterior forms.
    """
    _require_one_run(x, "sbc")
    if num_simulations < 1:
        raise ValueError(f"sbc: num_simulations must be >= 1, got {num_simulations}")
    if num_guide_draws < 1:
        raise ValueError(f"sbc: num_guide_draws must be >= 1, got {num_guide_draws}")
    observed = frozenset(observed_names)
    host = dict(data or {})
    master = torch.Generator().manual_seed(seed)
    seeds: list[list[int]] = torch.randint(
        0, _INT64_MAX, (num_simulations, 3), generator=master
    ).tolist()

    statistic_names: list[str] | None = None
    truths: list[torch.Tensor] = []
    kept: list[torch.Tensor] = []
    intervals: list[int] = []
    for simulation_seed, fit_seed, _ in seeds:
        with _seeded_rng(simulation_seed):
            with torch.no_grad():
                simulated = trace(model, x, observations=dict(host))
                missing = sorted(observed - simulated.sites.keys())
                if missing:
                    raise ValueError(
                        f"sbc: observed names {missing} are not sample sites "
                        "of the model"
                    )
                observations = {
                    name: simulated.sites[name].value.detach() for name in observed
                }
                fit_data = {**host, **observations}
                registry = LatentRegistry.from_trace(
                    model, trace(model, x, observations=dict(fit_data))
                )
        truth = {
            site.name: _site_value(simulated, site.name, site.constrained_shape)
            for site in registry.sites.values()
        }
        with _seeded_rng(fit_seed):
            posterior = _posterior_draws(
                fit(dict(fit_data), fit_seed), x, num_guide_draws
            )
        names, truth_values = _evaluate_statistics(
            statistics,
            registry,
            {name: value.unsqueeze(0) for name, value in truth.items()},
            fit_data,
        )
        num_chains, num_draws = next(iter(posterior.values())).shape[:2]
        flat_posterior = {
            name: values.reshape(num_chains * num_draws, *values.shape[2:])
            for name, values in posterior.items()
        }
        _, draw_values = _evaluate_statistics(
            statistics, registry, flat_posterior, fit_data
        )
        if statistic_names is None:
            statistic_names = names
        elif names != statistic_names:
            raise ValueError(
                "sbc: the simulations disagree on the latent sites or statistics: "
                f"{names} against {statistic_names}"
            )
        if not bool(torch.isfinite(truth_values).all()):
            raise ValueError("sbc: a prior draw of a statistic is not finite")
        if not bool(torch.isfinite(draw_values).all()):
            raise ValueError("sbc: the fit returned a non-finite posterior draw")
        chains = draw_values.reshape(num_chains, num_draws, -1)
        interval = _thinning_interval(chains, thin)
        intervals.append(interval)
        truths.append(truth_values[0])
        kept.append(chains[:, ::interval].reshape(-1, chains.shape[2]))

    retained = [int(draws.shape[0]) for draws in kept]
    common = min(retained)
    ranks: list[torch.Tensor] = []
    for (_, _, tie_seed), truth_row, draws in zip(seeds, truths, kept, strict=True):
        positions = (torch.arange(common) * draws.shape[0]) // common
        ranks.append(
            sbc_ranks(
                truth_row,
                draws[positions],
                generator=torch.Generator().manual_seed(tie_seed),
            )
        )
    return SBCResult(
        ranks=torch.stack(ranks),
        num_draws=common,
        statistic_names=tuple(statistic_names or ()),
        truths=torch.stack(truths),
        thinning=torch.tensor(intervals, dtype=torch.int64),
        retained=torch.tensor(retained, dtype=torch.int64),
        seed=seed,
    )

Cited sources

Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating Bayesian Inference Algorithms with Simulation-Based Calibration. arXiv preprint arXiv:1804.06788, 2018. doi:10.48550/arXiv.1804.06788.

Teemu Säilynoja, Paul-Christian Bürkner, and Aki Vehtari. Graphical Test for Discrete Uniformity and Its Applications in Goodness-of-Fit Evaluation and Multiple Sample Comparison. Statistics and Computing, 32(2):32, 2022. URL: https://doi.org/10.1007/s11222-022-10090-6, doi:10.1007/s11222-022-10090-6.

C. J. Clopper and E. S. Pearson. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika, 26(4):404–413, 1934. URL: https://doi.org/10.1093/biomet/26.4.404, doi:10.1093/biomet/26.4.404.

Martin Modrák, Angie H. Moon, Shinyoung Kim, Paul-Christian Bürkner, Niko Huurre, Kateřina Faltejsková, Andrew Gelman, and Aki Vehtari. Simulation-Based Calibration Checking for Bayesian Computation: The Choice of Test Quantities Shapes Sensitivity. Bayesian Analysis, 20(2):461–488, 2025. URL: https://doi.org/10.1214/23-BA1404, doi:10.1214/23-BA1404.