quivers.diagnostics.elpd¶
An ELPDEstimate holds one estimate of the expected log pointwise predictive
density (ELPD) as the R loo package stores it (Vehtari et al., 2017): the estimate,
the effective number of parameters, the deviance-scale criterion, their
standard errors, the pointwise terms, and Pareto diagnostics for PSIS-based
estimates (Vehtari et al., 2024). ArviZ computes PSIS, comparisons, and stacking
weights (Yao et al., 2018). This module adds the difference estimator of
subsampled PSIS-LOO (Magnusson et al., 2020), PSIS-weighted expectations with
precomputed weights, and the leave-one-out R-squared of brms
(Gelman et al., 2019). The analysis guide
shows the fit methods that produce these estimates.
elpd
¶
Typed expected-log-predictive-density estimates and their comparisons.
ELPDEstimate is the counterpart of a loo object of the R loo
package [@vehtari2017loo] as brms stores it: one estimate of the
expected log pointwise predictive density (ELPD) from PSIS-LOO, WAIC,
k-fold cross-validation, or subsampled PSIS-LOO, with its pointwise terms
and diagnostics. ArviZ computes PSIS, WAIC weights, comparisons, and
stacking weights; this module converts ArviZ's results into retained
tensors and implements what ArviZ lacks: the difference estimator of
subsampled PSIS-LOO [@magnusson2020leave], PSIS-weighted expectations with
precomputed weights, and the leave-one-out R-squared of brms.
NO_CRITERIA
module-attribute
¶
NO_CRITERIA: Mapping[str, ELPDEstimate | Tensor] = MappingProxyType({})
The stored criteria of a fit without any.
A fit compares its stored criteria by identity, so every fit without criteria holds this one read-only mapping and such fits agree on the field.
Criterion
¶
Predictive criterion of an ELPDEstimate.
ELPDEstimate
didactic-model
¶
ELPDEstimate(*, criterion: Criterion, elpd: float, se_elpd: float, p: float, se_p: float, pointwise_elpd: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), pointwise_p: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), pareto_k: Tensor | None = field(default=None, opaque=True, extras=OPAQUE_BY_IDENTITY), num_samples: int, good_k: float | None = None, responses: tuple[str, ...], response_hash: str, moment_matched: tuple[int, ...] = (), refitted: tuple[int, ...] = (), subsample: tuple[int, ...] = (), subsampling_se: float | None = None, subsampling_se_p: float | None = None, approximation: Tensor | None = field(default=None, opaque=True, extras=OPAQUE_BY_IDENTITY), approximation_method: str | None = None, folds: tuple[int, ...] = (), joint: str | None = None)
Bases: ValueModel
One ELPD estimate with pointwise terms, as a brms loo object.
| PARAMETER | DESCRIPTION |
|---|---|
criterion
|
Estimator that produced the values.
TYPE:
|
elpd
|
ELPD estimate and its standard error over observations.
TYPE:
|
se_elpd
|
ELPD estimate and its standard error over observations.
TYPE:
|
p
|
Effective number of parameters and its standard error.
TYPE:
|
se_p
|
Effective number of parameters and its standard error.
TYPE:
|
pointwise_elpd
|
Float64 terms per scored unit. Subsampled estimates hold exact terms on the subsample and NaN elsewhere.
TYPE:
|
pointwise_p
|
Float64 terms per scored unit. Subsampled estimates hold exact terms on the subsample and NaN elsewhere.
TYPE:
|
pareto_k
|
Pareto shape diagnostic per unit for PSIS estimates (NaN where not computed), None for WAIC and k-fold.
TYPE:
|
num_samples
|
Pooled posterior draws behind the estimate.
TYPE:
|
good_k
|
Pareto shape threshold,
TYPE:
|
responses
|
Scored response names.
TYPE:
|
response_hash
|
Hash of the scored responses, checked before comparisons.
TYPE:
|
moment_matched
|
Units whose terms were improved by moment matching.
TYPE:
|
refitted
|
Units whose terms come from exact refits.
TYPE:
|
subsample
|
Units scored exactly by subsampled PSIS-LOO.
TYPE:
|
subsampling_se
|
Standard errors of the ELPD and of
TYPE:
|
subsampling_se_p
|
Standard errors of the ELPD and of
TYPE:
|
approximation
|
Approximate pointwise ELPD of every unit for subsampled PSIS-LOO.
TYPE:
|
approximation_method
|
TYPE:
|
folds
|
Zero-based fold of every observation for k-fold estimates.
TYPE:
|
joint
|
k-fold predictive unit:
TYPE:
|
num_observations
property
¶
Number of scored units.
| RETURNS | DESCRIPTION |
|---|---|
int
|
Length of the pointwise terms. |
ic
property
¶
Deviance-scale criterion -2 * elpd (looic, waic, kfoldic).
| RETURNS | DESCRIPTION |
|---|---|
float
|
The information criterion. |
se_ic
property
¶
Standard error of the deviance-scale criterion.
| RETURNS | DESCRIPTION |
|---|---|
float
|
|
estimates
¶
estimates() -> SummaryTable
Return the loo package's estimates table.
| RETURNS | DESCRIPTION |
|---|---|
SummaryTable
|
Rows |
Source code in src/quivers/diagnostics/elpd.py
bad_observations
¶
List units whose Pareto shape exceeds a threshold.
| PARAMETER | DESCRIPTION |
|---|---|
threshold
|
Shape threshold; None uses
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
tuple[int, ...]
|
Zero-based unit positions, as loo's |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the estimate has no Pareto diagnostics. |
Source code in src/quivers/diagnostics/elpd.py
to_arviz
¶
Convert to ArviZ's ELPDData for ArviZ comparisons.
| RETURNS | DESCRIPTION |
|---|---|
ELPDData
|
The same estimate, pointwise terms and diagnostics. |
Source code in src/quivers/diagnostics/elpd.py
PSISWeights
didactic-model
¶
PSISWeights(*, log_weights: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), pareto_k: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY), r_eff: float, good_k: float)
Bases: ValueModel
Pareto smoothed importance weights for leave-one-out, as loo's psis.
| PARAMETER | DESCRIPTION |
|---|---|
log_weights
|
Float64 normalized smoothed log weights
TYPE:
|
pareto_k
|
Pareto shape diagnostic per unit [@vehtari2024psis].
TYPE:
|
r_eff
|
Relative efficiency used for the tail length.
TYPE:
|
good_k
|
Threshold
TYPE:
|
LooExpectation
didactic-model
¶
LooExpectation(*, table: SummaryTable = field(opaque=True, extras=OPAQUE_BY_VALUE), pareto_k: Tensor = field(opaque=True, extras=OPAQUE_BY_IDENTITY))
Bases: ValueModel
PSIS-weighted leave-one-out expectations, as brms loo_predict.
| PARAMETER | DESCRIPTION |
|---|---|
table
|
One row per unit and one column per statistic (
TYPE:
|
pareto_k
|
Function-specific Pareto shape per unit (and component).
TYPE:
|
response_hash
¶
response_hash(values: Sequence[Tensor]) -> str
Hash observed responses to check that criteria score the same data.
| PARAMETER | DESCRIPTION |
|---|---|
values
|
Observed response tensors in a fixed order.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
str
|
SHA-1 digest of the Float64 values, their shapes, and NaN positions. brms stores the same check as a hash of its response data. |
Source code in src/quivers/diagnostics/elpd.py
pointwise_lpd
¶
Return the log pointwise predictive density over pooled draws.
| PARAMETER | DESCRIPTION |
|---|---|
log_likelihood
|
Scores
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Tensor
|
Float64 |
Source code in src/quivers/diagnostics/elpd.py
from_arviz
¶
from_arviz(result: ELPDData, log_likelihood: Tensor, *, criterion: Criterion, responses: tuple[str, ...], hashed: str, moment_matched: tuple[int, ...] = (), refitted: tuple[int, ...] = ()) -> ELPDEstimate
Convert a pointwise ArviZ PSIS-LOO result to an ELPDEstimate.
| PARAMETER | DESCRIPTION |
|---|---|
result
|
Pointwise ArviZ result.
TYPE:
|
log_likelihood
|
The scores
TYPE:
|
criterion
|
Estimator label.
TYPE:
|
responses
|
Scored response names.
TYPE:
|
hashed
|
Response hash.
TYPE:
|
moment_matched
|
Units changed by moment matching or exact refits.
TYPE:
|
refitted
|
Units changed by moment matching or exact refits.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ELPDEstimate
|
The estimate with |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the result has no pointwise terms. |
Source code in src/quivers/diagnostics/elpd.py
waic_estimate
¶
waic_estimate(log_likelihood: Tensor, *, responses: tuple[str, ...], hashed: str) -> ELPDEstimate
Estimate WAIC with the loo package's conventions.
| PARAMETER | DESCRIPTION |
|---|---|
log_likelihood
|
Finite scores
TYPE:
|
responses
|
Scored response names.
TYPE:
|
hashed
|
Response hash.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ELPDEstimate
|
|
Source code in src/quivers/diagnostics/elpd.py
good_k
¶
Return the sample-size dependent Pareto shape threshold.
| PARAMETER | DESCRIPTION |
|---|---|
num_samples
|
Pooled draw count S.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
float
|
|
Source code in src/quivers/diagnostics/elpd.py
require_scores
¶
require_scores(log_likelihood: Tensor, *, criterion: str) -> None
Refuse an importance-sampling criterion over missing pointwise scores.
| PARAMETER | DESCRIPTION |
|---|---|
log_likelihood
|
Scores
TYPE:
|
criterion
|
Criterion named in the error.
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If a score is missing ( |
Source code in src/quivers/diagnostics/elpd.py
require_psis_draws
¶
Refuse Pareto smoothing with too few draws for a tail fit.
PSIS fits a generalized Pareto distribution to the largest fifth of the importance ratios of each unit and needs at least five of them [@vehtari2024psis].
| PARAMETER | DESCRIPTION |
|---|---|
count
|
Pooled posterior draws.
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If fewer than 25 draws are pooled. |
Source code in src/quivers/diagnostics/elpd.py
psis_weights
¶
psis_weights(log_likelihood: Tensor, *, r_eff: float) -> PSISWeights
Smooth leave-one-out importance ratios with ArviZ's PSIS.
| PARAMETER | DESCRIPTION |
|---|---|
log_likelihood
|
Scores
TYPE:
|
r_eff
|
Relative efficiency of the draws, one for independent draws.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
PSISWeights
|
Normalized smoothed log weights and Pareto diagnostics. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the scores are not three-dimensional, the efficiency is not positive, fewer than 25 draws are pooled, or a score is missing. |
Source code in src/quivers/diagnostics/elpd.py
loo_expectation
¶
loo_expectation(draws: Tensor, weights: PSISWeights, *, kind: Literal['mean', 'var', 'sd', 'median', 'quantile'], probs: Sequence[float] = (0.5,), labels: Sequence[str | int] | None = None, components: Sequence[str | int] | None = None) -> LooExpectation
Compute leave-one-out expectations of draws with ArviZ.
| PARAMETER | DESCRIPTION |
|---|---|
draws
|
Draws
TYPE:
|
weights
|
Leave-one-out weights of the same units.
TYPE:
|
kind
|
Weighted statistic. The weighted variance divides by
TYPE:
|
probs
|
Probabilities for
TYPE:
|
labels
|
Unit and component labels; 0-based positions by default.
TYPE:
|
components
|
Unit and component labels; 0-based positions by default.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
LooExpectation
|
Statistics and ArviZ's function-specific Pareto diagnostics. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If draws and weights disagree or probabilities are invalid. |
Notes
Cost is O(S U) for S pooled draws and U units, plus a sort per unit
for quantiles. The estimates assume that PSIS is reliable for the unit
(its Pareto shape below PSISWeights.good_k).
Source code in src/quivers/diagnostics/elpd.py
618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 | |
weighted_quantile
¶
Interpolate quantiles of a weighted sample as loo's E_loo.
| PARAMETER | DESCRIPTION |
|---|---|
values
|
Draws
TYPE:
|
weights
|
Nonnegative weights
TYPE:
|
probs
|
Probabilities in the open unit interval.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ndarray
|
Linear interpolation of the normalized cumulative weights at each probability, from the largest draw below it; R's type-7 quantile when all weights are equal. |
Source code in src/quivers/diagnostics/elpd.py
loo_r2_draws
¶
Draw brms's leave-one-out R-squared by the Bayesian bootstrap.
| PARAMETER | DESCRIPTION |
|---|---|
observed
|
Observed responses
TYPE:
|
loo_mean
|
Leave-one-out expected responses
TYPE:
|
generator
|
Stream of the Dirichlet weights.
TYPE:
|
num_draws
|
Bootstrap draws, the posterior draw count S in brms.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Tensor
|
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If fewer than two units are given. |
Source code in src/quivers/diagnostics/elpd.py
subsample_estimate
¶
subsample_estimate(exact: Tensor, approximation: Tensor, subsample: Sequence[int], lpd: Tensor, pareto_k: Tensor, *, num_samples: int, responses: tuple[str, ...], hashed: str, method: Literal['plpd', 'lpd']) -> ELPDEstimate
Combine exact subsample PSIS-LOO terms with an approximation of all units.
The difference estimator of [@magnusson2020leave] under simple random
sampling without replacement estimates the total ELPD as
sum_i a_i + N mean_j (e_j - a_j) over subsample units j, with
a the approximation and e the exact PSIS-LOO terms, and its
variance with the finite-population correction 1 - m / N.
| PARAMETER | DESCRIPTION |
|---|---|
exact
|
Exact PSIS-LOO terms of the subsample,
TYPE:
|
approximation
|
Approximate terms of every unit,
TYPE:
|
subsample
|
Zero-based subsample units, aligned with
TYPE:
|
lpd
|
Log predictive densities of the subsample,
TYPE:
|
pareto_k
|
Pareto shapes of the subsample,
TYPE:
|
num_samples
|
Pooled draw count.
TYPE:
|
responses
|
Scored response names.
TYPE:
|
hashed
|
Response hash.
TYPE:
|
method
|
Approximation behind
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ELPDEstimate
|
The subsampled estimate with total and subsampling-only standard
errors; |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the subsample has fewer than two units, repeats a unit, or terms are not finite. |
Source code in src/quivers/diagnostics/elpd.py
831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 | |
kfold_estimate
¶
kfold_estimate(elpd: Tensor, lpd: Tensor, *, num_samples: int, folds: tuple[int, ...], joint: Literal['obs', 'fold', 'group'], responses: tuple[str, ...], hashed: str) -> ELPDEstimate
Assemble brms's k-fold criterion from held-out and full-data scores.
| PARAMETER | DESCRIPTION |
|---|---|
elpd
|
Held-out log predictive density per predictive unit.
TYPE:
|
lpd
|
Full-data log predictive density of the same units.
TYPE:
|
num_samples
|
Pooled draws of the full-data fit.
TYPE:
|
folds
|
Zero-based fold of every observation.
TYPE:
|
joint
|
Predictive unit.
TYPE:
|
responses
|
Scored response names.
TYPE:
|
hashed
|
Response hash.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ELPDEstimate
|
|
Source code in src/quivers/diagnostics/elpd.py
compare_estimates
¶
compare_estimates(estimates: Mapping[str, ELPDEstimate]) -> SummaryTable
Rank models by ELPD with the loo package's loo_compare layout.
| PARAMETER | DESCRIPTION |
|---|---|
estimates
|
Named estimates of one criterion on the same responses.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
SummaryTable
|
Rows ordered from best to worst; columns |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If fewer than two models are given, criteria differ, or the models score different responses or unit counts. |
Source code in src/quivers/diagnostics/elpd.py
995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 | |
stacking_weights
¶
stacking_weights(estimates: Mapping[str, ELPDEstimate], *, method: Literal['stacking', 'pseudobma'], bayesian_bootstrap: bool, seed: int) -> dict[str, float]
Return ArviZ's stacking or pseudo-BMA weights, as loo_model_weights.
| PARAMETER | DESCRIPTION |
|---|---|
estimates
|
PSIS-LOO estimates of the candidate models on the same responses.
TYPE:
|
method
|
Stacking of predictive distributions [@yao2018stacking] or pseudo-BMA weights.
TYPE:
|
bayesian_bootstrap
|
Regularize pseudo-BMA weights with the Bayesian bootstrap (loo's
TYPE:
|
seed
|
Seed of the Bayesian bootstrap.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
dict[str, float]
|
Normalized weights in input order. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the estimates are not comparable PSIS-LOO estimates. |
Source code in src/quivers/diagnostics/elpd.py
information_weights
¶
information_weights(estimates: Mapping[str, ELPDEstimate]) -> dict[str, float]
Return Akaike-type weights exp(-ic_diff / 2) as brms model_weights.
| PARAMETER | DESCRIPTION |
|---|---|
estimates
|
Estimates of one criterion (loo, waic or kfold).
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
dict[str, float]
|
Normalized weights in input order. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If criteria differ. |
Source code in src/quivers/diagnostics/elpd.py
Cited sources¶
Aki Vehtari, Andrew Gelman, and Jonah Gabry. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27:1413–1432, 2017. URL: https://link.springer.com/article/10.1007/s11222-016-9696-4, doi:10.1007/s11222-016-9696-4.
Aki Vehtari, Daniel Simpson, Andrew Gelman, Yuling Yao, and Jonah Gabry. Pareto Smoothed Importance Sampling. Journal of Machine Learning Research, 25(72):1–58, 2024. URL: https://www.jmlr.org/papers/v25/19-556.html.
Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Using Stacking to Average Bayesian Predictive Distributions (with Discussion). Bayesian Analysis, 13(3):917–1007, 2018. URL: https://doi.org/10.1214/17-BA1091, doi:10.1214/17-BA1091.
Måns Magnusson, Aki Vehtari, Johan Jonasson, and Michael Andersen. Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, 341–351. 2020. URL: https://proceedings.mlr.press/v108/magnusson20a.html.
Andrew Gelman, Ben Goodrich, Jonah Gabry, and Aki Vehtari. R-squared for Bayesian Regression Models. The American Statistician, 73(3):307–309, 2019. URL: https://doi.org/10.1080/00031305.2018.1549100, doi:10.1080/00031305.2018.1549100.