Missing data and measurement error¶
A formula model can treat missing values and noisy measurements as latent
quantities, as brms does (Bürkner, 2017). A response declared with
mi() keeps its family law: its recorded values are observed, and each
missing value is a latent coordinate drawn from that law. A predictor term
mi(x) uses those completed values. A predictor me(x, sdx) replaces a
noisy covariate by a latent true value with a known measurement error.
These terms compile through the joint program of
multivariate formulas, so a model with them is
built with bf and fitted by the ordinary fit.
A formula string passed directly to fit with these terms is refused with
a pointer to bf.
Imputation during fitting with mi()¶
bf("x | mi() ~ z") models the response x from z, and
bf("y ~ mi(x)") uses the completed x in the model for y. The joint
density is brms's: with recorded values \(x_o\) and latent missing values
\(x_m\), the response law is evaluated on the completed vector, so the latent
values receive exactly the density of the model for x. Missing values of
x are thus imputed from that model and from their role in the model for
y, jointly in each posterior draw. An mi() term enters the predictor
linearly, also in interactions such as mi(x):z. A nonlinear use such as
I(mi(x)^2) is refused. Only continuous families admit mi() responses,
as in brms (Gaussian, Student, skew Normal, log-Normal, ex-Gaussian, Gamma,
Weibull, Fréchet, exponential, inverse Gaussian, Beta, von Mises,
asymmetric Laplace, and generalized extreme value).
import numpy as np
import pandas as pd
from quivers.formulas import bf, fit, set_rescor
data = pd.DataFrame(
{
"bmi": [22.0, 24.5, 21.0, 27.0, 25.5, 23.0, 26.0, 20.5],
"chl": [180.0, np.nan, 170.0, 230.0, np.nan, 190.0, 215.0, np.nan],
"age": [1.0, 2.0, 1.0, 3.0, 2.0, 1.0, 3.0, 1.0],
}
)
data["chl"] = data["chl"] / 100
spec = (
bf("bmi ~ age * mi(chl)")
+ bf("chl | mi() ~ age")
+ set_rescor(False)
)
fitted = fit(spec, data=data, method="svi", num_steps=50, seed=1)
names = set(fitted.variables(num_samples=4))
assert {"bsp_bmi_michl", "bsp_bmi_age:michl", "Ymi_chl[2]"} <= names
scores = fitted.log_likelihood(resp="chl", num_samples=4)["chl"]
assert bool(scores[..., 1].isnan().all())
The model has no residual correlation, which is the brms default when a
response uses mi(). Parameter names follow brms: bsp_<resp>_mi<x> for
mi() coefficients and Ymi_<resp>[<row>] for imputed values.
With set_rescor(True), Gaussian or Student responses share one
multivariate Normal
or
multivariate Student
law per row, and that law scores the completed vector of the row, brms's
multi_normal_cholesky_lpdf(Yl | Mu, LSigma). A missing value is then
informed by its own model, by every model that uses it through mi(), and
by the recorded responses of its row through the residual correlation. As
in brms, the missing values have no prior of their own, so they carry the
improper flat law. Operations that need a proper prior on every site
(sample_prior, prior draws, and SMC evidence) thus refuse such a model.
correlated = fit(
bf("bmi ~ age + mi(chl)") + bf("chl | mi() ~ age") + set_rescor(True),
data=data,
method="svi",
num_steps=50,
seed=1,
)
names = set(correlated.variables(num_samples=4))
assert {"rescor__bmi__chl", "Ymi_chl[2]", "bsp_bmi_michl"} <= names
joint = correlated.log_likelihood(num_samples=4)["bmi__chl"]
assert bool(joint[..., 1].isnan().all()) and bool(joint[..., 0].isfinite().all())
drawn = correlated.posterior_predict(num_samples=4)
assert drawn["bmi"].shape == drawn["chl"].shape == (1, 4, 8)
The pointwise unit of such a model is the row's vector event. Its score is
missing (nan) on a training row where a selected response has no recorded
value, as brms reports NA, and resp selects the responses whose exact
marginal is scored. posterior_predict draws each row's vector jointly at
the completed predictor values, and posterior_epred returns the locations
at those values.
An mi() term may also be the slope of a group-level term, as in
bmi ~ mi(chl) + (mi(chl) | clinic). The varying coefficient of each
group then multiplies the completed values, brms's
(bsp + r[J[n]]) * Yl[n], and its names use the mi spelling:
sd_clinic__bmi_michl and r_clinic__bmi[<level>,michl]. The same term must
appear among the population-level terms. Without it brms declares the
varying slope and leaves it out of the predictor, so QVR refuses the formula.
data["clinic"] = ["a", "b"] * 4
varying = fit(
bf("bmi ~ mi(chl) + (mi(chl) | clinic)")
+ bf("chl | mi() ~ age")
+ set_rescor(False),
data=data,
method="svi",
num_steps=20,
seed=1,
)
assert "sd_clinic__bmi_michl" in varying.variables(num_samples=2)
Known measurement error on a response¶
bf("x | mi(sdx) ~ z") declares known measurement standard deviations in
column sdx. Every true value is then latent, \(x^\ast_i\) drawn from the
response law, and each recorded value is a Normal measurement
\(x_i \sim \mathcal N(x^\ast_i, \mathrm{sd}_i)\). A missing value is a row
without a measurement. brms names the true values Yl_<resp>[<row>].
Measurement-error predictors with me()¶
me(x, sdx) adds a latent true covariate with brms's noncentered
parameterization \(X_{\mathrm{me}} = \mu_{\mathrm{me}} +
\sigma_{\mathrm{me}} z\), \(z \sim \mathcal N(0, 1)\), and each recorded
value is \(x_i \sim \mathcal N(X_{\mathrm{me},i}, \mathrm{sd}_i)\). With
gr = g, rows of one level share a latent value, and their recorded
values and errors must agree, as in brms. The me() terms of a response
that share a grouping are correlated through one
LKJ
Cholesky factor (Lewandowski et al., 2009) unless set_mecor(False) is added.
Their hyperparameters have the brms prior classes meanme, sdme, and
corme, which PriorSpec selects.
import pandas as pd
from quivers.formulas import PriorSpec, bf, fit, set_mecor
data = pd.DataFrame(
{
"y": [0.1, 0.4, -0.2, 0.3, 0.5, 0.0, 0.7, -0.1],
"x": [1.0, 0.1, 0.5, 0.3, 2.0, 1.5, 0.4, 0.2],
"z": [0.2, 0.1, -0.3, 0.4, 0.0, 0.3, -0.2, 0.6],
"s": [0.1, 0.2, 0.3, 0.1, 0.2, 0.2, 0.4, 0.3],
}
)
spec = bf("y ~ me(x, s) + me(z, s)") + set_mecor(True)
fitted = fit(
spec,
data=data,
priors=PriorSpec(distribution="Normal(0, 2)", class_="meanme"),
method="svi",
num_steps=50,
seed=2,
)
assert {"bsp_mexs", "meanme_mex", "sdme_mez", "corme__mex__mez"} <= set(
fitted.variables(num_samples=4)
)
brms gives meanme and sdme flat priors, and so does
default_priors="brms" (flat on the real line for meanme, on (0, inf)
for sdme), which also scales the default priors of an mi() response on
its recorded values, as brms does. A flat prior is improper, so the default
quivers policy uses the formula defaults instead (fixed_prior for
meanme, random_scale_prior for sdme, correlation_prior for
corme), which keep prior draws and sample_prior available.
Sparse imputation models¶
index(id) names the rows of an mi() response, and mi(x, idx) joins a
predictor to it by those identities, so the imputation model can have
fewer rows than the model that uses it. Together with subset(), this
expresses brms's sparse imputation models: x | mi() + index(id) +
subset(sub) ~ z models x on the rows where sub is true, and
y ~ mi(x, idx) reads the completed value whose id equals each row's
idx.
Prediction, refits, and cross-validation¶
On the training rows, predictions of a model that uses mi(x) replay the
imputed values of each posterior draw. On fresh rows the recorded values
of x in new_data are used directly. brms returns missing predictions
for fresh rows whose mi() or me() values are missing, so QVR refuses
such rows with that reason. Pointwise log-likelihoods of an mi()
response are missing (nan) at its missing rows, as brms reports NA.
refit_callback thus defines held-out units on the rows whose
mi() values used by the scored responses are recorded. The true values
of an mi(sdx) response are replayed from their posterior draws, which is
brms's behavior when they are saved with save_pars(latent = TRUE).
Without saved values brms redraws them from the measurement model. update,
save_fit, and load_fit work as for any joint fit.
Multiple imputation before fitting¶
fit_multiple
fits one formula to each of several imputed datasets, as brms
brm_multiple does, and pools the draws with
combine_models.
The pooled posterior is the equally weighted mixture of the per-dataset
posteriors. Chains fitted to different datasets need not overlap, so
convergence is checked within each dataset with MultipleFit.rhats.
import numpy as np
import pandas as pd
from quivers.formulas import fit_multiple
x = np.linspace(-1.0, 1.0, 10)
first = pd.DataFrame({"x": x, "y": 0.3 + 0.8 * x + 0.1 * np.sin(7 * x)})
second = first.assign(x=first["x"] + 0.05)
result = fit_multiple(
"y ~ x",
data=[first, second],
method="nuts",
num_warmup=20,
num_samples=10,
num_chains=2,
seed=3,
)
assert result.pooled.draws()["beta_x"].shape[:2] == (4, 10)
assert len(result.rhats()) == 2
brms treats the combined fit as an ordinary fit, and so does
MultipleFit: its
summaries, prediction methods, criteria, conditional_effects, pp_check,
and hypothesis are those of the pooled fit result.pooled, evaluated over
the pooled draws. Quantities that depend on the data use the first dataset,
as in brms. combine=False returns the separate fits instead, as brms's
combine = FALSE does.
assert result.nchains() == 4
assert result.fixef().rows == ("Intercept", "x")
assert result.posterior_epred(new_data=first.iloc[:3]).shape == (4, 10, 3)
separate = fit_multiple(
"y ~ x", data=[first, second], combine=False,
method="svi", num_steps=5, seed=3,
)
assert len(separate) == 2
MultipleFit.update refits every dataset through the recipe of the first
fit, as brms's update of a brm_multiple fit does. The datasets must be
given again, the remaining arguments are kept unless overridden, and the
number of datasets may change. data2 takes one mapping per dataset.
third = first.assign(x=first["x"] - 0.05)
updated = result.update([first, second, third], num_samples=8)
assert updated.pooled.draws()["beta_x"].shape[:2] == (6, 8)
Cited sources¶
Paul-Christian Bürkner. brms: An R Package for Bayesian Multilevel Models Using Stan. Journal of Statistical Software, 2017. doi:10.18637/jss.v080.i01.
Daniel Lewandowski, Dorota Kurowicka, and Harry Joe. Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100(9):1989–2001, 2009. URL: https://www.sciencedirect.com/science/article/pii/S0047259X09000876, doi:10.1016/j.jmva.2009.04.008.