Skip to content

Mixtures, category formulas, and count denominators

This guide covers three parts of the brms response interface (Bürkner, 2017): finite mixtures of response families, separate linear predictors for the categories of a categorical or compositional response, and the rate() and subset() response terms. Each compiles to one QVR program through the ordinary fit entry point, so SVI, HMC, NUTS, prediction, refits, cross-validation, and saved artifacts work as they do for any other formula.

Finite mixtures

In a finite mixture each observation comes from one of K component distributions, and the likelihood sums the weighted component densities (McLachlan and Peel, 2000):

\[ p(y_i) = \sum_{k=1}^{K} \theta_{ik} \, p_k(y_i \mid \mu_{ik}, \ldots). \]

mixture builds the family from its components, as brms mixture() does. Components may be different families with the same support (all real or all integer). The distributional parameters of component k carry the suffix k: mixture("gaussian", "student_t") has mu1, sigma1, mu2, sigma2, nu2, and the mixing proportions theta1 and theta2.

brms QVR
family = mixture(gaussian, student) family=mixture("gaussian", "student_t")
mixture(gaussian, nmix = 3) mixture("gaussian", nmix=3) or family="mixture", mixture_components=3
mixture(..., order = "none") mixture(..., order="none")
bf(y ~ 1, mu1 ~ x, mu2 ~ z) distributional={"mu1": "~ x", "mu2": "~ z"}
bf(y ~ x, theta2 ~ z) distributional={"theta2": "~ z"}
bf(y ~ x, theta1 = 1, theta2 = 3) fixed_parameters={"theta1": 1.0, "theta2": 3.0}
bf(y ~ x, sigma2 = "sigma1") mixture(..., equal={"sigma2": "sigma1"})
prior(normal(5, 2), Intercept, dpar = mu2) PriorSpec(distribution="Normal(5, 2)", class_="Intercept", dpar="mu2")
prior(dirichlet(2), theta) PriorSpec(distribution="Dirichlet(concentration=[2.0, 2.0])", class_="theta")
pp_mixture(fit, summary = FALSE) fitted.pp_mixture()

A parameter equated to a sampled one, such as sigma2 above, appears in draws() and variables() under its own name with the draws of its source, because brms declares it as a transformed parameter. A parameter equated to a predicted one varies by row and is not among the draws.

The main formula predicts every mu<k> without a formula of its own, and each component receives its own coefficients and group effects. A mu<k> ~ ... formula replaces the main formula for that component. Mixing proportions take one of three forms:

  • by default, one simplex theta with a Dirichlet prior (Dirichlet(1, ..., 1)), as in brms,
  • predicted by formulas for all but one proportion, as brms requires. The proportion without a formula is the reference with linear predictor zero, and the proportions are the softmax of the K linear predictors, row by row,
  • fixed for every component, then divided by their sum.
import numpy as np
import pandas as pd

from quivers.formulas import fit, mixture

rng = np.random.default_rng(1)
rows = 60
low = rng.random(rows) < 0.6
data = pd.DataFrame({"x": rng.normal(size=rows), "z": rng.integers(0, 2, rows)})
data["y"] = np.where(low, rng.normal(0.5 * data.x, 1.0), rng.normal(5.0, 1.0))

two_normals = fit(
    "y ~ x",
    data=data,
    family=mixture("gaussian", "gaussian"),
    distributional={"theta2": "~ z"},
    method="svi",
    num_steps=200,
    seed=2,
)
membership = two_normals.pp_mixture(num_samples=20)
assert membership.shape == (1, 20, rows, 2)
locations = two_normals.posterior_linpred(num_samples=20)
assert locations.shape == (1, 20, rows, 2)
expected = two_normals.posterior_epred(new_data=data.head(3), num_samples=20)
assert expected.shape == (1, 20, 3)

Identifying the components

Relabelling the components of a mixture leaves its likelihood unchanged, so an unconstrained posterior has K! symmetric modes and draws can switch labels (Stephens, 2000). With order="mu" (the default when all components share one family, as in brms) the location intercepts are ordered: the first intercept keeps its prior, and each later intercept is the previous one plus a positive gap,

\[ \alpha_1 < \alpha_2 < \cdots < \alpha_K, \qquad \alpha_k = \alpha_1 + \sum_{j<k} \delta_j, \quad \delta_j > 0 . \]

The gaps are sampled on the log scale by the same positive transform as ordinal threshold spacings, so the mirrored labellings are outside the parameter space rather than penalized. Two choices differ from brms, and both follow from conventions this package uses throughout:

  • brms places its intercept prior on each ordered intercept. QVR places the intercept prior on the first and a HalfNormal(5) prior on each gap (class mixture_intercept_spacing), because every latent coordinate carries a proper generative prior. A prior on the intercept of mu2 and later components is thus not available under ordering.
  • brms orders the intercepts of its centered design. QVR orders the intercept that the program samples: the centered one under default_priors="brms" (where intercepts are centered), and the uncentered one otherwise.

Ordering requires an intercept in every location formula. Mixtures of different families default to order="none".

fitted.draws() reports the derived intercepts of mu2, ..., muK beside the sampled sites, so fixed_effects(parameter="mu2") and the coefficient tables read them as ordinary coefficients.

Censoring, truncation, and prediction

cens() scores a censored row by the weighted component probabilities of its interval. Under trunc() each component is normalized on the truncation interval before mixing, as in brms, so the density is sum_k theta_k p_k(y) / P_k(lb <= Y <= ub). Predictions draw a component and then a value from that component's truncated law.

posterior_epred returns sum_k theta_k E_k[Y], posterior_predict draws a component and then a response, and log_likelihood returns the pointwise log of the weighted sum. posterior_linpred(parameter="mu") returns every component location on a last axis, and parameter="mu2" selects one. pp_mixture returns the posterior probability that each row belongs to each component, theta_k p_k(y) / sum_j theta_j p_j(y), with draws on the leading axes as brms returns with summary = FALSE.

Mixtures of Gaussian components compile to the native MixtureNormal kernel, and other mixtures to the general Mixture of component laws. A mixture can be one response of a multivariate model.

A group identifier shared by several component formulas, (1 | p | g) in the main formula for instance, correlates those group effects in one block across the components, as in brms. Categorical, simplex, continuous hurdle, Wiener, Cox, and custom families are refused as components. brms also excludes the categorical, simplex, and continuous hurdle families.

Mixtures of ordinal families

The cumulative, stopping-ratio (sratio), continuation-ratio (cratio), adjacent-category (acat), and hurdle cumulative families mix with one another but not with other families, as in brms. Each component is the ordinary ordinal model of its family, with its own thresholds, discrimination (disc1 ~ x), and category-specific cs() terms. The response is an ordered factor or integer codes (1..K as brms codes them, or 0..K-1), and posterior_epred returns the mixed category probabilities on a last category axis.

order selects how the thresholds are identified. The default for ordinal components is order="none", where every component has its own threshold vector. order="mu" gives every component one shared threshold vector, which is ordered when any component is cumulative. The components then differ only through their predictors and discriminations. Shared thresholds require flexible thresholds in every component and location formulas that all keep or all drop the intercept.

rating_rng = np.random.default_rng(7)
ratings = pd.DataFrame(
    {"x": rating_rng.normal(size=80), "z": rating_rng.normal(size=80)}
)
ratings["y"] = pd.Categorical(
    rating_rng.integers(1, 5, 80), categories=[1, 2, 3, 4], ordered=True
)

ordinal = fit(
    "y ~ x",
    data=ratings,
    family=mixture("cumulative", "sratio"),
    distributional={"mu2": "~ z"},
    method="svi",
    num_steps=50,
    seed=5,
)
probabilities = ordinal.posterior_epred(new_data=ratings.head(3), num_samples=4)
assert probabilities.shape == (1, 4, 3, 4)
assert ordinal.pp_mixture(num_samples=4).shape == (1, 4, 80, 2)

shared = fit(
    "y ~ x",
    data=ratings,
    family=mixture("sratio", "cumulative", order="mu"),
    distributional={"mu2": "~ z"},
    method="svi",
    num_steps=50,
    seed=5,
)
assert "dpar_mu1_cutpoint_offset" not in shared.draws()

Each ordinal component keeps the quivers threshold coordinates of its family: a location intercept and thresholds centered at zero, where brms samples the thresholds alone. The two describe the same likelihood, with brms's threshold k equal to the centered threshold minus the intercept, but the default priors differ as they do for a single ordinal family. Shared thresholds are not available with equidistant thresholds (as in brms) or hurdle components. Group-specific thresholds (thres(gr = g)) give every component its own vector per group.

Category-specific formulas

A categorical, multinomial, Dirichlet, Dirichlet-multinomial, or logistic-normal response has one location parameter per non-reference category, named mu followed by the category label, as in brms: levels A, B, C with reference A give muB and muC. The main formula predicts every category without a formula of its own. A formula for one category gives it its own coefficients and group effects, and the reference category's linear predictor is zero.

import numpy as np
import pandas as pd

from quivers.formulas import PriorSpec, fit

rng = np.random.default_rng(3)
rows = 60
choices = pd.DataFrame({"x": rng.normal(size=rows), "z": rng.normal(size=rows)})
choices["y"] = pd.Categorical(rng.choice(["A", "B", "C"], rows))

per_category = fit(
    "y ~ x",
    data=choices,
    family="categorical",
    distributional={"muC": "~ z"},
    priors=PriorSpec(distribution="Normal(0.0, 1.0)", class_="b", dpar="muC"),
    method="svi",
    num_steps=50,
    seed=4,
)
assert set(per_category.fixed_effects(parameter="muC", num_samples=5)) == {
    "Intercept",
    "z",
}
probabilities = per_category.posterior_epred(new_data=choices.head(4), num_samples=5)
assert probabilities.shape == (1, 5, 4, 3)

A per-category predictor accepts every term of an ordinary predictor. Monotonic mo() effects, gp() terms, multi-membership mm() factors, and the gr() options by, dist, and pw have scalar coefficients, so each free category takes its own simplex, Gaussian process hyperparameters, and group standard deviations (simo_muB_mok1, lscale_muC_gpz, sd_mmg1g2__muB_Intercept in brms's names). A main formula with one of these terms is copied to every free category, as brms copies it. A factor with dist = "student" shares one degrees-of-freedom parameter across the categories that use it.

term_rng = np.random.default_rng(8)
choices["dose"] = term_rng.integers(1, 5, rows)
choices["rater1"] = term_rng.choice(["p", "q", "r"], rows)
choices["rater2"] = term_rng.choice(["p", "q", "r"], rows)

special = fit(
    "y ~ mo(dose) + (1 | mm(rater1, rater2))",
    data=choices,
    family="categorical",
    distributional={"muC": "~ gp(z, k = 5, c = 5/4)"},
    method="svi",
    num_steps=50,
    seed=4,
)
assert {"simo_muB_modose1[1]", "sd_mmrater1rater2__muB_Intercept", "lscale_muC_gpz"} <= set(
    special.variables()
)
assert special.posterior_epred(new_data=choices.head(4), num_samples=5).shape == (
    1,
    5,
    4,
    3,
)

Measurement-error me() terms are not available in a per-category or other distributional formula, and mi() terms there compile through a joint formula. The shared logistic-normal scale formula sigma ~ ... keeps one coefficient per category and refuses mo(), mm(), and the gr() options.

Family.with_reference("C") (brms refcat) selects the reference, and the free categories are then muA and muB. When every category has the same formula, the model equals the shared emission with one coefficient vector over the free categories. Coefficient names follow brms (b_muC_z).

Category-specific scales

The logistic-normal family models the log ratios of each non-reference component to the reference as correlated Normal variables. Each non-reference component has its own scale, named sigma followed by the component label, as in brms (Bürkner, 2017): components a, b, c with reference a give sigmab and sigmac, the standard deviations of log(b / a) and log(c / a). A formula for one scale predicts it through the log link with its own coefficients and group effects. A scale without a formula stays one positive parameter, and fixed_parameters={"sigmac": 0.5} holds it constant.

shares = rng.dirichlet([2.0, 3.0, 4.0], size=rows)
parts = pd.DataFrame(
    {"a": shares[:, 0], "b": shares[:, 1], "c": shares[:, 2], "x": choices["x"]}
)

scaled = fit(
    "cbind(a, b, c) ~ x",
    data=parts,
    family="logistic_normal",
    distributional={"sigmab": "~ x"},
    priors=PriorSpec(distribution="HalfNormal(1.0)", class_="sigmac"),
    method="svi",
    num_steps=50,
    seed=4,
)
assert {"b_sigmab_Intercept", "b_sigmab_x", "sigmac"} <= set(scaled.variables())
spread = scaled.posterior_linpred(
    parameter="sigmab", transform=True, new_data=parts.head(4), num_samples=5
)
assert spread.shape == (1, 5, 4) and bool((spread > 0).all())

A scale formula is an ordinary scalar predictor, so it accepts the terms a per-category location accepts. Priors address it with dpar="sigmab", and the scalar scale has the class sigmac. The reference component has no log ratio and thus no scale. sigma ~ ... is the quivers spelling that predicts every scale with one formula, which brms does not have, and it cannot be combined with a sigma<c> formula or fixed value.

Count denominators, thresholds, and row subsets

y | rate(denom) ~ x models a count over denom units of exposure for the poisson, negative_binomial, negbinomial2, and geometric families. The mean becomes mu * denom for every link, and the negative binomial shape is multiplied by denom as well, so a count over denom units is the sum of denom independent unit counts. The geometric family thus becomes the negative binomial with shape denom, as in brms. posterior_epred includes the denominator, while posterior_linpred and mu do not. New rows require the denominator column. Denominators must be positive (brms admits zero, which gives a degenerate zero mean).

subset() accepts a Boolean column or numeric 0/1 indicators, as brms reads them as logical values.

y | thres(4) ~ x declares four thresholds (five categories) for an ordinal family even when some categories are not observed, and the older cat(5) declares the five categories directly. Integer responses are then codes 0, ..., 4, and a factor response must have exactly five levels. For the hurdle cumulative family, code zero is the hurdle and lies outside the thresholds, as in brms: thres(4) then declares the codes 0, ..., 5. The custom-family data arguments vreal() and vint() correspond to custom_family(..., data={"native": "column"}).

Group-specific thresholds

y | thres(gr = g) ~ x gives each level of g its own threshold vector with no pooling between groups, as brms does. The vectors may differ in length, which describes items rated on scales with different numbers of categories. A group's number of thresholds is its largest response code, or the value of n in thres(n, gr = g), where n is a number or a column that is constant within each group. thresholds_by= is the partially pooled alternative, with one common number of thresholds.

items = pd.DataFrame(
    {
        "rating": [0, 1, 2, 3, 1, 2, 0, 3, 1, 2, 0, 1, 2, 1, 0, 2, 1, 2, 0, 1],
        "x": np.linspace(-1.5, 1.5, 20),
        "item": ["long"] * 10 + ["short"] * 10,
    }
)
grouped = fit(
    "rating | thres(gr = item) ~ x",
    data=items,
    family="cumulative",
    method="svi",
    num_steps=50,
    seed=6,
)
names = set(grouped.variables(num_samples=4))
assert {"b_Intercept[long,3]", "b_Intercept[short,2]", "b_x"} <= names
assert "b_Intercept[short,3]" not in names
probabilities = grouped.posterior_epred(num_samples=4)
assert probabilities.shape == (1, 4, 20, 4)
assert bool((probabilities[..., 10:, 3] == 0).all())

The cumulative, sratio, cratio, acat, and hurdle cumulative families take threshold groups. variables() reports brms's thresholds b_Intercept[<group>,<k>], and a multivariate model reports b_<resp>_Intercept[<group>,<k>] for a response with threshold groups and b_<resp>_Intercept[<k>] for a cumulative response with one vector. The sampled coordinates are the quivers threshold coordinates in every group: a shape with mean zero and a location. With a formula intercept the first group's location is zero and the other locations (threshold_location) are contrasts with it. The map to brms's vectors is one to one, but the default priors differ, as they do for a single threshold vector. posterior_epred returns one column per category of the longest scale, with probability zero for the categories a group lacks. New rows must belong to a fitted group, because a group's thresholds are estimated from its own rows alone. Category-specific effects are refused with threshold groups, as in brms, and so are equidistant thresholds.

With the hurdle cumulative family a group's number of thresholds is its largest response code minus one, since code zero is the hurdle, and posterior_epred has the hurdle in its first column. In a mixture of ordinal families every component has its own vector for every group, and order="mu" gives the components one shared vector per group.

from quivers.formulas import mixture

visits = items.assign(rating=items["rating"] + 1)
visits.loc[[2, 7, 13, 18], "rating"] = 0
hurdle = fit(
    "rating | thres(gr = item) ~ x",
    data=visits,
    family="hurdle_cumulative",
    method="svi",
    num_steps=50,
    seed=6,
)
assert {"hu", "b_Intercept[long,3]", "b_Intercept[short,2]"} <= set(
    hurdle.variables(num_samples=4)
)
assert hurdle.posterior_epred(num_samples=4).shape == (1, 4, 20, 5)

mixed = fit(
    "rating | thres(gr = item) ~ x",
    data=items,
    family=mixture("cumulative", "acat"),
    method="svi",
    num_steps=50,
    seed=6,
)
mixed_probabilities = mixed.posterior_epred(num_samples=4)
assert bool((mixed_probabilities[..., 10:, 3] == 0).all())
assert mixed.pp_mixture(num_samples=4).shape == (1, 4, 20, 2)

A mixture keeps the quivers threshold coordinates of each component under their native names (dpar_mu1_cutpoint_log_spacing, dpar_mu1_threshold_location), as an ordinal mixture without groups does. brms fits this model but its log_lik and posterior_epred fail for it. The components of a mixture with threshold groups must all be hurdle cumulative or all lack the hurdle, because a group's number of thresholds is read from one coding of the response. brms accepts the mixed case.

update_adterms edits the addition terms of a formula as brms update_adterms does: update_adterms("y | weights(w) ~ x", "~ trials(10)") returns "y | weights(w) + trials(10) ~ x", replacing a term of the same kind, and action="replace" drops the existing terms first.

import numpy as np
import pandas as pd

from quivers.formulas import fit

rng = np.random.default_rng(5)
exposure = pd.DataFrame({"x": rng.normal(size=40), "hours": rng.uniform(1.0, 4.0, 40)})
exposure["events"] = rng.poisson(2.0 * exposure.hours)
exposure["keep"] = (np.arange(40) % 4 != 0).astype(int)

counts = fit(
    "events | rate(hours) + subset(keep) ~ x",
    data=exposure,
    family="negative_binomial",
    method="svi",
    num_steps=30,
    seed=6,
)
assert counts.posterior_epred(new_data=exposure.head(3), num_samples=4).shape == (1, 4, 3)

Cited sources

Paul-Christian Bürkner. brms: An R Package for Bayesian Multilevel Models Using Stan. Journal of Statistical Software, 2017. doi:10.18637/jss.v080.i01.

Geoffrey McLachlan and David Peel. Finite Mixture Models. Wiley, 2000. URL: https://onlinelibrary.wiley.com/doi/book/10.1002/0471721182, doi:10.1002/0471721182.

Matthew Stephens. Dealing with label switching in mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 62(4):795–809, 2000. URL: https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/1467-9868.00265, doi:10.1111/1467-9868.00265.