Skip to content

Log-Space Families

A Beta or Gamma variable with a small shape parameter keeps most of its mass within a few multiples of the smallest positive floating-point number, where scoring it through a sigmoid or an exponential loses its tails. These distributions live on the real line instead. LogitBeta is the distribution of the logit of a Beta variable, and LogGamma that of the log of a Gamma variable, each with its exact density and a sampler that stays in log space. The continuous families guide says when to use them.

log_space

Distributions of the logit of a Beta variable and the log of a Gamma variable.

A Beta or Gamma site with a small shape parameter puts most of its mass within a few multiples of the smallest floating-point number of zero (or of one). Scored on the constrained scale, through a sigmoid or an exponential and the Jacobian of that map, such a site loses its tails: the sigmoid saturates long before the density of the logit does, and a Gamma sampler that returns the smallest normal number in place of a smaller draw no longer samples the declared prior.

The two families here live on the real line and carry the density of the transformed variable directly.

LogitBeta The distribution of x = logit(p) for p ~ Beta(a, b), with density a * logsigmoid(x) + b * logsigmoid(-x) - log B(a, b). LogGamma The distribution of y = log(g) for g ~ Gamma(a, r), with density a * (log r + y) - r * exp(y) - log Gamma(a).

Both densities are exact for every finite argument: no intermediate quantity is formed on the constrained scale. Sampling stays in log space through the shape-augmentation identity: if G' ~ Gamma(a + 1) and U ~ Uniform(0, 1) are independent, then G' * U^(1/a) ~ Gamma(a) (Marsaglia and Tsang 2000), so log G' + log(U) / a is a log-Gamma draw that never forms the Gamma variable itself. A logit-Beta draw is the difference of two such draws, since G_a / (G_a + G_b) ~ Beta(a, b) for independent unit-rate Gamma variables. G' has shape at least one, so it never approaches the underflow threshold, and its implicit reparameterization gradient carries through, which makes both samplers reparameterized.

LogGamma

LogGamma(concentration: _Parameter, rate: _Parameter, validate_args: bool | None = None)

Bases: Distribution

The log of a Gamma variable: y = log(g) for g ~ Gamma(a, r).

The density on the real line is

log p(y) = a * (log r + y) - r * exp(y) - lgamma(a),

exact for every finite y and every positive shape. Draws are made in log space by shape augmentation (see log_standard_gamma), so a shape of 1e-8 yields draws near -1e8 rather than the clamped smallest normal number a constrained-scale sampler returns. TensorFlow Probability ships the same family as ExpGamma.

PARAMETER DESCRIPTION
concentration

The Gamma shape a, positive.

TYPE: Tensor or float

rate

The Gamma rate r, positive.

TYPE: Tensor or float

validate_args

Whether to check parameter constraints and the support of scored values.

TYPE: bool DEFAULT: None

Source code in src/quivers/continuous/log_space.py
def __init__(
    self,
    concentration: _Parameter,
    rate: _Parameter,
    validate_args: bool | None = None,
) -> None:
    self.concentration, self.rate = broadcast_all(concentration, rate)
    super().__init__(self.concentration.shape, validate_args=validate_args)

mean property

mean: Tensor

digamma(a) - log(r).

mode property

mode: Tensor

log(a / r), where a * y - r * exp(y) peaks.

variance property

variance: Tensor

trigamma(a), independent of the rate.

expand

expand(batch_shape: Size, _instance: LogGamma | None = None) -> LogGamma

Broadcast the parameters to a larger batch shape.

PARAMETER DESCRIPTION
batch_shape

The batch shape to expand to.

TYPE: Size

_instance

The instance torch's expand protocol fills.

TYPE: LogGamma DEFAULT: None

RETURNS DESCRIPTION
LogGamma

The expanded distribution.

Source code in src/quivers/continuous/log_space.py
def expand(
    self, batch_shape: torch.Size, _instance: LogGamma | None = None
) -> LogGamma:
    """Broadcast the parameters to a larger batch shape.

    Parameters
    ----------
    batch_shape : torch.Size
        The batch shape to expand to.
    _instance : LogGamma, optional
        The instance torch's ``expand`` protocol fills.

    Returns
    -------
    LogGamma
        The expanded distribution.
    """
    new = self._get_checked_instance(LogGamma, _instance)
    shape = torch.Size(batch_shape)
    new.concentration = self.concentration.expand(shape)
    new.rate = self.rate.expand(shape)
    Distribution.__init__(new, shape, validate_args=False)
    new._validate_args = self._validate_args
    return new

rsample

rsample(sample_shape: Size = Size()) -> Tensor

Draw reparameterized samples in log space.

PARAMETER DESCRIPTION
sample_shape

Leading sample dimensions.

TYPE: Size DEFAULT: Size()

RETURNS DESCRIPTION
Tensor

Draws of shape sample_shape + batch_shape.

Source code in src/quivers/continuous/log_space.py
def rsample(self, sample_shape: torch.Size = torch.Size()) -> Tensor:
    """Draw reparameterized samples in log space.

    Parameters
    ----------
    sample_shape : torch.Size
        Leading sample dimensions.

    Returns
    -------
    Tensor
        Draws of shape ``sample_shape + batch_shape``.
    """
    shape = self._extended_shape(sample_shape)
    concentration = self.concentration.expand(shape)
    return log_standard_gamma(concentration) - torch.log(self.rate.expand(shape))

log_prob

log_prob(value: Tensor) -> Tensor

The log density at value.

PARAMETER DESCRIPTION
value

Points on the real line.

TYPE: Tensor

RETURNS DESCRIPTION
Tensor

a * (log r + y) - r * exp(y) - lgamma(a).

Source code in src/quivers/continuous/log_space.py
def log_prob(self, value: Tensor) -> Tensor:
    """The log density at ``value``.

    Parameters
    ----------
    value : Tensor
        Points on the real line.

    Returns
    -------
    Tensor
        ``a * (log r + y) - r * exp(y) - lgamma(a)``.
    """
    if self._validate_args:
        self._validate_sample(value)
    log_rate = torch.log(self.rate)
    return (
        self.concentration * (log_rate + value)
        - torch.exp(value + log_rate)
        - torch.lgamma(self.concentration)
    )

entropy

entropy() -> Tensor

lgamma(a) - a * digamma(a) + a, independent of the rate.

RETURNS DESCRIPTION
Tensor

The differential entropy, one per batch element.

Source code in src/quivers/continuous/log_space.py
def entropy(self) -> Tensor:
    """``lgamma(a) - a * digamma(a) + a``, independent of the rate.

    Returns
    -------
    Tensor
        The differential entropy, one per batch element.
    """
    a = self.concentration
    return torch.lgamma(a) - a * torch.digamma(a) + a + torch.zeros_like(self.rate)

LogitBeta

LogitBeta(concentration1: _Parameter, concentration0: _Parameter, validate_args: bool | None = None)

Bases: Distribution

The logit of a Beta variable: x = logit(p) for p ~ Beta(a, b).

The density on the real line is

log p(x) = a * logsigmoid(x) + b * logsigmoid(-x) - log B(a, b),

exact for every finite x: at a = 1e-3, b = 1 and x = -800 it is about -7.71, where scoring sigmoid(x) under a Beta density with the sigmoid's Jacobian saturates and returns about -99. A draw is the difference of two log-Gamma draws made in log space (see log_standard_gamma), so it is finite and reparameterized for shapes far below where a Beta sampler returns exactly zero or one.

PARAMETER DESCRIPTION
concentration1

The Beta shape a on p, positive.

TYPE: Tensor or float

concentration0

The Beta shape b on 1 - p, positive.

TYPE: Tensor or float

validate_args

Whether to check parameter constraints and the support of scored values.

TYPE: bool DEFAULT: None

Source code in src/quivers/continuous/log_space.py
def __init__(
    self,
    concentration1: _Parameter,
    concentration0: _Parameter,
    validate_args: bool | None = None,
) -> None:
    self.concentration1, self.concentration0 = broadcast_all(
        concentration1, concentration0
    )
    super().__init__(self.concentration1.shape, validate_args=validate_args)

mean property

mean: Tensor

digamma(a) - digamma(b).

mode property

mode: Tensor

log(a / b), where the logit density peaks.

variance property

variance: Tensor

trigamma(a) + trigamma(b).

expand

expand(batch_shape: Size, _instance: LogitBeta | None = None) -> LogitBeta

Broadcast the parameters to a larger batch shape.

PARAMETER DESCRIPTION
batch_shape

The batch shape to expand to.

TYPE: Size

_instance

The instance torch's expand protocol fills.

TYPE: LogitBeta DEFAULT: None

RETURNS DESCRIPTION
LogitBeta

The expanded distribution.

Source code in src/quivers/continuous/log_space.py
def expand(
    self, batch_shape: torch.Size, _instance: LogitBeta | None = None
) -> LogitBeta:
    """Broadcast the parameters to a larger batch shape.

    Parameters
    ----------
    batch_shape : torch.Size
        The batch shape to expand to.
    _instance : LogitBeta, optional
        The instance torch's ``expand`` protocol fills.

    Returns
    -------
    LogitBeta
        The expanded distribution.
    """
    new = self._get_checked_instance(LogitBeta, _instance)
    shape = torch.Size(batch_shape)
    new.concentration1 = self.concentration1.expand(shape)
    new.concentration0 = self.concentration0.expand(shape)
    Distribution.__init__(new, shape, validate_args=False)
    new._validate_args = self._validate_args
    return new

rsample

rsample(sample_shape: Size = Size()) -> Tensor

Draw reparameterized samples as a difference of log-Gamma draws.

PARAMETER DESCRIPTION
sample_shape

Leading sample dimensions.

TYPE: Size DEFAULT: Size()

RETURNS DESCRIPTION
Tensor

Draws of shape sample_shape + batch_shape.

Source code in src/quivers/continuous/log_space.py
def rsample(self, sample_shape: torch.Size = torch.Size()) -> Tensor:
    """Draw reparameterized samples as a difference of log-Gamma draws.

    Parameters
    ----------
    sample_shape : torch.Size
        Leading sample dimensions.

    Returns
    -------
    Tensor
        Draws of shape ``sample_shape + batch_shape``.
    """
    shape = self._extended_shape(sample_shape)
    return log_standard_gamma(self.concentration1.expand(shape)) - (
        log_standard_gamma(self.concentration0.expand(shape))
    )

log_prob

log_prob(value: Tensor) -> Tensor

The log density at value.

PARAMETER DESCRIPTION
value

Points on the real line.

TYPE: Tensor

RETURNS DESCRIPTION
Tensor

a * logsigmoid(x) + b * logsigmoid(-x) - log B(a, b).

Source code in src/quivers/continuous/log_space.py
def log_prob(self, value: Tensor) -> Tensor:
    """The log density at ``value``.

    Parameters
    ----------
    value : Tensor
        Points on the real line.

    Returns
    -------
    Tensor
        ``a * logsigmoid(x) + b * logsigmoid(-x) - log B(a, b)``.
    """
    if self._validate_args:
        self._validate_sample(value)
    return (
        self.concentration1 * F.logsigmoid(value)
        + self.concentration0 * F.logsigmoid(-value)
        - _log_beta(self.concentration1, self.concentration0)
    )

entropy

entropy() -> Tensor

log B(a, b) - a (digamma(a) - digamma(a + b)) - b (digamma(b) - digamma(a + b)).

RETURNS DESCRIPTION
Tensor

The differential entropy, one per batch element.

Source code in src/quivers/continuous/log_space.py
def entropy(self) -> Tensor:
    """``log B(a, b) - a (digamma(a) - digamma(a + b)) - b (digamma(b) - digamma(a + b))``.

    Returns
    -------
    Tensor
        The differential entropy, one per batch element.
    """
    a, b = self.concentration1, self.concentration0
    total = torch.digamma(a + b)
    return (
        _log_beta(a, b)
        - a * (torch.digamma(a) - total)
        - b * (torch.digamma(b) - total)
    )

log_standard_gamma

log_standard_gamma(concentration: Tensor) -> Tensor

Draw log(G) for G ~ Gamma(concentration, 1) in log space.

The draw is log G' + log(U) / concentration with G' ~ Gamma(concentration + 1, 1) and U ~ Uniform(0, 1], which has the law of the log of a Gamma variable for every positive concentration and never forms that variable. Gradients reach the concentration through the implicit reparameterization of G' and through the division.

PARAMETER DESCRIPTION
concentration

Positive shape parameters, one per draw.

TYPE: Tensor

RETURNS DESCRIPTION
Tensor

One log-Gamma draw per entry of concentration, finite for every concentration at or above 64 times the smallest normal number of its dtype.

Source code in src/quivers/continuous/log_space.py
def log_standard_gamma(concentration: Tensor) -> Tensor:
    """Draw ``log(G)`` for ``G ~ Gamma(concentration, 1)`` in log space.

    The draw is ``log G' + log(U) / concentration`` with ``G' ~
    Gamma(concentration + 1, 1)`` and ``U ~ Uniform(0, 1]``, which has
    the law of the log of a Gamma variable for every positive
    concentration and never forms that variable. Gradients reach the
    concentration through the implicit reparameterization of ``G'`` and
    through the division.

    Parameters
    ----------
    concentration : Tensor
        Positive shape parameters, one per draw.

    Returns
    -------
    Tensor
        One log-Gamma draw per entry of ``concentration``, finite for
        every concentration at or above 64 times the smallest normal
        number of its dtype.
    """
    boosted = torch._standard_gamma(concentration + 1.0)
    return torch.log(boosted) + _log_uniform(concentration.shape, concentration) / (
        concentration
    )