Skip to content

Sequence functions

These compiler helpers elaborate learned functions over whole sequences, a [role=function] morphism with param_source=self_attention or param_source=positions(N), to comprehensions of the checked core over a sequence's open extent. Their runtime form is SequenceFunction.

Internal module

This module is an implementation detail of the compiler and its tooling. Its __all__ is empty, so none of the names below are part of the public API, and they may change without notice.

sequence_functions

Checked elaboration of learned functions over whole sequences.

A [role=function] morphism whose param_source is self_attention or positions(N) maps a sequence of rows to a sequence of rows: a program calls it in a let on a table Tensor[Real]([T, d]), the sequence a scan with outputs=all binds or a host datum, and the value is the table Tensor[Real]([T, m]). The position extent T may be open, so the elaboration builds the map from comprehensions over it rather than from factor binders over declared objects.

Self-attention with h heads of width e = m / h elaborates to

  • the query, key, and value tables, one affine map per row: q[i] = Wq x[i] + bq, and likewise k and v;
  • per head g and row i, the scores s[j] = (1 / sqrt(e)) * sum_u q[i, g e + u] k[j, g e + u] and the attention weights, softmax(s) or, when causal, the weights w[j] = [j <= i] exp(s[j] - max_{j' <= i} s[j']) normalized to sum to one, which equal the softmax over the positions j <= i and are zero after i;
  • the context c[i, col] = sum_j a[col div e, i, j] v[j, col];
  • the output projection Wo c[i] + bo at every row.

A position table elaborates to x[i, k] + P[i, k]. The weights are program inputs with roles weight and bias, named after the function as a learned function's are, so the reference machine runs a call with the compiled program's weights supplied as data.

sequence_source_kind

sequence_source_kind(morphism: MorphismDecl) -> str | None

The sequence network a learned function's param_source names.

PARAMETER DESCRIPTION
morphism

The [role=function] declaration.

TYPE: MorphismDecl

RETURNS DESCRIPTION
str | None

self_attention or positions, else None.

Source code in src/quivers/dsl/sequence_functions.py
def sequence_source_kind(morphism: MorphismDecl) -> str | None:
    """The sequence network a learned function's ``param_source`` names.

    Parameters
    ----------
    morphism : MorphismDecl
        The ``[role=function]`` declaration.

    Returns
    -------
    str | None
        ``self_attention`` or ``positions``, else ``None``.
    """
    for entry in morphism.options:
        if entry.key != "param_source":
            continue
        value = entry.value
        kind = value.func if isinstance(value, OptionCall) else None
        if kind is None:
            kind = str(getattr(value, "value", ""))
        return kind if kind in SEQUENCE_SOURCE_KINDS else None
    return None