Sequence functions¶
These compiler helpers elaborate
learned functions over whole sequences,
a [role=function] morphism with param_source=self_attention or param_source=positions(N),
to comprehensions of the checked core over a sequence's open extent. Their runtime form is
SequenceFunction.
Internal module
This module is an implementation detail of the compiler and its
tooling. Its __all__ is empty, so none of the names below are
part of the public API, and they may change without notice.
sequence_functions
¶
Checked elaboration of learned functions over whole sequences.
A [role=function] morphism whose param_source is
self_attention or positions(N) maps a sequence of rows to a
sequence of rows: a program calls it in a let on a table
Tensor[Real]([T, d]), the sequence a scan with outputs=all binds
or a host datum, and the value is the table Tensor[Real]([T, m]). The
position extent T may be open, so the elaboration builds the map from
comprehensions over it rather than from factor binders over declared
objects.
Self-attention with h heads of width e = m / h elaborates to
- the query, key, and value tables, one affine map per row:
q[i] = Wq x[i] + bq, and likewisekandv; - per head
gand rowi, the scoress[j] = (1 / sqrt(e)) * sum_u q[i, g e + u] k[j, g e + u]and the attention weights,softmax(s)or, when causal, the weightsw[j] = [j <= i] exp(s[j] - max_{j' <= i} s[j'])normalized to sum to one, which equal the softmax over the positionsj <= iand are zero afteri; - the context
c[i, col] = sum_j a[col div e, i, j] v[j, col]; - the output projection
Wo c[i] + boat every row.
A position table elaborates to x[i, k] + P[i, k]. The weights are
program inputs with roles weight and bias, named after the
function as a learned function's are, so the reference machine runs a
call with the compiled program's weights supplied as data.
sequence_source_kind
¶
sequence_source_kind(morphism: MorphismDecl) -> str | None
The sequence network a learned function's param_source names.
| PARAMETER | DESCRIPTION |
|---|---|
morphism
|
The
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
str | None
|
|