Pygments lexer

The Pygments lexer for .qvr source, registered under the qvr alias through the pygments.lexers entry point.

pygments_lexer

Pygments lexer for the QVR domain-specific language.

The lexer drives on the in-tree tree-sitter parser (compiled via quivers.dsl._grammar_build) so it always reflects the authoritative grammar; there is no regex approximation. When the shared library cannot be loaded, lexer construction raises with a typed diagnostic so the failure is visible at the rendering site rather than silently producing a degraded highlight.

Registers the qvr language alias so that code blocks tagged qvr in MkDocs, Sphinx, or any Pygments-based renderer get proper syntax highlighting.

Registration via the [project.entry-points] table in pyproject.toml::

[project.entry-points."pygments.lexers"]
qvr = "quivers.dsl.pygments_lexer:QvrLexer"

QvrLexer

Bases: Lexer

Pygments lexer for .qvr (quivers DSL) files.

The lexer is a thin walker over the in-tree tree-sitter parse; the grammar is the single source of truth. There is no regex approximation: when the shared library can't be loaded, the lexer raises a typed exception so the failure is visible at the rendering site rather than silently emitting a degraded highlight.

get_tokens_unprocessed

get_tokens_unprocessed(text: str) -> Iterator[tuple[int, _TokenType, str]]

Yield (index, token_type, text) tuples for text.

PARAMETER	DESCRIPTION
text	The .qvr source to tokenize. TYPE: str

YIELDS	DESCRIPTION
tuple[int, _TokenType, str]	The character offset, Pygments token type, and text of each leaf of the parse and of the whitespace between leaves.

RAISES	DESCRIPTION
OSError	If the in-tree tree-sitter parser library cannot be loaded.

Source code in src/quivers/dsl/pygments_lexer.py

def get_tokens_unprocessed(
    self, text: str
) -> Iterator[tuple[int, _TokenType, str]]:
    """Yield ``(index, token_type, text)`` tuples for ``text``.

    Parameters
    ----------
    text : str
        The ``.qvr`` source to tokenize.

    Yields
    ------
    tuple[int, _TokenType, str]
        The character offset, Pygments token type, and text of each
        leaf of the parse and of the whitespace between leaves.

    Raises
    ------
    OSError
        If the in-tree tree-sitter parser library cannot be loaded.
    """
    parser, _, _ = _load_parser()
    src_bytes = text.encode("utf-8")
    tree = parser.parse(src_bytes)

    # Walk leaf-first in source order, emitting tokens for
    # each leaf and reproducing inter-leaf whitespace as Text.
    leaves: list[tuple[tree_sitter.Node, str | None, str | None]] = []

    def walk(
        node: tree_sitter.Node,
        parent_kind: str | None,
        field_name: str | None,
    ) -> None:
        if not node.children:
            leaves.append((node, parent_kind, field_name))
            return
        for index, child in enumerate(node.children):
            walk(child, node.type, node.field_name_for_child(index))

    walk(tree.root_node, None, None)

    cursor = 0
    for node, parent_kind, field_name in leaves:
        start = node.start_byte
        end = node.end_byte
        if start > cursor:
            gap = src_bytes[cursor:start].decode("utf-8")
            if gap:
                yield (_byte_to_char_offset(src_bytes, cursor), Text, gap)
        node_text = src_bytes[start:end].decode("utf-8")
        token = _node_kind_to_pygments_token(
            node.type,
            node_text,
            parent_kind,
            field_name,
        )
        yield (_byte_to_char_offset(src_bytes, start), token, node_text)
        cursor = end
    if cursor < len(src_bytes):
        tail = src_bytes[cursor:].decode("utf-8")
        if tail:
            yield (_byte_to_char_offset(src_bytes, cursor), Text, tail)