The leaf tokens

LoomTokens defines the AST's leaf nodes — every span the parser recognises inside a line. Each token is built by loomNode, so it carries the same type, position, and health as any container, and most carry a Probe: the regular expression the classifier and tokeniser recognise it by. The compound tokens — Specifier, Warp — expose their parts as named subnodes, so a diagnostic can land on the exact part at fault.

The Probe annotation

A Probe is a regular expression annotated onto a token schema. The classifier and tokeniser recognise a token by its Probe match, then assemble the typed token from that match. getProbe reads the annotation back off a schema.

TypeScript

export const Probe = 'loom/Probe'

export const getProbe = (schema: Schema.Top): Option.Option<RegExp> =>
  Option.fromNullishOr(schema.ast.annotations?.[Probe] as RegExp | undefined)

Heading markers

HeadingStart marks a heading line — one to six # and the mandatory space after. It is position-only: the marker text is its source, and the heading level is the hash count, recorded for the reader, since every heading makes a flat section.

TypeScript

export const HeadingStartTokenSchema = loomNode('HeadingStart', {}).annotate({
  [Probe]: /^#{1,6} /,
})
export type HeadingStartToken = typeof HeadingStartTokenSchema.Type

Specifiers

A Specifier is {Lang}, a heading's language label — {Bash}, {json}, {Prose}. Its open, label, and close are separate subnodes, each with its own health, so a missing } lands on the close. The label's rule is health-aware: a real label matches [a-zA-Z0-9_-]+, and an empty value is allowed only on a non-ok node — the placeholder an earlier pass leaves before the tokeniser fills it.

TypeScript

export const SpecifierOpenTokenSchema = loomNode('SpecifierOpen', {
  value: Schema.Literal('{'),
}).annotate({
  [Probe]: /\{/g,
})
export type SpecifierOpenToken = typeof SpecifierOpenTokenSchema.Type

export const SpecifierLabelTokenSchema = loomNode('SpecifierLabel', {
  value: Schema.String,
}).check(
  Schema.makeFilter((t) => {
    if (t.value === '' && t.health.status === 'ok') {
      return 'empty `value` requires non-ok `health.status`'
    }
    if (t.value !== '' && !/^[a-zA-Z0-9_-]+$/.test(t.value)) {
      return `specifier label value must match [a-zA-Z0-9_-]+, got \`${t.value}\``
    }
    return undefined
  }),
)
export type SpecifierLabelToken = typeof SpecifierLabelTokenSchema.Type

export const SpecifierCloseTokenSchema = loomNode('SpecifierClose', {
  value: Schema.Literal('}'),
}).annotate({
  [Probe]: /\}/g,
})
export type SpecifierCloseToken = typeof SpecifierCloseTokenSchema.Type

export const SpecifierTokenSchema = loomNode('Specifier', {
  open: SpecifierOpenTokenSchema,
  label: SpecifierLabelTokenSchema,
  close: SpecifierCloseTokenSchema,
}).annotate({
  [Probe]: /\{[a-zA-Z0-9_-]+\}/g,
})
export type SpecifierToken = typeof SpecifierTokenSchema.Type

Sinks

A Sink is the bracketed file a section tangles to, written [file]. A section carrying one names a file to write, whether or not it also wears a {Tangle} specifier. Brackets set a sink apart from a language Specifier, which wears braces. When a chapter's Package names a directory, each {Tangle} section marks the file it emits there — [LoomNode.ts] under packages/loom-ast/src/ast/ — so the file travels with its section, not with the frontmatter. A sink exposes its open, file, and close as separate subnodes, each carrying its own health, so a diagnostic lands on the part at fault.

TypeScript

export const SinkOpenTokenSchema = loomNode('SinkOpen', {
  value: Schema.Literal('['),
}).annotate({
  [Probe]: /\[/g,
})
export type SinkOpenToken = typeof SinkOpenTokenSchema.Type

export const SinkCloseTokenSchema = loomNode('SinkClose', {
  value: Schema.Literal(']'),
}).annotate({
  [Probe]: /\]/g,
})
export type SinkCloseToken = typeof SinkCloseTokenSchema.Type

export const SinkFileLabelTokenSchema = loomNode('SinkFileLabel', {
  value: Schema.String,
}).check(
  Schema.makeFilter((t) => {
    if (t.value === '' && t.health.status === 'ok') {
      return 'empty `value` requires non-ok `health.status`'
    }
    if (t.value !== '' && !/^[a-zA-Z0-9_\-./]+$/.test(t.value)) {
      return `sink file label must match [a-zA-Z0-9_-./]+, got \`${t.value}\``
    }
    return undefined
  }),
)
export type SinkFileLabelToken = typeof SinkFileLabelTokenSchema.Type

export const SinkTokenSchema = loomNode('Sink', {
  open: SinkOpenTokenSchema,
  file: SinkFileLabelTokenSchema,
  close: SinkCloseTokenSchema,
}).annotate({
  [Probe]: /\[[^\]]*\]/g,
})
export type SinkToken = typeof SinkTokenSchema.Type

Frontmatter tokens

A chapter opens with a frontmatter block — a --- fence, a run of field lines, and a closing --- — that declares where the chapter belongs and what it produces. These tokens are its parts. FrontmatterFence is a fence line; its Probe is how the classifier finds it. A chapter always names itself, and its part is named once, by the chapter that opens it: that opener announces the part on its own line and its chapter on the next — Part III: The shape of a loom, then Chapter 1: The node foundation. A later chapter carries the part by its Roman numeral alone and names itself after the comma — Part III, Chapter 3: Wefts. FrontmatterPart holds the Roman numeral, FrontmatterPartName the part's name where the opener declares it, FrontmatterChapter the chapter's number, and FrontmatterTitle its title. Every other line is a key: value pair — a FrontmatterKey, either Package or Language, and the FrontmatterValue it binds. Only the fence carries a Probe; the tokeniser reads each remaining token from its line by position, the way it reads a heading's title.

TypeScript

export const FrontmatterFenceTokenSchema = loomNode('FrontmatterFence', {}).annotate({
  [Probe]: /^---\s*$/,
})
export type FrontmatterFenceToken = typeof FrontmatterFenceTokenSchema.Type

export const FrontmatterPartTokenSchema = loomNode('FrontmatterPart', {
  value: Schema.String,
})
export type FrontmatterPartToken = typeof FrontmatterPartTokenSchema.Type

export const FrontmatterPartNameTokenSchema = loomNode('FrontmatterPartName', {
  value: Schema.String,
})
export type FrontmatterPartNameToken = typeof FrontmatterPartNameTokenSchema.Type

export const FrontmatterChapterTokenSchema = loomNode('FrontmatterChapter', {
  value: Schema.String,
})
export type FrontmatterChapterToken = typeof FrontmatterChapterTokenSchema.Type

export const FrontmatterTitleTokenSchema = loomNode('FrontmatterTitle', {
  value: Schema.String,
})
export type FrontmatterTitleToken = typeof FrontmatterTitleTokenSchema.Type

export const FrontmatterKeyTokenSchema = loomNode('FrontmatterKey', {
  value: Schema.String,
})
export type FrontmatterKeyToken = typeof FrontmatterKeyTokenSchema.Type

export const FrontmatterValueTokenSchema = loomNode('FrontmatterValue', {
  value: Schema.String,
})
export type FrontmatterValueToken = typeof FrontmatterValueTokenSchema.Type

Table-of-contents tokens

The table of contents lists the book's parts and their chapters, and these tokens are its entries. A TocPart is a part's line — the #### Part III — The shape of a loom heading that opens a part in the listing. A TocChapter is a chapter's number and a TocTitle its name, both read from a numbered item like 5. The multi-file AST: the list number is the chapter's Chapter N, and the text is its title. None carries a Probe. Like the frontmatter tokens, each is read from its line by position, once the classifier has marked the line as table-of-contents content.

TypeScript

export const TocPartTokenSchema = loomNode('TocPart', {
  value: Schema.String,
})
export type TocPartToken = typeof TocPartTokenSchema.Type

export const TocChapterTokenSchema = loomNode('TocChapter', {
  value: Schema.String,
})
export type TocChapterToken = typeof TocChapterTokenSchema.Type

export const TocTitleTokenSchema = loomNode('TocTitle', {
  value: Schema.String,
})
export type TocTitleToken = typeof TocTitleTokenSchema.Type

Transition markers

An Arrow opens a code chunk and a Tilde opens prose. Both are position-only: the marker is recognised here, and the content after it elsewhere.

An arrow comes in two strengths. The plain arrow, =>, opens code that runs until the next heading or tilde. The explicit arrow, ~>, opens code that only a tilde closes, so a line that would otherwise read as a heading stays code inside it. A Markdown file, a shell script, or a configuration file with # comments needs that, because each of them writes lines that look like headings. The explicit arrow names its own closer, which is how a reader can tell the two apart at a glance.

TypeScript

export const ArrowTokenSchema = loomNode('Arrow', {}).annotate({
  [Probe]: /^\s*[=~]>/,
})
export type ArrowToken = typeof ArrowTokenSchema.Type

A tilde closes code only when it stands at the start of its line, followed by the end of the line or by a space and some prose. Content can begin with a tilde too: a home-relative path such as ~/bin, a Markdown fence of three tildes, a lock-file pattern in an ignore list. None of these is followed by a space, so each one stays code. The explicit arrow begins with a tilde as well, and the > after it keeps it from reading as a closer.

TypeScript

export const TildeTokenSchema = loomNode('Tilde', {}).annotate({
  [Probe]: /^~(?=\s|$)/,
})
export type TildeToken = typeof TildeTokenSchema.Type

Heading title

HeadingTitle is the human-readable title of a heading — the text between the marker and the first structural token, trimmed. It is position-only and computed from positions, not scanned, so it carries no Probe.

TypeScript

export const HeadingTitleTokenSchema = loomNode('HeadingTitle', {})
export type HeadingTitleToken = typeof HeadingTitleTokenSchema.Type

Code and prose content

Code is the content after an arrow, and Prose the content after a tilde. Both are position-only; each one's Probe picks up the run of text following the marker.

TypeScript

export const CodeTokenSchema = loomNode('Code', {}).annotate({
  [Probe]: /(?<=^\s*[=~]>\s*)\S.*$/,
})
export type CodeToken = typeof CodeTokenSchema.Type

export const ProseTokenSchema = loomNode('Prose', {}).annotate({
  [Probe]: /(?<=^~\s+)\S.*$/,
})
export type ProseToken = typeof ProseTokenSchema.Type

Warps and anchors

A Warp is a declaration, {{ name [: type] = value }}, written in a preamble line. A WarpAnchor is a reference, ::[name], written in a code line. The two use different delimiters on purpose: a warp sits in prose, where {{ is safe, but an anchor sits inside product code, where {{ would collide with templating languages that own that syntax — ::[ does not. So a warp opens with {{ and closes with }}, an anchor opens with ::[ and closes with ]; both expose open, close, and name as subnodes. An anchor may also carry an optional target — a (path) after its close, the .loom file a directory anchor places as a chapter. The path is opaque here, for a later pass to resolve relative to the anchor's own file. An anchor may instead carry an optional specifier after its close — a {Lang} label or a [sink] an author wrote by mistake. A member names a chapter and never overrides, so a later pass faults any specifier an anchor wears; the grammar keeps it only to name that fault precisely rather than misread the trailing token as code. A declaration's name must be a TypeScript identifier; an anchor's name is looser — any text up to the closing delimiter, since it may name either a binding or a section by heading. A declaration also carries an optional annotation — the : type before the value — and a default — the = value it binds. Both are opaque text here; a later pass reads their structure and decides how the name binds.

An anchor's open and close are plain strings rather than fixed literals, because a package may set its own delimiter pair in loom.json when ::[ would collide with its product language. AnchorDelims is that pair, and defaultAnchorDelims holds the built-in ::[ and ] — written as a concatenation so this corpus, which the tokeniser scans like any other, holds no bare ::[ of its own. For the same reason, this section writes the warp braces as the numeric character references { and }: a literal {{ on a preamble line would tokenise as a Warp.

TypeScript

export const WarpOpenTokenSchema = loomNode('WarpOpen', {
  value: Schema.Literal('{{'),
}).annotate({
  [Probe]: /\{\{/g,
})
export type WarpOpenToken = typeof WarpOpenTokenSchema.Type

export const WarpCloseTokenSchema = loomNode('WarpClose', {
  value: Schema.Literal('}}'),
}).annotate({
  [Probe]: /\}\}/g,
})
export type WarpCloseToken = typeof WarpCloseTokenSchema.Type

export const AnchorOpenTokenSchema = loomNode('AnchorOpen', {
  value: Schema.String,
}).annotate({
  [Probe]: /::\[/g,
})
export type AnchorOpenToken = typeof AnchorOpenTokenSchema.Type

export const AnchorCloseTokenSchema = loomNode('AnchorClose', {
  value: Schema.String,
})
export type AnchorCloseToken = typeof AnchorCloseTokenSchema.Type

export interface AnchorDelims {
  readonly open: string
  readonly close: string
}

export const defaultAnchorDelims: AnchorDelims = { open: '::' + '[', close: ']' }

export const WarpNameTokenSchema = loomNode('WarpName', {
  value: Schema.String,
}).check(
  Schema.makeFilter((t) => {
    if (t.value === '' && t.health.status === 'ok') {
      return 'empty `value` requires non-ok `health.status`'
    }
    if (t.value !== '' && !/^[a-zA-Z_][a-zA-Z0-9_]*$/.test(t.value)) {
      return `warp name must be a TS identifier, got \`${t.value}\``
    }
    return undefined
  }),
)
export type WarpNameToken = typeof WarpNameTokenSchema.Type

export const WarpAnnotationTokenSchema = loomNode('WarpAnnotation', {
  value: Schema.String,
})
export type WarpAnnotationToken = typeof WarpAnnotationTokenSchema.Type

export const WarpDefaultTokenSchema = loomNode('WarpDefault', {
  value: Schema.String,
})
export type WarpDefaultToken = typeof WarpDefaultTokenSchema.Type

export const WarpAnchorNameTokenSchema = loomNode('WarpAnchorName', {
  value: Schema.String,
}).check(
  Schema.makeFilter((t) => {
    if (t.value === '' && t.health.status === 'ok') {
      return 'empty `value` requires non-ok `health.status`'
    }
    if (t.value !== '' && t.value.includes(']')) {
      return `anchor name must not contain ], got \`${t.value}\``
    }
    return undefined
  }),
)
export type WarpAnchorNameToken = typeof WarpAnchorNameTokenSchema.Type

export const WarpAnchorTargetTokenSchema = loomNode('WarpAnchorTarget', {
  value: Schema.String,
})
export type WarpAnchorTargetToken = typeof WarpAnchorTargetTokenSchema.Type

export const WarpAnchorTokenSchema = loomNode('WarpAnchor', {
  open: AnchorOpenTokenSchema,
  name: WarpAnchorNameTokenSchema,
  close: AnchorCloseTokenSchema,
  target: Schema.optional(WarpAnchorTargetTokenSchema),
  specifier: Schema.optional(Schema.Union([SpecifierTokenSchema, SinkTokenSchema])),
}).annotate({
  [Probe]: /::\[[^\]]*\](?:\([^)]*\))?/g,
})
export type WarpAnchorToken = typeof WarpAnchorTokenSchema.Type

export const WarpTokenSchema = loomNode('Warp', {
  open: WarpOpenTokenSchema,
  name: WarpNameTokenSchema,
  annotation: Schema.optional(WarpAnnotationTokenSchema),
  default: Schema.optional(WarpDefaultTokenSchema),
  close: WarpCloseTokenSchema,
}).annotate({
  [Probe]: /\{\{[^{}]*\}\}/g,
})
export type WarpToken = typeof WarpTokenSchema.Type

Validating the anchor delimiters

A package may choose its own anchor delimiter pair, so a bad pair must fail the way a mixed line terminator does: a parse-level error the editor and the tangler both report, never a silent misparse. checkAnchorDelims is that gate, and it raises a distinct, self-describing error for each fault. It rejects an empty delimiter, an open equal to its close, and whitespace inside either. It holds the open to more, since the open is what triggers anchor scanning: the open may not be one of Loom's own markers — the warp braces, the => arrow, the ~ prose line, the # heading — nor a bracket or angle that pervades product code: <, >, [, or ]. The close escapes that list, so the built-in ] stays a valid close. Each error carries its own message naming the fault and suggesting a sound pair, so the recovery reads emptyDocument(text, err.message) and needs no separate message builder.

TypeScript

const suggestAnchorDelims = `Choose a distinct open that Loom does not use and that does not occur in your product code — for example \`${defaultAnchorDelims.open}\` and \`${defaultAnchorDelims.close}\` — in this package's loom.json.`

const reservedOpen: ReadonlyArray<string> = ['{{', '}}', '=>', '~', '#', '<', '>', '[', ']']

export class EmptyAnchorDelims extends Data.TaggedError('EmptyAnchorDelims')<{
  readonly open: string
  readonly close: string
}> {
  get message(): string {
    return `Anchor delimiters cannot be empty (got \`${this.open}\` and \`${this.close}\`). ${suggestAnchorDelims}`
  }
}

export class IdenticalAnchorDelims extends Data.TaggedError('IdenticalAnchorDelims')<{
  readonly delim: string
}> {
  get message(): string {
    return `Anchor open and close must differ; both are \`${this.delim}\`. A symmetric pair cannot be paired unambiguously. ${suggestAnchorDelims}`
  }
}

export class WhitespaceAnchorDelims extends Data.TaggedError('WhitespaceAnchorDelims')<{
  readonly open: string
  readonly close: string
}> {
  get message(): string {
    return `Anchor delimiters cannot contain whitespace (got \`${this.open}\` and \`${this.close}\`). ${suggestAnchorDelims}`
  }
}

export class ReservedAnchorDelims extends Data.TaggedError('ReservedAnchorDelims')<{
  readonly marker: string
}> {
  get message(): string {
    return `Anchor open \`${this.marker}\` is one of Loom's reserved markers. ${suggestAnchorDelims}`
  }
}

export type InvalidAnchorDelims =
  EmptyAnchorDelims | IdenticalAnchorDelims | WhitespaceAnchorDelims | ReservedAnchorDelims

export const checkAnchorDelims = (
  delims: AnchorDelims,
): Effect.Effect<AnchorDelims, InvalidAnchorDelims> => {
  if (delims.open === '' || delims.close === '')
    return Effect.fail(new EmptyAnchorDelims({ open: delims.open, close: delims.close }))
  if (delims.open === delims.close)
    return Effect.fail(new IdenticalAnchorDelims({ delim: delims.open }))
  if (/\s/.test(delims.open) || /\s/.test(delims.close))
    return Effect.fail(new WhitespaceAnchorDelims({ open: delims.open, close: delims.close }))
  if (reservedOpen.includes(delims.open))
    return Effect.fail(new ReservedAnchorDelims({ marker: delims.open }))
  return Effect.succeed(delims)
}

The token union

LoomToken is the union of every leaf token — the type the tokeniser emits.

TypeScript

export const LoomTokenSchema = Schema.Union([
  HeadingStartTokenSchema,
  SpecifierTokenSchema,
  SinkTokenSchema,
  FrontmatterFenceTokenSchema,
  FrontmatterPartTokenSchema,
  FrontmatterPartNameTokenSchema,
  FrontmatterChapterTokenSchema,
  FrontmatterTitleTokenSchema,
  FrontmatterKeyTokenSchema,
  FrontmatterValueTokenSchema,
  TocPartTokenSchema,
  TocChapterTokenSchema,
  TocTitleTokenSchema,
  ArrowTokenSchema,
  TildeTokenSchema,
  HeadingTitleTokenSchema,
  CodeTokenSchema,
  ProseTokenSchema,
  WarpTokenSchema,
  WarpAnchorTokenSchema,
])
export type LoomToken = typeof LoomTokenSchema.Type

What this module draws on

LoomTokens draws Schema for the token definitions, with Option wrapping the Probe annotation getProbe reads back off a schema. Effect and Data build the delimiter check and the tagged error it fails with. loomNode from the foundation gives each token its shared shape.

TypeScript

import { Data, Effect, Option, Schema } from 'effect'
import { loomNode } from '#ast/LoomNode'

The module

TypeScript

::[What this module draws on]

::[The Probe annotation]

::[Heading markers]

::[Specifiers]

::[Sinks]

::[Frontmatter tokens]

::[Table-of-contents tokens]

::[Transition markers]

::[Heading title]

::[Code and prose content]

::[Warps and anchors]

::[Validating the anchor delimiters]

::[The token union]