The classifier stage

WeftClassifier labels each line with its weft kind by mode. It is a Mealy machine: the state is the previous weft and whether a heading has been seen, the input is a pattern probe of the line, and the output weft also feeds the next state. The wefts it emits are partial — the leading marker filled, the weft incomplete — and the tokeniser settles the rest.

The classifier service

The state opens before the first heading — the Document Preamble — and tracks whether a heading has been emitted, whether the line sits inside a frontmatter block, whether it sits inside a table-of-contents section, and whether it sits inside a fenced block. classifyWefts threads that state through the line-range stream with Stream.mapAccum, classifying each line and carrying the result forward. A --- toggles the frontmatter flag: the first one opens the block, the next closes it. A {TOC} heading opens a contents section, and the next top-level # heading closes it — so the toc flag follows every heading, set when the heading bears {TOC} and cleared by any other. A code fence — a line of three backticks — toggles the fence flag the same way, but only in prose. Inside a fenced block a person quotes a loom verbatim, so a #, =>, or ~ there reads as text, not as a mark. A note is a third such region: :::[Note] opens it and a lone ::: closes it, and the Markdown body stays inert between them, so a reader's note never reads as loom structure. A note and a fence never open inside each other.

One more flag follows an explicit block. An explicit arrow, ~>, sets it, and the next tilde clears it. While it is set, the block is code that no heading can close. A Markdown file or a commented script can then keep its # lines, because the classifier reads them as code rather than as the start of a new section.

TypeScript

type ClassifierState = {
  readonly prev: Option.Option<LoomWeft>
  readonly seenHeading: boolean
  readonly inFrontmatter: boolean
  readonly inToc: boolean
  readonly inFence: boolean
  readonly inNote: boolean
  readonly inExplicit: boolean
}

const initialState: ClassifierState = {
  prev: Option.none(),
  seenHeading: false,
  inFrontmatter: false,
  inToc: false,
  inFence: false,
  inNote: false,
  inExplicit: false,
}

export class WeftClassifier extends Context.Service<WeftClassifier>()('WeftClassifier', {
  make: Effect.succeed({
    classifyWefts:
      (text: string) =>
      (source: Stream.Stream<LineRange>): Stream.Stream<LoomWeft> =>
        Stream.mapAccum(
          source,
          () => initialState,
          (state, range) => {
            const lineText = text.slice(range[0], range[1])
            const weft = probeWeft(lineText, range, state)
            const frontmatterFence = weft.type === 'FrontmatterWeft' && weft.fence !== undefined
            const next: ClassifierState = {
              prev: Option.some(weft),
              seenHeading: state.seenHeading || weft.type === 'HeadingWeft',
              inFrontmatter: state.inFrontmatter !== frontmatterFence,
              inToc: weft.type === 'HeadingWeft' ? isTocHeading(lineText) : state.inToc,
              inFence: nextFence(state.inFence, state.inNote, weft),
              inNote: nextNote(state.inNote, state.inFence, weft),
              inExplicit: nextExplicit(state.inExplicit, weft),
            }
            return [next, [weft]]
          },
        ),
  }),
}) {
  static readonly layer = Layer.effect(this, this.make)
}

Classifying one line

probeWeft is the decision table. A --- on the first line opens a frontmatter block, and every line through the closing --- is a frontmatter weft; outside that block the rest of the table takes over. A code fence opens a fenced block the same way. Every line inside it is verbatim until the fence closes, so a #, =>, or ~ there stays text — the block quotes a loom, it does not compose one. A note region — :::[Note] to ::: — is inert in just the same way, so a reader's Markdown body never composes. A fenced line is a preamble weft in the document preamble, where nothing else may appear, and a prose weft inside a section. Inside a table-of-contents section every line is a toc weft — a deeper ##–###### heading and a 1. list item alike — so a part heading never opens a section and a chapter entry never reads as prose; only the next top-level # closes the section and reads as an ordinary heading again. A # line is a heading in any other mode, and opens a section in preamble mode. Before the first heading every line is a preamble weft, since no transition fires there. After it, the previous weft's mode decides. In preamble, => opens code and ~ opens prose. In code, ~ returns to prose and every other line stays code. In prose, => re-enters code and every other line stays prose. A line of dashes elsewhere is ordinary content, classified by the current mode like any other line — there is no separator weft.

An explicit block outranks every rule after the fence and the note. Inside it a tilde returns to prose, and every other line is code: a # line, an arrow, a line of dashes. The check runs before the heading rule, since a heading is exactly what the block exists to hold. A fence or a note still comes first, so an explicit arrow quoted in prose stays prose.

TypeScript

const probeWeft = (lineText: string, range: LineRange, state: ClassifierState): LoomWeft => {
  const line = Option.match(state.prev, {
    onNone: () => 1,
    onSome: (w) => w.position.end.line + 1,
  })

  if (Option.isNone(state.prev) && isFrontmatterFence(lineText))
    return makeFrontmatterFence(lineText, line, range)

  if (state.inFrontmatter) {
    if (isFrontmatterFence(lineText)) return makeFrontmatterFence(lineText, line, range)
    return makeFrontmatterField(lineText, line, range)
  }

  if (state.inFence || state.inNote)
    return state.seenHeading
      ? makeProseWeft(lineText, line, range)
      : makePreambleWeft(lineText, line, range)

  const probe = probeOf(lineText)

  if (state.inExplicit)
    return probe.kind === 'tilde'
      ? makeTildeWeft(lineText, line, range, probe.m)
      : makeCodeWeft(lineText, line, range)

  if (probe.kind === 'heading') {
    if (state.inToc && headingLevel(probe.m) > 1) return makeTocWeft(lineText, line, range)
    return makeHeadingWeft(lineText, line, range, probe.m)
  }

  if (state.inToc) return makeTocWeft(lineText, line, range)

  if (!state.seenHeading) return makePreambleWeft(lineText, line, range)

  return pipe(
    Match.value(modeOf(state.prev)),
    Match.when('preamble', () =>
      pipe(
        Match.value(probe),
        Match.when({ kind: 'arrow' }, (p) => makeArrowWeft(lineText, line, range, p.m)),
        Match.when({ kind: 'tilde' }, (p) => makeTildeWeft(lineText, line, range, p.m)),
        Match.orElse(() => makePreambleWeft(lineText, line, range)),
      ),
    ),
    Match.when('code', () =>
      pipe(
        Match.value(probe),
        Match.when({ kind: 'tilde' }, (p) => makeTildeWeft(lineText, line, range, p.m)),
        Match.orElse(() => makeCodeWeft(lineText, line, range)),
      ),
    ),
    Match.when('prose', () =>
      pipe(
        Match.value(probe),
        Match.when({ kind: 'arrow' }, (p) => makeArrowWeft(lineText, line, range, p.m)),
        Match.orElse(() => makeProseWeft(lineText, line, range)),
      ),
    ),
    Match.exhaustive,
  )
}

Line positions

linePos is a whole-line span; span is an arbitrary offset range on a line. Both stamp the line number the classifier tracks.

TypeScript

const linePos = (line: number, range: LineRange): Position => ({
  start: { line, offset: range[0] },
  end: { line, offset: range[1] },
})

const span = (line: number, start: number, end: number): Position => ({
  start: { line, offset: start },
  end: { line, offset: end },
})

Mode and probe

Two axes drive the table. modeOf derives the mode — preamble, code, or prose — from the previous weft's type; a frontmatter weft and a toc weft both read as preamble, since neither sits in a section body. probeOf recognises the line as a heading, arrow, or tilde marker, or plain text, using the same Probe regexes the token schemas carry. isFrontmatterFence tests one more marker: the --- that opens or closes the frontmatter block. headingLevel counts a heading's # marks, so the top-level heading that closes a contents section is told from a deeper one, and isTocHeading tests a heading line for a {TOC} specifier, the marker that opens that section. isCodeFence tests a line for the three backticks that bound a fenced block, and nextFence toggles the fence flag from it — opening only on a prose weft, which isProseWeft identifies, so a fence inside a code region is left to that region's own language. isNoteOpen and isNoteClose test the :::[Note] and ::: markers that bound a note, and nextNote toggles its flag the same way; each takes the other's flag and refuses to open while it is set, so a note and a fence never nest. isExplicitArrow tells an explicit arrow from a plain one by its marker, and nextExplicit sets the explicit flag on that arrow and clears it on the next tilde.

TypeScript

type Mode = 'preamble' | 'code' | 'prose'

const modeOf = (prev: Option.Option<LoomWeft>): Mode =>
  Option.match(prev, {
    onNone: () => 'preamble' as const,
    onSome: (w) =>
      pipe(
        Match.value(w),
        Match.when({ type: 'HeadingWeft' }, () => 'preamble' as const),
        Match.when({ type: 'PreambleWeft' }, () => 'preamble' as const),
        Match.when({ type: 'ArrowWeft' }, () => 'code' as const),
        Match.when({ type: 'CodeWeft' }, () => 'code' as const),
        Match.when({ type: 'TildeWeft' }, () => 'prose' as const),
        Match.when({ type: 'ProseWeft' }, () => 'prose' as const),
        Match.when({ type: 'FrontmatterWeft' }, () => 'preamble' as const),
        Match.when({ type: 'TocWeft' }, () => 'preamble' as const),
        Match.exhaustive,
      ),
  })

type Probe =
  | { readonly kind: 'heading'; readonly m: RegExpMatchArray }
  | { readonly kind: 'arrow'; readonly m: RegExpMatchArray }
  | { readonly kind: 'tilde'; readonly m: RegExpMatchArray }
  | { readonly kind: 'plain' }

const headingProbe = Option.getOrThrow(getProbe(HeadingStartTokenSchema))
const arrowProbe = Option.getOrThrow(getProbe(ArrowTokenSchema))
const tildeProbe = Option.getOrThrow(getProbe(TildeTokenSchema))
const frontmatterFenceProbe = Option.getOrThrow(getProbe(FrontmatterFenceTokenSchema))

const isFrontmatterFence = (lineText: string): boolean => frontmatterFenceProbe.test(lineText)

const isCodeFence = (source: string): boolean => /^\s*```/.test(source)

const isNoteOpen = (source: string): boolean => /^\s*:::\[Note\]/.test(source)

const isNoteClose = (source: string): boolean => /^\s*:::\s*$/.test(source)

const isProseWeft = (weft: LoomWeft): boolean =>
  weft.type === 'PreambleWeft' || weft.type === 'ProseWeft' || weft.type === 'TildeWeft'

const nextFence = (inFence: boolean, inNote: boolean, weft: LoomWeft): boolean =>
  inFence ? !isCodeFence(weft.source) : !inNote && isProseWeft(weft) && isCodeFence(weft.source)

const nextNote = (inNote: boolean, inFence: boolean, weft: LoomWeft): boolean =>
  inNote ? !isNoteClose(weft.source) : !inFence && isProseWeft(weft) && isNoteOpen(weft.source)

const isExplicitArrow = (weft: LoomWeft): boolean =>
  weft.type === 'ArrowWeft' && weft.arrow.source === '~>'

const nextExplicit = (inExplicit: boolean, weft: LoomWeft): boolean =>
  inExplicit ? weft.type !== 'TildeWeft' : isExplicitArrow(weft)

const headingLevel = (m: RegExpMatchArray): number => m[0].replace(/[^#]/g, '').length

const tocSpecifier = /\{TOC\}/i

const isTocHeading = (lineText: string): boolean => tocSpecifier.test(lineText)

const probeOf = (lineText: string): Probe => {
  const h = headingProbe.exec(lineText)
  if (h) return { kind: 'heading', m: h }
  const a = arrowProbe.exec(lineText)
  if (a) return { kind: 'arrow', m: a }
  const t = tildeProbe.exec(lineText)
  if (t) return { kind: 'tilde', m: t }
  return { kind: 'plain' }
}

Building the wefts

A constructor emits the partial weft for each kind, carrying incompleteHealth for the tokeniser to settle. makeHeadingWeft fills the headingStart marker, and makeArrowWeft and makeTildeWeft fill the arrow and tilde markers from the probe match. The arrow keeps whichever two characters it was written with, => or ~>, so the flag that follows an explicit block can read the marker back.

TypeScript

const makeHeadingWeft = (
  lineText: string,
  line: number,
  range: LineRange,
  m: RegExpMatchArray,
): HeadingWeft =>
  HeadingWeftSchema.make({
    position: linePos(line, range),
    source: lineText,
    health: incompleteHealth,
    headingStart: HeadingStartTokenSchema.make({
      position: span(line, range[0], range[0] + m[0].length),
      source: m[0],
      health: okHealth,
    }),
  })

const makeArrowWeft = (
  lineText: string,
  line: number,
  range: LineRange,
  m: RegExpMatchArray,
): ArrowWeft => {
  const start = range[0] + m[0].length - 2
  const end = range[0] + m[0].length
  return ArrowWeftSchema.make({
    position: linePos(line, range),
    source: lineText,
    health: incompleteHealth,
    arrow: ArrowTokenSchema.make({
      position: span(line, start, end),
      source: m[0].slice(-2),
      health: okHealth,
    }),
    anchors: [],
  })
}

const makeTildeWeft = (
  lineText: string,
  line: number,
  range: LineRange,
  m: RegExpMatchArray,
): TildeWeft => {
  const run = /~+$/.exec(m[0])![0]
  const start = range[0] + m[0].length - run.length
  const end = range[0] + m[0].length
  return TildeWeftSchema.make({
    position: linePos(line, range),
    source: lineText,
    health: incompleteHealth,
    tilde: TildeTokenSchema.make({
      position: span(line, start, end),
      source: run,
      health: okHealth,
    }),
    anchors: [],
  })
}

A plain line bears no marker, so its constructor takes none. makePreambleWeft, makeCodeWeft, and makeProseWeft build the ordinary lines of the three modes.

TypeScript

const makePreambleWeft = (lineText: string, line: number, range: LineRange): PreambleWeft =>
  PreambleWeftSchema.make({
    position: linePos(line, range),
    source: lineText,
    health: incompleteHealth,
    warps: [],
    anchors: [],
  })

const makeCodeWeft = (lineText: string, line: number, range: LineRange): CodeWeft =>
  CodeWeftSchema.make({
    position: linePos(line, range),
    source: lineText,
    health: incompleteHealth,
    anchors: [],
  })

const makeProseWeft = (lineText: string, line: number, range: LineRange): ProseWeft =>
  ProseWeftSchema.make({
    position: linePos(line, range),
    source: lineText,
    health: incompleteHealth,
    anchors: [],
  })

A frontmatter line splits two ways. makeFrontmatterFence builds a ---, already whole, and makeFrontmatterField a membership or key: value line, left for the tokeniser to fill. makeTocWeft builds one line of a contents section the same way — a part heading, a chapter item, or a blank — its parts left for the tokeniser.

TypeScript

const makeFrontmatterFence = (lineText: string, line: number, range: LineRange): FrontmatterWeft =>
  FrontmatterWeftSchema.make({
    position: linePos(line, range),
    source: lineText,
    health: okHealth,
    fence: FrontmatterFenceTokenSchema.make({
      position: span(line, range[0], range[0] + 3),
      source: '---',
      health: okHealth,
    }),
  })

const makeFrontmatterField = (lineText: string, line: number, range: LineRange): FrontmatterWeft =>
  FrontmatterWeftSchema.make({
    position: linePos(line, range),
    source: lineText,
    health: incompleteHealth,
  })

const makeTocWeft = (lineText: string, line: number, range: LineRange): TocWeft =>
  TocWeftSchema.make({
    position: linePos(line, range),
    source: lineText,
    health: incompleteHealth,
  })

What this module draws on

Effect supplies Stream for the accumulating scan and Match for the dispatch. The LineRange type comes from the line scanner, the marker probes and schemas from LoomTokens, the weft schemas from Weft, and incompleteHealth, okHealth, and Position from the foundation.

TypeScript

import { Context, Effect, Layer, Match, Option, Stream, pipe } from 'effect'
import type { LineRange } from './LineRanges'
import { incompleteHealth, okHealth, type Position } from '@athrio/loom-ast/LoomNode'
import {
  ArrowTokenSchema,
  FrontmatterFenceTokenSchema,
  HeadingStartTokenSchema,
  TildeTokenSchema,
  getProbe,
} from '@athrio/loom-ast/LoomTokens'
import {
  type ArrowWeft,
  ArrowWeftSchema,
  type CodeWeft,
  CodeWeftSchema,
  type FrontmatterWeft,
  FrontmatterWeftSchema,
  type HeadingWeft,
  HeadingWeftSchema,
  type LoomWeft,
  type PreambleWeft,
  PreambleWeftSchema,
  type ProseWeft,
  ProseWeftSchema,
  type TildeWeft,
  TildeWeftSchema,
  type TocWeft,
  TocWeftSchema,
} from '@athrio/loom-ast/Weft'

The module

TypeScript

::[What this module draws on]

::[The classifier service]

::[Classifying one line]

::[Line positions]

::[Mode and probe]

::[Building the wefts]