The document builder

LoomAstBuilder is the final parse stage: build turns a stream of wefts into a LoomDocument. A weft before the first heading stands at the document's top level, as frontmatter or preamble; a heading gathers the body wefts up to the next heading into a section. build groups the wefts along that line, then folds each group into the document. The module also holds emptyDocument, the empty error-carrying tree a caller falls back to when it cannot produce a real one.

The build

Grouping a heading with its body means looking ahead to the next heading, so build gathers the whole weft stream first with Stream.runCollect. faultUnclosedExplicit first marks an explicit block that never closes, and warnHeadingAgainstCode marks a heading that most likely belonged to the code above it. groupNodes then splits the wefts into the document's top-level nodes, appendToDocument folds each node into a growing document, and makeDocument finishes it.

TypeScript

export class LoomAstBuilder extends Context.Service<LoomAstBuilder>()('LoomAstBuilder', {
  make: Effect.succeed({
    build: (source: Stream.Stream<LoomWeft>): Effect.Effect<LoomDocument> =>
      Stream.runCollect(source).pipe(
        Effect.map((wefts) =>
          makeDocument(
            Array.reduce(
              groupNodes(warnHeadingAgainstCode(faultUnclosedExplicit(wefts))),
              initialDocument,
              appendToDocument,
            ),
          ),
        ),
      ),
  }),
}) {
  static readonly layer = Layer.effect(this, this.make)
}

An explicit block that never closes

An explicit arrow, ~>, holds its code until a tilde. The classifier honours that line by line, so when no tilde comes, every line to the end of the file reads as code, headings included. Nothing in the classifier can say so, because it has already emitted the arrow by the time the file runs out. The builder sees the whole weft list at once, so the check lives here.

Only the last explicit arrow can be open. Inside an explicit block every line is code, so no second arrow can start while one is open. unclosedExplicit finds that last arrow and keeps it only when no tilde follows it. faultUnclosedExplicit then gives the arrow's own marker an UnclosedExplicitBlock fault. The fault is an error, so a tangle refuses the file rather than writing a section that swallowed the rest of it, and the editor underlines the ~> where the block began.

TypeScript

const isExplicitArrow = (weft: LoomWeft): weft is ArrowWeft =>
  weft.type === 'ArrowWeft' && weft.arrow.source === '~>'

const unclosedExplicit = (wefts: ReadonlyArray<LoomWeft>): Option.Option<number> =>
  pipe(
    Array.findLastIndex(wefts, isExplicitArrow),
    Option.filter(
      (at) => !Array.some(Array.drop(wefts, at + 1), (weft) => weft.type === 'TildeWeft'),
    ),
  )

const faultedArrow = (weft: ArrowWeft): ArrowWeft => ({
  ...weft,
  arrow: {
    ...weft.arrow,
    health: faulty(UnclosedExplicitBlock(), weft.arrow.position),
  },
})

const faultUnclosedExplicit = (wefts: ReadonlyArray<LoomWeft>): ReadonlyArray<LoomWeft> =>
  Option.match(unclosedExplicit(wefts), {
    onNone: () => wefts,
    onSome: (open) =>
      Array.map(wefts, (weft, at) =>
        at === open && isExplicitArrow(weft) ? faultedArrow(weft) : weft,
      ),
  })

A heading against the code

A plain arrow, =>, holds its code until the next heading. A # comment in a script, or a heading inside a Markdown file, reads as that heading, and the code stops short without a word. The explicit arrow is the cure, but an author has to know to reach for it.

The tell is the blank line. A real heading has one above it, and a comment usually sits flush against the code it follows. So a heading whose previous weft is a non-blank code line gets a HeadingAgainstCode warning on its marker. A warning leaves the tangle running, because a heading flush against code may be what the author meant. It still reaches the editor, at the line where the code was cut.

TypeScript

const isFlushCode = (weft: LoomWeft): boolean =>
  weft.type === 'CodeWeft' && weft.source.trim() !== ''

const warnedHeading = (weft: HeadingWeft): HeadingWeft => ({
  ...weft,
  headingStart: {
    ...weft.headingStart,
    health: faulty(HeadingAgainstCode(), weft.headingStart.position),
  },
})

const warnHeadingAgainstCode = (wefts: ReadonlyArray<LoomWeft>): ReadonlyArray<LoomWeft> =>
  Array.map(wefts, (weft, at) =>
    isHeading(weft) && Option.exists(Array.get(wefts, at - 1), isFlushCode)
      ? warnedHeading(weft)
      : weft,
  )

Grouping the wefts

groupNodes splits the flat weft list at the first heading. What comes before stands at the document's top level — its frontmatter and preamble — so isTopLevelWeft keeps those two kinds. From the first heading on, Array.chop walks the rest heading by heading: each sectionChunk takes a heading and the body wefts up to the next heading, and buildSection turns that pair into a LoomSection. Every chunk leads with a heading, so sectionChunk guards on isHeading; a well-formed file never trips that guard. The isHeading, isSectionBody, isPreamble, and isToc predicates classify each weft.

TypeScript

const isHeading = (w: LoomWeft): w is HeadingWeft => w.type === 'HeadingWeft'

const isSectionBody = (w: LoomWeft): w is SectionBodyWeft =>
  w.type === 'ArrowWeft' ||
  w.type === 'CodeWeft' ||
  w.type === 'TildeWeft' ||
  w.type === 'ProseWeft'

const isPreamble = (w: LoomWeft): w is PreambleWeft => w.type === 'PreambleWeft'

const isToc = (w: LoomWeft): w is TocWeft => w.type === 'TocWeft'

const isTopLevelWeft = (w: LoomWeft): w is FrontmatterWeft | PreambleWeft =>
  w.type === 'FrontmatterWeft' || w.type === 'PreambleWeft'

type TopLevelNode = FrontmatterWeft | PreambleWeft | LoomSection

const sectionChunk = (
  group: Array.NonEmptyReadonlyArray<LoomWeft>,
): readonly [Option.Option<LoomSection>, ReadonlyArray<LoomWeft>] => {
  const [body, rest] = Array.span(Array.drop(group, 1), (w) => !isHeading(w))
  const heading = group[0]
  return [isHeading(heading) ? Option.some(buildSection(heading, body)) : Option.none(), rest]
}

const groupNodes = (wefts: ReadonlyArray<LoomWeft>): ReadonlyArray<TopLevelNode> => {
  const [preamble, sections] = Array.span(wefts, (w) => !isHeading(w))
  return [
    ...Array.filter(preamble, isTopLevelWeft),
    ...Array.getSomes(Array.chop(sections, sectionChunk)),
  ]
}

Folding into the document

The fold routes each peeled node. appendToDocument appends a LoomSection to the document's sections, a PreambleWeft to its preamble, and a FrontmatterWeft to its frontmatter, starting from the empty initialDocument.

TypeScript

type DocumentBuilder = {
  readonly frontmatter: ReadonlyArray<FrontmatterWeft>
  readonly preamble: ReadonlyArray<PreambleWeft>
  readonly sections: ReadonlyArray<LoomSection>
}

const initialDocument: DocumentBuilder = {
  frontmatter: [],
  preamble: [],
  sections: [],
}

const appendToDocument = (doc: DocumentBuilder, node: TopLevelNode): DocumentBuilder =>
  pipe(
    Match.value(node),
    Match.when({ type: 'LoomSection' }, (section) => ({
      ...doc,
      sections: [...doc.sections, section],
    })),
    Match.when({ type: 'PreambleWeft' }, (weft) => ({
      ...doc,
      preamble: [...doc.preamble, weft],
    })),
    Match.when({ type: 'FrontmatterWeft' }, (weft) => ({
      ...doc,
      frontmatter: [...doc.frontmatter, weft],
    })),
    Match.exhaustive,
  )

Building a section

buildSection turns a heading and its body chunk into a LoomSection: it partitions the body into preamble, code, and toc-entry wefts, then derives the section's source and span. Only a {TOC} section carries entries; every other section leaves that run empty. headingOf lifts the heading weft to a LoomHeading; sourceOf concatenates the children's sources to reconstruct the byte range; spanFrom runs the span from the heading's start to the last child's end.

TypeScript

const buildSection = (heading: HeadingWeft, body: ReadonlyArray<LoomWeft>): LoomSection => {
  const h = headingOf(heading)
  const preamble = body.filter(isPreamble)
  const code = body.filter(isSectionBody)
  const entries = body.filter(isToc)
  return LoomSectionSchema.make({
    position: spanFrom(h.position, preamble, code, entries),
    source: sourceOf(h, preamble, code, entries),
    health: okHealth,
    heading: h,
    preamble,
    code,
    entries: entries.length > 0 ? entries : undefined,
  })
}

const headingOf = (weft: HeadingWeft): LoomHeading =>
  LoomHeadingSchema.make({
    position: weft.position,
    source: weft.source,
    health: weft.health,
    unexpected: weft.unexpected,
    headingStart: weft.headingStart,
    title: weft.title,
    specifier: weft.specifier,
    sink: weft.sink,
  })

const sourceOf = (
  ...groups: ReadonlyArray<{ readonly source: string } | ReadonlyArray<{ readonly source: string }>>
): string =>
  pipe(
    groups,
    Array.flatMap((g) => Array.ensure(g)),
    Array.map((n) => n.source),
    Array.join(''),
  )

const spanFrom = (
  headingPos: Position,
  ...groups: ReadonlyArray<ReadonlyArray<{ readonly position: Position }>>
): Position => {
  const flat = groups.flat()
  return {
    start: headingPos.start,
    end: flat.length === 0 ? headingPos.end : flat[flat.length - 1].position.end,
  }
}

Building the frontmatter

makeFrontmatter gathers a file's frontmatter wefts into one LoomFrontmatter, or nothing when the file opened with no block. The membership comes from up to two lines — the part from one, the chapter from the other — and each is numbered or not: a part carries a number and a name, or a name alone; a chapter a number and a title, or a title alone. The Language, Status, and Priority fields join the tangle target, each read by name. The target answers to Target, or to Package — the older spelling it is replacing, still honoured so a file may use either during the move.

TypeScript

const makeFrontmatter = (wefts: ReadonlyArray<FrontmatterWeft>): Option.Option<LoomFrontmatter> => {
  if (wefts.length === 0) return Option.none()
  const partFields = Option.match(
    Option.fromNullishOr(wefts.find((w) => w.part !== undefined || w.partName !== undefined)),
    {
      onNone: () => ({}),
      onSome: (w) => ({ part: w.part, partName: w.partName }),
    },
  )
  const chapterFields = Option.match(
    Option.fromNullishOr(wefts.find((w) => w.chapter !== undefined || w.title !== undefined)),
    {
      onNone: () => ({}),
      onSome: (w) => ({ chapter: w.chapter, title: w.title }),
    },
  )
  return Option.some(
    LoomFrontmatterSchema.make({
      position: {
        start: wefts[0].position.start,
        end: wefts[wefts.length - 1].position.end,
      },
      source: sourceOf(wefts),
      health: okHealth,
      ...partFields,
      ...chapterFields,
      package: Option.getOrUndefined(
        Option.orElse(frontmatterField(wefts, 'Target'), () => frontmatterField(wefts, 'Package')),
      ),
      language: Option.getOrUndefined(frontmatterField(wefts, 'Language')),
      status: Option.getOrUndefined(frontmatterField(wefts, 'Status')),
      priority: Option.getOrUndefined(frontmatterField(wefts, 'Priority')),
    }),
  )
}

frontmatterField reads a key: value field's value by name — the Target (or Package) a {Tangle} section writes to, the Language its sections inherit, or the Status and Priority a roadmap reads — lifting the missing case to Option at the .find.

TypeScript

const frontmatterField = (
  wefts: ReadonlyArray<FrontmatterWeft>,
  key: string,
): Option.Option<FrontmatterValueToken> =>
  pipe(
    Option.fromNullishOr(wefts.find((w) => w.key !== undefined && w.key.value === key)),
    Option.flatMapNullishOr((w) => w.value),
  )

Assembling the document

makeDocument assembles the final LoomDocument from the folded frontmatter, preamble, and sections, building the frontmatter block through makeFrontmatter. documentSpan spans the whole, from the first node to the last. A document needs no lang Warp: a section without one inherits the workspace's primary language, so an absent declaration is no fault.

TypeScript

const documentSpan = (db: DocumentBuilder): Position => {
  const all: ReadonlyArray<{ readonly position: Position }> = [
    ...db.frontmatter,
    ...db.preamble,
    ...db.sections,
  ]
  return all.length === 0
    ? { start: { line: 1, offset: 0 }, end: { line: 1, offset: 0 } }
    : { start: all[0].position.start, end: all[all.length - 1].position.end }
}

const makeDocument = (db: DocumentBuilder): LoomDocument => {
  const position = documentSpan(db)
  return LoomDocumentSchema.make({
    position,
    source: sourceOf(db.frontmatter, db.preamble, db.sections),
    health: okHealth,
    frontmatter: Option.getOrUndefined(makeFrontmatter(db.frontmatter)),
    preamble: db.preamble,
    sections: db.sections,
  })
}

Recovering to an empty document

Some failures leave the chain with no tree to fold. emptyDocument is the recovery: an empty document — no preamble, no sections — whose root health carries one error diagnostic at the document's span. A caller that cannot produce a real document returns this one. So a .loom always parses to a tree, and the error rides in that tree's own health rather than being thrown.

emptyDocumentFor is the mixed-terminator case. The parse chain cannot pin a mixed line terminator to any node, so it raises MixedEOL. emptyDocumentFor names the two conventions and the lines they sit on, then builds the empty document from that message. eolName gives each convention — LF, CRLF, or CR — a name a person reads.

TypeScript

export const emptyDocument = (text: string, message: string): LoomDocument => {
  const position: Position = {
    start: { line: 1, offset: 0 },
    end: { line: 1, offset: text.length },
  }
  return {
    type: 'LoomDocument',
    position,
    source: text,
    health: {
      status: 'error',
      diagnostics: [{ message, position, severity: 'error' }],
    },
    preamble: [],
    sections: [],
  }
}

export const emptyDocumentFor = (text: string, err: MixedEOL): LoomDocument =>
  emptyDocument(
    text,
    `Mixed line terminators. Line ${err.primaryLine} has ${eolName(err.primary)}, but line ${err.foundLine} has ${eolName(err.found)}. Pick one and stick with it.`,
  )

const eolName = (kind: 'lf' | 'crlf' | 'cr'): string => {
  switch (kind) {
    case 'lf':
      return 'LF (Unix)'
    case 'crlf':
      return 'CRLF (Windows)'
    case 'cr':
      return 'CR (classic Mac)'
  }
}

What this module draws on

Effect supplies Stream to gather the weft stream, Array to group and fold it, and Match, Option, Context, and Layer besides. The container schemas come from LoomAst and the frontmatter value token from LoomTokens; Position and okHealth from the foundation; the MixedEOL type from the line scanner; and the weft types from Weft. The fault catalog lends faulty and the UnclosedExplicitBlock and HeadingAgainstCode faults.

TypeScript

import { Array, Context, Effect, Layer, Match, Option, Stream, pipe } from 'effect'
import {
  LoomDocumentSchema,
  LoomFrontmatterSchema,
  LoomHeadingSchema,
  LoomSectionSchema,
  type LoomDocument,
  type LoomFrontmatter,
  type LoomHeading,
  type LoomSection,
} from '@athrio/loom-ast/LoomAst'
import type { FrontmatterValueToken } from '@athrio/loom-ast/LoomTokens'
import { okHealth, type Position } from '@athrio/loom-ast/LoomNode'
import type { MixedEOL } from './LineRanges'
import { faulty, HeadingAgainstCode, UnclosedExplicitBlock } from './LoomFault'
import type {
  ArrowWeft,
  FrontmatterWeft,
  HeadingWeft,
  LoomWeft,
  PreambleWeft,
  SectionBodyWeft,
  TocWeft,
} from '@athrio/loom-ast/Weft'

The module

TypeScript

::[What this module draws on]

::[The build]

::[An explicit block that never closes]

::[A heading against the code]

::[Grouping the wefts]

::[Folding into the document]

::[Building a section]

::[Building the frontmatter]

::[Assembling the document]

::[Recovering to an empty document]