The weave pass

WeaveBuilder is weave's one pass. It reads the whole corpus and returns a WovenSite — the book as a page for each chapter and a menu over them all, ready for a presentation to render. It mirrors the product pass. Where ProductBuilder composes each document into the code it hides, WeaveBuilder projects the corpus into the literate document a reader browses.

The pass reads the parse, not the product. A woven page holds each section's prose and code as the author wrote them, in reading order, and turns every ::[…] anchor into a link rather than inlining what it names. So the pass keeps what tangle discards, and leaves referenced what tangle would resolve.

What this module draws on

The pass reads the Loom AST and writes the woven model, so it draws from both. From effect come the array and option combinators, Match for dispatching on a section's runs, Order for the reading order the pages come out in, and pipe for the spine. From the Loom side come the section a page is built from, the preamble and section-body wefts its runs are made of, and the anchor token a code link resolves. The corpus supplies titleIndexes and resolveAnchor — the same lexical lookup the editor navigates by — so a code anchor reaches the very section a bare ::[…] composes. The table-of-contents pass supplies the book's structure: buildContents reads the corpus's frontmatter into parts and chapters, which the navigation projects into its tree. From the woven side come the schema constructors for every node the pass builds: the corpus, its parts and entries, its pages, the source anchor each block records, and the blocks and links a page holds.

TypeScript

import { Array, Context, Effect, Layer, Match, Option, Order, pipe } from 'effect'
import type { LoomSection } from '@athrio/loom-ast/LoomAst'
import type { PreambleWeft, SectionBodyWeft } from '@athrio/loom-ast/Weft'
import type { WarpAnchorToken } from '@athrio/loom-ast/LoomTokens'
import {
  type LoomCorpusAst,
  type LoomModule,
  resolveAnchor,
  titleIndexes,
  type TitleIndexes,
} from '@athrio/loom-ast/LoomCorpusAst'
import {
  type AnchorLink,
  AnchorLinkSchema,
  CodeBlockSchema,
  HeadingBlockSchema,
  NoteBlockSchema,
  ProseBlockSchema,
  type SourceAnchor,
  SourceAnchorSchema,
  type WovenBlock,
  type WovenCorpus,
  WovenCorpusSchema,
  type WovenNavEntry,
  WovenNavEntrySchema,
  type WovenPage,
  WovenPageSchema,
  type WovenPart,
  WovenPartSchema,
} from './WovenCorpus'
import { buildContents, type TocChapter, type TocPart } from './TableOfContentsBuilder'
import { sectionLanguage } from '../ast/ProductBuilder'

The builder and its spine

WeaveBuilder is the service the rest of the pipeline calls. Its one method, build, wraps the pure buildCorpus in an Effect so callers compose it like any other pass.

buildCorpus is the spine. It reads the corpus once for the title index a code link resolves through, projects the table of contents into the navigation tree and its loose chapters, maps each chapter to its page, and pairs them into a WovenCorpus. A page carries the name of the part it sits under, so the spine reads that from the tree it just built.

Then it sorts. A corpus is a map, and a map hands back its values in the order they were put in — which is the order the loader finished reading them, not an order anyone chose. Left alone, the pages come out in whatever sequence the filesystem answered that morning, and a loader running alongside other work answers differently from one running by itself. The navigation is already in reading order, so the pages take theirs from it. A woven corpus is committed to a repository and read by a site that opens on its first page; it has to come out the same way twice.

TypeScript

export class WeaveBuilder extends Context.Service<WeaveBuilder>()('WeaveBuilder', {
  make: Effect.succeed({
    build: (corpus: LoomCorpusAst): Effect.Effect<WovenCorpus> =>
      Effect.sync(() => buildCorpus(corpus)),
  }),
}) {
  static readonly layer = Layer.effect(this, this.make)
}

export const buildCorpus = (corpus: LoomCorpusAst): WovenCorpus => {
  const modules = Array.fromIterable(corpus.modules.values())
  const indexes = titleIndexes(corpus.modules)
  const toc = buildContents(corpus)
  const nav = Array.map(toc.parts, wovenPart)
  const loose = Array.map(toc.loose, wovenEntry)
  const partOfSlug = partIndex(nav)
  const pages = pipe(
    Array.getSomes(Array.map(modules, (module) => buildPage(module, indexes, partOfSlug))),
    Array.sort(inReadingOrder(navOrder(nav))),
  )
  return WovenCorpusSchema.make({ nav, loose, pages })
}

Slugs and identifiers

A page is addressed by a slug, and a link points at a slug and a heading id. slugFor builds the page slug from its file path — the last two segments without the .loom extension, so 03-the-shape-of-a-loom/06-the-corpus.loom addresses as 03-the-shape-of-a-loom/06-the-corpus. slugify folds a heading title into the fragment a link targets, lowercasing it and joining its words with dashes. A heading and every anchor that names it slugify the same way, so the link and its target agree.

sliceContent reads a block's text straight from the module source between two offsets, dropping only trailing blank space. Reading the source itself, rather than rejoining wefts, keeps a block byte-for-byte as the author wrote it and keeps every offset true, so an anchor's place in the block is its place in the source, shifted by the block's start.

TypeScript

type Span = { readonly start: number; readonly end: number }

const slugFor = (path: string): string =>
  pipe(path.replace(/\.loom$/, '').split('/'), Array.takeRight(2), Array.join('/'))

const slugify = (text: string): string =>
  text
    .toLowerCase()
    .replace(/[^a-z0-9]+/g, '-')
    .replace(/^-+|-+$/g, '')

const isBlankSource = (source: string): boolean => source.trim().length === 0

const sliceContent = (text: string, span: Span): string =>
  text.slice(span.start, span.end).replace(/\s+$/, '')

The navigation is the table of contents, projected into the woven model. buildContents has already read the corpus's frontmatter into ordered parts and their chapters, so the weave only turns each into its woven node. wovenPart maps a part — its number, empty for a numberless one, its name, and its chapters. wovenEntry maps a chapter to a WovenNavEntry: its number, its title, the slug its path opens, the status and priority a board reads, and its language. partIndex inverts the finished tree into a slug-to-part-name lookup, the breadcrumb each page shows, and navOrder inverts it again into the position each slug holds. inReadingOrder sorts pages by that position and falls back to the slug, so a chapter the navigation does not list still lands somewhere fixed.

TypeScript

const wovenEntry = (chapter: TocChapter): WovenNavEntry =>
  WovenNavEntrySchema.make({
    number: chapter.number,
    title: chapter.title,
    slug: slugFor(chapter.path),
    status: chapter.status,
    priority: chapter.priority,
    language: chapter.language,
  })

const wovenPart = (part: TocPart): WovenPart =>
  WovenPartSchema.make({
    number: part.number,
    name: part.name,
    chapters: Array.map(part.chapters, wovenEntry),
  })

const partIndex = (nav: ReadonlyArray<WovenPart>): ReadonlyMap<string, string> =>
  new Map(
    Array.flatMap(nav, (part) =>
      Array.map(part.chapters, (chapter) => [chapter.slug, part.name] as const),
    ),
  )

const navOrder = (nav: ReadonlyArray<WovenPart>): ReadonlyMap<string, number> =>
  new Map(
    Array.map(
      Array.flatMap(nav, (part) => Array.map(part.chapters, (chapter) => chapter.slug)),
      (slug, at) => [slug, at] as const,
    ),
  )

const inReadingOrder = (order: ReadonlyMap<string, number>): Order.Order<WovenPage> =>
  Order.combineAll([
    Order.mapInput(Order.Number, (page: WovenPage) =>
      Option.getOrElse(Option.fromNullishOr(order.get(page.slug)), () => order.size),
    ),
    Order.mapInput(Order.String, (page: WovenPage) => page.slug),
  ])

A chapter's title

A page is titled from its loom. chapterTitleOf reads that title from the frontmatter, or from the first heading when the frontmatter declares none, so a work-in-progress loom with no membership still names its page. firstHeadingTitle finds that heading. isBookRoot marks the one module that carries the {TOC}, the contents itself, which is no chapter and so yields no page.

TypeScript

const isBookRoot = (module: LoomModule): boolean =>
  Array.some(module.doc.sections, (section) => section.entries !== undefined)

const firstHeadingTitle = (module: LoomModule): Option.Option<string> =>
  pipe(
    Array.findFirst(module.doc.sections, (section) => section.heading.title !== undefined),
    Option.flatMap((section) => Option.fromNullishOr(section.heading.title)),
    Option.map((title) => title.source),
  )

const chapterTitleOf = (module: LoomModule): Option.Option<string> =>
  Option.orElse(
    Option.map(Option.fromNullishOr(module.doc.frontmatter?.title), (title) => title.value),
    () => firstHeadingTitle(module),
  )

A page and its blocks

buildPage builds a chapter's page. Every module is a chapter except the book root, so a loom with no frontmatter title still yields a page, its title read from the first heading instead. The page takes its slug from its path, its title from chapterTitleOf, its part from the navigation tree — absent for a loose chapter — its frontmatter from the source text the frontmatter node covers, and its blocks from its sections.

buildBlocks walks a chapter's sections in order, and sectionBlocks turns one section into its blocks: its heading, then a block for each run of its body. Every block a section yields shares one source anchor, which sectionSource builds from the chapter's slug and the section's heading id — the pin back to the loom. headingBlock reads the heading's level from the count of its # marks, and its title straight from the source. Cutting the body into runs is the next section's work.

TypeScript

const buildPage = (
  module: LoomModule,
  indexes: TitleIndexes,
  partOfSlug: ReadonlyMap<string, string>,
): Option.Option<WovenPage> =>
  isBookRoot(module)
    ? Option.none()
    : Option.map(chapterTitleOf(module), (title): WovenPage => {
        const slug = slugFor(module.path)
        return WovenPageSchema.make({
          slug,
          title,
          part: partOfSlug.get(slug),
          frontmatter: module.doc.frontmatter?.source,
          blocks: buildBlocks(module, indexes),
        })
      })

const buildBlocks = (module: LoomModule, indexes: TitleIndexes): ReadonlyArray<WovenBlock> =>
  Array.flatMap(module.doc.sections, sectionBlocks(module, indexes))

const sectionSource = (module: LoomModule, section: LoomSection): SourceAnchor =>
  SourceAnchorSchema.make({
    chapter: slugFor(module.path),
    section: slugify(titleSource(section)),
  })

const sectionBlocks =
  (module: LoomModule, indexes: TitleIndexes) =>
  (section: LoomSection): ReadonlyArray<WovenBlock> => {
    const source = sectionSource(module, section)
    return [
      headingBlock(section, source),
      ...pipe(
        splitRuns(sectionBody(section)),
        Array.flatMap(runBlocks(module, indexes, section, source)),
      ),
    ]
  }

const titleSource = (section: LoomSection): string => section.heading.title?.source ?? ''

const headingLevel = (section: LoomSection): number => {
  const marks = section.heading.headingStart.source.replace(/[^#]/g, '').length
  return marks === 0 ? 1 : marks
}

const headingBlock = (section: LoomSection, source: SourceAnchor): WovenBlock =>
  HeadingBlockSchema.make({
    source,
    level: headingLevel(section),
    title: titleSource(section),
    id: slugify(titleSource(section)),
  })

Splitting a section into runs

A section's body is its preamble and its code together. sectionBody joins the two, so a fenced block that opens in the leading prose and runs on stays one prose block, not two split at the seam. splitRuns cuts that body at each => and ~: a run is one unbroken stretch of one mode, and a run that opens with no marker is prose — the leading paragraphs, or a fence the classifier left verbatim.

runBlocks turns a run into its blocks, and stamps each with the section's source anchor. A run that => opened is one code block, holding its verbatim text as code; a prose run is its prose, with any :::[Note] … ::: region split out into a note block by proseBlocks, so a reader's annotation stands apart from the text around it. runSpan bounds the content from the first non-blank line to the last, keeping a marker's own inline text but dropping the => or ~ itself, so a block reads without the frame around it. markerInline reads the code or prose written on an arrow or tilde line; pieceOfWeft reads a plain line. codeLanguage names a code block's language for the highlighter by the product pass's own rule, sectionLanguage: the section's specifier, else the extension of the file its sink names, else the chapter's language. A {Tangle} section takes the extension of the file it writes, never the mark itself. So a section that writes extension.toml in a chapter of Rust is labelled TOML, as the tangler composes it.

TypeScript

type BodyWeft = PreambleWeft | SectionBodyWeft

type Piece = {
  readonly start: number
  readonly end: number
  readonly blank: boolean
}

const sectionBody = (section: LoomSection): ReadonlyArray<BodyWeft> => [
  ...section.preamble,
  ...section.code,
]

const splitRuns = (body: ReadonlyArray<BodyWeft>): ReadonlyArray<ReadonlyArray<BodyWeft>> =>
  Array.isReadonlyArrayNonEmpty(body)
    ? Array.groupWith(body, (a, _b) => a.type !== 'ArrowWeft' && a.type !== 'TildeWeft')
    : []

const isMarker = (weft: BodyWeft): boolean => weft.type === 'ArrowWeft' || weft.type === 'TildeWeft'

const markerInline = (marker: BodyWeft): Option.Option<Piece> =>
  Match.value(marker).pipe(
    Match.when({ type: 'ArrowWeft' }, (a) =>
      Option.map(Option.fromNullishOr(a.code), (code) => ({
        start: code.position.start.offset,
        end: marker.position.end.offset,
        blank: false,
      })),
    ),
    Match.when({ type: 'TildeWeft' }, (t) =>
      Option.map(Option.fromNullishOr(t.prose), (prose) => ({
        start: prose.position.start.offset,
        end: marker.position.end.offset,
        blank: false,
      })),
    ),
    Match.orElse(() => Option.none<Piece>()),
  )

const pieceOfWeft = (weft: BodyWeft): Piece => ({
  start: weft.position.start.offset,
  end: weft.position.end.offset,
  blank: isBlankSource(weft.source),
})

const runSpan = (run: ReadonlyArray<BodyWeft>): Option.Option<Span> => {
  const pieces = Array.flatMap(run, (weft, index) =>
    index === 0 && isMarker(weft) ? Option.toArray(markerInline(weft)) : [pieceOfWeft(weft)],
  )
  const kept = Array.filter(pieces, (piece) => !piece.blank)
  return Array.isReadonlyArrayNonEmpty(kept)
    ? Option.some({
        start: Array.headNonEmpty(kept).start,
        end: Array.lastNonEmpty(kept).end,
      })
    : Option.none()
}

const codeLanguage = (module: LoomModule, section: LoomSection): string => {
  const frontmatter = Option.fromNullishOr(module.doc.frontmatter)
  return sectionLanguage(
    section,
    pipe(
      frontmatter,
      Option.flatMapNullishOr((f) => f.language),
      Option.map((language) => language.value.trim().toLowerCase()),
      Option.getOrElse(() => 'plaintext'),
    ),
    pipe(
      frontmatter,
      Option.flatMapNullishOr((f) => f.package),
      Option.map((pkg) => pkg.value),
    ),
  )
}

const notePattern = /^[ \t]*:::\[Note\][^\n]*\n([\s\S]*?)\n[ \t]*:::(?:[ \t])*$/gm

const proseBlockOf = (markdown: string, source: SourceAnchor): ReadonlyArray<WovenBlock> =>
  markdown.trim().length === 0 ? [] : [ProseBlockSchema.make({ source, markdown: markdown.trim() })]

const proseBlocks = (markdown: string, source: SourceAnchor): ReadonlyArray<WovenBlock> => {
  const walked = Array.reduce(
    Array.fromIterable(markdown.matchAll(notePattern)),
    { blocks: [] as ReadonlyArray<WovenBlock>, cursor: 0 },
    (acc, match) => {
      const at = match.index ?? 0
      return {
        blocks: [
          ...acc.blocks,
          ...proseBlockOf(markdown.slice(acc.cursor, at), source),
          NoteBlockSchema.make({ source, markdown: (match[1] ?? '').trim() }),
        ],
        cursor: at + match[0].length,
      }
    },
  )
  return [...walked.blocks, ...proseBlockOf(markdown.slice(walked.cursor), source)]
}

const codeBlock = (
  module: LoomModule,
  indexes: TitleIndexes,
  section: LoomSection,
  source: SourceAnchor,
  run: ReadonlyArray<BodyWeft>,
  span: Span,
): WovenBlock =>
  CodeBlockSchema.make({
    source,
    language: codeLanguage(module, section),
    code: sliceContent(module.text, span),
    links: pipe(
      Array.flatMap(run, (weft) => weft.anchors),
      Array.filter(
        (anchor) =>
          anchor.position.start.offset >= span.start && anchor.position.end.offset <= span.end,
      ),
      Array.map((anchor) => resolveLink(module, indexes, span.start, anchor)),
      Array.getSomes,
    ),
  })

const runBlocks =
  (module: LoomModule, indexes: TitleIndexes, section: LoomSection, source: SourceAnchor) =>
  (run: ReadonlyArray<BodyWeft>): ReadonlyArray<WovenBlock> =>
    Array.matchLeft(run, {
      onEmpty: (): ReadonlyArray<WovenBlock> => [],
      onNonEmpty: (first) =>
        Option.match(runSpan(run), {
          onNone: (): ReadonlyArray<WovenBlock> => [],
          onSome: (span) =>
            Match.value(first).pipe(
              Match.withReturnType<ReadonlyArray<WovenBlock>>(),
              Match.when({ type: 'ArrowWeft' }, () => [
                codeBlock(module, indexes, section, source, run, span),
              ]),
              Match.orElse(() => proseBlocks(sliceContent(module.text, span), source)),
            ),
        }),
    })

Resolving a code anchor

resolveLink turns one ::[…] anchor into a link. It resolves the anchor the way the composition does — through the corpus title index, in the anchor's own file or the file its path names — and reads the section it reaches. The link carries the name the author wrote, the page and heading that section becomes, and the anchor's place within the block, so the presentation makes just those characters a link. An anchor that resolves to nothing yields no link, and stands in the code as the author's own text.

TypeScript

const resolveLink = (
  module: LoomModule,
  indexes: TitleIndexes,
  blockStart: number,
  anchor: WarpAnchorToken,
): Option.Option<AnchorLink> =>
  pipe(
    resolveAnchor(indexes, module.path, anchor),
    Option.flatMap((ref) =>
      Option.map(Option.fromNullishOr(ref.section.heading.title), (title): AnchorLink =>
        AnchorLinkSchema.make({
          name: anchor.name.value,
          targetSlug: slugFor(ref.module),
          targetId: slugify(title.source),
          offset: anchor.position.start.offset - blockStart,
          length: anchor.position.end.offset - anchor.position.start.offset,
        }),
      ),
    ),
  )

The module

TypeScript

::[What this module draws on]

::[The builder and its spine]

::[Slugs and identifiers]

::[The navigation tree]

::[A chapter's title]

::[A page and its blocks]

::[Splitting a section into runs]

::[Resolving a code anchor]