Packaging the grammar

The tree-sitter-loom package holds the grammar, its scanner, its queries, and the parser tree-sitter generates from them. Editors do not install it from a registry. Neovim's nvim-treesitter, Helix, and Zed each fetch a grammar from a Git repository, from a subdirectory when the grammar shares its repository with other code, and compile the C sources they find there. So the package stays private, and the repository is how it ships.

Building the package is two steps. Tangling writes grammar.js, src/scanner.c, and the queries from the book. Then tree-sitter generate compiles grammar.js into src/parser.c with its tables in src/grammar.json and src/node-types.json. The generated files are committed beside the tangled ones, because an editor compiles the parser without running the generator.

The manifest

The generate script runs the generator, and the parse script parses a file with the grammar to show its tree. The typecheck script checks the test's TypeScript, as every package's does. The tree-sitter-cli package provides the tree-sitter command at a pinned version, so every checkout generates the same parser. The test compares the grammar with Loom's own classifier, so @athrio/loom-lang and the AST types of @athrio/loom-ast are development dependencies, with Effect and the workspace's test helper.

The package's version is the grammar's own. It comes from the grammar variable of the workspace configuration, not from the version that every other Loom package shares. tree-sitter copies the grammar's version into the parser it generates, so a version shared with Loom's releases would make every release regenerate a parser whose grammar had not changed. Neovim, Helix, and Zed fetch the grammar by commit, so none of them reads its version. The grammar's version rises only when the grammar changes, and the parser is regenerated with it.

{{ version = ${grammar} }} {{ effect = ${effect} }}

JSON

{
  "name": "tree-sitter-loom",
  "version": "::[version]",
  "description": "Loom grammar for tree-sitter",
  "license": "MIT",
  "private": true,
  "repository": {
    "type": "git",
    "url": "https://github.com/athrio-com/loom.git",
    "directory": "packages/tree-sitter-loom"
  },
  "scripts": {
    "generate": "tree-sitter generate",
    "parse": "tree-sitter parse",
    "typecheck": "tsc -p tsconfig.json --noEmit"
  },
  "devDependencies": {
    "@athrio/effect-test": "workspace:*",
    "@athrio/loom-ast": "workspace:*",
    "@athrio/loom-lang": "workspace:*",
    "effect": "::[effect]",
    "tree-sitter-cli": "0.26.8"
  }
}

The grammar's description

The tree-sitter command reads tree-sitter.json to learn what the grammar is. It names the language loom, the scope source.loom, and the .loom extension, and points at the two queries. The injection pattern lets another grammar hand text to this one: a Markdown fence written loom is parsed as a loom, so a loom quoted in the docs is coloured as one. Its version is the grammar's own, the one the package carries, and the one tree-sitter writes into the parser.

{{ version = ${grammar} }}

JSON

{
  "grammars": [
    {
      "name": "loom",
      "camelcase": "Loom",
      "scope": "source.loom",
      "path": ".",
      "file-types": ["loom"],
      "highlights": "queries/highlights.scm",
      "injections": "queries/injections.scm",
      "injection-regex": "^loom$"
    }
  ],
  "metadata": {
    "version": "::[version]",
    "license": "MIT",
    "description": "Loom literate programming documents",
    "links": {
      "repository": "https://github.com/athrio-com/loom"
    }
  }
}

The TypeScript config

The package's only TypeScript is its test. The tsconfig.json extends the workspace-root tsconfig.base.json and covers the test directory, and nothing is ever compiled to ship.

JSON

{
  "extends": "../../tsconfig.base.json",
  "compilerOptions": {
    "outDir": "out",
    "rootDir": "test"
  },
  "include": ["test"]
}

Generated files, marked

The generated parser is thousands of lines no one reviews. .gitattributes marks it and its tables as generated, so a diff on GitHub folds them away.

gitattributes

src/parser.c linguist-generated
src/grammar.json linguist-generated
src/node-types.json linguist-generated
src/tree_sitter/** linguist-generated

Compiled parsers, ignored

An editor that builds the grammar from a local checkout, as nvim-treesitter does from a path, leaves the compiled library in the package folder, and so does tree-sitter build. The library is specific to one machine, so Git ignores it in each form a platform gives it, and in WebAssembly.

Zed and Gram build the extension in editors/zed in place, the same way. They leave Cargo's build folder and lockfile there, the compiled extension, and a clone of the grammar under grammars, and Git ignores all of it.

gitignore

*.so
*.dylib
*.dll
*.wasm
editors/zed/target/
editors/zed/grammars/
editors/zed/Cargo.lock