The store of installed services

Loom resolves an id to a package name by one rule: the id python names the package @athrio/loom-service-python. The id the author activates, the package the store installs, and the id the loaded service declares are one string from end to end, so finding a language's package is a substitution, not a lookup table to keep in step.

export const servicePackage = (id: string): string => `@athrio/loom-service-${id}`

The store lives in the workspace, under .loom/services/ — the directory loom add fills, beside the committed config.yaml. Git ignores the store, since Loom rebuilds it on demand, while the configuration is committed. Loom installs each language's service there once, to serve the whole workspace, and never under the user's home directory: a project's tools belong to the project, not to the machine.

A path belongs to the nearest workspace above it. A workspace is a folder that holds a configuration, the .loom/config.yaml file that loom init writes, and isWorkspace asks that of a folder. A .loom folder alone does not make one. The install script keeps the standalone executable in ~/.loom/bin, and a loose loom keeps the lock of its last tangle in a .loom folder beside it. Neither folder holds a configuration, so neither turns the folder above it into a workspace. The rule keeps the home folder, which holds ~/.loom/bin once Loom is installed, from becoming a workspace that every loose loom under it falls into.

workspaceOf walks up from a directory to the first workspace, and yields nothing when it reaches the filesystem's root without finding one. Workspaces can nest. A Git worktree checked out inside a repository carries a configuration of its own. The walk stops at the first configuration it meets, so a loom in the worktree belongs to the worktree, not to the repository around it.

workspaceRoot names a workspace for any directory. It returns the workspace that holds the directory, or the directory itself when no workspace does. storeDir names the store, the services folder within that workspace. The language server and loom format pass the workspace of the loom they serve, and the commands that add and remove a language pass the working directory. Either way the walk reaches the workspace the path belongs to, so a service the loader looks for is the service loom add placed there. LOOM_HOME overrides the search with an explicit store, the hook a test or a shared setup uses to point every resolver at one directory.

export const isWorkspace = (dir: string): boolean => existsSync(join(dir, '.loom', 'config.yaml'))

export const workspaceOf = (dir: string): Option.Option<string> => {
  if (isWorkspace(dir)) return Option.some(dir)
  const parent = dirname(dir)
  return parent === dir ? Option.none() : workspaceOf(parent)
}

export const workspaceRoot = (dir: string): string => Option.getOrElse(workspaceOf(dir), () => dir)

export const storeDir = (dir: string): string =>
  process.env.LOOM_HOME === undefined
    ? join(workspaceRoot(dir), '.loom', 'services')
    : join(process.env.LOOM_HOME, 'services')

installedServices lists the languages a store has installed, each by its id. It reads the @athrio scope of the store's node_modules, keeps the loom-service- packages, and strips the prefix back to the id — the inverse of servicePackage. A store that does not yet exist holds nothing.

const servicePrefix = 'loom-service-'

export const installedServices = (dir: string): Effect.Effect<ReadonlyArray<string>> =>
  Effect.sync(() => {
    const scope = join(storeDir(dir), 'node_modules', '@athrio')
    if (!existsSync(scope)) return []
    return readdirSync(scope)
      .filter((name) => name.startsWith(servicePrefix))
      .map((name) => name.slice(servicePrefix.length))
  })

Every package in the store carries a manifest, the package.json its author published. Loom reads three of its fields. The version field says which release of the service the store holds. The exports and main fields say which file a host imports to load the service.

ServiceManifestSchema is that much of a manifest, every field optional, since a field the author left out is no error. The exports field may take several shapes, so the schema admits any value there and leaves its reading to the loader. serviceDir names the folder where the store keeps a service, and serviceManifest reads the manifest in that folder and decodes it. It yields nothing when the store does not hold the service, or when the file is not a manifest at all.

export const ServiceManifestSchema = Schema.Struct({
  version: Schema.optional(Schema.String),
  main: Schema.optional(Schema.String),
  exports: Schema.optional(Schema.Unknown),
})

export type ServiceManifest = typeof ServiceManifestSchema.Type

export const serviceDir = (id: string, dir: string): string =>
  join(storeDir(dir), 'node_modules', servicePackage(id))

export const serviceManifest = (id: string, dir: string): Option.Option<ServiceManifest> =>
  pipe(
    Option.liftThrowable(() => readFileSync(join(serviceDir(id, dir), 'package.json'), 'utf8'))(),
    Option.flatMap(Option.liftThrowable((text: string): unknown => JSON.parse(text))),
    Option.flatMap(Schema.decodeUnknownOption(ServiceManifestSchema)),
  )

Loom releases every package at one version, the command line and each language service alike, so a service works with the Loom that shares its version. serviceVersion is that version, the workspace's own, which the release sets for every package at once. A store can therefore check its services by a plain comparison: a service at serviceVersion belongs to this Loom, and a service at any other version was installed by another.

The comparison matters because a store outlives the Loom that filled it. A service installed by an older release stays in the store when Loom is upgraded, and a service built against an older Effect fails inside the newer host, so a stale store breaks a language's checks in every editor.

installedVersion reads the version of a service the store holds, from the service's manifest. It yields nothing when the store does not hold the service.

{{ version = ${version} }}

export const serviceVersion = '::[version]'

export const installedVersion = (id: string, dir: string): Option.Option<string> =>
  Option.flatMap(serviceManifest(id, dir), (manifest) => Option.fromNullishOr(manifest.version))

Loom ships in two forms. The npm package is a JavaScript bundle that Node or Bun runs, so whoever installs it already has a JavaScript runtime and a package manager. The standalone executable is that same bundle compiled by Bun into one file, with Bun's runtime inside it, for a person who has no JavaScript tooling at all.

The executable also carries files that are not code, which its build adds beside the bundle: the TypeScript compiler, and the grammar library an editor loads. Bun lists such files in Bun.embeddedFiles, each a Blob that knows its own name. embeddedFiles reads that list, and it is empty in the npm package, where Bun may not be the runtime at all and nothing is embedded. embeddedFile finds one file in it by name.

interface EmbeddedFile extends Blob {
  readonly name?: string
}

const embeddedFiles: ReadonlyArray<EmbeddedFile> =
  (globalThis as { readonly Bun?: { readonly embeddedFiles?: ReadonlyArray<EmbeddedFile> } }).Bun
    ?.embeddedFiles ?? []

export const embeddedFile = (name: string): Option.Option<EmbeddedFile> =>
  Array.findFirst(embeddedFiles, (file) => file.name === name)

The two forms differ in what they can find around them, and a few steps of Loom must know which form is running. standalone answers that. Bun has no flag that says a program was compiled, so standalone reads two signs of it. The first is the list of embedded files: only a compiled executable has any, and Loom's always carries two.

The second sign covers an executable compiled without them. Bun gives every module of a compiled executable an address in a file system of its own, under /$bunfs/ on macOS and Linux and under ~BUN on Windows. The exact form of that address differs between platforms, so the test looks for either name anywhere in the address, with or without its first character percent-encoded.

export const standalone: boolean =
  embeddedFiles.length > 0 || /(\$|%24)bunfs|(~|%7E)BUN/i.test(String(import.meta.url))

The store is a package root, a folder with a private package.json for a package manager to install into. prepareStore makes that folder and that file the first time anything is installed.

installPackage installs one package into the store at a range of versions, and the form Loom runs in picks the package manager. The npm package runs npm install. The standalone executable has no npm to run, but it carries Bun, and Bun answers as itself when the BUN_BE_BUN variable is set. So the executable starts a second copy of itself as Bun and runs bun add.

Both leave out peer dependencies. Effect, Volar's contracts, and the service interface are a service's peers, and the host that loads the service supplies all three, so the store need not hold them.

An install that fails, whether the registry is unreachable, the package is not published, or npm is absent, fails with a StoreError that names the package and carries the cause. The command that called it can then report the reason rather than a stack trace.

export class StoreError extends Data.TaggedError('StoreError')<{
  readonly packageName: string
  readonly cause: unknown
}> {}

const prepareStore = (store: string): void => {
  mkdirSync(store, { recursive: true })
  const manifest = join(store, 'package.json')
  if (!existsSync(manifest))
    writeFileSync(
      manifest,
      `${JSON.stringify({ name: 'loom-services', private: true }, null, 2)}\n`,
    )
}

interface PackageManager {
  readonly command: string
  readonly args: ReadonlyArray<string>
  readonly env: NodeJS.ProcessEnv
}

const packageManager = (store: string, spec: string): PackageManager =>
  standalone
    ? {
        command: process.execPath,
        args: ['add', spec, '--cwd', store, '--omit', 'peer'],
        env: { ...process.env, BUN_BE_BUN: '1' },
      }
    : {
        command: 'npm',
        args: ['install', spec, '--prefix', store, '--omit=peer', '--no-audit', '--no-fund'],
        env: process.env,
      }

const installPackage = (
  dir: string,
  packageName: string,
  range: string,
): Effect.Effect<void, StoreError> =>
  Effect.gen(function* () {
    const store = storeDir(dir)
    yield* Effect.sync(() => prepareStore(store))
    const manager = packageManager(store, `${packageName}@${range}`)
    yield* Effect.tryPromise({
      try: () => run(manager.command, [...manager.args], { env: manager.env }),
      catch: (cause) => new StoreError({ packageName, cause }),
    })
  })

installService brings a language's package in the store to serviceVersion. A store that already holds the service at that version is left alone, so the call is quick and needs no network. A store that lacks the service gets it installed, and a store that holds it at another version gets it replaced. ServiceInstall records which of the three happened, with the replaced version when there was one, so the command that called the function can say what it did. Turning a language on and bringing an active language up to date both come down to this one call.

export type ServiceInstall = Data.TaggedEnum<{
  Current: {}
  Installed: {}
  Updated: { readonly from: string }
}>
export const ServiceInstall = Data.taggedEnum<ServiceInstall>()

export const installService = (id: string, dir: string): Effect.Effect<ServiceInstall, StoreError> =>
  Effect.gen(function* () {
    const installed = installedVersion(id, dir)
    if (Option.contains(installed, serviceVersion)) return ServiceInstall.Current()
    yield* installPackage(dir, servicePackage(id), serviceVersion)
    return Option.match(installed, {
      onNone: () => ServiceInstall.Installed(),
      onSome: (from) => ServiceInstall.Updated({ from }),
    })
  })

Two hosts load services, the language server and loom format, and each lends the services it loads the TypeScript compiler. The language server also checks every TypeScript section against that compiler. Both hosts look for one in the project and then beside their own install, and the language server first takes the one an editor sends. The store is the last place either looks, and the one place Loom can fill itself, so a project with no TypeScript of its own still gets one.

The host loads TypeScript through its JavaScript API, the lib/typescript.js file that TypeScript 6 and every earlier release ship. TypeScript 7 is a native program with no such file, so the store holds a release of TypeScript 6, the range typeScriptRange names. storeTypeScript finds that file in the store, and yields nothing when the store holds no TypeScript or holds a release without it.

export const typeScriptRange = '^6'

export const storeTypeScript = (dir: string): Option.Option<string> => {
  const file = join(storeDir(dir), 'node_modules', 'typescript', 'lib', 'typescript.js')
  return existsSync(file) ? Option.some(file) : Option.none()
}

The standalone executable carries TypeScript itself, so it never downloads it. Its build packs the compiler and the declaration files of the standard library, lib.*.d.ts, into one gzipped file, typescript.json.gz, which holds the package's own package.json and a map from each file's path in the package to its text. PackedTypeScriptSchema is that map. unpackTypeScript writes those files into the store, where storeTypeScript finds them, as though a package manager had installed them.

It writes them into a folder of its own beside the store's packages first, and then renames that folder to typescript in one step. A host that starts while the unpack runs finds either no TypeScript or a whole one, never a compiler without its standard library. The package manager leaves the folder alone when it later installs a service, since it touches only the packages it was asked for.

const typeScriptAsset = 'typescript.json.gz'

const PackedTypeScriptSchema = Schema.Struct({
  files: Schema.Record(Schema.String, Schema.String),
})

const typeScriptError = (cause: unknown): StoreError =>
  new StoreError({ packageName: 'typescript', cause })

const unpackTypeScript = (dir: string, packed: Blob): Effect.Effect<void, StoreError> =>
  Effect.gen(function* () {
    const archive = yield* Effect.tryPromise({
      try: async () => new Uint8Array(await packed.arrayBuffer()),
      catch: typeScriptError,
    })
    const json = yield* Effect.try({
      try: (): unknown => JSON.parse(gunzipSync(archive).toString('utf8')),
      catch: typeScriptError,
    })
    const { files } = yield* Schema.decodeUnknownEffect(PackedTypeScriptSchema)(json).pipe(
      Effect.mapError(typeScriptError),
    )
    const store = storeDir(dir)
    const modules = join(store, 'node_modules')
    const partial = join(modules, `.typescript-${process.pid}`)
    yield* Effect.sync(() => prepareStore(store))
    yield* Effect.forEach(
      Object.entries(files),
      ([path, text]) =>
        Effect.try({
          try: () => {
            mkdirSync(dirname(join(partial, path)), { recursive: true })
            writeFileSync(join(partial, path), text)
          },
          catch: typeScriptError,
        }),
      { discard: true },
    )
    yield* Effect.try({
      try: () => {
        rmSync(join(modules, 'typescript'), { recursive: true, force: true })
        renameSync(partial, join(modules, 'typescript'))
      },
      catch: typeScriptError,
    })
  })

installTypeScript brings TypeScript into the store and returns the path of its typescript.js. The executable unpacks the copy it carries. The npm package has none to unpack, so it installs TypeScript from the registry with its package manager, which needs the network once per workspace. Either way, a store that still holds no lib/typescript.js afterwards fails with a StoreError, just as a failed install does.

export const installTypeScript = (dir: string): Effect.Effect<string, StoreError> =>
  Option.match(embeddedFile(typeScriptAsset), {
    onSome: (packed) => unpackTypeScript(dir, packed),
    onNone: () => installPackage(dir, 'typescript', typeScriptRange),
  }).pipe(
    Effect.flatMap(() =>
      Option.match(storeTypeScript(dir), {
        onNone: () => Effect.fail(typeScriptError('the store holds no lib/typescript.js')),
        onSome: Effect.succeed,
      }),
    ),
  )

removeService deletes a language's package from the store. It removes the package directory directly rather than asking the package manager to, so it undoes an install however the install arrived — and removing a language the store never held is harmless.

export const removeService = (id: string, dir: string): Effect.Effect<void> =>
  Effect.sync(() => rmSync(serviceDir(id, dir), { recursive: true, force: true }))

The store works in Effect: Effect runs each read and write, Data gives StoreError and ServiceInstall their tags, Schema decodes a service's manifest and the TypeScript the executable carries, Array finds a file the executable embeds, and Option with pipe carries a value that may be missing, such as the workspace above a directory or the version of a service the store lacks. Node's modules supply the rest, and Bun provides them too. child_process runs the package manager, which promisify turns into a promise Effect.tryPromise can await. fs reads the store's contents and a package's manifest, and adds, removes and renames packages. zlib unpacks the TypeScript the executable carries. join and dirname form and walk the store path up to the workspace root, and existsSync tells whether a folder holds a configuration.

import { Array, Data, Effect, Option, Schema, pipe } from 'effect'
import { execFile } from 'node:child_process'
import {
  existsSync,
  mkdirSync,
  readdirSync,
  readFileSync,
  renameSync,
  rmSync,
  writeFileSync,
} from 'node:fs'
import { dirname, join } from 'node:path'
import { promisify } from 'node:util'
import { gunzipSync } from 'node:zlib'

const run = promisify(execFile)