Extending the Builder

How to extend tbdocs — a new pipeline task, a markdown-it plugin, a render-worker sub-stage, a verification gate or a build-time count — and how to change one that already exists. Read tbdocs Builder first for the architecture and Pipeline Stages for the data contracts each task uses.

The extension points

Pipeline task — a new entry in the static TASKS graph in tbdocs.mjs. The task declares its predecessors and either runs on the main thread or dispatches to a worker handler. There is no plugin registry or hook system.

Markdown-it plugin — a function that configures the shared markdown-it instance, registered in createMarkdownIt in render.mjs. Each render worker builds its own markdown-it from that factory, so the plugin runs on every page.

Render-worker sub-stage — a transformation slotted into the per-chunk render handler in cpu-worker.mjs, between two existing sub-stages (renderPhase → computeChunkSeo → templatePhase → offline → deriveSearchEntries). Use it for per-page CPU work that should run in parallel with the page render.

Verification gate — a script under scripts/, run after the build by check.bat or test.bat, that decides whether what the build produced is acceptable. It changes nothing in TASKS or the plugin chain; see Adding a verification gate, which also says which wrapper it belongs in.

Build-time count — a named number the build derives and substitutes into prose, written {{tbdocs:<name>}} on a page. A new one is an entry in deriveCounts in counts.mjs plus a row in the table on Authoring Pages; see Adding a build-time count.

Styling is not an extension point. Style rules live under docs/_sass/ and need no change to the task graph or the plugin chain. See Project styling for where each rule belongs, the two-compilation model, and the dark-mode specificity trap: a rule that loses it applies in light mode and silently does not in dark.

Why serve.bat does not show a builder change

Note

Changes to task definitions, worker handlers, or markdown-it plugins are not hot-reloaded by serve mode. The worker pool is persistent: after editing any of these, stop serve.bat (Ctrl+C) and re-run to load the new code. SCSS and page content are watched and rebuilt.

Two things produce the same symptom, and neither reports anything.

The watcher never sees the edit. serve.bat watches docs/. builder/ is at the repository root, so editing a builder module fires no rebuild.

A forced rebuild still runs the old code. If you touch a page to provoke a rebuild, the workers still have the previous module imported. A new task does not appear; a changed task goes on running its old body, which looks like the change having no effect.

So: Ctrl+C and re-run after every builder edit. There is no partial reload. Page content and docs/_sass/ are watched, so styling work needs no restart.

When test.bat says a regex can backtrack exponentially

check_regex_safety.mjs reads every regex in the .mjs files under builder/, scripts/, lib/, book/, eval/, wisdom/ and indexer/ (vendored code excluded), and refuses one that can take exponential time on some input. It runs in test.bat and in both CI workflows, so a regex can pass the build and check.bat and still be refused. The report starts with

FAIL: 1 regex(es) can backtrack exponentially:

and names the file and line, the pattern, and a witness — an input that makes that pattern take exponential time.

The gate’s own entry explains the cause and the rewrite in full. The short form:

  1. Keep the witness. Put the pattern and the witness in a scratch script and call re.test(witness): it hangs. Only this shows the rewrite worked; the site’s pages pass both before and after.
  2. Find the two parts of the pattern that can match the same character. Narrowing a character class usually leaves the same ambiguity one level down.
  3. Stop describing the structure between the delimiters. Match <tag([^>]*)>, and take the attributes apart afterwards in ordinary JavaScript, which cannot backtrack.
  4. Run the gate again, and re.test(witness) once more. It must return at once.

Such a regex reaching a build does not fail it: it stops it, and the build prints its last line and waits. When a build stops instead of failing covers that.

When test.bat fails in check_code_regions

check_code_regions.mjs runs every page through the real pre-render rewrite chain and fails when a rewrite changed the inside of a fenced code block, an indented code block or an inline code span. It names each page:

FAIL  Reference/Core/Example.md: 1 code region(s) altered by a pre-render rewrite

Run node scripts/check_code_regions.mjs --verbose to see each region before and after, which usually shows which rewrite did it.

The rewrites in render.mjs run over raw markdown, before markdown-it parses it, so a rewrite cannot tell prose from code. lib/markdown.mjs decides what is code, by a block parse:

  • blockRegions lists the fences, code blocks and HTML blocks with their lines.
  • maskCode hides the code around a rewrite.
  • splitCodeSpans cuts a line into prose and code spans.
  • splitOnMarker and mapLines split and rewrite by line.

A rewrite, or a tool that scans page source, uses these instead of a regex of its own. The fix is in where the rewrite runs:

  1. For a fence or an inline span, move the rewrite behind the mask. In applyPreRenderRewrites, code regions are replaced by placeholders before the rewrites run and put back afterwards. A rewrite added between the two never sees code:

    const code = maskCode(source, { md });
let work = code.masked;
work = rewriteTripleAsteriskEmphasis(work);
// ... the other masked rewrites ...
work = yourRewrite(work);
return rewriteAdmonitions(code.restore(work), md);

  2. For an indented code block, the mask cannot help. It does not cover indented blocks. Narrow the rewrite so it cannot match the block’s lines, and let the gate say when it no longer does. A rewrite that has to run outside the mask can skip the lines blockRegions reports, as rewriteAdmonitions does.

  3. Do not widen the mask to make the report go away. No rewrite reaches text the mask hides, so a mask that hides prose stops the rewrites firing on it — and the region comparison cannot see that, because the hidden text comes back unchanged.
  4. Read a built page the rewrite is meant to change. The gate’s probes assert what the existing rewrites must do, and nothing asserts what a new one must do.

A failed probe that names a raw <pre>, such as FAIL probe: a void tag inside a raw <pre> and <code>, is a rewrite over rendered HTML reaching into code. Pass its pattern to replaceOutsideCode from builder/code-guard.mjs, as applyPostRenderRewrites in render.mjs does, instead of calling replace over the whole page.

Changing what is already there

Most builder work changes something that exists. The walkthroughs below run forwards, from nothing to a working task; this section indexes them for a reader who has already changed something.

Find the row for what changed. The middle column is the step that explains the mechanism; the right-hand column is what else moves with it in code. What a change obliges in the documentation applies to every row alike — see What a change obliges in the documentation.

Changed Mechanism Also moves, in code
A task’s expected list Pick the right flags Nothing. The scheduler derives successor edges from expected. The exception is the dynamic edges: render:i and flush:i are wired in dispatch.submit, not in TASKS.
A task’s execute() or handler return shape Define the task in TASKS The task’s own submit(), and every downstream task that reads the field back off state.
Which thread a task runs on Decide where the work runs Swap runOnMain: true + execute for handler: plus a cpu-worker.mjs entry, and add the name to HANDLERS in sab-scheduler.mjs. A task moved onto a worker loses direct access to state: everything it reads must arrive through the payload SAB, and everything it produces must come back through its return value.
Which Gantt section a task charts under — GANTT_SECTION in gantt.mjs, not the task definition. A ganttSection on the definition wins where it is set, which is why the walkthroughs below set one, but only the render:i and flush:i tasks that dispatch registers carry one. Of the 33 static tasks, 31 are in the map; the other two, warmInit and renderEnvInit, are unique_per_worker tasks, whose per-lane timings chart as Boot. Any other task with no section, or with a section the chart does not draw, fails the build.
A field on the per-page render delta Worked example B The pages.map(…) projection at the end of the render handler and the merge in dispatch.submit’s render:i callback. Without both, the field never reaches main.
What a markdown-it rule emits Write the plugin Nothing in the chain, if the rule is self-contained. But changing the emitted HTML changes what the accessibility scan sees — read the construct-family note at the end of Verify.

A running serve.bat does not pick up an edit to a handler or a task definition; see the NOTE at the top of this page.

An expected list says what must have merged, not what the body reads. Removing a predecessor because execute() ignores its input reintroduces a race. writePdf lists renderJoin and never touches it: the book is assembled from page.renderedContent, and renderJoin guarantees every chunk’s content has been merged into the master pages. The scheduler checks only a barrier’s expected list before letting it proceed.

The link check has two front ends: the build runs it over the HTML it holds in memory, and scripts/check_links.mjs runs it over a tree on disk. Both call the same functions in builder/check.mjs, over the core in builder/link-check.mjs. A change to any of the three can make the two disagree, and a checker that silently checks less reports a clean pass.

After changing builder/link-check.mjs, builder/check.mjs or scripts/check_links.mjs, run the full comparison by hand:

node scripts/check_links_diff.mjs --a script --b fused

It builds everything it compares — the site into the usual trees, a copy under a base path, and the three-page fixture — so no build.bat is needed first. It exits 1 if the two sides disagree in any category, or if a fixture stops finding what it is there to provoke.

Nothing else runs this comparison. build.bat, check.bat and test.bat never call it. CI runs only fixture cases, so a green pull request says only that the two sides agree over the fixtures, not over the pages you will publish. A change meant to alter what the checker finds also moves the counts asserted after every fixture run: FIXTURE_EXPECTED, FIXTURE_BUILT_ONLINE and FIXTURE_BUILT_OFFLINE in check_links_diff.mjs. test/README.md says what each fixture page provokes.

If you changed the harness, or check_links.mjs (the harness’s script side), also run:

node scripts/check_links_diff.mjs --self-test

It runs check_links.mjs’s own regression guards, then compares the script with a deliberately corrupted copy of itself and fails unless the difference is reported.

The link-checker parity fixtures explains what CI runs, and check_links_diff.mjs is the full reference: every case, every side, and what each fixture holds the comparison to.

Adding a pipeline task

1. Decide where the work runs

The first question is whether the task body needs the main thread.

Pick runOnMain when… Pick a worker handler when…
The body mutates state.pages or state.site and other tasks must see the result. The body is pure compute.
The body coordinates filesystem layout (a mkdir race or a sequenced clean). The body reads files but does not coordinate destination layout.
The output is small and downstream tasks consume it via the inputs object. The output is per-page and you want to fan out across cores.
The task should join multiple worker outputs together. The task should run multiple times in parallel (render:i, flush:i).

For per-page CPU compute, prefer a sub-stage inside the existing render handler over a new task: the render fan-out already runs at full parallelism, and a sub-stage avoids an extra postMessage round-trip. The render-worker sub-stage walkthrough below covers that case.

2. Pick the right flags

The scheduling primitives compose. The common patterns:

You want… Set
A regular dependency. expected: ["predName"]
A seed that does not auto-start; runs when a successor needs it. on_demand: true
Per-worker setup that must run on each lane before another task is claimable on that lane. unique_per_worker: true, declared as a perWorkerDeps on the dependent task.
Speculative execution during idle time (e.g. warmup that overlaps the main spine). run_when_idle: true
Must run on the lane that ran a specific predecessor. pinnedTo set by submit() via the SAB pinnedTo array (see dispatch.submit for the pattern).
Per-lane done flags survive serve-mode rebuilds. survives_reset: true (unique_per_worker tasks only).
Higher-number priority claims first when multiple tasks are READY. priority: N
Combine per-lane timings into one Gantt swimlane. consolidate: true

The full reference is in the Scheduler-level concepts section of Pipeline Stages.

3. Write the worker handler (if worker-resident)

Worker handlers live in cpu-worker.mjs. Two edits:

Step 3a. Add an entry to HANDLERS in sab-scheduler.mjs (the IDs are arbitrary integers, just pick the next free one):

// builder/sab-scheduler.mjs
export const HANDLERS = {
  warmInit: 0, renderEnvInit: 1, flush: 2,
  scssLight: 3, scssDark: 4, dot: 5,
  buildInfo: 6, render: 7,
  myHandler: 8,                          // ← new
};

Step 3b. Add the handler function in the handlers object in cpu-worker.mjs:

// builder/cpu-worker.mjs
const handlers = {
  // ... existing handlers ...

  async myHandler(taskIdx) {
    // taskIdx is the SAB slot index for this task; useful when reading
    // per-task payload via _payloadSAB + payloadOffset/payloadLength.
    // For a parameterless task you can ignore it.

    // Workers have access to ctx (srcRoot, destRoot, opts, workerCount),
    // _sharedSAB (from dispatch.broadcastDynamicData), and any module-scope
    // state previous handlers stashed (e.g. _renderEnv from renderEnvInit).
    const { srcRoot } = ctx;
    const result = await someComputation(srcRoot);
    return { result };
  },
};

The handler’s return value comes back to main as { done: taskIdx, output, timing, lane }. The scheduler stores output in results and passes it to your task’s submit().

4. Define the task in TASKS

Edit the TASKS object in tbdocs.mjs:

// builder/tbdocs.mjs
const TASKS = {
  // ... existing tasks ...

  myTask: {
    expected: ["write"],                 // run after the main write pass
    handler: "myHandler",                // worker dispatch (omit + add runOnMain for a main task)
    ganttSection: "Write",               // appears under "Write" in the chart
    submit(out, state) {
      // Merge the worker's output into shared state, or pass it to
      // downstream tasks via state.site / state.<custom>. For a
      // terminal task you can leave this as a no-op.
      state.site.myResult = out.result;
    },
  },
};

For a main-thread task, drop handler and add runOnMain: true plus an execute:

myMainTask: {
  expected: ["renderJoin", "prepDest"],
  runOnMain: true,
  ganttSection: "Write",
  async execute({ renderJoin: _ }, ctx, state) {
    // inputs is { [predName]: predOutput } for every predecessor in expected.
    // ctx carries srcRoot, destRoot, opts, workerCount.
    // state is the SharedState (pages, staticFiles, site, pageByDest, …).
    const manifest = state.pages.map(p => ({ url: p.permalink, title: p.frontmatter.title }));
    if (!ctx.opts.dryRun) {
      const dest = path.join(ctx.destRoot, "pages-manifest.json");
      await writeFileMkdirp(dest, JSON.stringify(manifest));
    }
    return { entries: manifest.length };
  },
  submit() {},
},

Important

If the task writes to disk, check ctx.opts.dryRun and skip the writes when it is true. Every existing task honours the flag.

5. Worked example A: main-thread auxiliary task

A task that emits pages-manifest.json listing every page’s URL and title. It depends on flushJoin (the in-memory page set is complete) and prepDest (the destination tree exists).

// builder/tbdocs.mjs

import { writeFileMkdirp } from "./write.mjs";

const TASKS = {
  // ... existing tasks ...

  pagesManifest: {
    expected: ["flushJoin", "prepDest"],
    runOnMain: true,
    ganttSection: "Write",
    async execute(_, ctx, state) {
      if (ctx.opts.dryRun) return { entries: 0 };
      const manifest = state.pages.map(p => ({
        url:   p.permalink,
        title: p.frontmatter.title ?? null,
      }));
      const dest = path.join(ctx.destRoot, "pages-manifest.json");
      await writeFileMkdirp(dest, JSON.stringify(manifest, null, 2));
      return { entries: manifest.length };
    },
    submit() {},
  },
};

No other edits are needed. The task appears under “Write” in the Gantt chart on the next build.

6. Worked example B: distributed compute

A per-page word count, computed inside the render fan-out (so it scales with cores), with the consolidated result written by a main-thread task. searchData and per-page SEO work the same way.

Step 6a. Compute the word count per page on the worker, inside the existing render handler in cpu-worker.mjs, between templatePhase and the offline rewrite:

// builder/cpu-worker.mjs, inside the render handler

await templatePhase(chunk, env.site, env.initData);

// ── New: per-page word count over the rendered body ──
for (const p of chunk) {
  if (p.renderedContent) {
    p.wordCount = p.renderedContent
      .replace(/<[^>]+>/g, " ")        // strip tags
      .split(/\s+/)
      .filter(Boolean)
      .length;
  }
}

Step 6b. Add wordCount to the per-page delta the handler returns, so it travels back to main:

return {
  pages: chunk.map(p => ({
    destPath:        p.destPath,
    renderedContent: p.renderedContent,
    offlineMisses:   p.offlineMisses,
    wordCount:       p.wordCount,           // ← new
  })),
  searchEntries,
};

Step 6c. Merge the new field into the master Page objects in dispatch.submit. Find the submit(renderOut, state) callback dispatch registers on each render:i and extend it:

// builder/tbdocs.mjs, in dispatch.submit's render:i registration

scheduler.tasks.set(rName, {
  expected: [],
  consolidate: true,
  ganttSection: "Render",
  submit(renderOut, state) {
    for (const r of renderOut.pages) {
      const p = state.pageByDest.get(r.destPath);
      // A miss is a bug, not a condition to tolerate -- see below.
      if (!p) {
        throw new Error(
          `render:${i} returned a page the build does not know: ${r.destPath}`,
        );
      }
      p.renderedContent = r.renderedContent;
      if (r.offlineMisses !== undefined) p.offlineMisses = r.offlineMisses;
      if (r.wordCount !== undefined)     p.wordCount     = r.wordCount;   // ← new
    }
    state.searchChunks[i] = renderOut.searchEntries;
  },
});

Important

Throw on the merge path; never skip. pageByDest is built from the same page list the chunks were sliced from, so a page the callback cannot find means a chunk produced something the build never dispatched. Skipping it drops that page’s renderedContent, and the search index and the PDF book skip a page that has none without complaining, so the page vanishes from search-data.json with no error. Every skip on the chunk-merge path throws; new fan-out code must do the same.

Step 6d. Write a consolidator task that runs after renderJoin:

// builder/tbdocs.mjs, in TASKS

writeWordCounts: {
  expected: ["renderJoin", "prepDest"],
  runOnMain: true,
  ganttSection: "Write",
  async execute(_, ctx, state) {
    if (ctx.opts.dryRun) return { entries: 0 };
    const data = state.pages
      .filter(p => p.wordCount !== undefined)
      .map(p => ({ url: p.permalink, words: p.wordCount }));
    const dest = path.join(ctx.destRoot, "assets/js/word-counts.json");
    await writeFileMkdirp(dest, JSON.stringify(data));
    return { entries: data.length };
  },
  submit() {},
},

The pattern: per-chunk compute on the render workers, merge into the master pages in the render submit(), consolidate in a main-thread task after renderJoin.

7. Verify

build.bat shows the new task in the timing summary line and in the Gantt chart on the Build Info page. A “Write” task that runs during the spine usually means a missing predecessor. Run check.bat and test.bat too: a new task is a change under builder/.

Adding a markdown-it plugin

Background

createMarkdownIt in render.mjs builds the configured markdown-it instance. The plugins are applied in a fixed order: three from npm (markdown-it-attrs, markdown-it-deflist, markdown-it-footnote) interleaved with those defined in render.mjs, and countPlugin from counts.mjs registered last. A new plugin becomes part of that order. Pipeline Stages tabulates the chain in registration order, with what each plugin does.

The in-tree plugins transform the token stream (figurePlugin wraps a lone image in a figure and embeds a diagram’s SVG, headingLevelNormalizePlugin repairs legacy pages that skip from h1 to h3) or provide link, slug and typography helpers. Most close a gap between markdown-it and the kramdown dialect the content was authored against.

The factory is called once on main (the shared site-level SEO instance, via markdownInit), once per render worker (via renderEnvInit), and once by each script that renders outside the build, such as check_code_regions.mjs. A plugin therefore must not keep mutable closure-captured state: each worker has its own module-scope instance.

1. Write the plugin

A markdown-it plugin is a function that receives the md instance and mutates it. Two common shapes follow. The first is shown as it ships, because it is live in the tree and must not be written a second time.

Renderer override — replace the function markdown-it uses to emit one token type. The site’s live example is the table wrapper. createMarkdownIt applies it inline rather than through md.use(), which suits a pair of rules that no other plugin has to be ordered against:

// builder/render.mjs, inside createMarkdownIt

md.renderer.rules.table_open = (tokens, idx, opts, _env, slf) =>
  slf
    .renderToken(tokens, idx, opts)
    .replace(/<table>/, `<div class="table-wrapper" tabindex="0"><table>`);

md.renderer.rules.table_close = (tokens, idx, opts, _env, slf) =>
  `</table></div>` + slf.renderToken(tokens, idx, opts).replace(/<\/table>/, "");

Those six lines hold three decisions:

  • The default renderer runs first and its output is rewritten, rather than the tag being built by hand. renderToken keeps the leading newline the parser emits when the table opens a list item, a definition or a blockquote child; a wrapper concatenated around a literal <table> string loses it.
  • tabindex="0" is an accessibility fix. .table-wrapper has overflow-x: auto, and a scroll container that cannot take focus cannot be scrolled without a pointer (axe’s scrollable-region-focusable rule). The attribute is unconditional because whether a table overflows depends on the viewport, which is unknown at render time. custom/custom.scss gives the focus a visible ring.
  • The wrapper exists because tbdocs renders no Liquid. The vendored _sass keys its table rules on .table-wrapper, so the renderer must produce the div.

Important

Every table the site publishes is already wrapped, and a plugin must not wrap them again. A second table_open override composes with the shipped rule, because the “original” it captures is the rule above. Each table then sits inside two .table-wrapper divs, which the stylesheet makes two nested scroll containers, the outer one with no tabindex. Write the markdown; the wrapper and its focusability are automatic.

Block rule — a new fenced syntax that emits a <div class="callout">. Nothing in the tree registers a md.block.ruler rule today, so this one is unbuilt; the GFM admonitions the pages use are a pre-render text rewrite.

// builder/callout-plugin.mjs

export function calloutPlugin(md) {
  md.block.ruler.before("fence", "callout", (state, startLine, endLine, silent) => {
    const pos = state.bMarks[startLine] + state.tShift[startLine];
    const max = state.eMarks[startLine];
    if (state.src.slice(pos, pos + 3) !== ":::") return false;
    if (silent) return true;

    const label = state.src.slice(pos + 3, max).trim();

    // Locate the closing fence before emitting anything. An unterminated
    // callout runs to the end of the enclosing block, as an unclosed ``` does.
    let closeLine = startLine + 1;
    for (; closeLine < endLine; closeLine++) {
      const p = state.bMarks[closeLine] + state.tShift[closeLine];
      if (state.src.slice(p, state.eMarks[closeLine]).trim() === ":::") break;
    }

    state.push("callout_open", "div", 1).attrSet("class", `callout callout-${label}`);

    // Tokenize the body. Advancing state.line past those lines instead
    // consumes them: the div renders empty and the content is gone, with
    // nothing to report because no rule failed.
    const parentType = state.parentType;
    const lineMax = state.lineMax;
    state.parentType = "callout";
    state.lineMax = closeLine;
    state.md.block.tokenize(state, startLine + 1, closeLine);
    state.parentType = parentType;
    state.lineMax = lineMax;

    state.push("callout_close", "div", -1);
    state.line = closeLine + 1;
    return true;
  }, { alt: ["paragraph", "blockquote", "list"] });
}

The alt list names the parse modes the rule is also consulted in. Without "paragraph", a ::: line after a line of prose is absorbed into that paragraph as literal text instead of opening a callout.

For the full rule API see the markdown-it documentation and the in-tree plugins in render.mjs.

2. Register in createMarkdownIt

A plugin in its own module is registered in two steps. A renderer override written inline, like the table rules above, needs neither.

Add an import at the top of render.mjs:

import { calloutPlugin } from "./callout-plugin.mjs";

Find createMarkdownIt and add md.use(calloutPlugin) in the plugin chain. Order matters: place the new plugin after any plugin it depends on and before any plugin that could interfere with its token types. Registration order decides execution order only for a rule added with md.core.ruler.push(). .before(name) and .after(name) insert at a named position whenever the plugin is registered, which is how headingLevelNormalizePlugin runs ahead of headerIdPlugin although it is registered after it.

export function createMarkdownIt(ctx) {
  const md = new MarkdownIt({ /* ... */ });
  // ... existing npm plugins ...
  // ... existing in-tree plugins ...
  md.use(calloutPlugin);
  return md;
}

3. Verify

Run build.bat and open an affected page. serve.bat will not show a plugin change: its worker pool is persistent, so a running preview keeps the old plugin (see the NOTE at the top of this page). Ctrl+C and re-run it first. A plugin that traverses the full token stream on every page runs N+1 times per build (main thread plus N workers), so check the per-task render timing in the summary or the Gantt chart for a spike.

Then write a unit test in test/render.test.mjs, which builds the site’s own createMarkdownIt and renders small inputs, one plugin per describe block. The build compares whole pages, so a plugin that is wrong only on input no page holds passes it; the test pins that input. Run it alone with node --test test/render.test.mjs; test.bat and both CI workflows run it too.

Then run test.bat, which judges a render.mjs edit. Two of its gates read this file: check_regex_safety.mjs refuses a regex that can backtrack exponentially, and check_code_regions.mjs refuses a pre-render rewrite that alters the contents of a code fence or code span.

A pattern assembled from string constants is still checked by the regex gate, when the source decides what the constants are. If it does not (a function parameter, a let built in a loop), node scripts/check_regex_safety.mjs --census names the pattern and you must reason about it yourself. See check_regex_safety.mjs for the shapes it resolves.

If the plugin emits markup the site has not carried before — a new wrapper element, a widget, a figure, a control — register a construct family for it in scripts/pick_a11y_sample.mjs in the same change. The accessibility scan audits fourteen sample pages out of ~1,160, and no axe rule keyed on a construct runs unless a sample page carries it. Without a family the gate keeps reporting a clean pass while covering less. node scripts/pick_a11y_sample.mjs --census shows the existing families and their pages; --check names the gaps and the cheapest page that closes each. A new task or sub-stage that changes the emitted HTML, or a template change, carries the same obligation.

Adding a render-worker sub-stage

For per-page CPU compute, a sub-stage in the existing render handler is the most direct path: it skips the per-task overhead and rides the same fan-out.

Where to plug in

The handler in cpu-worker.mjs runs five sub-stages in order:

chunk = parseFromPayloadSAB(taskIdx)
await renderPhase(chunk, env.site)              // markdown-it body render
computeChunkSeo(chunk, ...)                     // per-page SEO fields
await templatePhase(chunk, env.site, ...)       // layout wrap
if (env.offlineBase) { … deriveOfflinePage per page … }
searchEntries = deriveSearchEntries(chunk, env.site)
return { pages: chunk.map(…), searchEntries }

A new transformation goes between two existing sub-stages, depending on what it reads and writes:

Insertion point Use when…
Between renderPhase and computeChunkSeo The transformation needs renderedContent but not seoTitle.
Between computeChunkSeo and templatePhase The transformation needs SEO fields but should run before the layout wrap.
Between templatePhase and the offline pass The transformation needs the final html (e.g. extracting outbound links from the wrapped page).
After the offline pass The transformation needs both html and offlineHtml.
Inside the chunk-mapping return Per-page output to thread back to main (extend the pages.map(…) projection).

Worked example: per-page outbound link count

After templatePhase has run, count outbound links per page and emit them in the delta:

// builder/cpu-worker.mjs, inside the render handler

await templatePhase(chunk, env.site, env.initData);

for (const p of chunk) {
  if (p.html) {
    const matches = p.html.match(/href="https?:\/\//g);
    p.outboundLinks = matches ? matches.length : 0;
  }
}

// ... offline pass ...

return {
  pages: chunk.map(p => ({
    destPath:        p.destPath,
    renderedContent: p.renderedContent,
    offlineMisses:   p.offlineMisses,
    outboundLinks:   p.outboundLinks,    // ← new
  })),
  searchEntries,
};

Then extend dispatch.submit’s render:i callback to merge the new field, exactly as the worked example B above does.

Note

Anything you mutate on a worker’s chunk page must travel back through the delta to be visible on main. The worker’s chunk is a structured-clone copy, and the pages.map(…) projection at the end of the render handler is the only channel back.

Adding a verification gate

A gate runs after the build and decides whether what the build produced is acceptable. Two batch wrappers run them: check.bat and test.bat. The entries on Tools and Scripts are the authoritative list of which gate each wrapper runs, in order. Give a new gate its own entry on that page in the same change.

First decide whether it is a gate at all. The link and integrity check runs inside the build because both trees’ final HTML is already decoded in worker memory when flush:i runs; checking it on disk would mean writing ~270 MB out to read it back. A check is a gate if it needs something the build does not hold: a browser, a real font, a second implementation to compare against, a tree from an earlier run. Otherwise it is a pipeline task, and the walkthroughs above apply.

Which wrapper it goes in

A gate belongs in test.bat rather than check.bat if it would still mean something with no documentation in the tree.

The rule is about what the gate interrogates, not what it opens. check_axe_patch_equiv.mjs loads a built page and needs Chromium, which makes it look like a check.bat gate. The page is only a document for the probe to run inside; the gate tests the vendored axe source patch. Against an empty docs/ it would still be worth running, so it goes in test.bat.

Two applications of the rule:

  • A regression test for a build-time rewrite that has stopped firing interrogates the rewrite, not the corpus, so it goes in test.bat. check_code_regions.mjs has this shape, and its admonition probes are the pattern to copy: each asserts a defect that must stay fixed, in the normal run rather than behind a flag.
  • A gate that compares this build’s output against an earlier build’s interrogates the built tree and says nothing about an empty docs/, so it goes in check.bat.

An edit confined to docs/ therefore usually needs check.bat only. Two test.bat gates can still go red on a content edit: check_code_regions.mjs tokenises every markdown file under docs/, and check_gate_lists.mjs reads README.md and every page under docs/Documentation/. Both CI workflows run every gate from both wrappers, each as its own step, so the choice of wrapper changes only what a local edit costs.

Conventions

The command line goes through lib/cli.mjs. parseCli(argv, { options, positionals, stopAt }) is a strict parse over node:util’s parseArgs. Each of these is a CliError: an unknown option, a boolean given a value, a value flag without its value or with an empty one, and a positional the tool does not take. options is keyed by long name ({ type: "string" | "boolean", short, multiple, default, empty }), positionals is a count or { min, max }, and the result’s values is keyed by the camelCase of each name. check_tree_fresh.mjs and check_dot_fit.mjs show the shape a new tool follows:

  • Help first. Declare help: { type: "boolean", short: "h" } and pass stopAt: ["help"], so nothing after --help is read or checked. Answer it straight after the parse with printHelpAndExit(USAGE), which prints to standard output and exits 0.
  • Refusals exit 2. Wrap the parse in withUsageError(() => parseCli(...)), which prints a CliError’s message to standard error and exits 2.
  • Check values straight after the parse, before the tool does any work: numberOption, choiceOption, regexOption, urlOption and dateOption read one value and throw a CliError that names the option, and refuseTogether(values, names) refuses options that exclude each other. Do not leave a value to fail later as NaN.
  • Install exitOnCrash() at the entry point, before anything else runs (see the exit codes below).
  • Stop with die(code, message) when the tool cannot go on: it prints the message to standard error and exits with the code. A tool that must put something back first, as sweep_attributes.mjs does the registry, defines its own.
  • End the usage text with an Exit codes: block, one <code> <meaning> line per code, a long meaning wrapped under itself.

scripts/check_cli.mjs holds every tool to this; its cases are in scripts/lib/cli-cases.mjs. Add a new tool to that file’s HELP_TOOLS list (the tool must answer --help and -h with exit 0 and end its usage with exactly one Exit codes: table) and to its REFUSALS list (a case for an unknown flag and for an empty value). The gate throws on start if the two lists disagree. A tool that checks a value after the parse gets a case of its own in the file’s bad-value list.

Exit codes. A new gate in either wrapper uses three values:

Code Means
0 The checked thing is fine.
1 The checked thing failed. This is the finding.
2 The harness or the environment failed — a command line the tool refuses (an unknown flag, a flag without its value or with an empty one, an unexpected argument, a value it cannot use), an absent tree, a crash. Nothing was checked.

Separating 1 from 2 stops a broken gate reading as a clean site, and it must hold at the top level too. Node’s own exit for an uncaught exception is 1, which reads as a finding. So a new tool calls exitOnCrash() from lib/cli.mjs before it does anything else. It makes an uncaught exception, an unawaited rejection or a rejected top-level await print the error and exit 2. It installs when called, never on import, so a module that can also be imported calls it only when it is the entry point. exitOnCrash(cleanup) runs cleanup(err) first for a tool that holds something the exit would lose. A script with a main() can instead end with main().catch((err) => { console.error(err); process.exit(2); }), as check_a11y.mjs does.

The rule applies to every tool under builder/, scripts/, book/, eval/ and wisdom/: 1 for a finding, 2 for anything that stops the tool doing its job. A tool with more outcomes to tell apart adds codes above 2 — tbbuild uses 3 and 4, tbrun 3 to 6, addin_test and wisdom use 3 — and impexp keeps a table of its own, shared with its Python edition. Each tool ends its --help text with an Exit codes: block, one line per code, and that block is the source of truth: Tools and Scripts states the same codes in each tool’s entry, and a change to one goes into the other.

Say what a pass covered. A gate silent on success does not show that it examined anything, and a gate that has quietly stopped working looks the same. So report coverage: check_dot_fit.mjs gives the diagram count, pick_a11y_sample.mjs --check the sample size and construct families in use, check_a11y.mjs the page × theme × viewport product it audited.

Name the artifact and the remedy. check_dot_fit.mjs names the failing diagram, then builder/dot-metrics.mjs and an @hpcc-js/wasm-graphviz bump as the usual cause, then build.bat as the fix. check_tree_fresh.mjs names the source file that is newer than the tree and says to run build.bat. pick_a11y_sample.mjs --check names the uncovered construct and the cheapest page that would cover it.

Resolve paths from the repository root in lib/repo-paths.mjs. Import REPO_ROOT, or DOCS_DIR for docs/, rather than deriving either from import.meta.url, path.resolve(__dirname, ...) or the working directory; the gate then works from any directory. A path given on the command line still resolves against the working directory. check_publish_policy.mjs is the one gate whose default is working-directory-relative (--src defaults to docs), which is why the batch wrappers pushd to the repository root first.

Find markdown through lib/markdown-files.mjs. markdownFiles(root) lists the markdown under a tree and never enters the build’s output trees (_site*, _serve*, _pdf*), which a running serve.bat rewrites. A walk that must read docs/ some other way uses isOutputTree(name) from the same module, as check_tree_fresh.mjs does. lib/markdown.mjs decides what is code inside a markdown file (see When test.bat fails in check_code_regions).

Be explicit about what may already have run. Both wrappers stop at the first failure, so a gate’s position decides what it can assume. check_tree_fresh.mjs runs first in check.bat, so every later gate there may assume _site-offline/ is current. test.bat has no freshness gate; its one gate that opens a built page, check_axe_patch_equiv.mjs, never reads that page’s DOM. Neither wrapper may assume the build’s own link check passed: a link failure sets the build’s exit code without aborting the build, so a tree that failed it is still on disk and fresh.

Register it everywhere a gate is listed. A node --test test/<name>.test.mjs file is a gate too, registered the same way, named by its path.

  • The wrapper it belongs in (check.bat or test.bat), at the position its assumptions need, with a comment saying what the gate protects against. Each step is followed by if errorlevel 1 goto :fail.
  • .github/actions/run-gates/action.yml, the one list of gates both CI workflows run, so a gate goes into CI once. Give it a step named after the gate, with the same arguments and in the same order as the wrapper.
  • Tools and Scripts: the gate’s own entry under CLI tools (with its {: #check-... } anchor and its exit codes), a line in the wrapper’s numbered list in check.bat or test.bat, the step count that section states, and the POSIX command block under the list.
  • WIP.md, the maintainers’ file: the gate’s row in its table of gates, and the wrapper’s bullet under Build / preview.

A gate left out of CI is listed with its reason in check_ci_workflows.mjs’s allowed differences: check_tree_fresh.mjs is left out because CI builds in the same job and cannot have a stale tree.

Two checks enforce the registration; neither reads the POSIX command blocks or WIP.md, so take care with those. check_gate_lists.mjs compares both wrappers against Tools.md’s two numbered lists — membership, order, and the step count each section states — so a gate added without the entry fails test.bat. It also fails on a gate count stated in prose in README.md or any page under docs/Documentation/: do not write one. check_ci_workflows.mjs checks that both workflows, read through the action, run every wrapper gate with the same arguments and in the same order, apart from its list of allowed differences.

It must be able to fail

A gate that silently checks less reports exactly what a clean site reports. A green run is not evidence about the site until something independent shows the gate can still go red. A hand-picked accessibility sample can report a clean pass while pages outside it carry violations, and an optimisation that makes axe see fewer nodes can report a pass too.

So a new gate needs a second assertion of the opposite sign. Four shapes in the repository to copy:

  • Named probes on both sides. check_publish_policy.mjs asserts that thirteen file types stay refused and that six stay published, because a policy that refuses everything also reports a clean sweep.
  • A deliberately corrupted side. check_links_diff.mjs --self-test diffs the checker against a mutated copy of itself and fails unless the difference is reported.
  • A fixture that provokes one fault of each kind, with the count asserted afterwards. The real site is clean, so without one every category compares empty against empty, and a category that has stopped being checked looks identical to a category with nothing to find.
  • An A/A control. Running check_a11y_fingerprint.mjs with the same scheme on both sides shows whether the harness is stable, before any A/B result from it is believed.

A self-test made of named probes, such as check_page_baseline.mjs, records them with createProbes(tool, onFailure) from scripts/lib/gate-probes.mjs. check(name, ok, detail) records one probe. report() prints a line for each, with a failed probe’s detail under it, then a summary, and returns the exit code (0, or 1 if any probe failed). The optional onFailure text ends the summary of a failed run, to say what a failed probe means. A drift guard whose baseline is a JSON file under builder/ is probed against a scratch copy, through baselineFixture(name) from the same module, so no probe touches the committed baseline.

When the gate cannot assert its own correctness from the inside, put the proof in a sibling script and name it in the gate’s header comment.

Adding a build-time count

A count name is a number the build works out and substitutes into prose, so a sentence stating it cannot go stale. Readers write {{tbdocs:<name>}}; this is the other end.

A name must be a derivation over build state, never a stored constant. A registry holding pages: 908 only moves the stale figure from a page a contributor reads into a module nobody opens. A number that cannot be derived gets no name and stays a digit.

Two shapes qualify. Most names count things discover already found: pages under a prefix, static files, packages. attributeAnchors and enumerations scan one page’s rawContent for the pattern that makes an entry. That is legitimate because the page is the list, but changing that page’s list formatting moves the number. Both scan only the lines outside code, which proseLines in counts.mjs gives them, so an example of the pattern in a fence is not counted; a new scan should do the same.

  1. Add the entry. A function beside the others in counts.mjs, returning a number from the state passed to deriveCounts, and a line in the object it returns. That object’s keys are the names a page may use, so nothing else registers it.
  2. Add the row to the name table on Authoring Pages. An undocumented name is one nobody will use.
  3. Use it, or do not. A name with no call sites is fine, and cheaper to add now than to retrofit when a figure goes stale.

Verify by building. Two checks guard the feature, and a green build exercises both:

  • validateCountNames runs on main before any worker renders and rejects an unknown name, naming the file, the line and the nearest match.
  • findSurvivingPlaceholder scans the rendered HTML for a placeholder that got through outside <code> and <pre>. Source validation cannot catch this case: a raw HTML block is one opaque token, so a placeholder inside one has a valid name and would publish verbatim.

Then read the built page and check the number against the thing it counts. Nothing compares a derivation against reality: a name that returns the wrong number is as green as one that returns the right one.

What a change obliges in the documentation

Nothing in Verify or Testing compares a documentation page against the task graph it describes. A contributor who follows this page exactly can ship a correct task with a green build.bat, check.bat and test.bat, and a pipeline reference that describes a build which no longer exists. Nothing will report it.

Four files under docs/ model the task graph: Pipeline-Stages.md, Builder.md, this page, and scheduler-dag.dot. None is generated from TASKS; every surface below is maintained by hand.

Pipeline-Stages.md

Three surfaces, and the second is the one that gets missed.

The task’s own section. One ### heading per task, under the numbered section matching its Gantt section. It opens with a fenced block giving its expected array — and its execute() return shape where that matters — followed by prose for what submit() merges into SharedState. A per-lane start-up task, which the chart draws in the worker rows rather than in a section, goes under Render, as warmInit and renderEnvInit do. A new predecessor, a new field on the returned delta, a new key written to state: each is an edit here.

The reverse edges. A task’s position in the graph is stated in its own section and again in the expected line of every task that depends on it. Add myTask to writeAux.expected in the code and writeAux’s section goes on printing the old list. Pipeline-Stages.md names renderJoin on many lines; three of them are the expected declarations of searchData, symbolIndex and writePdf. Grep the task name across the whole file rather than editing only its own section.

The module export table. Module export tables gives the signature of the function each task calls, return shape included. A return shape is thus documented twice on one page, and the two copies go stale independently.

Builder.md

Four surfaces, none derived from TASKS:

  • The static-task count in Task DAG by section — the literal number in “The pipeline has N named static tasks plus 2N dynamic ones”.
  • The section-membership lists immediately below it, one line per Gantt section.
  • The per-task bullet under the matching ### Seeds / ### Spine / ### Render / ### Write subsection, plus the task’s row in What runs where.
  • The two ASCII DAG sketches, one under Spine and one under Render.

These four are not kept in step with each other: a task can be named in a membership list and absent from both the per-task bullets and the “What runs where” table. Do not treat a neighbouring task’s coverage as proof of the full set; take the union of two, as the completeness test below does.

scheduler-dag.dot

docs/assets/images/dot/scheduler-dag.dot is the DAG rendered on Builder.md, written by hand node by node and edge by edge, against two-letter aliases declared at the top of the file. A new task needs a node (fill #fef7e0 with stroke #f9ab00 for main-thread, #e8f0fe with #4285f4 for worker), one outgoing edge per entry in its expected, and one new incoming edge on every task that now depends on it.

check_dot_fit.mjs does not guard this. It renders each committed .svg in a browser and asks whether every label sits inside the box Graphviz drew for it, a geometry question. A diagram whose edges contradict TASKS passes that check and the build. Read the diagram against TASKS by eye.

What not to update

A grep for a task name also hits builder/, which holds design documents at three life stages. builder/README.md’s Documentation list says which is which, and marks two classes as not maintained reference:

  • REVIEW-*.md and PLAN-REVIEW-*.md — frozen audit snapshots of a commit range. Editing one to match a later change destroys its value.
  • PLAN-scheduler.md — the superseded push-based scheduler design, kept for its reasoning. PLAN-sab-pull-scheduler.md is the current one, and it does need updating.

Read the README entry for a builder/ file before editing it.

The completeness test

No gate fails when a task’s documentation goes stale, so run a grep before committing:

grep -rn "myTask" docs/ builder/

Triage every hit by the file it is in. Under docs/ the legitimate homes are the four above; a hit anywhere else is a page that has grown a dependency on the task graph, or an ordinary English word (dispatch and render collide that way). For writePdf, expect one more diagram: pdf-render-pipeline.dot names it in a cluster label without modelling the graph.

The inverse is more useful, and worth running before writing anything. Grep two existing tasks you did not touch, and take the union of the files they appear in:

grep -rl "searchData" docs/ builder/
grep -rl "writePdf"   docs/ builder/

One task is not a template, because an existing task can itself be missing from a surface. Where two agree is the obligatory set; where they disagree, read both before deciding which is wrong.

Testing

Five commands cover the loop:

  1. build.bat — full pipeline, including the link and integrity check over both trees. A clean exit and a sensible Gantt placement is the bar.
  2. serve.bat — live-reload dev server for visual checks. Ctrl+C and restart after handler-code or task-graph changes. Check both themes if the change touches anything visible.
  3. check.bat — the gates that read the built site. Tools and Scripts lists them in order.
  4. test.bat — the gates that test the toolchain, listed at Tools and Scripts. Every change under builder/, scripts/, lib/, book/, eval/, wisdom/, indexer/ or test/, or to a wrapper or workflow, needs it — that is every change this page describes. Most content edits do not, except that check_code_regions.mjs sweeps every markdown file under docs/ and check_gate_lists.mjs reads README.md and every page under docs/Documentation/.
  5. book.bat — re-renders the PDF if your change affects _site-pdf/ or any chapter body.

A clean run of all five is the bar for “ready to commit”.

Three gates are the ones a builder change is most likely to trip:

  • pick_a11y_sample.mjs --check, in check.bat, fails when a construct family the site uses is covered by no page in the sample, normally because a build added or moved pages and the cheapest page for some family is no longer in the list. Add the page it names; do not widen the sample by hand. It cannot report a genuinely new construct. The gate iterates the registered FAMILIES only, so markup no family describes produces silence, and the axe rule keyed on that construct then runs nowhere. Adding a family is the fix; Site Checks shows the shape of one.
  • check_publish_policy.mjs, in test.bat, fails when a new emitted file type is not on the allowlist in builder/publish-policy.mjs. Add it to BUILD_EXTENSIONS, a separate set from SOURCE_EXTENSIONS so that allowing a generated type does not also allow a stray one a contributor drops into docs/.
  • check_code_regions.mjs, also in test.bat, fails when a new pre-render rewrite alters the contents of a code fence, an indented code block or a code span. For a fence or a span, move the rewrite inside applyPreRenderRewrites in render.mjs, between maskCode and its restore, rather than widening the mask; an indented block is not masked at all. It also fails when a rewrite over rendered HTML changes a <pre> or <code> written as raw HTML. When test.bat fails in check_code_regions has each case.

Note

Both check.bat and test.bat want build.bat to have run first. check.bat reads the built tree throughout, and check_tree_fresh.mjs refuses one older than its sources. test.bat needs a built tree only for its last gate, check_axe_patch_equiv.mjs, and does not care how old it is.

See Also

  • Pipeline Stages – full data model, per-task interface reference, per-module export tables, and the markdown-it plugin chain in registration order.
  • Project styling – where a CSS rule goes, the light/dark two-compilation model, and the specificity trap that makes a dark-mode override silently do nothing.
  • tbdocs Builder – architectural tour and design rationale.
  • Tools and Scripts – every gate check.bat and test.bat run, and what each one is protecting.
  • Building and Deployment – the day-to-day build workflow for content contributors.