ask-kit’s tests are plain Node scripts that exit non-zero when an assertion fails. They use
node:assert/strict and no test framework, they need no model and no network, and they run in
about two seconds.
Source: package.json (scripts),
test/ and
.github/workflows/ci.yml.
Run them
You need Node 22.13 or newer. From the repository root:
npm ci
npm test # the four checks CI runs
npm run test:machine # this machine's real transcripts; not run in CI
Each check prints ok when it passes. npm test runs these in order and stops at the first
failure:
node test/fixtures.mjs && node test/setup.mjs && node test/repl.mjs && node test/bundle.mjs
To run one check, run its file: node test/setup.mjs.
What each file covers
| File | Covers |
|---|---|
test/fixtures.mjs |
The transcript adapters, against made-up stores. For each of Claude Code, Codex, Copilot CLI, Gemini CLI and opencode: the session is found, and its cwd, project and title are read (and model, for opencode); events come out in the right roles and order, with i equal to position; every tool call has exactly one result with the same toolId, after it, with ok set correctly; usage matches the file’s totals (Claude Code counts a message split over two lines once; Codex takes the last running total). Then makeHistory over all five: the investigator’s own session (working directory ending /warm/cwd) is left out, even by id; filters by source and project; grep needs all words in one event, takes a RegExp, respects roles, searches every source including the ones with no file pre-filter; id prefixes resolve, and an ambiguous or unknown one throws. |
test/setup.mjs |
Setup without touching the machine. With a made-up detection result (Codex and Copilot present, no API key, Ollama running) it checks the question ids and suggested answers, that embedding models are not offered as investigators, backendName, that an API choice with no key throws, and plan(). It then runs apply() twice into throwaway folders: the config is written without the pasted key, keys.env has the key and (not on Windows) mode 0600, there is exactly one [mcp_servers.ask] block in Codex’s config.toml with tool_timeout_sec = 600, the user’s other servers are kept, and Copilot’s mcp-config.json has the ask server. |
test/repl.mjs |
The server over MCP with no model. The investigator face offers exactly help, load, repl and reset; help lists the sessions scripts; a cell runs a script and a variable survives into the next cell; a restarted server restores it from the snapshot (restored ... r); the calling face offers only ask. |
test/bundle.mjs |
The bundle installer against throwaway agent homes. Every agent present gets its skills, tuner agent and rule in its own format (Copilot drops the model line; opencode gets mode: subagent; Copilot’s rule has its front matter); an absent agent (Gemini here) gets nothing; a reinstall changes nothing and rule blocks are replaced in place, not added twice; a file you edited, or one that was never ask-kit’s, is kept; uninstall removes only what ask-kit wrote and leaves your own text in AGENTS.md; the guide is written with this machine’s paths. |
test/adapters.mjs |
The adapters against this machine’s real transcripts (test:machine). For every store that exists: it lists sessions, reads the newest sessions’ events and usage, and runs one search. It fails when a store exists but yields no sessions, or none of the five newest has a user event. It prints skip and exits 0 when the machine has no store. |
What the tests do not cover: src/ask.mjs (the four backends and the loop), src/router.mjs,
src/report.mjs, src/gallery.mjs, bin/ask-kit.mjs and the domains’ scripts and repo.mjs.
Check those by running the real thing: ask-kit ask, the benchmark, and
ask-kit report.
How the tests stay off your machine
Every test builds a temporary folder with fs.mkdtempSync(path.join(os.tmpdir(), ...)), sets the
environment variables that move ask-kit’s and each agent’s folders into it, and only then imports
the code, which reads those variables when it loads.
| Variable set by the tests | Moves |
|---|---|
ASK_KIT_CONFIG, ASK_KIT_STATE |
ask-kit’s config and state folders |
CLAUDE_CONFIG_DIR, CODEX_HOME, COPILOT_HOME, GEMINI_CLI_HOME, XDG_CONFIG_HOME |
each agent’s folders (bundle.mjs, setup.mjs, fixtures.mjs) |
OPENCODE_DB |
opencode’s database (fixtures.mjs) |
Keep this order in a new test: set the variables, then await import(...). A static import at
the top would read the real folders. Files and environment variables
lists every variable.
test/repl.mjs starts real child servers, with ASK_KIT_CONFIG and ASK_KIT_STATE in their
environment and a config whose only backend is an Ollama entry that is never called.
Fixtures
The fixtures are made-up transcripts in
test/fixtures/, one folder per
agent, in the agent’s own layout:
| Path | Format |
|---|---|
claude-code/projects/-work-shop-api/cc-0001.jsonl, cc-warm.jsonl |
Claude Code JSONL. cc-warm is an investigator session, which the sessions domain must leave out. |
codex/sessions/2026/11/17/rollout-...-cx-0001.jsonl |
Codex rollout. |
copilot-cli/session-state/cp-0001/events.jsonl |
Copilot CLI events. |
gemini-cli/tmp/<hash>/chats/session-...-gm0001.json |
Gemini CLI, one JSON document. |
There is no opencode file: its SQLite database is built in test/fixtures.mjs with node:sqlite,
with the columns the adapter reads, one archived session that must be skipped and one that must be
listed.
The fixtures hold two placeholders. __CWD__ becomes a real path on the system running the test
(<tmp>/work/shop-api), JSON-escaped, so a Windows path works. __WARM__ becomes a working
directory ending /warm/cwd, which is how the sessions domain recognises its own investigators.
The adapters read the working directory as it is written, so the test copies the fixtures to the
temporary folder with the replacement applied.
Rules for fixtures (CONTRIBUTING):
- Names are made up. New ones come from Terry Pratchett’s Discworld (
Sam Vimes,ankh-morpork/watch,WATCH-12); the existing ones useshop-api. Hosts areexample.comor.invalid. - A real transcript never enters the repository. Write a synthetic file with the same structure.
Write a test
- Put the data under
test/fixtures/if it is a file, or build it in the test, as opencode’s database is. - Follow the pattern of the nearest test: temporary folder, environment variables, then dynamic imports.
- Assert the specific values the code must produce, as
test/fixtures.mjsdoes with roles, citations and usage totals. A test that only checks “no error” would pass on an empty result. - If it needs a new file, add it to
package.json’stestscript and toci.yml. They list the four checks separately. - Run it on its own, then
npm test.
A change to what a new agent’s adapter reads comes with a fixture and a block in test/fixtures.mjs
(Support a new coding agent). A change to a registration comes with an
assertion in test/setup.mjs. A change to bundle/ comes with one in test/bundle.mjs.
CI
.github/workflows/ci.yml
runs on every push and pull request:
| Setting | Value |
|---|---|
| Systems | ubuntu-latest, macos-latest, windows-latest |
| Node | 22.x and 24.x |
| Matrix | six jobs; fail-fast: false, so one failing job does not stop the others |
| Steps | actions/checkout@v4, actions/setup-node@v4 with the npm cache, npm ci, then node test/fixtures.mjs, node test/setup.mjs, node test/repl.mjs and node test/bundle.mjs |
The bundle step runs only when test/bundle.mjs exists. CI does not run test:machine, because
it needs real transcripts, and does not lint or type-check.
Failures that appear on one system only are usually paths (backslashes in JSON, a temporary folder
behind a symlink: the tests call fs.realpathSync on it), file modes (Windows has no POSIX modes,
and setup.mjs skips that check there) or shims (claude, codex and copilot are .cmd files
on Windows, handled in src/platform.mjs).
After you change a dependency, regenerate the notices:
node scripts/notices.mjs > THIRD-PARTY-NOTICES.md.