Version
0.1.0
Status
No release yet
Licence
MIT
Language
JavaScript (Node.js)

Why I built it

On the morning of the 17th I asked Claude about the new code mode in Claude Code. It tickled a part of my brain, and after a benchmark or two it made me wonder whether we were missing the forest for the trees in our own MCP for mike. I started digging and found Cloudflare’s Code Mode. It turns an MCP server’s tools into a TypeScript API, and the model writes code against that API, which runs in a sandbox. A model can do more by writing code than by calling tools one at a time. What I wanted to add was a model behind that endpoint as well, and ask-kit went its own way from there.

When I looked at how code mode could save me context, it became apparent that the cost moves into the code the model writes up front. Before, it would write a script, get the result, send it off, get a response, write the next script, and so on. With code mode it writes all the steps it might need and runs that set of JS in one go. The intermediate steps and results stay out of the context, and only the end result comes back. It is efficient, but it still leaves a lot on the floor: all the code it wrote still lands in the session, and the session didn’t need that to answer what I asked. From there I thought… what if it didn’t need that?

What bothered me was where the work goes. My main session is the expensive one, and it’s the one I want thinking: deciding what to build, writing the code, holding the plan. But before it can do any of that it has to find things out. Where is this set, what did we decide last week, which service is failing, what does the log say. Every search result and every file it reads on the way lands in its context and stays there for the rest of the session. A lot of the main context ends up being grunt work it has already finished with.

mike made the problem easy to see, because it records every request. A session that wanted to know something would call a tool, read the result, call the next, and the investigation cost more than the answer it came back with.

The experiment

Taking mike’s MCP as the example, I thought, let’s put the whole MCP behind one MCP that only exposes an “ask” tool. All the tooling sits behind that one ask, with an LLM behind the endpoint to work out what was asked. Done smartly, that LLM is a cheap, fast model, and the call goes to a session that already knows how to work the MCP behind the scenes. It writes the JS, and it has a set of ready-made scripts and tooling it can extend. Done well, the same setup also takes follow-up questions: the code has already run and its results are still loaded, so a follow-up uses what you’ve already generated.

So I built mike-ask as a proof of concept on top of mike. It is one tool, ask, with a little model inside it. The main session asks a question in plain words. The small model writes small programs for the task, runs them in a REPL sandbox, and hands back a short answer with the evidence it relied on.

So… is it worth it?

Well, here’s the cool thing. mike carries every session I’ve had since I started using it: the requests, the responses, how many tokens were spent, what was done in each session. And mike-ask keeps its own journal of every ask it answers, so I can see exactly what each one cost.

In real use on 17 and 18 November, 25 of my sessions called mike-ask 105 times. These numbers come from its journal, with its benchmark runs left out.

Per ask Median 10th percentile 90th percentile
Tokens the investigator worked through 125,025 7,495 359,542
Tokens returned to the session that asked 445 43 1,117
Programs it ran (cells) 10 0 20
Time 36 s 10 s 75 s
Cost $0.011 $0.0015 $0.033

103 of the 105 asks came back with an answer, and 17 of them were follow-ups to an earlier ask. All 105 together cost $2.11. To compare that with a session digging by itself, ask-kit’s benchmark put 21 real questions to ask and to Sonnet 5.5 reading the files itself. “Measuring it” below says how it was run.

First run ask (glm-5.3-flash) Sonnet digs itself
Score, out of 21 20 (21 on the second run) 21
Context added to the calling agent, per question (median) 391 tokens on sessions, 333 on code 10,000 on sessions, 4,800 on code
Context added over all 21 8.3k 225k
Cost per question, as run $0.002–0.008 $0.06–0.15
Median time, question names a date 35 s 25 s
Median time, question names nothing 111 s 172 s
Median time, code 12 s 11 s
Follow-up with continue: true 3 s, $0.0006

The context added to the calling agent is where the two differ. Per question, ask added a median of 391 tokens on the session questions and 333 on the code questions, and Sonnet added 10,000 and 4,800. Over all 21 questions that came to 8.3k with ask and 225k with Sonnet.

Tuning it like a dark city

There are some things I built into it from the get-go. It monitors itself so it can get better: it needs to know what was asked and how to make that work faster and more efficiently. It also needed a warm session to start new work from. A technique I started using in mike and its fleets is to “warm up” a seat. With Claude and a lot of other models, you pay the startup cost every time you start a session. Pay it once in a warm session, and new work can be forked from that seat: the forks share its context and prompt cache up to the point where they diverge, and pay only for their own work from there. New sessions start faster because they’re already loaded with the right context. Think of a robot with its paperclip instructions pre-loaded.

Okay cool story bro, what now?

Now the obvious question: could it work without mike? Most people don’t run a proxy that records everything their agents do. I know, that’s a me problem. But the idea never needed mike. It needs something to investigate and a cheap model to do it. mike-ask proved it works, and ask-kit is the general version: anyone can install it without mike and point it at whatever their agents keep investigating. Everything below is about ask-kit.

What it does

Your agent gets one tool, ask. It takes a question in plain words and returns a short answer with the evidence it rests on. Behind it sit investigators, one per domain. An investigator is a cheap model with a JavaScript REPL. For each step of the question it writes a small program and runs it there: loops over files, joins one search with another, counts, compares, parses, filters, and prints only the result. The programs are written fresh for each question. A domain gives them a way in to the data, instructions on what counts as evidence, and scripts for the questions people keep asking.

It comes with two domains to start: code, the repository you’re in, answered with path:line citations, and sessions, the transcripts Claude Code, Codex, GitHub Copilot CLI, Gemini CLI and opencode already write. The screen above is one real answer from the benchmark, about a Codex session from two days earlier: 35 seconds, less than a tenth of a cent, and the turns it rests on.

One ask. The agent sends a question and gets back the answer; the reading stays with the investigator.

My mike-ask also answers about the machines (is a service failing, what do its logs say) and about Claude Code’s own documentation, all behind its one ask. The ask-kit-domain skill writes a domain with you for a database, a service, a docs folder or a pile of logs, and the guide shows the shape.

A domain you put on GitHub with the topic ask-kit-domain and a RECIPE.md another agent could rebuild it from shows in ask-kit gallery, and others can add it with one line. ask-kit add shows you the files before it installs anything, because a domain is code that runs with your rights.

How I built it

ask-kit is JavaScript on Node. It is an MCP server, so any agent that takes MCP servers can call it. The parts are the ones mike-ask is made of, and most of the work was taking out the places where mike-ask leaned on mike.

From mike-ask to ask-kit

mike-ask answers from what mike records. ask-kit has no recording to lean on, so it reads the transcripts the agents already write, where they already are. There is one reader per agent: Claude Code’s JSONL files under ~/.claude/projects, Codex’s rollout files, Copilot CLI’s events.jsonl for each session, Gemini CLI’s session files and opencode’s SQLite database. Each reader turns its agent’s format into one event shape, so the sessions domain searches all five the same way. A small cache of session metadata saves it reading the start of every file on every question, and that cache is all ask-kit keeps about the sessions.

mike-ask only ever had to be set up on my machines, by me. ask-kit has to set itself up in whatever agents someone has, and that became the setup conversation. I took the shape from Herdr’s gallery: every layout and mod there has one line to copy, and your agent reads the instructions and does the rest. The copy button on this page holds ask-kit’s line, which sends your agent to AGENTS.md. From there ask-kit detect looks at what the machine has, ask-kit questions turns that into a few questions with a suggested answer each, and ask-kit apply writes the config and registers ask in each agent’s own MCP config, with a dry run first. ask-kit init puts the skills, the tuner agent and the when-to-ask rule into every agent in that agent’s own format. It records a hash of each file it writes and leaves alone any file you’ve edited since. The setup page of the guide walks through what the agent does and asks.

ask-kit setup from the terminal, in a throwaway home with a dummy key. The agent asks the same questions.

The investigators had to come apart from the code as well. In ask-kit a domain is a folder: domain.mjs with the REPL’s globals and prompts, api.d.ts with a typed description of what its programs can reach, instructions.md on how to investigate and cite, and a scripts/ folder. code and sessions ship with it. Your own go in ~/.config/ask-kit/domains/, and one there wins over a built-in of the same name. A router in front picks the domain. The caller can name it with about, a follow-up goes back to the one that answered last, and anything else gets one short call to a cheap model that replies with a domain’s name, with keyword rules when there is no model to ask.

Then the model. ask-kit has to run on whatever the person already has, so setup offers three kinds. Any OpenAI-compatible chat API, where ask-kit runs the loop itself. A subscription they already pay for, where Claude Code, Codex or Copilot CLI runs headless with the REPL as its only MCP server. Or a local model through Ollama or LM Studio. Each domain can have its own.

The ask tool and the REPL

Besides the question, ask takes an optional domain or backend, and continue: true for a follow-up. The answer comes back with the session turns or path:line it rests on.

Behind that, the investigator has one tool of its own: a JavaScript REPL in a Node vm, where its programs run with the domain’s way in to the data and a set of ready-made scripts (how it works has the details). What a program prints is all that goes back to the investigator’s model, and the investigator’s answer is all that goes back to the calling agent. The REPL’s variables are saved after every cell, so a follow-up with continue: true reaches an investigator that still holds what it found.

The exporter example from the guide's how-it-works page: four tool calls one after another, each adding its result to the context, then the same step as one program in one cell.

The warm seat from mike carries over. The instructions, the typed API and the scripts are the same for every question, so they’re loaded once. With Claude Code or Codex as the investigator, ask-kit starts one session with them loaded and forks it for each question. With an API, the provider’s prompt cache holds them. Copilot CLI can’t fork, so it gets the instructions in the prompt each time. The warm session carries a fingerprint of the instructions and scripts, so an edit to either warms a fresh one on the next ask, with nothing to restart.

The scripts are where the tuning goes. They start with the common questions (find-sessions, session-digest, where), and the investigator extends them: when it writes something useful by hand it can save it as a draft script, and a draft can be promoted to a script. Every ask and every cell goes into a journal with its time, cost and steps. ask-kit report lists the slow, wrong and expensive asks, and the searches the investigator keeps writing by hand. The ask-kit-improve skill and its tuner agent take it from there: find what caused a bad answer (the instructions, a script or the backend), fix it there, and ask the same question again to prove it.

The language matters more than I first thought. The investigator writes its own code every time it answers, and the scripts it runs are plain files it can read, change and save new ones next to. A scripting language like JavaScript or Python makes that cheap: there is no build step, a changed file is the whole change, and these models have seen an enormous amount of both, so even a cheap one writes it well enough. mike is Rust, and it stays Rust. The code ask-kit runs is written on the fly by a cheap model to answer one question, and JavaScript suits that.

Measuring it

The results table is under “So… is it worth it?” above. A session of mine benchmarked ask against Sonnet 5.5 reading the files itself with Read, Grep and Bash; code mode wasn’t part of the comparison. There were 21 real questions about the work in my desktop-config repo (Hyprland, the bar and dock, Herdr layouts, the Mac keyboard), its code and ask-kit’s own code, each expected answer checked by hand against the transcript or source line. 18 more questions about client work and other private repos were run and left out of what’s published. The full results are in the repo.

Context added to the calling agent, question by question: 8,363 tokens with ask against 225,662.

On accuracy they were level. Costed inside a session already holding 50k tokens, Sonnet digging comes to an estimated $0.044–0.122 a question, and ask to $0.014–0.020. When a question named the day and the agent, Sonnet went straight to the right folder, which is why it was faster on those.

The subscriptions are slower. Codex on a ChatGPT plan scored 12.5 out of 13 on the session questions, the best of the ask backends, at 34 seconds a question. Copilot CLI scored 12 at 87 seconds, and once spent 10 minutes and handed back its progress note (“Searching the session transcripts…”) as the answer. A small local model did worst: qwen3.5 9B on Ollama scored 7.5 out of 13 and said “could not find” about things that were there.

Scores out of the 13 session questions, by the model doing the investigating.

The benchmark also found two bugs in ask-kit, which is what I hoped it would do. Ollama cut every prompt at 4,096 tokens while ask-kit doctor said all was well. And the session grep only looked back 14 days, so on a count question glm answered 9 where the answer was 12; with a 30-day default it got 12 both times, in about 20 seconds for $0.002.

The part I liked most was that on 4 of the 21 questions the expected answer was the wrong one. The investigators found sessions that had been missed, and on one code question the code contradicts its own comment. The expected answer had come from the comment; haiku and Sonnet answered from the code.

What doesn’t work yet

The Gemini CLI. I don’t have a good login for it. Probably a bunch of other things.

ask-kit is MIT and has no release yet. Set it up, or read the source.