Jev in front of Memoir: the 200-millisecond gate

A model that only answers questions

You send Jev some state and a map of typed questions. A Noul returns the probability that a yes/no question is yes. A Choice picks from options you define and returns the full distribution. Everything in one request is evaluated in parallel, nothing is generated, and the probabilities are trained to be calibrated, which is the part that turns out to matter.

{
  "state": { "transcript": "[Human]\nFrom now on, always use pytest ..." },
  "model": "jev-latest",
  "questions": {
    "durable_any": {
      "type": "noul",
      "instructions": "Would a senior engineer writing onboarding notes
        record anything from this turn as a durable fact, rule,
        preference, or decision? Answer no for routine Q&A, one-off
        debugging, code reading, tool mechanics, thanks, small talk,
        or unresolved discussions."
    }
  }
}
→ { "answers": { "durable_any": { "noul": 0.90 } } }

Input tokens cost $0.042 per million, output is free, and our calls came back in 180 to 330 ms. It cannot write text, it reads questions literally, and a Choice holds at most 255 options. Those edges shaped both experiments.

Experiment 1: the gate

The plugin's prompt harness had 16 labelled transcripts, each must capture or must produce nothing. We sent each to Jev with the question above, picked a threshold of 0.3 that split all 16, then wrote 25 more before touching the number: 12 capture cases and 13 ignore cases built to be tricky, including the word always inside a factual question and a convention the agent proposed but the user declined.

The 25 held-out cases came out 12 of 12 and 13 of 13. Across all 41, every capture case is above the line and every ignore case below it, with the closest calls at 0.43 and 0.25. For comparison we ran the same 25 through the production Haiku prompt: it got 24, and the one it missed was the parked pnpm decision, which it filed as a backlog item despite a paragraph in its own prompt saying not to. Jev put that turn at 0.19.

Jev is about 5x faster and 30x cheaper than the call it guards. It cannot replace that call, because when the answer is yes someone still has to write the fact. But the answer is no on most turns, and the gate removes that call from those. The capture prompt stays exactly as it is for the turns that matter.

Experiment 2: filing the memory

275 paths is more than one Choice holds, so we borrowed the shape from TypeSafe's hierarchical classification cookbook: one request with a Choice over the 17 top-level categories plus 17 speculative leaf Choices, one per category, and code multiplies P(category) by P(leaf | category). About 5,000 tokens, 0.33 s. The gold set is the taxonomy's own 207 examples; the baseline is Haiku given the same descriptions and path list.

Jev alone is two points under Haiku. What it has that Haiku lacks is a probability you can act on: above 0.8 it is right 87% of the time, above 0.9, 92%. Haiku's self-reported confidence sits above 0.8 on 202 of the 207 memories, so there is nothing to threshold. Route on Jev's number and the hybrid lands a point ahead of Haiku alone while making 40% fewer LLM calls. No degradation, and the common case gets fast.

Two smaller findings. The biggest single gain came from rewriting the 17 category descriptions to say what each bucket is not for, which lifted Jev's exact-path accuracy from 61% to 67% and should help Haiku equally, since it reads the same file. And on the 18 real memories in this project's own store, Jev cleared the 0.8 threshold on 3 and matched Haiku's path on exactly those 3. It declines rather than guesses, which is the behaviour a gate needs.

A model that only answers questions is useless for writing and excellent for deciding whether to write.

Try Memoir

Most turns are easy to judge. That is where the time was going.