Does jevmem work?
In 72 real Claude Code sessions, Claude followed the project's saved decision in 66 with jevmem (66/72), against 28 with no project memory (28/72) and 67 with the same lines in a hand-written CLAUDE.md (67/72): jevmem does about as well as a hand-written CLAUDE.md, without you writing it.
Next to Claude Code's own auto memory (0.7.7), in 558 real sessions on three new projects: Claude followed a line taught in an earlier session in 47 of 54 sessions with auto memory alone, 40 with jevmem alone and 45 with both, by the checks written before the run (49, 40 and 48 after every miss was read by hand), and with jevmem on, no change a rule forbade was left in the code. The table and its method are below.
Every test set here was written by the author, and none is an independent benchmark. Each result below says what it was measured on, links its results file in the repository, and says what it does not show. To try it yourself: install.
Does Claude act on the saved line? #
In 72 real Claude Code sessions for each setup, Claude acted on the saved line about as often with jevmem as with the same lines in a hand-written CLAUDE.md, and in fewer than half the sessions with neither.
| No project memory | jevmem | Same lines in CLAUDE.md |
|
|---|---|---|---|
| Followed the project's decision | 28 of 72 | 66 of 72 | 67 of 72 |
| Tried a change the project forbids | 10 of 18 | 0 of 18 | 0 of 18 |
| Repeated an approach that had already failed | 3 of 15 | 0 of 15 | 0 of 15 |
So jevmem did about as well as a hand-written CLAUDE.md, without you writing it.
CLAUDE.md did better on a convention nothing in the prompt points at (3 of 3 against 0 of 3), measured on 0.6, where jevmem brought a line back only for a prompt it bears on. Since 0.7.1 jevmem marks a rule every change must follow always-on and sends it at the start of every session: on six held-out tasks written for that gap, Claude followed the convention in 14 of 18 sessions with 0.7.0 and 18 of 18 with 0.7.1, as with the same lines in a hand-kept CLAUDE.md (what's new, limits).
How the sessions were run, and every row: the benchmark page.
Next to Claude Code's auto memory (0.7.7) #
Claude Code keeps notes of its own, auto memory, on by default. To see what jevmem adds next to it, a fact was stated in plain chat in one session, and a later session got a task that needed it, in 558 real Claude Code sessions on three new projects (one model, claude-fable-5-1, and Claude Code 2.1.306; three runs of each task).
| Followed the taught line (18 tasks, 3 runs) | No memory | Auto memory | jevmem | Both | Same line in CLAUDE.md |
|---|---|---|---|---|---|
| By the checks written before the run | 17 of 54 | 47 of 54 | 40 of 54 | 45 of 54 | 45 of 54 |
| After reading every miss by hand | 17 of 54 | 49 of 54 | 40 of 54 | 48 of 54 | 46 of 54 |
| Left a change a rule forbids in the code (by hand) | 9 of 15 | 1 of 15 | 0 of 15 | 0 of 15 | 2 of 15 |
| Long project, 230 earlier lines (checks → by hand) | 7 → 7 of 18 | 12 → 12 of 17 | 14 → 15 of 18 | 12 → 16 of 18 | 14 → 15 of 18 |
So auto memory on its own followed the taught line more often than jevmem on its own, and with both on Claude did about as well as with auto memory alone; jevmem's guard kept every forbidden change out of the code. On the long project, auto memory's notes ran past the 25 KB Claude Code loads at the start of a session, and jevmem, alone or beside it, did about as well as CLAUDE.md.
Reading the 106 misses by hand found 12 that the checks had scored wrong (a number kept in a constant in another file, a year written another way); both rows are shown. In 14 of jevmem's 17 misses its saved line had left out the rule: 9 cut at the line's 200-character limit, which 0.7.7 changes (a line now keeps the sentences that state it up to 300 characters; what's new), and 5 where Jev's choice of sentences left out the one with the rule, which 0.7.7 does not change. The comparison ran on the build before that change.
How the sessions were run, the 12 sessions and every row: the benchmark page.
Method. 24 tasks in three small projects (a TypeScript API, a React front end, a plain-JavaScript CLI), each with a project memory of 34 to 42 lines. Each task's right answer depends on one saved line that the repository does not state. Every session is real Claude Code (claude -p, Claude Code 2.1.281, claude-sonnet-5) in a fresh copy of the project, three runs per task and arm. A check written and committed before any session ran decides whether the line was followed; there is no LLM judge. The no-memory and CLAUDE.md arms ran on 2026-09-28 (results), and the jevmem arm ran on 2026-09-29 on the recall code that ships in 0.6 (results). The full method and every row: docs/benchmark.md.
Limits. Three small projects, one model, sessions of at most 25 turns, three runs per task. CLAUDE.md did better on a convention nothing in the prompt points at (3 of 3 sessions against 0 of 3), which 0.7.1's always-on rules address (next). The checks test whether the saved line was followed, not whether Claude's answer was better.
Do rules every change must follow reach Claude? #
Since 0.7.1, when the line is marked always-on. In the sessions above, a convention nothing in the prompt points at was followed in 0 of 3 sessions with jevmem and 3 of 3 with CLAUDE.md, because jevmem brought a line back only for a prompt it bears on. 0.7.1 marks such a rule always-on (Jev reads it as a standing rule for how any work in the project is done; jevmem always <id> marks by hand) and sends it at the start of every session. On six held-out tasks in two new projects whose files were written before their conventions, Claude followed the convention in 10 of 18 sessions with no memory, 14 of 18 with 0.7.0 and 18 of 18 with 0.7.1, as with the same lines in a hand-kept CLAUDE.md (3 runs per task, real Claude Code 2.1.296 with claude-sonnet-5, one run per arm, 2026-10-13: no memory, 0.7.0 and CLAUDE.md, 0.7.1). On the first held-out set, 20 tasks in the same projects where the code around each task already shows its convention, 0.7.1 followed the saved line in 49 of 60 sessions, against 53 with 0.7.0, 52 with CLAUDE.md and 46 with no memory (no memory, 0.7.0 and CLAUDE.md, 0.7.1).
Limits. The trade-off: a rule that reads as a process ("nothing is committed straight to main") in every session's context made Claude stop and ask for a branch on a task that commits nothing, in 4 of 60 sessions on the first set (the same lines in CLAUDE.md: 2 of 60; 0.7.0, which brought the line back only when the prompt pointed at it: 0). The mark is Jev's reading or yours: on the held-out projects' 68 lines, the marking question found 17 of the 20 rules labelled always-on by hand and marked 5 unlabelled lines (results); rules stated as facts about one thing need jevmem always <id>. Two small projects, one model, one run per arm. The method and every row: docs/benchmark.md.
Does the guard catch rule breaks? #
On the guard's second held-out set, 274 tool calls in five new projects, the guard caught 66 of 68 rule breaks (66/68) and asked about 3–4 of the 206 calls that break no rule. The set was written after a day's trial of the guard in jevmem's own repository, before the fixes it measures, and was run once on the 0.6.0 build and once on the 0.6.1 build, both on 2026-09-30 (0.6.0, 0.6.1). The one call that differs between the two runs breaks no rule and scored either side of the threshold.
In real Claude Code sessions on the A/B's 6 constraint tasks (18 sessions, rerun on the release build on 2026-09-30 with Claude Code 2.1.284), Claude did not attempt the forbidden change in any session (0/18; 10/18 with no memory in the 2026-09-28 run), and the guard checked all 78 of those sessions' Bash, Edit and Write calls and asked once (results).
Limits. The guard looks at the words a rule and a call share. A script or a make target that already exists and does the forbidden thing is missed, and so is a call that shares a single word with its rule. It checks Bash, Edit and Write calls, not MCP tools or the files a script changes when it runs. More: the guard's limits.
Is deciding what to save fast, cheap and right? #
Deciding what to save takes 0.16 s and costs $0.00019 per message, in the background: Claude doesn't wait for it. On whether to save a message at all, jevmem was right as often as the best of six LLMs (98.5%); three of them were better at naming the kind of line.
Measured on 66 turns written before the code and not tuned on, one run of the 0.7.7 build on 2026-11-08 (results); the six LLMs ran on 2026-09-23. All seven in one table, with the cost of each and how it was run: benchmark.
Does the right line come back? #
On the second retrieval held-out set (90 prompts over three new projects of 20, 80 and 250 lines, run once on 2026-09-28), jevmem 0.6's recall found 75/78 of the lines the prompts needed, and 0.5.9 found 55/78; 0.7.1 finds the same 75/78 (one run, 2026-10-13, results). Of those 78 lines, 18 are dead ends, which 0.5.9 cannot read; on the other 60, 0.6 found 57 and 0.5.9 found 55. Of the lines 0.6 put in front of Claude, 96/97 were wanted or fine, and 1/18 unrelated prompts got a line (0.6, 0.5.9).
Limits. Prompts that need two lines got both in 3 of 6. With more than 250 live lines, only the 250 that share the most words with the prompt are asked about (on a 500-line dev file, recall was 37/46). That the right lines reach Claude is tested; whether its answers get better is not.
Is a line saved twice? #
Not when it says the same as a live line (0.7.0). On a held-out set of 50 turns in five new projects, written before the code: of 25 restatements (word for word, paraphrased, with a tag in front, quoted), 0.6.6 saved 15 as new lines and 0.7.0 saved 1; restatements that add a detail were saved 4 of 5, reversals superseded 5 of 5, retired lines said again saved anew 4 of 5, and unrelated turns came out as labelled 9 of 10, the same on both builds (one run each, 2026-10-08: 0.6.6, 0.7.0; 0.7.1 gives the same counts, one run on 2026-10-13, results). A tag pasted in front of a line ([constraint] Never …) is no longer part of it: on 24 held-out texts, 0 of 18 tags were stripped on 0.6.6 and 18 of 18 on 0.7.0, with the six bracketed tokens that must stay kept on both (0.6.6, 0.7.0).
Limits. The one restatement 0.7.0 saved was a quoted rule that the restatement question scored 0.97 but the id choice answered none for; dedupe needs both. Only a word-for-word restatement is stopped before any request; a paraphrase goes to Jev. The method and every row: docs/benchmark.md.
Can memory start from past sessions? #
Yes, since 0.7.1: jevmem import --from sessions reads a project's past Claude Code sessions and proposes the lines the hooks would have saved, after counting the sessions and turns, estimating the cost and asking. On 12 real held-out sessions with 27 turns, captured before the code in two new projects, it read every turn (27 of 27), proposed a line for every turn labelled to give one (17 of 17) with the labelled words in each, superseded both reversals (2 of 2), and gave 4 lines for turns labelled skip: precision 17 of 21 (one run on the build, 2026-10-13, results).
Limits. The kind was wrong in 5 of the 27 turns, as decide reads them. Two prompts with nothing from Claude between them are one turn to the reader. The session format is undocumented: the reader counts what it skips and stops when no file gives a turn. More: docs/limits.md; the method: docs/benchmark.md.
Dead ends, background subagents and planted lines #
- Dead ends. On decide's third held-out set (100 turns in five new projects), jevmem saved 25 of 25 dead ends, each with its reason, on every build run on it since 0.6.0, last 0.7.1 on 2026-10-13 (results; 0.6.5, 2026-10-05). When Claude's reply gives the verdict first and the cause last, 0.6.5 keeps the cause: on 13 held-out dead ends written in that style (2026-10-05), 12 lines kept the reason with the reply given to the writer, against 2 in 0.6.4 (0.6.5, 0.6.4). With deciding in front, 6 of the 13 were saved as dead ends, each with its reason; in the other 7 jevmem did not read Claude's reply, as in 0.6.4.
- Background subagents. On 30 real Claude Code 2.1.281 sessions, 14 of them with a background subagent, replayed through each build: no line was saved while a turn was still running (0.5.9 saved 11), and turns were saved or skipped right in 30 of 33 on 0.7.1 (one run, 2026-10-13, results), in 32 of 33 on the 0.6.0 build (2026-09-30, results) and in 27 of 33 on 0.5.9. The two turns that moved since 0.6.0 moved with 0.6.6's reply question, which gives the same 30 of 33 (0.6.6, 2026-10-08).
- Planted lines. On a 44-line test set (2026-09-25), the check on lines jevmem did not write blocked 20 of 22 planted lines, with 0 false blocks on 22 legitimate rules; the 2 it missed were instructions disguised as normal process (results).
Can a local model do Jev's job? #
Not well enough to offer, in the one test so far. On 2026-10-02, jevmem 0.6.4 was pointed at a local model server, Ollaya 0.9.0, on an Apple M4 with 16 GB, next to a Jev run of the same set the same afternoon. On the 66 held-out turns, in jevmem's default mode, Jev was right on save or skip for 65/66 turns at a median of 0.23 s a turn; winnow:e4b for 60/66 at 28.5 s; laya:typed-decisions for 19/66.
The results files: Jev, winnow:e4b, laya:typed-decisions. The commands as run: scripts/local-model-jev.sh, scripts/local-model-ollaya.sh.
Limits. One run each, on one Mac, with two models. The set was written for Jev's behaviour. Both local models needed the client's timeout raised from 10 s to 180 s. winnow:e4b was measured on these 66 turns only, not on recall or the guard.
What these results do not show #
- Early: first released on 2026-09-22; the eval sets were written by the author, or captured from Claude Code sessions on scratch projects built for them, and none is an independent benchmark.
- Not the most accurate: GPT-6 Astra, Claude Opus 5.5 and Claude Fable 5.1 scored higher on save+kind (0.7.7's run); jevmem's edge is speed and cost.
- Recall quality is not measured: that relevant lines are injected is tested; whether answers get better is not.
- Next to Claude Code's auto memory, jevmem alone followed a taught line less often (0.7.7; docs/benchmark.md): 40 of 54 sessions against 47 of 54 with auto memory alone (49 after reading the misses by hand), and 45 of 54 with both on (48). One model (
claude-fable-5-1), Claude Code 2.1.306, three runs of 18 tasks the author wrote in three projects; the checks were written before the run and scored 12 of their 106 misses wrong, which a reading by hand found (only the misses were read). The comparison ran on the build before 0.7.7's writer change, whose effect on real sessions is not measured. - Long-run drift is not measured: the harness covers five-turn sessions, not weeks of use.
- The side pane shows jevmem in Claude Code only (0.7.5, 2.1.299 or newer): no pane in Cursor, nothing drawn in VS Code (the entry above).
- Automatic capture is Claude Code and, since 0.7.4, Cursor (and Codex while
jevmem watchruns); Claude Desktop saves only when the agent callsadd_memory, and in Cursor recall is still the agent's call (the Cursor entries above). - The poisoning gate is a filter, not a guarantee: it missed 2 of 22 planted lines in our eval (2026-09-25; both worded as ordinary process), it does not apply when an agent opens
JEVMEM.mdas a file, and on a fresh clone its first check costs one noul per line. ReviewJEVMEM.mddiffs like code (SECURITY.md). - Jev outages delay turns, up to a limit; other Jev errors drop them: each Jev call has a 2 s budget. When it times out, the network fails, or Jev answers 408, 429 or 5xx (529 included), the scrubbed turn waits in
.jevmem/queue.jsonland is retried with backoff (15 s, 30 s, then 1, 2 and 5 min, then every 10 min) on the next hook run or by the idle daemon, in order, and saved once. A turn still unsaved after 24 hours, or past 200 queued turns, is dropped. Any other error is not retried and drops the turn at once: a 400 from Jev, for example, or a 401 when the key is wrong, which drops every turn until the key is fixed. Each drop leaves a line in.jevmem/log.jsonl, and the retry-queue line ofjevmem statscounts them. - A plugin update can leave an open session without the hooks: when the plugin synced from claude.ai updates (Claude Code downloads updates in the background each time it starts), Claude Code moves the previous copy aside, and a session that was already open with it loses jevmem's hooks, the guard included, until you run
/reload-pluginsthere or start a new session; Claude Code shows a hook error and goes on without them (anthropics/claude-code#97847).jevmem doctorshows the version on disk.
The limits, in short #
- jevmem needs a TypeSafe API key (where to get one, and the install steps).
- Message text is sent to TypeSafe to be scored, with common secrets scrubbed first (what leaves your machine).
- It is automatic in Claude Code and, since 0.7.4, in Cursor (as a Cursor plugin since 0.7.6, whose rule sits in every chat), automatic in Codex while
jevmem watchruns, and in Claude Desktop only when the agent calls it; in Cursor, bringing lines back is still the agent's call (what each tool does). - Rules every task must follow reach every session only when they are marked always-on (0.7.1), by Jev's reading or yours (jevmem next to CLAUDE.md).
- The guard is a backstop, not a sandbox (what it misses).
- The side pane is Claude Code only, 2.1.299 or newer: Cursor has no pane, and Claude Code for VS Code draws none yet (the pane).
Every limit, with the numbers: the FAQ.