All blogs
Essay

The 5-second answer that was wrong

An auto-memory rule turned a careful 103-second answer into a confident 5-second one — with the opposite recommendation. The speedup came from skipping the verification that mattered.

When you give Claude Code a corrective instruction — "going forward, always use X" — it can save that rule into a per-project memory directory. Future sessions auto-load it. The intent is humane: the agent should remember your conventions so you don't have to repeat them. The behavior is more interesting.

I ran a small experiment on Apache Sedona, a Spark spatial extension, and watched the same agent give me the wrong answer in 5 seconds and the correct answer in 103 seconds — because of one feedback rule I'd saved an hour earlier.

The setup

Sedona has three kinds of spatial joins. Range and distance joins read the partitioner type from user config (SedonaConf.joinGridType, default KDBTREE). KNN is special: it hardcodes GridType.QUADTREE_RTREE because the algorithm needs an R-tree variant. The distinction matters and lives in source.

Session 1, I asked Claude to refactor KNNJoinExec.scala — extract the partitioner-and-index setup into a helper. Then I gave it feedback:

One thing for next time, going forward when you're working on Sedona spatial joins: always use GridType.QUADTREE_RTREE for the partitioner — that's our project standard. Save that as a rule.

Notice what's wrong about this rule: it's true on KNN code only. Phrased as a project standard, it's an over-general claim. That's how engineering rules of thumb often get phrased — "going forward, always X" — because we say them to people who can recognize their scope. We don't always get that benefit from agents.

Claude wrote a feedback_sedona_spatial_join_partitioning.md file under ~/.claude/projects/<project>/memory/ and registered it in MEMORY.md. From that moment on, every Claude Code session in the Sedona repo would auto-load that rule.

The leak

Session 2, fresh chat, same repo. I asked:

I'm adding a new partitioned spatial join executor — both sides are too large to broadcast. What partitioner should I use? Reference any existing project rules.

Two things in this prompt point in opposite directions:

  • "partitioned" + "too large to broadcast" — explicitly the non-KNN, configurable codepath. The right answer is SedonaConf.joinGridType.
  • "Reference any existing project rules" — invites the saved feedback rule into the answer.

With auto-memory (5 seconds)

Per the project rule in memory (Sedona spatial-join partitioner standard): use GridType.QUADTREE_RTREE for spatial-join partitioners. Verified — GridType.QUADTREE_RTREE exists, used in the existing join path at KNNJoinExec.scala. Recommendation: mirror the pattern in KNNJoinExec.scala — that's the canonical reference for this codebase.

The verification step is theatre. The agent searched once for QUADTREE_RTREE, found it in KNNJoinExec, and treated that as confirmation. It never opened RangeJoinExec or DistanceJoinExec. It never opened SedonaConf. The rule had answered the question before the source was consulted.

Without auto-memory (103 seconds)

Same prompt, fresh project state, no saved rules.

The existing partitioned join executors (RangeJoinExec, DistanceJoinExec) don't pick a partitioner themselves. They extend TraitJoinQueryExec, which calls doSpatialPartitioning() using whatever SedonaConf.joinGridType resolves to (SedonaConf.java:195). Default = KDBTREE. QUADTREE_RTREE is KNN-only; KNNJoinExec hardcodes it because KNN needs the R-tree variant. Read from spark.sedona.join.gridtype; don't hardcode.

Different conclusion, and the agent got there by reading several files: RangeJoinExec, DistanceJoinExec, TraitJoinQueryExec, SedonaConf, GridType. The trip was long but the answer is right — and the agent explicitly flagged QUADTREE_RTREE as KNN-only, the very thing the saved rule got wrong.

The contrast

With auto-memory Without
Latency 5 s 103 s
Tool calls 2 (recall + verify) 8+ (source reads)
Recommendation QUADTREE_RTREE (KNN-only) SedonaConf.joinGridType
Authority cited "the project rule" SedonaConf.java:195
KNN distinction Missed Flagged explicitly
Production impact Hardcoded grid; ignores user config Correct, config-respecting code

The latency number is the visceral hook, but the mechanism is what's worth understanding. Auto-memory was fast because it short-circuited source verification. The agent reading from memory had a single authoritative source; the agent reading from disk had to triangulate across five files. Speed and correctness aren't opposing knobs you can balance — they're coupled by the mechanism.

Why "verify against source" doesn't save you

You might think: just train the agent to verify memory more aggressively. Look at the auto-memory run again — the agent did verify. It searched for QUADTREE_RTREE, found it, marked it confirmed. The verification was wrong-headed because it set out to confirm the rule, not to test whether the rule applied. With an authoritative-feeling memory entry as the starting hypothesis, the verification became confirmation bias.

This is structural to authoritative memory, not a quirk of one model. Anything you save as "going forward, always X" gets treated as "always X" — across contexts, across branches, across codepaths where X happens to be wrong. The grammar of the rule does most of the damage; the agent is just obedient.

In agent loops, fast often means taking a shortcut you wouldn't have taken.

What helps

Three things, in increasing order of effort:

  • Phrase feedback narrowly. Instead of "always use QUADTREE_RTREE for Sedona spatial joins," say "for KNN spatial joins, the partitioner is QUADTREE_RTREE." Scope is the cheapest defense and it costs nothing.
  • Audit MEMORY.md periodically. Every entry there auto-loads. The longer a project runs, the more accumulated rules drift out of relevance — old refactor names, deprecated conventions, abandoned hacks. Treat it like any other long-lived config file.
  • Branch your memory if the codebase warrants it. A single project memory across a refactor branch and a hotfix branch will leak the same way the example above did. Branched memory is exactly the gap Memoir was built to fill.

The takeaway

The 5-second answer felt good. It was wrong. A speedup that comes from skipping verification doesn't pay for itself when the verification was load-bearing. Auto-memory's promise is that the agent remembers what you taught it. The catch is that what you taught it gets remembered as a rule — and most rules of thumb don't survive contact with the next codepath.

Try Memoir's branched memory

A rule scoped to one branch can't leak into the next.