Persistent Memory for LLM Agents

View on GitHub → Updated 2026-07 · v4.0.0

98 experiments, 35 case studies, 5 benchmarks, 6 benchmark experiments, 20 approaches evaluated. Now at v4.0.0: the staged retrieval lanes are finally live on the production path -- temporal spine, intentional clustering, structural HRR, junk demotion -- plus read-only cross-project federation, layered locks, passive conversational capture, and 6,009 tests.

Jump to the latest update ↓ -- What changed since the original write-up.

Abstract

LLM agents have no memory across conversations. Corrections are lost when the context window closes, and the agent repeats the same mistakes. Existing memory systems treat this as a retrieval problem, but the harder unsolved problems are write correctness and governance: what gets stored, how conflicts are resolved, and whether wrong memories can be corrected. We present aelfrice, a persistent memory system built on FTS5 keyword search, typed knowledge graphs, and entity-index retrieval, with no embeddings. The system detects user corrections at 92% accuracy without LLM calls, recovers 99.5% of the 31% of stored directives unreachable by keyword search, and reduces injected tokens by 55% with zero retrieval loss. Across five benchmarks, aelfrice achieves 66.1% F1 on LoCoMo (+14.5pp over GPT-4o), 90% on MemoryAgentBench single-hop (+45pp), 60% on multi-hop (8.6x the published 7% ceiling), 100% on StructMemEval state tracking, and 59.0% on LongMemEval (-1.6pp vs GPT-4o pipeline, different judge). A controlled A/B test showed 31% token savings and 41% fewer tool calls, though correction rates did not decrease. All code, benchmarks, and experiment data are open source under MIT license. (aelfrice was developed and benchmarked under the working name "agentmemory"; the two names refer to the same system. The benchmark figures above are from the v1 substrate; a v3.0.1 re-run revised several of them — see §12, which includes a corrected account of what did and did not drive the LoCoMo change.)

1. Introduction

LLM agents have no memory across conversations. Every session starts from zero. When a user corrects the agent, that correction is lost the moment the context window closes. The next session, the agent makes the same mistake. The user corrects again. To make matters worse, the agent will often ignore corrections in the exact same context window session.

Memory failures are the largest single category of LLM behavioral failures documented in this project's failure taxonomy: 7 of 38 cataloged patterns. The problem compounds across sessions: the MemoryAgentBench benchmark (ICLR 2026) tested multi-hop conflict resolution and found a ceiling of 7% accuracy across all tested methods.

Existing approaches overwhelmingly treat memory as a retrieval problem. Across the architectures cataloged by Zhang et al. (2024), Hu et al. (2025), and Leonard Lin's independent analysis of 35+ papers and 14+ community systems, the dominant pattern is: store text, embed it, retrieve by similarity. StructMemEval found that retrieval-only systems built on this pattern score near zero on state tracking at scale. They can't tell you what's currently true vs. what was superseded.

Lin concluded:

"The biggest differentiator is not vector DB vs SQLite. It is write correctness and governance: provenance, write gates, conflict handling, reversibility."

Current memory systems are write-only: content goes in, but the system never learns whether what it retrieved was actually helpful. On every turn, the memory system retrieves stored content, the LLM reads it and generates a response, and the turn ends. There is no feedback path. The memory system cannot reinforce directives the user found helpful, weaken directives the user has overridden, or distinguish a user correction from an LLM inference at storage time. None of the architectures in the 47-author survey by Hu et al. or Lin's 14+ system analysis include such a feedback path.

The LoCoMo benchmark (ACL 2024) showed that a simple filesystem with grep achieves 74%. That's the bar.

Now for my Fosbury Flop.

I built this system because I got sick and tired of asking Claude for the latest on my test runs, which were burning CPU time on cloud compute, only for Claude to tell me "huh? what test dispatches? oh those. yeah they've been hanging for 2 hours because I didn't follow the runbook you told me to follow."

2. Related work

Before building, I surveyed prior work: 4 survey papers (Zhang et al. 2024, Hu et al. 2025, Yang et al. 2026, "Memory in the LLM Era" 2026), 6 benchmarks, 14+ community systems, and Leonard Lin's independent benchmark reproduction of 35+ papers which verified (and in some cases refuted) published claims.

Four findings shaped the project direction:

1. Human memory is the wrong target. Zhang et al. (2024) and Hu et al. (2025) show that the dominant design paradigm maps psychological memory models onto LLM architectures. Human memory is notoriously unreliable: Ebbinghaus (1885; replicated by Murre & Dros, 2015) showed ~56% of learned material is forgotten within one hour. Eyewitness misidentification contributed to roughly 69% of the 375+ wrongful convictions overturned by DNA evidence (Innocence Project). Computer memory is literally perfect at storage; the hard problem is retrieval. Where human memory is useful: retrieving gists. Brainerd & Reyna (2005) showed that gist traces are far more durable than verbatim traces. We're trying to replicate that associative retrieval: connecting things that share no surface-level vocabulary.

2. A simple filesystem achieves 74% on the most-used benchmark. Letta's best result with no special memory architecture, just gpt-4o-mini writing to files, hits 74% on LoCoMo (Maharana et al., ACL 2024). Any memory system must beat "gpt-4o-mini writes it to a file."

3. Multi-hop conflict resolution tops out at 7%. MemoryAgentBench (ICLR 2026) tested the ability to follow a chain of related decisions across sessions and determine which is currently in effect. The ceiling was 7% accuracy across all tested methods.

4. Lin's independent analysis pointed to an underexplored axis. The majority of systems focus their novelty on retrieval: better embeddings, better similarity metrics, better ranking. The harder unsolved questions are about what gets stored, how conflicts are resolved, and whether wrong memories can be corrected. Lin's concept of "write correctness and governance" breaks into four components:

   • Provenance: Where did this memory come from? A user prompt, an LLM inference, or derived content? Every stored entry should carry its lineage.
   • Write gates: Not everything should be stored. Expanding a 586-node graph to 16,463 nodes without quality filtering dropped retrieval coverage from 92% to 69% (Exp 48).
   • Conflict handling: When two memories contradict, which wins? StructMemEval found that vector stores score near zero on state tracking at scale.
   • Reversibility: None of the 14+ systems Lin tested had a working rollback mechanism.

Full benchmark comparison tables are in Appendix B.

3. Approach

Four questions drove the design:

1. How do you detect user corrections, stated preferences, and behavioral rules without extra LLM inference?
2. How do you retrieve relevant content when the query shares zero vocabulary with the stored content?
3. How do you track whether a retrieved memory was actually useful?
4. How do you distinguish "the LLM should consider this" from "the LLM must obey this"?

Correction detection: 92% accuracy without any LLM calls, across five codebases (Exp 39-41). When LLM classification is enabled (~$0.005/session), accuracy reaches 99%. The zero-LLM pipeline runs on every conversation turn with zero marginal cost.

Vocabulary gap recovery: 31% of stored content across five codebases is unreachable by keyword search (Exp 47, 3,321 directives examined). The query and stored directive share zero vocabulary. For example: a user says "never mock the database in tests." Later, the agent is about to write a test with unittest.mock.patch('db.connect'). Zero overlapping words, but a human immediately sees the connection. The system recovers 99.5% of these gapped directives through a structural graph traversal that doesn't require embeddings or LLM inference.

Entity-index retrieval: To address multi-hop conflict resolution, the system extracts structured triples (entity, property, value, serial_number) from ingested text using regex relation-family patterns, then chains through entity relationships at query time. This layer (L2.5, between FTS5 and HRR) took the MemoryAgentBench multi-hop score from 6% to 35% chain-valid, 5x the published 7% ceiling (see Section 5.3).

Confidence tracking: The system tracks retrieval outcomes and updates confidence accordingly (Exp 66: +22% MRR gain over 10 feedback rounds; Bayesian calibration ECE 0.066, target < 0.10). Memories that help get stronger. Memories that hurt get weaker. Memories irrelevant to the current task get no update, because absence of evidence is not evidence of absence.

Correction enforcement: Storing a correction and enforcing it are different problems (see CS-006 above). The system distinguishes between content the LLM should consider and constraints the LLM must obey (Exp 84: 10/10 locked directives retrieved and enforced across 5 sessions).

The architecture uses keyword search as the primary retrieval layer and wraps it with structural gap recovery, confidence tracking, and constraint injection. The combined system handles both the 69% grep reaches and the 31% it misses. Grep won on the keyword retrieval benchmark (Exp 47, 92% coverage vs. 85% for the prototype), and we accepted that: grep is fast, precise, and costs no LLM calls. But grep cannot bridge the vocabulary gap. The locked-directive Mean Reciprocal Rank improved from 0.589 to 0.867 after retrieval tuning (Exp 63).

Token reduction: Type-aware token reduction achieves 55% savings with zero measured retrieval loss (Exp 42). Constraints survive verbatim; rationale compresses to 0.4x; metadata compresses to 0.3x. Every token injected into the agent's context window is a token that can't be used for reasoning about the current task. At 19K+ stored nodes, naive injection would consume the entire context window before the agent reads the user's message.

Scale effects: When the graph expanded from 586 to 16,463 nodes without filtering, retrieval coverage dropped from 92% to 85% for grep and from 85% to 69% for prototypes (Exp 48). Decision-level directives were 3.6% of the expanded graph. The fix: filter at ingestion time, not after expansion. Only decision-level directives pass the write gate.

4. Evaluation

The project uses a four-layer evaluation architecture.

Layer 1 (programmatic checkers): Deterministic checks like "does the output contain the required section headings?" These catch structural violations but can't evaluate semantic correctness.

Layer 2 (structural validators): Added after discovering that LLMs satisfy individual constraints while violating the relationships between them (CS-007b): correct sections in the wrong order, or referencing a decision that contradicts an earlier one.

Layer 3 (LLM-as-judge with anti-contamination): The most important design decision. Standard LLM-as-judge approaches (Zheng et al., 2023) give the evaluating LLM the input prompt and the response. The problem: when the evaluating LLM can see the system's reasoning, it tends to find that reasoning plausible and rationalize the violation (CS-005). Our approach isolates the evaluating LLM. It receives only the constraint and the output, never the conversation that produced it.

Layer 4 (adversarial follow-up): Added after CS-024 (sycophantic collapse). An LLM can pass all three previous layers and still fail when a user pushes back.

For agent memory retrieval, precision matters more than recall. A false negative (relevant directive missed) is invisible. A false positive (irrelevant directive injected) causes the LLM to act on wrong context, requiring the user to notice, diagnose, and correct. The scale experiment (Exp 48) demonstrated this directly: expanding the graph without filtering retrieved more content, but the wrong content.

5. Results

5.1 Core pipeline metrics

Correction detection achieves 92% accuracy without LLM calls across five codebases (Exp 39-41), reaching 99% with LLM classification at ~$0.005/session. The vocabulary gap recovery layer handles the 31% of directives keyword search misses, recovering 99.5% of them through graph traversal (Exp 47). Token reduction saves 55% with zero retrieval loss (Exp 42). The feedback loop improves MRR by +22% over 10 feedback rounds (Exp 66), with Bayesian calibration at ECE 0.066.

5.2 Single-hop results

LoCoMo

The LoCoMo benchmark (Maharana et al., ACL 2024) tests whether a system can answer questions about past conversations across five categories. Setup: 10 conversations (5,882 turns, 272 sessions, 1,986 QA pairs) ingested through the standard onboarding pipeline. Retrieval used FTS5+HRR+BFS with a 2,000-token budget. Scoring followed LoCoMo's exact F1 methodology.

The initial run was contaminated (agents had access to ground truth; see Appendix A for the full contamination narrative and protocol). The protocol-correct results:

Single-hop is strongest (69.4%); adversarial is near-perfect (97.5%). Multi-hop and temporal are weaker (42-45%), requiring cross-session reasoning and date arithmetic. Open-ended is weakest (30.5%), requiring synthesis the retrieval pipeline doesn't directly support. Ingest time for all 10 conversations: ~25s. Average query latency: ~16ms.

MAB single-hop

MemoryAgentBench (Hu et al., ICLR 2026) tests conflict resolution: when facts change over time, can the system track which version is current? Single-hop asks direct questions: "What is X's current Y?"

The improvement from v1.0 (60%) to v1.1 (90%) came from triple extraction in the ingestion pipeline. SUPERSEDES edges are created automatically. Haiku still beats GPT-4o-mini (62% vs 45%), so the improvement comes from retrieval, not the reader model.

LongMemEval

LongMemEval (Wu et al., ICLR 2025) is a 500-question benchmark spanning six categories. The published best is 60.6% using a GPT-4o pipeline with embeddings.

Strengths: single-session recall (91.4%) and knowledge updates (70.5%). Weakness: multi-session (24.1%). Failure analysis of 101 incorrect multi-session answers: 67% were retrieval misses, 33% were reasoning failures. Of the retrieval misses, 84% were counting/aggregation questions. Budget and top_k sweeps did not help; the bottleneck for this category is FTS5's BM25 ranking. Methodological note: scoring uses Opus as judge rather than GPT-4o; the comparison carries an asterisk.

5.3 Multi-hop and state tracking

MAB multi-hop

Multi-hop chains entity relationships: "What is the Z of X's current Y?" This requires following a chain of updated properties. The entity-index (described in Section 3) extracts structured triples and chains through them at query time.

Both Opus and Haiku score identically at 35% chain-valid. This is the strongest evidence that the entity-index retrieval, not the LLM reader, drives the improvement. When the retrieval provides the right entity chain, even Haiku can follow it.

Experiment 6 resolved the retrieval question: by branching through all historical values at each hop, 96 of 100 ground truth answers became reachable. The remaining 40pp gap is entirely a reader chain resolution problem.

StructMemEval

StructMemEval (Shutova et al., 2026) tests state tracking: given location updates across sessions, can the system answer "where is X now?"

The fix: assign narrative timestamps (30 days apart per session) and enable temporal_sort=True so the reader sees the most recent session content first. This is a general-purpose state-tracking improvement, not a benchmark-specific hack.

5.4 Scale and onboarding

Scan phase scales sublinearly: 5.4x more nodes in 2.3x the time. AST parsing is the bottleneck (32-38% of scan time). Full pipeline (scan + ingest + edge storage + vault sync) measured at 6.5s for a 10,872-node repo (249 commits, 112 docs) after v1.2.1 performance fixes batching edge inserts and deferring per-belief FTS5 checks during bulk ingestion. Temporal decay validated on the larger codebase: 2-day-old beliefs score 0.92, 18-day-old beliefs score 0.43, 14-month-old beliefs score ~0.

6. Discussion

What doesn't work yet

What remains unmeasured

A/B test: status report with vs without memory

To test whether the system actually helps, we ran a controlled A/B comparison: two fresh Claude Code sessions given the identical prompt ("generate a comprehensive project status report"), one with aelfrice active, one without.

The first attempt was invalid: the experimental agent was coached, sub-agents don't receive hook injections, and the control wasn't isolated (it found the SQLite database and queried it directly). Despite being invalid, it surfaced a real bug: 25% of memory search results were relevant, 75% were cross-project noise.

The valid test used two live Claude Code sessions in separate terminals with identical prompts. The control had .mcp.json deleted and ran in an isolated worktree. The experimental session had hooks firing automatically with no special instructions.

The experimental session was 31% more token-efficient and used 41% fewer tool calls. The experimental agent barely used aelfrice directly (one status() call, zero searches). The benefit was passive: hook-injected context at session start reduced cold-start exploration. Both reports were comparable in quality. The value was efficiency.

Longitudinal analysis

A second analysis examined 44 qualifying sessions across 1 week (23 before aelfrice activation, 21 after).

The LLM does roughly twice as much autonomous work per instruction. Tasks complete in 33% fewer turns. But correction rates went up, not down. Two confounds likely explain this: the "after" sessions were refinement/debugging work (inherently more correction-heavy), and possible model version changes during the period. A rigorous future test would need matched task design, randomized condition assignment, at least 20 tasks per condition, manual annotation, cross-session measurement, and blind evaluation. This experiment has not been run yet.

7. Failure taxonomy

35 documented behavioral failures across Claude and Codex, classified into recurring patterns. Each case study includes verbatim exchange, root cause analysis, pattern classification, what the memory system should do to prevent it, and a concrete acceptance test with pass/fail criteria.

Each case study maps to acceptance tests. The suite runs against the live SQLite store and retrieval pipeline: 62/65 tests pass (29 files, 1.65s), with 3 skipped for capabilities requiring behavioral hooks not yet implemented (CS-012: PostEdit hook, CS-024: sycophantic collapse detection, CS-026: permission-gated intent completion).

8. What was abandoned (and why)

The negative findings shaped the architecture as much as the positive ones.

SimHash clustering was the first attempt at deduplication. SimHash works well for near-duplicate text, but stored directives are short and semantically dense -- "always use mocks" vs. "never use mocks" differ by one word that inverts the meaning.

Mutual information re-ranking was supposed to improve retrieval by scoring candidates on their statistical relationship to the query. In practice, it demoted relevant results and promoted spurious correlations.

Global holographic superposition was theoretically promising and spectacularly failed. At 775 edges, the representation exceeded its information-theoretic capacity by 7.6x and produced pure noise (Exp 50).

Pre-prompt compilation attempted to pre-compute relevant directives for common query patterns. It performed worse than random selection (23% vs. 33%, Exp 52) because directive value is context-dependent.

9. Conclusion

aelfrice demonstrates that persistent memory for LLM agents does not require embeddings, vector databases, or expensive inference. A pipeline built on FTS5 keyword search, typed knowledge graphs, and entity-index retrieval achieves competitive or superior results across five benchmarks while running at 0.7s average retrieval latency on a 19K-node production database.

The core contributions are: (1) correction detection at 92% accuracy without LLM calls, (2) vocabulary gap recovery that handles the 31% of directives keyword search cannot reach, (3) entity-index retrieval that breaks the published 7% multi-hop ceiling by 5-8x, and (4) a confidence tracking loop that improves retrieval quality over time.

Four limitations remain. LongMemEval multi-session accuracy is 24.1%, bottlenecked by FTS5's inability to aggregate scattered mentions. Correction rates did not decrease in the longitudinal analysis, though confounds prevent attribution. Contradiction detection during retrieval does not work yet. Cross-project noise in shared databases needs tighter scoping enforcement.

The failure taxonomy and its 62 passing acceptance tests may be the most practically useful contribution: a catalog of the specific ways LLM agents fail at memory, each with a reproducible test that blocks recurrence. All code, experiment data, and benchmark adapters are available at github.com/robotrocketscience/aelfrice under MIT license. Version 1.2.1, research frozen 2026-04-16.

10. Research breadth

The project drew on multiple fields:

• Information theory: The information bottleneck (Tishby et al., 1999) was applied to context compression, producing the 55% token savings. Mutual information was tested for retrieval re-ranking and abandoned. Rate-distortion theory was explored for optimal token budget allocation but proved unnecessary.

• Bayesian inference: Beta-Bernoulli conjugate pairs for confidence tracking. Thompson sampling for the exploration/exploitation tradeoff in retrieval. Calibration measured at ECE 0.066 (target < 0.10).

• Cognitive architectures: SOAR's impasse-driven substates informed retrieval failure escalation. CLARION's meta-cognitive subsystem inspired confidence tracking. ACT-R's declarative/procedural distinction mapped to the system's separation of factual content from behavioral constraints. The design borrows structure without inheriting human-like decay and distortion.

• Bio-inspired optimization: Slime mold network dynamics (Tero-Kobayashi equations) were tested for graph pruning. Evolutionary algorithms were tested for edge set optimization. Both showed promise in simulation but were not adopted.

• Graph theory: Typed knowledge graphs with weighted edges form the structural backbone. Multi-hop traversal enables the vocabulary gap recovery.

11. Technical details

• Source: github.com/robotrocketscience/aelfrice, MIT license.
• Language: Python with strict typing enforced by pyright in strict mode.
• Storage: SQLite with WAL mode. The entire memory store is a single file.
• Dependencies: Minimal by design. No PyTorch, no TensorFlow, no embedding models. LLM classification (~$0.005/session) brings accuracy to 99% and is the recommended configuration.
• Deployment: MCP server with 19 tools, integrating with Claude Code, Cursor, Windsurf, and other MCP-compatible tools. Also ships as a CLI with 23 commands.
• Modules: 87 production modules in src/aelfrice/, plus benchmark adapters and scoring scripts.
• Scale tested: 600 to 90,000+ nodes across five codebases. Largest production deployment: 0.7s average retrieval latency on 19K-node graph.
• Benchmarks: 5 benchmarks tested with contamination-proof protocol. Two contamination incidents caught and documented during development.
• Test suite: 362 passing tests plus 62 acceptance tests (29 files, 1.65s).
• Experiments: 85+ during core development, plus 6 benchmark-phase experiments with pre-registered hypotheses. Negative findings documented with the same rigor as positive findings.
• Case studies: 35 documented LLM behavioral failures, each with verbatim transcripts, root cause analysis, and derived acceptance tests.
• Version: 1.2.1 (research frozen 2026-04-16)

12. Update: v1.2 to v4.0 (2026)

Everything above documents the v1 research. The system kept evolving, and was renamed aelfrice. This section covers what changed through v4.0.0 (June–July 2026).

What the v1 write-up got right

The core architecture held. FTS5 keyword search, typed knowledge graphs, Bayesian confidence tracking, correction detection, and the feedback loop all survived into v4.0 without fundamental redesign, and the failure taxonomy still maps accurately to observed failures. The benchmarks were re-run at v3.0.1, and the scores changed (see Benchmarks re-run at v3.0.1 below).

What changed

The shipped retrieval pipeline stayed at four lanes. What ships in v3.5.0 is four lanes: locked beliefs (L0), an entity-index lane (L2.5), FTS5 keyword search (L1, now BM25F with anchor-augmented scoring, default-on since v1.7), and an optional typed-edge graph walk (BFS, off by default), plus a structural HRR lane (Plate-FFT bind/probe, default-on since v2.1) that answers structural queries such as CONTRADICTS:<belief-id>. Two ideas from the research line landed here in their shipping form:

• Intention clustering ships as a packing stage. Default-on since v3.0, it replaces the L1 pack loop with a diversity-aware greedy fill that biases the returned set toward distinct graph-connected clusters, so the token budget is not spent on near-duplicates. Locked and L2.5 results are pre-included unchanged.
• HRR ships as a structural-query lane. The deterministic Plate-FFT codec binds role-filler structure rather than learned embeddings, and surfaces structurally-related beliefs with no LLM call and no vector index.

Confidence stayed Beta-Bernoulli. Each belief carries a Beta-Bernoulli (α, β) posterior, and L1 scoring blends BM25 with the posterior mean. A regime classifier (SUPERSEDE / IGNORE / MIXED / INSUFFICIENT_DATA) ships as a read-only health audit surfaced through aelf health / aelf regime.

Beliefs can cross project boundaries, read-only. v3 added cross-project federation as a read-only mechanism. A project lists peer stores in knowledge_deps.json; peer databases are opened read-only (mode=ro&immutable=1), and any attempt to mutate a foreign belief id is rejected at the API surface (ForeignBeliefError). Shared scopes live under ~/.aelfrice/shared/{scope}/memory.db, and a scope column (project / global / shared:<name>) tracks each belief's visibility.

This directly addresses the cross-project noise problem documented in Section 6 of the v1 write-up (25% cross-project noise in the A/B test). Exp 97 measured: 100% recall, 0% top-5 contamination, 1.06x latency overhead.

Wonder and reason turned research into conversation. Two new capabilities let the agent investigate open questions using the memory graph as context. These are invoked as slash commands (/aelf:wonder, /aelf:reason) but they're used in context during natural conversational turns, not as separate formal operations. The user types something like "please /aelf:wonder about X" mid-discussion, and the system launches the research pipeline with the full conversational context already loaded. A wonder query spawns parallel research agents; a reason query builds evidence chains. Both save their findings as beliefs that persist across sessions.

A case study (in the project's internal docs, which didn't make the public-repo cut) documents a real session where wonder + reason produced a marketing strategy for the project itself: 4 parallel research agents, findings synthesized, README rewritten, all grounded in beliefs accumulated over prior weeks.

Updated numbers

Benchmarks re-run at v3.0.1

The suite was re-run at v3.0.1 (captured 2026-05-22, single-run). The scores moved in both directions. Neither version used embeddings, so that is not the variable. An earlier revision of this section attributed the LoCoMo movement to a retrieval-lane change — the write-up ran a deterministic HRR FFT vocabulary-bridge expansion lane that aelfrice gates off under the #605 ratification. A 2026-07 forensic audit of the archived v1 code and the later controlled ablations revised that attribution; see the LoCoMo row and the paragraph below the table.

An earlier revision of this section got the lane story wrong. The v1 write-up hit 66.1% with a deterministic, embeddings-free HRR vocabulary-bridge expansion lane: circular-convolution FFT binding over fixed-seed vectors, seeded from the top keyword hits, traversing typed edges written at ingest (the semantics live in those edges; HRR is the deterministic index over them). aelfrice gates that lane off under the #605 ratification, which classifies HRR-class associative recall as the same semantic-relatedness signal aelfrice delegates to the consuming agent. So this is not embeddings versus no-embeddings (HRR is deterministic FFT, not learned vectors), and it is not a determinism-forced loss (the lane is deterministic); it is a settled scope decision about what retrieval is for.

The earlier text said how much LoCoMo the lane would recover "needs a controlled ablation, not an assumption." Two ablations and a forensic audit later, the answer is: almost nothing. v3.7.0 wired the lane back in behind a flag (#981, use_hrr_expand, backed by an hrr_expand_neighbors cache) and ablated it over LoCoMo: +0.13pp — 2 of 1,540 queries — so it ships off by default. A 2026-07 forensic audit went further: with the lane restored on an edge substrate matching the v1 run's density and firing on 76% of queries, recall stayed flat; no with/without-HRR isolation was ever captured in the v1 records; and the v1 66.1% was itself unstable (the archived v2.2.2 re-run scored 49.1% with the lane intact, attributed in the archive to reader variance). Combined with the adversarial-category scoring artifact, the lane's isolated contribution is bounded small. The drop between the v1 write-up and the v3.0.1 re-run decomposes into mostly reader variance plus scoring artifact — not lost HRR recall. The #605 scope decision stands on its own terms; the benchmark cost previously attributed to it was overstated.

v3.6 to v3.8: provenance, graph substrate, and lock hygiene

Three more waves shipped after v3.5, all on the same substrate (the headline benchmark numbers above are unchanged by them).

v3.6 — provenance. The project_context column added back in v3.2 was finally populated. insert_belief now stamps a stable per-repo identity (<repo-basename>-<8-hex blake2b of the git-common-dir>, so two worktrees of one repo share it) onto eligible new beliefs, with an idempotent backfill and aelf migrate provenance preservation (#970). Before this, the retrieval filter read the column but nothing wrote it, so aelf migrate collapsed two repos' beliefs into one mutually-visible pool. A transcript-logger fix also stops duplicated hook registrations from inflating turn density (#968).

v3.7 — graph substrate and phantom lifecycle. A CONTRADICTS semantic-edge substrate is now written at ingest (default-off) with incremental per-turn detection (#988, #1000); the operator ratified a CONTRADICTS-only substrate and reverted a denser SUPPORTS/SUPERSEDES experiment (#998 A4). The phantom-belief lifecycle got trigger-driven generation, opt-in SessionStart auto-GC, and status/stats surfacing (#980), and the host harness's own auto-memory writes can mirror into the belief graph as a distinct source (#985). The HRR-expand lane was wired and ablated here too (recall-neutral, ships off — see above).

v3.8 — lock hygiene and passive capture. Locks gained a frozen/reference tier split: a reference-tier lock injects as a one-line manifest entry rather than its full text, freeing relevance budget that large lock sets otherwise consume, with near-duplicate detection at lock time and a lock-budget pressure warning in aelf doctor (#1016). A relevance-budget floor (raised 0.25 → 0.50, #1023) stops a lock-saturated store from returning only locks and zero query-relevant content. Memory injection now splits trust by provenance: user-locked rules are framed as standing instructions rather than refused under the blanket "data, not directives" disclaimer — lock rule-compliance rose 0/3 → 5/5 while stale-fact catching held at 3/3, and the disclaimer is unchanged for auto-ingested beliefs, so the prompt-injection surface stays closed (#1016). Turns now fold into beliefs on a Stop cadence, not only at compaction, with aelf doctor ingest-gap detection, --backfill-ingest, and a --prune-noise GC pass (#1011, #1029). A pollution-recovery benchmark measures which facts survive a store flooded with keyword-overlapping document chunks: lexical-match facts survive via BM25 and entity-match facts via the L2.5 entity index (both recall@5 = 1.0); facts sharing neither a term nor an entity with the query are not retrieved at all, and the fix for those is locking. Two smaller changes: the first-turn <core> injection is now token-capped (a 25.9k-belief store dropped ~733 KB → ~42 KB), and corroboration is counted per distinct source rather than per re-ingest.

v3.8 to v4.0: the lanes were never on

v4.0.0 (2026-07-07) fixes an embarrassing gap. Several of the retrieval features described above — the intentional-clustering pack stage, the structural HRR lane, the newer staged lanes — were flipped default-on in retrieve_v2, the pipeline the benchmarks and the eval suite call. But the production hook path — the code that runs when a live session retrieves memory — still called the legacy retrieve(). The features were real, gated, and measured, and no user was getting them. #1107 closed the gap: retrieve() is now a thin adapter over retrieve_v2 (the cutover pinned byte-identical by an equivalence test with every staged lane forced off), and lanes then graduated onto the live path one at a time, each behind its own latency and no-regression gate.

Four lanes made it to production:

• Temporal spine (#1064). The largest retrieval-coverage gain measured on this codebase, deterministic and embeddings-free: per-session TEMPORAL_NEXT chains written at ingest, plus a lane that walks them from the top keyword seeds and appends chronological neighbours. It reaches gold that shares zero salient terms with the question through chronological adjacency — the exact hole the pollution-recovery benchmark (v3.8, above) left open. Confirmed on a fresh LoCoMo sample: +14.6pp gold-set coverage (0.460 → 0.606; temporal questions +17.2pp, multi-hop +10.4pp), 10× a seeded shuffled control, and the out-of-sample gain exceeded dev (+12.7pp on LongMemEval) — the opposite of winner's curse.
• Junk demotion (#1096). A mild log-additive rerank penalty for beliefs whose only referential grounding is transient coordination tokens (bare PR numbers, version tags) rather than durable entities (file paths, error codes, symbols). On 118 hand-labeled coordination beliefs, the grounding score separates durable content from ephemeral status at mean 0.56 vs 0.06, lifting the durable-above-ephemeral ranking AUC from 0.48 (posterior-only) to 0.87. Pure demotion: well-grounded and entity-free beliefs are never touched.
• Intentional clustering (#436). The multi-fact pack stage, now actually reaching users: cluster coverage 0.500 → 1.000 on the public ablation with recall non-degrading, p99 latency 0.328 ms against a 5 ms budget.
• Structural HRR (#152). Marker queries (CONTRADICTS:<belief-id>) now route through the graph lane on the production path. The public ablation is constructed so the answers share no vocabulary with the query: recall 0.000 → 1.000, because keyword search cannot reach a structural answer by definition.

And two lanes deliberately did not. Origin-priority reranking was refuted on LoCoMo (#1013) — the failure there was BM25 recall (the trusted fact never became a candidate), which reranking cannot fix — so it survives only as a within-tier tie-break, default-off. And HRR-expand stays off (the +0.13pp ablation in the section above).

The same distrust of recurrence drove the release's biggest scoring fix: retrieval exposure no longer updates the posterior (#1086). Being retrieved had counted as positive evidence, so whatever kept getting retrieved kept rising. On a real 24,883-belief store, junk (session scaffolding, fragments) accumulated ~3× the exposure of clean beliefs and outscored it (mean μ 0.554 vs 0.446). Exposure is now audit-only: the event still lands in feedback history, but the posterior moves only on real signal. The controlled check: a belief retrieved 50 times no longer outranks a belief seen once. Under the old behaviour it inflated to μ 0.857.

The rest of the release, with numbers where they exist: curation grew honest tooling (aelf introspect surfaces per-session posterior, recurrence, grounding, and noise signals; aelf retire / restore make soft-delete reversible end-to-end, #1081); the host harness's own memory files now reconcile into the belief graph by default (#1089); dispatched subagents inherit memory context instead of running blind to the store (#1068); valence propagation finally turned on (#1058 — shipped in v0.1.0, never invoked by any production path until now; mean fan-out 1.7 recipients per explicit event, walk p99 < 1 ms); and Codex joined Claude as a first-class host (#1052–#1055). One more result from the wide-retrieval knobs (#1045): raising the BM25 candidate cap from 50 to 200 with an 8,000-token budget takes LongMemEval-S from 58.8% to 68.6%, past the GPT-4o paper baseline of 60.6%. The multi-hop misses were a recall-cap problem, not a ranking problem — budget alone is inert because candidates cap before the pack trim. Test count: 4,757 → 6,009.

One research thread closed negative this cycle. A posterior-informed reranking term (ζ) showed a real directional signal on dev data (permutation p ≈ 5×10⁻⁴, +0.023 MRR). The pre-registered confirmatory run on fresh samples came back −0.001 MRR. The direction was real; the effect size was winner's curse. ζ stays off, the PR is parked, and the thread reopens only on a new signal.

What's still not solved

A full canonical benchmark re-run has not been done since v3.0.1 (above). The v4.0 cycle ran targeted fresh-data measurements instead — the temporal-spine coverage gate on a fresh LoCoMo sample, the wide-retrieval knob on LongMemEval-S, the fully-fed HRR ablation — but suite-wide numbers on the current substrate remain outstanding. A v2 reproducibility harness reproduces 6 of 11 adapters within tolerance.

Known limitations that remain from Section 7:

• Contradiction detection during retrieval still isn't wired into scoring. A CONTRADICTS semantic-edge substrate is written at ingest (v3.7, default-off, with incremental per-turn detection), and as of v4.0 an explicit structural query (CONTRADICTS:<belief-id>) reaches those edges on the production path — but free-text retrieval scoring still doesn't consume them for internal consistency.
• Correction rates were not re-measured. The v1 longitudinal analysis found no decrease; whether the v3/v4 changes affect this is unknown.
• LongMemEval multi-session was 24.1% in v1 and 56.4% at the v3.0.1 re-run; the wide-retrieval knobs later took LongMemEval-S overall to 68.6% (opt-in configuration, not the hook-path default). What the default configuration scores on the current substrate is unmeasured.

Cross-project scoping was solved (v3 shared scopes with content-hash dedup). The A/B test's 25% cross-project noise is no longer expected.

Appendix A: benchmark methodology

This appendix documents the exact protocol used for each benchmark. Any deviation invalidates the results. The protocol was developed after two contamination incidents during development. The full protocol, contamination verification script, and all benchmark adapters are available in the public repository.

Contamination protocol

Three contamination modes were identified during development:

1. Ground truth in retrieval output. The retrieval JSON contained answer fields. The LLM reader saw correct answers while generating predictions. This produced the invalid 87.8% LoCoMo score. The contamination was not immediately obvious: the first Opus score (61.6% F1) was plausible. It was only discovered when slow-finishing agents overwrote the merged predictions file, producing 87.8% F1 near the human ceiling of 87.9%. Exact-match analysis confirmed: 9 of 10 batches showed 43-87% exact-match rates. Four additional isolation failures were identified: a renamed _ground_truth field, pre-computed prediction and f1 fields, category_name labels leaking evaluation strategy, and no separation between question-context and scoring metadata. All results from this run were retracted. Prevention: Adapter code writes two separate files. A mandatory contamination check (verify_clean.py) scans for 30 banned keys before any reader touches the data.

2. LLM self-judging with answer visible. Prevention: Generation and judging are strictly separate passes.

3. World knowledge override. The LLM reader used real-world knowledge instead of retrieved context, particularly on counterfactual benchmarks. Mitigation: Reader prompts include explicit instructions to use only the provided context. Documented as a known limitation (~17% of MAB failures).

General protocol

Every benchmark run follows these steps:

Step 1: Data acquisition. Download from published source. Verify row counts and field names.

Step 2: Retrieval. Run adapter in --retrieve-only mode, producing retrieval and ground truth files. Each test case uses a fresh SQLite database.

uv run python benchmarks/<adapter>.py \
  --retrieve-only /tmp/benchmark_<name>.json

Step 3: Contamination check. Mandatory before any reader touches the data:

uv run python benchmarks/verify_clean.py /tmp/benchmark_<name>.json

Step 4: Answer generation. LLM reader receives only the retrieval file. Never sees ground truth.

Step 5: Scoring. Reads predictions and ground truth. Metrics follow exact published formulas.

Step 6: Reporting. Includes exact commands, contamination check output, adapter commit hash, dataset version, reader model, scoring metric, published baselines, and known limitations.

Per-benchmark specifics

LoCoMo ([Maharana et al., ACL 2024](https://snap-research.github.io/locomo/))

• Dataset: locomo10.json, 10 conversations, 5,882 turns, 1,986 QA pairs across 5 categories.
• Ingestion: All 10 conversations through standard onboarding pipeline. Session boundaries preserved.
• Retrieval: FTS5 + HRR + BFS, 2,000-token budget, batch size 1.
• Reader model: Claude Opus 4.6.
• Prompts: Exact LoCoMo protocol prompts. Categories 1/3/4: "Based on the above context, write an answer in the form of a short phrase..." Category 2 appends: "Use DATE of CONVERSATION to answer with an approximate date." Category 5: forced-choice "(a) Not mentioned (b) [adversarial_answer]" with randomized option order (seed=42).
• Scoring: Token-level F1 with Porter stemming and article removal.
• Score: 66.1% F1.

MemoryAgentBench FactConsolidation ([Hu et al., ICLR 2026](https://arxiv.org/abs/2507.05257))

• Dataset: HuggingFace ai-hyz/MemoryAgentBench, Conflict_Resolution split.
• Ingestion: Context chunked at 4,096 tokens using NLTK sent_tokenize and tiktoken gpt-4o encoding.
• Retrieval (single-hop): FTS5 with triple extraction. SUPERSEDES edges created automatically.
• Retrieval (multi-hop): Entity-index adapter. 41 regex patterns. 4-hop chaining with breadth cap of 30.
• Reader models: Claude Opus 4.6 and Claude Haiku 4.5.
• Scoring: substring_exact_match per the paper. Chain validation for multi-hop.
• Scores: SH: 90% Opus, 62% Haiku. MH: 60% Opus (raw SEM), 35% chain-valid.

StructMemEval ([Shutova et al., 2026](https://github.com/yandex-research/StructMemEval))

• Dataset: GitHub yandex-research/StructMemEval, location/small_bench, 14 cases.
• Ingestion: Narrative timestamps (30 days apart per session). Standard pipeline.
• Retrieval: FTS5 with temporal_sort=True.
• Disclosure: temporal_sort was developed after seeing the initial 29% result.
• Score: 14/14 (100%).

LongMemEval ([Wu et al., ICLR 2025](https://arxiv.org/abs/2410.10813))

• Dataset: HuggingFace xiaowu0162/longmemeval-cleaned, 500 questions across 6 categories.
• Retrieval: FTS5 + HRR + BFS, 2,000-token budget, top_k=50.
• Judge: Claude Opus 4.6 binary judge (non-standard; paper specifies GPT-4o).
• Disclosure: Using Opus as judge instead of GPT-4o means the comparison is not apples-to-apples.
• Score: 59.0% (295/500).

Reproducibility

All benchmark adapters, scoring scripts, and the contamination verification script are in the benchmarks/ directory of the public repository. To reproduce any result:

# Clone and install
git clone https://github.com/robotrocketscience/aelfrice
cd aelfrice
uv sync

# Run retrieval (example: MAB single-hop)
uv run python benchmarks/mab_adapter.py \
  --split Conflict_Resolution \
  --source factconsolidation_sh_262k \
  --retrieve-only /tmp/mab_sh.json

# Verify clean
uv run python benchmarks/verify_clean.py /tmp/mab_sh.json

# Score (after running reader)
uv run python benchmarks/exp6_score.py /tmp/mab_sh_preds.json /tmp/mab_sh_gt.json

Complete per-benchmark commands and adapter documentation are in docs/BENCHMARK_PROTOCOL.md.

Appendix B: literature tables