Skip to content

ANALYSIS-joelhooks-adr-0077-memory-system-next-phase.md

Latest commit

 

History

History
165 lines (130 loc) · 9.94 KB

ANALYSIS-joelhooks-adr-0077-memory-system-next-phase.md

File metadata and controls

165 lines (130 loc) · 9.94 KB
Raw
title Analysis — joelclaw ADR-0077 (Memory System: Next Phase)
date 2026-02-21
type analysis
system joelclaw-memory-adr-0077
source
references/joelhooks-adr-0077-memory-system-next-phase.md
related
ANALYSIS-jumperz-agent-memory-stack.md

Analysis — joelclaw ADR-0077 (Memory System: Next Phase)

This file is a critical deep dive on @joelhooks’ ADR-0077 for joelclaw’s memory system “next phase” (Feb 2026). It is intended as synthesis input and builds on references/joelhooks-adr-0077-memory-system-next-phase.md.

Context & Sources

  • Primary: joelclaw ADR-0077 document and the joelclaw repo (implementation context).
  • Local capture: references/joelhooks-adr-0077-memory-system-next-phase.md (summary + mapping to @jumperz checklist).
  • Evidence posture: a mix of E5 (ADR planning) and E4 (implementation exists), but key performance claims are mostly self-reported.

Stage 1 — Descriptive (what the ADR proposes)

Baseline system (as described)

The ADR assumes a memory pipeline already running in production since 2026-02-14 with:

  • An Observer extracting structured observations from transcripts on compaction/shutdown.
  • Qdrant as semantic store (reported scale: 1,343 points, 768d embeddings).
  • A Reflector that proposes updates to a canonical MEMORY.md (staged in Redis).
  • A Promotion step that merges approved proposals into MEMORY.md, using a human review surface (Todoist).
  • Auto-triage to reduce review burden (only “needs-review” creates tasks).
  • A Friction job that clusters patterns daily (deployed but not yet producing actionable tasks).
  • A recall tool used mid-session by agents to retrieve memories from Qdrant.
  • Daily append-only logs under ~/.joelclaw/workspace/memory/YYYY-MM-DD.md.

Gap framing

The ADR uses @jumperz’s 31-piece stack as a checklist and highlights gaps mainly in:

  • Retrieval intelligence (score decay, query rewriting, tiered search, inject caps)
  • Storage intelligence (dedup, write gating, strength/sentiment tags)
  • Maintenance (nightly jobs, staleness handling)
  • Feedback loops (echo/fizzle, behavior loop)

The decision: three increments

The ADR proposes three sequential increments:

  1. Retrieval quality (highest impact / lowest risk)
  • Add score decay to retrieval results.
  • Add query rewriting before search.
  • Enforce an explicit inject cap (e.g., 10).
  1. Storage quality (reduce noise)
  • Add dedup at observation time (cosine similarity merge).
  • Add nightly maintenance (merge duplicates, identify orphans).
  • Add staleness tagging (old + unused → deprioritize).
  1. Feedback loop (compounding improvement)
  • Add echo/fizzle tracking (used memories up; ignored down).
  • Formalize a behavior loop (extract corrections/preferences into lessons and load at session start).

Several “advanced” items are explicitly deferred (categories, knowledge graphs, cross-agent, forward triggers, dual search).

Stage 2 — Evaluative (coherence, feasibility, risk)

Coherence: good prioritization, but risk concentrates in “implicit semantics”

The incremental plan is coherent: it targets retrieval first (fast wins), then storage quality, then feedback loops.

The weak point is that each increment depends on semantics that are easy to wave away in an ADR but hard in code:

  • What is a “memory” in Qdrant? Observation granularity matters for dedup, decay, and echo/fizzle.
  • What counts as “used”? Echo/fizzle is only as good as attribution; naïve string overlap is gameable.
  • What constitutes “stale”? “0 recall hits in 90 days” is a retrieval-policy-dependent proxy, not a property of the memory itself.

Evidence quality: self-report + partial grounding

Some aspects are grounded by implementation reality (observer/reflector/promotion exist), but:

  • The impact of score decay / query rewriting is not supported by measured retrieval gains yet.
  • The dedup threshold (e.g., 0.85) and decay constant (e.g., 0.01/day) appear as plausible defaults, not validated constants.

Key risks and likely failure modes

1) Score decay can harm evergreen facts

If decay is applied uniformly, evergreen truths (identity facts, stable preferences, long-term decisions) will be suppressed. Mitigations to consider:

  • Decay on last_accessed or last_confirmed, not created_at.
  • Add an importance/pinning field and apply decay only within a type/category.

2) Query rewriting can regress recall and increase leakage

Query rewriting is valuable, but failure modes are common:

  • Rewriter can omit a crucial proper noun, time constraint, or negation.
  • It adds an LLM call to retrieval (latency/cost).
  • It can leak sensitive context to the rewriter model/provider.

Mitigations:

  • Cache rewritten queries per turn.
  • Keep the original query as a fallback and compare result sets (A/B within a single request).
  • Redact or summarize context sent to the rewriter.

3) Dedup-at-write risks destructive merges

Cosine similarity merges can erase distinctions:

  • Two similar “lessons” can differ in preconditions (“X works only if Y”).
  • “Fresh wording” overwrites provenance and time context.

Mitigations:

  • Restrict dedup to same type/category/entity.
  • Store merges as new records with “supersedes” links (don’t overwrite in place).
  • Keep an audit trail and allow manual un-merge.

4) Staleness tagging is sensitive to retrieval-policy changes

A memory can have zero hits because retrieval is poor, not because it’s stale. Mitigations:

  • Separate “low usage” from “low quality”; keep a “quality suspicion” flag for later review.
  • Use “stale” to mean “unconfirmed” rather than “unused”.

5) Echo/fizzle optimization can create perverse incentives

If the model can increase a memory’s priority by referencing it, it may start name-dropping injected facts. Mitigations:

  • Require explicit citation IDs and use a post-hoc judge for “actually needed”.
  • Penalize “gratuitous citation” patterns.
  • Consider counterfactual evaluation (withheld-memory re-run) for a small sample to calibrate.

What’s missing (for synthesis-grade confidence)

  • A benchmark harness (query set + ground-truth expectations) to measure retrieval changes and prevent regressions.
  • Clear definitions for:
    • memory record types (fact/decision/lesson/etc.),
    • confidence semantics,
    • “stale” vs “archived” vs “pinned”.
  • A privacy/security story for any raw transcript retention (even if “only” in logs).

Stage 3 — Dialectical (steelman + counterarguments)

Steelman

ADR-0077 is a good example of translating a broad conceptual checklist (jumperz) into deployable increments. It respects the reality of an already-running system and picks improvements that can plausibly be shipped without refactoring everything.

Counterarguments

  • If write quality is the primary bottleneck, retrieval improvements may surface more noise, not more signal.
  • Human-in-the-loop promotion via Todoist can become a throughput ceiling as the system scales.
  • Staleness and echo/fizzle are second-order features: if first-order extraction is noisy, these can optimize the wrong objective.

Fit with other systems

This ADR is a “middle path” between:

  • @jumperz’s full blueprint (ANALYSIS-jumperz-agent-memory-stack.md), and
  • a fully productized system like ClawVault (ANALYSIS-versatly-clawvault.md) or OpenClaw’s layered architecture (ANALYSIS-coolmanns-openclaw-memory-architecture.md).

Practical validation plan (recommended next work)

To make ADR-0077 synthesis-ready, measure changes with a minimal harness:

  • Retrieval benchmark: 40–80 natural queries + expected “correct memory IDs” (or file/proposal IDs) with top-K scoring.
  • A/B for query rewriting: run retrieval on (raw query) vs (rewritten query) and compute:
    • recall@K, MRR, and “bad inject rate” (memories injected but irrelevant by judge).
  • Decay study: segment by memory type; report how often evergreen items are suppressed.
  • Dedup audit: sample merges weekly; estimate “false merge rate”.

Claims (for later synthesis / registration)

Claim Type Evidence Credence Layer Actor Scope Quantifier Notes / verification
The joelclaw memory system has been running in production since 2026-02-14 and has proven the core pipeline. [F] E5 0.55 Ops @joelhooks joelclaw “since” Self-report; verified that ADR-0077 contains this statement.
Qdrant contains ~1,343 memory points (768d embeddings) after ~6 days. [F] E5 0.5 Storage @joelhooks joelclaw “~1,343” Self-report; verified that ADR-0077 contains this statement (still not independently validated).
Score decay + query rewriting + inject cap are the highest-impact, lowest-risk next improvements. [H] E5 0.6 Retrieval @joelhooks joelclaw “highest impact” Plausible; needs benchmark to confirm.
Dedup at observation time and nightly maintenance reduce long-term noise. [H] E5 0.55 Maintenance @joelhooks joelclaw “over time” Plausible; false-merge risk requires audit.
Echo/fizzle tracking produces compounding retrieval quality improvements. [H] E5 0.45 Feedback @joelhooks joelclaw “over time” Directionally plausible; attribution and gaming risk are non-trivial.

Corrections & Updates

  • 2026-02-21: Verified key baseline claims and the “three increments” plan against the ADR-0077 web page (production since 2026-02-14; Qdrant 1,343 points; Increment 1–3 text). This is still self-report, but the local summary matches the published ADR.
  • The joelclaw repository code has not been audited in this repo; if we later find code/behavior diverges from the ADR, update this file and record the delta explicitly (what changed, and on what date).