| title | Analysis — Nested Learning (Behrouz et al., 2025) | ||
|---|---|---|---|
| date | 2026-02-22 | ||
| type | analysis | ||
| paper_id | arxiv:2512.24695 | ||
| paper_title | Nested Learning: The Illusion of Deep Learning Architectures | ||
| source |
|
||
| related |
|
Nested Learning (NL) is not an “agent memory service” paper (no vector DB / RAG pipeline), but it is very relevant to memory systems because it treats memory as update dynamics: which parts of a system change, on what schedule, with what consolidation behavior. The Continuum Memory System (CMS) is especially useful as a conceptual template for multi-timescale consolidation and “corrections without erasing history”.
- Problem: Modern LLMs are mostly static after training; they adapt only via in-context learning and struggle with continual learning/memorization without expensive retraining or external components.
- Core idea: Reframe learning as nested optimization problems with different context flows and update frequencies (“nested learning”), yielding higher-order in-context learning and a path toward continual learning.
- Memory contribution: Replace the binary “short-term vs long-term memory” framing with a Continuum Memory System (CMS): multiple memory blocks updated at different frequencies.
- Self-modification: Proposes self-referential/self-modifying Titans where multiple components can adapt in-context, including signals analogous to learning rate and decay.
- Hope: Combines self-modifying Titans + CMS into a continual learning module (“Hope”) with reported gains on continual learning and long-context tasks (as reported).
- Main caveat: For
shisad, this is mainly conceptual guidance (consolidation semantics + scheduling), not a drop-in implementation for external memory storage.
NL treats a “model” as a set of nested learning modules where each module:
- has its own parameters/state,
- has its own optimization/update process,
- and sees its own context flow (data stream).
From this view:
- classical deep learning compresses information into parameters via its context flow (training data),
- in-context learning emerges naturally in large models, and
- continual learning requires architectural/algorithmic designs that support updates at multiple time scales without catastrophic forgetting.
CMS formalizes memory as a chain of MLP blocks {MLP^(f_1), …, MLP^(f_k)} whose parameters update at different frequencies:
- The forward pass is a composition of these blocks.
- Each block’s parameters are updated every
C^(ℓ)steps, with “no update” in between.
The paper outlines several variants that are useful conceptual knobs:
- Nested CMS: higher-level blocks’ initial states are meta-learned in lower levels; blocks can be re-initialized at the end of their context flow. This is framed as enabling higher-order in-context learning.
- Sequential CMS: levels are sequentially composed but initial states are jointly meta-learned in the lowest-frequency level, tying long-term knowledge across levels.
- Independent/head-wise CMS: multiple blocks process the same input in parallel (different frequencies) and an aggregator combines outputs (weighted sum as a simple case).
Why this matters for memory: the authors explicitly argue that if one block forgets (due to being updated), other blocks at different frequencies can still retain the knowledge, and cross-level transfer can reintroduce it.
The paper critiques Transformers as having static projections/MLPs within a context; it proposes “self-referential Titans” where:
- components that produce keys/values/queries (and related signals) can themselves be adaptive memories updated in-context,
- and the update dynamics can be more expressive (including chunk-wise parallelizable updates).
This is a very different “memory” axis than external RAG memory: it is about internal parametric/state updates as a kind of associative memory.
Hope combines:
- self-modifying Titans (smaller capacity but more expressive learning/update rule), and
- CMS (larger capacity memory with simpler rules, spread across time scales).
The paper also discusses initializing CMS blocks from pretrained model weights and controlling internal learning rates to trade off “stay close to pretrained” vs “adapt”.
The paper reports experiments across:
- continual learning settings (e.g., class-incremental text classification),
- long-context understanding benchmarks,
- and language modeling / knowledge incorporation tasks.
For our purposes, the key takeaway is not any single metric, but that they attempt to evaluate:
- continual update behavior,
- long context retention,
- and transfer across tasks.
- Different memory substrate: NL/CMS/Hope focus on internal updates; external agent memory (like shisad) is explicitly about storing/retrieving artifacts outside model weights.
- Operationalization gap: the paper provides a strong conceptual framing, but external systems still need:
- explicit schemas,
- retrieval correctness guarantees,
- poisoning defenses,
- and auditability across tenants.
- Consolidation semantics are underspecified for products: CMS says “knowledge remains in other levels” but doesn’t directly tell you what to persist, when, and how to represent corrections for user-facing memory.
The most valuable transfer is to treat external memory as a continuum of stores with different update frequencies, rather than a single bucket:
- high-frequency: per-turn scratch/working notes (high churn; low trust)
- medium-frequency: per-session episodic summaries/events (stable across a day/week)
- low-frequency: user model / preferences / “identity” memory (rare updates; high trust)
- very-low-frequency: governance/policies/procedural rules (updates via explicit review)
And crucially: when something is corrected, don’t “delete”; move it across time scales (preserve history) and mark validity/recency.
Concrete additions to consider in shisad’s roadmap (conceptual primitives, not implementation details):
- Make “consolidation schedule” first-class: define memory tiers by update frequency and allowed ops (append/supersede/invalidate).
- Encode “corrections without forgetting”:
- preserve prior beliefs as historical entries,
- store a supersedes chain,
- maintain validity intervals (aligns strongly with Zep/Graphiti).
- Treat write-policy evolution as procedural memory:
- store extraction/update prompts and their revisions,
- keep change history and tie updates to evaluation regressions.
- v0.7 core memory overhaul: implement multi-tier memory with explicit consolidation jobs (daily/weekly) and versioned corrections.
- later: explore learned write policies / adaptive gating (a “self-modifying write policy” analogue), but only after benchmarks and safety constraints exist.
- Capture date: 2026-02-22.
- Paper version reviewed: arXiv v1 (2025-12-31).