Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents

TL;DR

Introduces LoCoMo-Plus, a benchmark and evaluation framework targeting beyond-factual “cognitive memory”: retaining and applying latent constraints (state/goals/values/causal context) across long conversations even when later queries are not semantically similar to the original cue (“cue–trigger semantic disconnect”).
Argues existing conversational memory benchmarks (including LoCoMo-style factual QA) over-index on explicit factual recall and surface-form scoring.
Proposes a constraint-consistency evaluation framing: instead of a single gold response, a response is correct if it lies in the set of outputs consistent with an induced constraint.
Uses LLM-as-a-judge for constraint consistency and reports agreement/stability checks vs human annotators and across judge backbones.

Level-2 cognitive memory: evaluates implicit behavioral constraints, not just recallable facts.
Cue–trigger construction: generates cue dialogues and later trigger queries with intentionally low semantic similarity.
Evaluation protocol shift: from overlap metrics (“did you match the reference?”) to behavioral validity (“are you consistent with the constraint?”).

Generate “cue dialogues” that imply a latent constraint (e.g., diet goal, relationship value, state).
Human verification to keep only memory-worthy cues (persistent, behaviorally constraining, non-trivial).
Construct trigger queries that require applying the cue but remain underspecified in isolation.
Apply semantic filtering + a “memory elicitation” check to ensure the trigger genuinely requires recalling the cue.
Embed validated cue–trigger pairs into long LoCoMo trajectories with a specified time gap.

LoCoMo-Plus decomposes cognitive memory constraints into four components:

Highlights evaluation artifacts in older protocols:
- prompt bias when explicitly disclosing task type to the model,
- length bias in overlap metrics (models penalized/favored by style/length, not semantic validity).
Quantifies judge reliability:
- agreement between two human annotators and an LLM judge,
- stability across different judge backbones.
Reports a qualitative “length sensitivity” result:
- object memory is robust,
- episodic memory decays steadily,
- cognitive memory collapses rapidly with long-context interference.

“Memory” that matters for user experience is often constraint-shaped, not fact-shaped.
Systems need to represent and retrieve latent constraints (values/goals/state) and apply them consistently across time.
For safety-first systems, “cognitive/procedural memory” is instruction-shaped; it needs stricter governance than factual memory.