feat(replay): replay-buffer env — continue/recheck/judge tasks from saved rollouts#2941
Draft
faresobeid wants to merge 13 commits into
Draft
feat(replay): replay-buffer env — continue/recheck/judge tasks from saved rollouts#2941faresobeid wants to merge 13 commits into
faresobeid wants to merge 13 commits into
Conversation
…aved rollouts
Turn saved rollouts (<run>/rollouts/step_*/train_rollouts.jsonl) into new
training tasks served by a new v1 taskset plugin, environments/replay_v1
(id "replay-v1"), mixed into training as a normal [[orchestrator.train.env]]
entry with a ratio:
- continue: resume from a context-compaction point — the seed is the
post-compaction prompt recovered from the trace's message graph; scored
by the original (inner) taskset.
- recheck: the finished conversation plus an appended check-your-work turn;
scored by the original taskset via a task-swapped trace view.
- judge: a rendered (truncated, label-balanced) transcript and "was this
correct?"; reward = verdict matches the reward the rollout received.
Buffers index (path, offset) handles and bind tasks lazily through the new
verifiers Taskset.resolve_task hook; offline buffers bind deterministically
per index, online buffers (buffer_dir = "self") rescan barrier-complete
steps and sample fresh per request, forcing whole-group dispatch so GRPO
groups share one source rollout.
Rollout.to_record() now stamps {kind, env_name, group_id, policy_version,
is_filtered} into info["prime_rl"] so saved rollouts stay attributable;
the orchestrator resolves the "self" buffer sentinel before env servers
spawn (children never learn the output dir).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- tests/unit/test_replay_v1.py: trace surgery on a synthetic forest,
verdict parsing, config validation, buffer scan/barrier/balance/pick.
- docs/replay.md (+ mint.json index, configuration.md pointer; also fixes
the Environments example that set a ratio on only one env, which the
all-or-none validator rejects).
- skills/configs + skills/training/monitor-run: replay env config surface
and the info.prime_rl rollout attribution stamp.
- configs/debug/replay/{offline_recheck,online_judge}.toml + README, both
validated with `rl --dry-run`.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Label balancing is now a derived view over the retained index, not a destructive rewrite: majority-label candidates dropped from one balanced view pair up in later ones (online rescans no longer bleed the judge buffer down to the earliest pairs). - Replay-derived records (task carries kind + source_task) are never candidates — an online "self" buffer would otherwise judge its own judge rollouts, a compounding feedback loop; the shipped online example also pins env_name. - Empty online buffers fail requests after a bounded wait instead of blocking forever — blocked groups were hoarding dispatch permits before step 0's barrier existed, which could deadlock the run. - Index swaps are atomic (rescan threads no longer mutate the list the event loop samples from); online sampling mixes the worker pid into the rng; vanished source files (a resumed producer run cleaning future steps) discard the candidate and redraw. - buffer_dir="self" resolution also covers eval envs, and the taskset errors clearly when the sentinel reaches it unresolved (standalone servers). - parse_verdict ignores lines quoting both verdict options (an echoed instruction is not an answer). - docs: startup fail-and-retry semantics, self-replay exclusion, allow_container runtime requirement, shared-toolset failure mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ained derivations Replace the single env_name filter and the hard never-replay-a-replay rule with one mechanism: source_envs, an allowlist over the info.prime_rl env-name stamp. Unset, it replays every env except replay envs (the old ban, now a default); listing env names replays exactly those — and naming a replay env is the deliberate opt-in for chained derivations, so depth is expressed as env topology (recheck-2 sources [recheck-1]) instead of a counter. Chained records nest their lineage; scoring, tools, and container provisioning always resolve to the innermost original task (unwrap_source_task). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Buffer: rescans prune already-scanned and out-of-window steps by dirname before any file stat (was ~2 stats per historical step every 10s); the scan early-breaks once a pass gathers max_candidates — newest-first order makes everything older permanently evictable — so a cold online buffer no longer reads a resumed run's entire history; no-op rescans skip the sort/rebalance; the balanced view is derived in one place (_set_index); timing knobs become module constants; the rescan lock is built eagerly. Taskset: resolve reads+materializes off the event loop in one thread; the rebuilt inner task is validated once per resolve and cached on the task (was revalidated by every per-rollout hook); mode/inner conditions collapse to `self.inner is None` (config validation ties them); continue skips the graph walk it never used. Also: is_replay_derived tightened to the actual lineage shape (an unrelated taskset with a `kind` field is no longer silently excluded), and the orchestrator's "self" comment states the real constraint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ess, not null Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…h varying num_slots (online sampling ignores the index — any size works), seed (each pool worker seeds from OS entropy; cross-worker determinism never existed), balance_labels (unbalanced judge reward is gameable; always on, degrades with a warning on one-sided data), max_message_chars (a rendering detail — the total transcript budget is the real knob), and max_candidates (a safety cap on 60-byte handles; now an internal constant). 16 fields -> 11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…_steps_back
Restructure the config so knobs exist only where they mean something:
- `mode` + flat per-mode fields become a `derivation` discriminated union
({type = "continue" | "recheck" | "judge"}): `inner`/`allow_container`
live on the delegating derivations, `instruction`/`success_threshold`/
`max_transcript_chars` on the derivations that use them. The judge-
forbids-inner and recheck-requires-inner validators disappear — judge's
config has no `inner` field and the delegating ones require it.
- `stop_conditions` becomes the first entry type of an opt-in `filters`
list (absent unless you add it) — the landing spot for future selection
ideas (recency, reward bands, group-relative) without new core fields.
- `online` is auto-set for buffer_dir = "self" (it must be pinned before
the orchestrator rewrites the sentinel); explicit use remains only for
watching another still-running run's dir.
- `max_steps_back` is removed: the buffer's capacity (newest-first scan +
cap) is the eviction policy, as in any replay buffer.
- Online buffers advertise a single virtual task instead of 1024 — the
index is ignored by sampling, so 1 is the honest count (replay envs need
explicit ratios regardless; documented).
- Judge per-message truncation derives from the total budget (1/20).
Core config: buffer_dir, derivation, source_envs, filters, online.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Source selection beyond source_envs is not core config: a derivation that wants it (e.g. recheck only agent_completed sources) owns that choice in its own code/config when derivations become user-extensible. Core config is now buffer_dir, derivation, source_envs, online. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…echeck Drop the judge derivation for now — with it go verdict parsing, transcript rendering, label balancing, success_threshold, and the balanced-view machinery (~150 lines). Both remaining derivations delegate to a now- required `inner`, so the delegation guards simplify too. Continue gains `anchor = "compaction" | "tool-call"`: tool-call anchoring resumes right after a tool result, at points where every issued call has its result (no dangling calls in the seed); tool calls are numerous, so one resume point per source rollout is sampled deterministically (same choice across pool workers). The debug online example becomes online_recheck.toml. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e thin packages Following the harbor/textarena pattern, the buffer, trace surgery, and delegation machinery move to verifiers.v1.tasksets.replay (with their tests), and each derivation is now a thin subclass package binding its own flat config: - environments/replay_continue_v1 (id "replay-continue-v1"): anchor = "compaction" | "tool-call" (~45 lines) - environments/replay_recheck_v1 (id "replay-recheck-v1"): instruction (~30 lines) The derivation union disappears from config — the derivation IS the taskset id, `inner` is a required base field, and a new derivation is a new small package implementing record_anchors + build_prompt. Replay- derived detection becomes structural (source_task + source_id lineage keys), so it holds for any future derivation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Legacy envs already get max_seq_len injected at config-resolve time; v1 envs had no equivalent, so any long-prompt env (replayed conversations especially) needed hand-set token budgets kept in sync with seq_len. resolve_env_config now defaults an unset max_total_tokens to seq_len for v1 train envs — rollouts stop at the training budget instead of producing samples that can't fit. Explicit values (including "None" = unlimited) are respected; eval envs are untouched. Drops the budget boilerplate from the replay debug configs and docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Whole-rollout derivations (recheck) now implement a single hook, build_prompt; only mid-rollout derivations (continue) override record_anchors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns saved rollouts into new training tasks. Every run already writes each rollout to
<output_dir>/rollouts/step_N/train_rollouts.jsonl; replay envs index those files as a buffer and serve derived tasks from them, mixed into training as normal[[orchestrator.train.env]]entries with aratio.Depends on PrimeIntellect-ai/verifiers#1919 (submodule pinned to its tip), which now hosts the shared machinery as
verifiers.v1.tasksets.replay— a base taskset in the harbor/textarena mold. This PR contributes the derivation packages, the orchestrator/persistence glue, docs, and configs.The derivations (thin packages over the verifiers base)
replay-continue-v1anchor = "compaction"; one task per compaction) or right after a tool result (anchor = "tool-call"; one deterministically sampled resumable point per source — only points where every issued call has its result).replay-recheck-v1Each is ~30-45 lines: a flat config subclass + two hooks (
record_anchors,build_prompt). All derivations are scored by the requiredinner(original) taskset — inner rewards land with their original names and weights via a task-swapped trace view; tools/setup/finalize delegate too. Each drawn source spawns a fresh GRPO group. Adding a new derivation = a new small package (see docs/replay.md "Writing a New Derivation").Buffers are offline (a prior run's rollouts, indexed once, deterministic per task index) or online (
buffer_dir = "self", inferred online: rescans barrier-complete steps, samples fresh per request, forces whole-group dispatch so the group shares one source).source_envsselects sources by the newinfo.prime_rl.env_namestamp; by default replay-derived records (structural lineage check) are skipped, and explicitly naming a replay env is the opt-in for chained derivations (recheck-a-recheck as env topology, scoring always unwrapping to the innermost original task).Core changes outside the packages
Rollout.to_record()stamps{kind, env_name, group_id, policy_version, is_filtered}intoinfo["prime_rl"]so saved rollouts stay attributable (round-trips throughWireTrace.model_validate).buffer_dir = "self"sentinel to<output_dir>/rolloutsbefore env servers spawn (train + eval envs).max_total_tokensnow defaults toseq_lenat config-resolve time (v1 parity with the legacymax_seq_leninjection) — long-prompt envs like replay no longer need hand-set token budgets; explicit values are respected.resolve_task, forced group dispatch, large-Messages-prompt file handoff).Design notes
(file, offset)handles (~1MB lines are never held in memory). Online buffers advertise a single virtual task (sampling ignores the index), so replay envs need explicit ratios — the all-or-none rule makes that loud.allow_container) until sandbox snapshotting lands.buffer_dir, requiredinner,source_envs,online-usually-inferred,allow_container) + the package's own knob.Verification
tests/v1/test_replay.py: surgery on synthetic forests, buffer scan/barrier/eviction/pick determinism, lineage skipping) + 8 package tests here (loader resolution, config validation, deterministic tool-call anchor sampling).configs/debug/replay/) passrl --dry-run.configs/debug/replay/online_recheck.toml, 20 steps): env ready with 1 virtual task + forced group dispatch; empty-buffer fail-fast freed permits with no deadlock (its error markers saved to step_0 as designed); 176 recheck rollouts across steps 6-15, zero errored, scored by the inner verifier (lcs, real reward distribution) withreplay/source_reward/source_stepmetrics; seeded prefixes fully masked, only the sampled revision trainable; no recheck-of-recheck (lineage exclusion held); trainer completed 20/20 clean.🤖 Generated with Claude Code