feat: train on replayed rollouts — continue (compaction/tool-call) + recheck envs

[faresobeid]

What

Train on replayed rollouts: point a train env at saved rollout records (<output_dir>/rollouts/step_*/*_rollouts.jsonl) and each record re-enters training as a fresh task, scored by the original env's verifier with its real toolkit. Zero orchestrator changes — the whole feature is the new built-in verifiers v1 replay taskset (PrimeIntellect-ai/verifiers#1933, which this PR's submodule bump pins), driven purely by env config:

# Continue: resume each recorded compaction from its handoff summary. String prompt →
# any harness, including rlm; use the source run's harness config.
[[orchestrator.train.env]]
name = "my-env-continue"
taskset = { id = "replay", records = "/data/run1/rollouts/step_*/train_rollouts.jsonl", mode = "continue", source = { id = "my-env-v1" } }
harness = { id = "rlm" }

# Recheck: replay each finished attempt with a "check your work" turn appended.
# Full-conversation seed → needs the default/null harness.
[[orchestrator.train.env]]
name = "my-env-recheck"
taskset = { id = "replay", records = "/data/run1/rollouts/step_*/train_rollouts.jsonl", mode = "recheck", source = { id = "my-env-v1" } }
harness = { id = "null" }

• Continue (anchor = "compaction" default, or "tool-call") trains dense mid-trajectory credit: a failed 40-turn rollout that gave one sparse 0.0 becomes a fresh GRPO group seeded exactly at the state where the policy was struggling — the compaction variant additionally trains working from a handoff summary. Compaction restarts are detected structurally from the trace's branch graph (no compaction-prompt matching), so records from different harnesses mix freely.
• Recheck trains self-verification: the model sees its finished attempt plus a verification turn, and the original verifier scores the revision. Seeded context is loss-masked and invisible to last_reply-style rewards.

Group rollouts of one seeded task form a regular GRPO group; uniform-outcome groups drop via the existing zero-advantage filter.

Changes in this repo

• deps/verifiers bump → the replay taskset + the framework seams it needs (see the verifiers PR for the shared-surface details: Taskset capability/discovery seams, Messages-prompt file transport for default/null).
• docs/configuration.md: new "Replaying Past Rollouts (Continue / Recheck)" section (worked TOML, knobs, harness-support matrix, glob-freezing and mixed-file caveats).
• skills/configs/SKILL.md: replay-env config pattern.

Notes for review

• records globs are frozen to concrete files at config validation so every env-server pool worker builds the identical task list — point them at a finished run's records, not a live producer's output dir.
• Sandbox state is not replayed on re-entry (fresh setup); the snapshot seam exists in the taskset (records carrying trace.info["snapshots"] refs restrict resume points to snapshotted anchors and restore during setup) but no runtime implements capture yet.
• Other recycling schemes are a subclass away: override ReplayTaskset.seeds(trace).

Verification

• Verifiers side: 18 unit tests + full local suite green (87 passed; details in feat(v1): replay taskset — re-enter saved rollouts as continue/recheck tasks verifiers#1933).
• prime-rl config path exercised end-to-end: TrainEnvConfig narrows taskset → ReplayConfig and nested source → the source taskset's config type; recheck+anchor conflicts rejected; resolved-config dump→re-validate round trip (the env-server spawn path) is clean; Environment construction under the rlm harness passes the capability gates.
• Not yet done (next steps, tracked): a load_tasks pass over real rlm prod records (the /beegfs eval files were rotated away mid-development) and a first live GRPO run with a replay env.

🤖 Generated with Claude Code

Update: online self-replay

The replay env can now feed on the run's own records as they are produced (no prior run needed): point records at this run's rollouts/step_*/train_rollouts.jsonl. Orchestrator side: TrainSource.refresh() re-polls env info at each ship (one in-flight refresh, off the ship path) and extends the sampling cycle with new task indices; save_rollouts writes tmp+rename so a follower never reads a half-written step file; next_example tolerates a follow env with no tasks yet (records appear with the first steps).

Update: per-env records + live validation

Train rollout records are now written per env (train_rollouts_.jsonl, mirroring eval). Replay adds min/max_source_reward and max_anchors knobs. Three live debug runs validate the loop end-to-end: online recheck (17.6k tasks mined), tool-call continue (mid-conversation seeds through the file transport, 0 errors, 100 steps), and incorrect-only recheck (repair rate improved 31.4%→36.9% early→late while base env rose 0.75→0.84). Debug configs: configs/debug/v1/gsm8k_recheck*.toml, gsm8k_continue_tool.toml.

Update: rollout record index

save_rollouts now returns each line's (offset, length) byte span, and finalize_train_batch writes a derived index_<env>.jsonl beside every train_rollouts_<env>.jsonl (same tmp+rename atomicity): one row per record with trace id, task idx/name, reward, policy version, step, branch/turn counts, stop condition, and the record's byte span. The replay taskset (verifiers bump) filters records index-side and parses only the selected spans — in a synthetic 500-record file with a 10% pass rate, index-selected loading parsed 10% of the bytes. Record files stay byte-identical; the index is optional and derived — a record source without one is fully parsed as before. Docs: docs/configuration.md replay section + configs skill.

Update: cleanup pass

Record + index writing moved into one save_train_records helper (orchestrator/utils.py): per-env to_record dump, records file, index rows via the replay taskset's index_row/index_path contract, and index file all run in a single asyncio.to_thread per env (gathered across envs), taking to_record off the event loop. save_rollouts stays the generic atomic-JSONL primitive. Also fixes the stale launch comment in gsm8k_recheck_incorrect.toml and the record glob in the configs skill + docs (*_rollouts_*.jsonl). Verifiers bumped to the consolidated iter_selected_records API.