feat(v1): multi-agent topologies — imperative go(), agent graphs, deferred cross-agent rewards#1941
feat(v1): multi-agent topologies — imperative go(), agent graphs, deferred cross-agent rewards#1941eligotts wants to merge 16 commits into
Conversation
…erred cross-agent rewards A Topology composes episodes (one agent x one task -> one trace) into a multi-agent interaction: plain imperative control flow in go() (loops = rounds, gather = fan-out, awaiting = fan-in), trace->task forward arrows, and judgement declared as @vf.reward(agent=...)/@vf.metric(agent=...) methods run over the finished instance. Each instance persists as one AgentGraph record (parent-linked traces, nested). Core: task-first contract (all episode behavior on the Task class; Taskset = pure factory), Agent = harness + routing only (per-agent model/client/sampling/trainable), pinned-harness AgentConfig subclassing with deep-merged partial overrides, seed slot XOR with load_tasks overrides, TopologyRunner as the run-lifetime resolver, in-process `direct` harness, judge plugin tier deleted in favor of the vf.Judge single-call utility + judge-as-agent topologies. Ships four instantiations, all verified live (deepseek/deepseek-v4-flash): - llm-judge (built-in): any taskset, solver + fixed direct-harness judge - agentic-judge (built-in): full solver trace uploaded into a real judge agent's runtime (verified on harbor in a prime sandbox) - proposer-solver-v1: proposer invents code-checkable puzzles via a submit tool; tool-less direct solvers race them; difficulty reward - writer-editors-v1: rounds + fan-in; one vf.Judge verdict rewards every trace Hardening from the live runs: markdown-tolerant verdict parsing, PEP 723 header for model-submitted scripts, bounded MCP connect in the null program (was a silent 0-turn hang), retried-away errors parked in trace.info, shared tool servers under topologies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restore main's agent-guidance docs (AGENTS.md, assets/agents, assets/lab root) and the eval dashboard pager (our branch had clobbered #1912's origin-anchored paging); drop a stray scratch file. environments/AGENTS.md and assets/lab/environments/AGENTS.md remain regenerated because main's checked-in copies are stale relative to their source (docs/environments.md lost the "v1 Env Shape" section; scripts/sync.py output enforced by pre-commit). API docs (GUIDE/README/ARCHITECTURE) stay updated to match the shipped contracts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # verifiers/v1/GUIDE.md # verifiers/v1/cli/output.py # verifiers/v1/cli/validate.py # verifiers/v1/judge.py # verifiers/v1/taskset.py
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| if len(rewards) < 2 | ||
| else await asyncio.gather(*(invoke(fn, available) for fn in rewards)) | ||
| ) | ||
| for fn, scores in zip(rewards, results): |
There was a problem hiding this comment.
🟡 Medium v1/task.py:232
Task.score_group() zips traces with each group reward's scores without checking that the two lists have the same length. When a @group_reward returns fewer scores than traces, the trailing traces silently get no reward from that function; when it returns too many, the extra scores are silently discarded. This produces incorrect rewards without any error, so the mismatch is easy to miss. Consider asserting len(scores) == len(traces) before the zip and raising a TaskError on mismatch.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/task.py around line 232:
`Task.score_group()` zips `traces` with each group reward's `scores` without checking that the two lists have the same length. When a `@group_reward` returns fewer scores than traces, the trailing traces silently get no reward from that function; when it returns too many, the extra scores are silently discarded. This produces incorrect rewards without any error, so the mismatch is easy to miss. Consider asserting `len(scores) == len(traces)` before the zip and raising a `TaskError` on mismatch.
| if trace.stop_condition is not None: | ||
| break | ||
| raise | ||
| message = completion.choices[0].message |
There was a problem hiding this comment.
🟡 Medium direct/harness.py:67
When the interception server injects hidden user-simulator turns, DirectHarness.launch drops that conversation history before the tool-call bounce retry. The loop only appends completion.choices[0].message to its local messages list, so on the next iteration the model answers against an incomplete prompt — missing the user-sim turns the server already added — and produces incorrect recovery responses. Consider reconstructing messages from the full trace (or the server's response) rather than only the final assistant message, so the bounce retry preserves the prior conversation context.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/direct/harness.py around line 67:
When the interception server injects hidden user-simulator turns, `DirectHarness.launch` drops that conversation history before the tool-call bounce retry. The loop only appends `completion.choices[0].message` to its local `messages` list, so on the next iteration the model answers against an incomplete prompt — missing the user-sim turns the server already added — and produces incorrect recovery responses. Consider reconstructing `messages` from the full trace (or the server's response) rather than only the final assistant message, so the bounce retry preserves the prior conversation context.
| TopologyError, f"topology {self.topology.config.id!r} scoring" | ||
| ): | ||
| await self.topology.score(run.graph) | ||
| except TopologyError as e: |
There was a problem hiding this comment.
🟡 Medium v1/topology.py:566
When a @reward or @metric method in Topology.score raises, run_instance catches the TopologyError and records graph.error, but traces that were already mutated by earlier scoring methods keep their partial rewards and metrics. The persisted AgentGraph for a failed instance ends up with a silently incomplete, inconsistent set of topology scores that downstream code can mistake for complete judgement.
This happens because score mutates traces in place as it iterates, and run_instance has no rollback of those mutations when a later scoring method crashes. Consider either scoring into a side structure and committing only after all methods succeed, or clearing partial topology rewards/metrics on the traces when the TopologyError is caught.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/topology.py around line 566:
When a `@reward` or `@metric` method in `Topology.score` raises, `run_instance` catches the `TopologyError` and records `graph.error`, but traces that were already mutated by earlier scoring methods keep their partial rewards and metrics. The persisted `AgentGraph` for a failed instance ends up with a silently incomplete, inconsistent set of topology scores that downstream code can mistake for complete judgement.
This happens because `score` mutates traces in place as it iterates, and `run_instance` has no rollback of those mutations when a later scoring method crashes. Consider either scoring into a side structure and committing only after all methods succeed, or clearing partial topology rewards/metrics on the traces when the `TopologyError` is caught.
| taskset = vf.load_taskset(config.taskset) | ||
| # `WireTask` reads any taskset's saved task without a runtime or its Task type (see WireTask). | ||
| traces = read_traces(source, Trace[WireTask, state_cls(type(taskset))]) | ||
| task_cls = vf.task_type(config.taskset.id) |
There was a problem hiding this comment.
🟡 Medium cli/replay.py:62
--taskset.* overrides passed to replay are silently ignored: the run re-scores with the original task deserialized from results.jsonl instead of the taskset built from the (possibly overridden) config. A replay with changed judge settings produces the old scoring, so users believe their overrides took effect when they didn't. This happens because run_replay now calls await trace.task.score(trace) on the deserialized trace.task, bypassing vf.load_taskset(config.taskset) entirely. Consider scoring through a taskset built from config.taskset so config overrides apply, or document that replay ignores --taskset.* overrides if that is intentional.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/cli/replay.py around line 62:
`--taskset.*` overrides passed to `replay` are silently ignored: the run re-scores with the original task deserialized from `results.jsonl` instead of the taskset built from the (possibly overridden) config. A replay with changed judge settings produces the old scoring, so users believe their overrides took effect when they didn't. This happens because `run_replay` now calls `await trace.task.score(trace)` on the deserialized `trace.task`, bypassing `vf.load_taskset(config.taskset)` entirely. Consider scoring through a taskset built from `config.taskset` so config overrides apply, or document that replay ignores `--taskset.*` overrides if that is intentional.
| raise ValueError( | ||
| f"topology @{kind} {fn.__name__!r} declares no agent scope; " | ||
| f"topology judgement is per-agent — use @vf.{kind}(agent=...)" |
There was a problem hiding this comment.
🟡 Medium v1/topology.py:292
A subclass that overrides an inherited @vf.reward/@vf.metric method with an undecorated replacement fails at load time with a spurious scope error for the base-class method — even though score() would never run it. The validation in agents iterates every class in type(self).__mro__ and every entry in vars(klass), so it processes the base class's decorated method without checking whether a subclass overrides that name. discover_decorated, used by score(), follows normal MRO override semantics and would skip the shadowed base method. Consider skipping names overridden by a subclass during validation.
+ seen = set()
for klass in type(self).__mro__:
for fn in vars(klass).values():
+ name = getattr(fn, "__name__", None) or getattr(getattr(fn, "__func__", None), "__name__", None)
+ if name in seen:
+ continue
+ if name is not None:
+ seen.add(name)
fn = getattr(fn, "__func__", fn)Also found in 1 other location(s)
verifiers/v1/env.py:201
validate_pairing()treats any override ofTask.load_toolsorTask.load_useras meaning every instance of that class requires MCP or a user simulator. Because these methods are instance methods and may legitimately return[]/Nonebased on task data,Environment.__init__can now reject an otherwise valid eval before any tasks are loaded. A task class with conditional tools/user support will raise here even when the actual tasks in this run do not use those features.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/topology.py around lines 292-294:
A subclass that overrides an inherited `@vf.reward`/`@vf.metric` method with an undecorated replacement fails at load time with a spurious scope error for the base-class method — even though `score()` would never run it. The validation in `agents` iterates every class in `type(self).__mro__` and every entry in `vars(klass)`, so it processes the base class's decorated method without checking whether a subclass overrides that name. `discover_decorated`, used by `score()`, follows normal MRO override semantics and would skip the shadowed base method. Consider skipping names overridden by a subclass during validation.
Also found in 1 other location(s):
- verifiers/v1/env.py:201 -- `validate_pairing()` treats any override of `Task.load_tools` or `Task.load_user` as meaning *every* instance of that class requires MCP or a user simulator. Because these methods are instance methods and may legitimately return `[]`/`None` based on task data, `Environment.__init__` can now reject an otherwise valid eval before any tasks are loaded. A task class with conditional tools/user support will raise here even when the actual tasks in this run do not use those features.
| ) | ||
| if retry is not None: | ||
| return await run_with_retry(attempt, retry) | ||
| return await attempt.run() |
There was a problem hiding this comment.
🟡 Medium v1/agent.py:208
When Agent.run is called with both a borrowed runtime and a retry config, each retry attempt reuses the same runtime instance without resetting its state. A failed attempt can leave files, processes, or other side effects behind, so subsequent retries execute against contaminated state rather than starting fresh. This means run_with_retry can produce a passing trace that reflects leftover state from a previous failed attempt, or fail again for the same reason, instead of giving each retry a clean environment.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/agent.py around line 208:
When `Agent.run` is called with both a borrowed `runtime` and a `retry` config, each retry attempt reuses the same runtime instance without resetting its state. A failed attempt can leave files, processes, or other side effects behind, so subsequent retries execute against contaminated state rather than starting fresh. This means `run_with_retry` can produce a passing trace that reflects leftover state from a previous failed attempt, or fail again for the same reason, instead of giving each retry a clean environment.
| the config's `taskset` slot (`--topology.taskset.id <id>`); override for a | ||
| topology that constructs its own seeds.""" | ||
| if not self.config.taskset.id: | ||
| raise ValueError( |
There was a problem hiding this comment.
🟡 Medium v1/topology.py:329
A @staticmethod decorated @vf.reward(agent=...) or @vf.metric(agent=...) method passes the load-time validation in Topology.agents but is silently skipped during Topology.score: the traces it should have scored persist without the declared metric/reward, producing incorrect training/eval results with no error. score discovers methods via discover_decorated, which filters by inspect.ismethod, and a @staticmethod on an instance is a plain function, not a bound method — so it is filtered out. The validation in agents unwraps __func__ and catches static methods, but score does not apply the same unwrapping, so the two paths disagree. Consider making discover_decorated (or its call in score) unwrap __func__ the same way the validation does.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/topology.py around line 329:
A `@staticmethod` decorated `@vf.reward(agent=...)` or `@vf.metric(agent=...)` method passes the load-time validation in `Topology.agents` but is silently skipped during `Topology.score`: the traces it should have scored persist without the declared metric/reward, producing incorrect training/eval results with no error. `score` discovers methods via `discover_decorated`, which filters by `inspect.ismethod`, and a `@staticmethod` on an instance is a plain function, not a bound method — so it is filtered out. The validation in `agents` unwraps `__func__` and catches static methods, but `score` does not apply the same unwrapping, so the two paths disagree. Consider making `discover_decorated` (or its call in `score`) unwrap `__func__` the same way the validation does.
| ) -> Trace: | ||
| """Run this agent on `task` once and return its trace.""" | ||
| ctx = ctx or self.ctx | ||
| services = services if services is not None else self._services |
There was a problem hiding this comment.
🟡 Medium v1/agent.py:192
Calling Agent.run(task) without entering the agent via async with or passing services= leaves services as None, so any task that declares a shared tool server raises "task declares a shared tool server but no shared-server registry is serving" inside Rollout._shared_urls(). This breaks the documented standalone run(task) path: services is only populated from self._services, which stays None unless the caller separately entered the agent or supplied services. Consider provisioning a RunServices scope inside run() when none is available, or documenting that the caller must enter the agent context before calling run() for shared-server tasks.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/agent.py around line 192:
Calling `Agent.run(task)` without entering the agent via `async with` or passing `services=` leaves `services` as `None`, so any task that declares a shared tool server raises `"task declares a shared tool server but no shared-server registry is serving"` inside `Rollout._shared_urls()`. This breaks the documented standalone `run(task)` path: `services` is only populated from `self._services`, which stays `None` unless the caller separately entered the agent or supplied services. Consider provisioning a `RunServices` scope inside `run()` when none is available, or documenting that the caller must enter the agent context before calling `run()` for shared-server tasks.
…ns, close test gaps The Agent primitive stays public but sheds its duplicated ownership half: RunServices is the ONE context manager that owns pooled interception + shared MCP servers, and an Agent only ever borrows them (ctor-injected). Deleted: Agent.__aenter__/__aexit__/_owned_services/multiplex, the per-call run(services=)/run(ctx=) params, and the speculative public rollout() (folded into the retry attempt). A bare Agent(harness, ctx) .run(task) remains fully standalone via per-rollout interception — verified live. Supporting cleanups: - Rollout takes the one resolved `timeouts: TimeoutConfig` instead of four kwargs (env.py and agent.py stop unpacking by hand). - retries.run_with_retry is typed against a Runnable protocol, so the agent attempt no longer duck-types where Rollout was hinted. - The `Agent as ExecutableAgent` import alias is gone; TopologyAgent drops its dead ctx= plumbing and provision() gains its docstring and return annotation. - vf.DirectAgentConfig / vf.NullAgentConfig replace three identical per-example pin subclasses; llm_judge exposes the one SCORE-line parser (writer-editors' copy deleted); vf.RunServices exported. - shared-runtime example: defensive handoff read (a failed writer episode early-returns instead of KeyError-ing go), docstring is explicit that the rewards close over the runtime handoff, not the model replies. Test gaps closed: topology-level provision()/runtime= borrowing (fake runtime lifetime + str-arm parents), the shared-runtime handoff stubbed end-to-end, proposer-solver's happy-path fan-out (solve_rate 0.5 -> difficulty 1.0), and a key-gated live e2e for shared-runtime-v1. 53 unit tests pass; llm-judge, shared-runtime, and a bare-Agent script smoked live. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| } | ||
|
|
||
| @contextlib.asynccontextmanager | ||
| async def provision(self, task: Task | None = None) -> AsyncIterator[Runtime]: |
There was a problem hiding this comment.
🟡 Medium v1/agent.py:211
When Agent.run() is called with a borrowed runtime=, runtime_for() returns runtime.config directly (line 147) and skips resolve_runtime_config(...), so task-specific overrides for image, workdir, and resources are silently dropped. A task that requires a specific container image can run inside an unrelated borrowed runtime with no error. Consider still resolving task-specific settings against the borrowed runtime's config, or documenting that borrowed runtimes intentionally ignore task requirements.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/agent.py around line 211:
When `Agent.run()` is called with a borrowed `runtime=`, `runtime_for()` returns `runtime.config` directly (line 147) and skips `resolve_runtime_config(...)`, so task-specific overrides for `image`, `workdir`, and `resources` are silently dropped. A task that requires a specific container image can run inside an unrelated borrowed runtime with no error. Consider still resolving task-specific settings against the borrowed runtime's config, or documenting that borrowed runtimes intentionally ignore task requirements.
| stripped = line.strip().strip("*`").strip() | ||
| if not stripped.upper().startswith("SCORE:"): | ||
| continue | ||
| try: |
There was a problem hiding this comment.
🟡 Medium topologies/llm_judge.py:71
parse_score returns nan for a SCORE: nan line — float('nan') succeeds, and min/max leave NaN untouched, so the clamp at the end doesn't catch it. The caller (JudgeTask.committed) treats this as a committed verdict and judge() records a NaN reward, poisoning trace.rewards/trace.reward and downstream aggregations. Consider rejecting non-finite values (e.g. math.isnan) and returning None.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/topologies/llm_judge.py around line 71:
`parse_score` returns `nan` for a `SCORE: nan` line — `float('nan')` succeeds, and `min`/`max` leave NaN untouched, so the clamp at the end doesn't catch it. The caller (`JudgeTask.committed`) treats this as a committed verdict and `judge()` records a NaN reward, poisoning `trace.rewards`/`trace.reward` and downstream aggregations. Consider rejecting non-finite values (e.g. `math.isnan`) and returning `None`.
`run(task)` composes completed episodes; back-and-forth interaction (chess, negotiation, debate — each agent effectively the other's user) needs episodes held OPEN. `Agent.interact(task)` yields a `Session`: the episode runs in the background, suspending between turns, and `go` converses with it. Three members, deliberately: `turn(message) -> reply`, `end()` (idempotent; scope exit calls it), `.trace` (live handle). N sessions + imperative routing in `go` compose into round-robins, simultaneous moves (asyncio.gather over turns), moderated rooms, and hidden-information views — no scheduler, no message bus, no ordering DSL. Mechanically a session is a user simulator without the server: the existing interception user loop suspends the episode between turns awaiting the user seat's `Respond`; a session supplies that callable over an in-process queue handshake (Rollout gains `user=`), so the turn loop, budgets, @Stops, trace building, and graph land unchanged. The safety contract is loud, not patient: `turn()` on a dead episode raises `SessionEnded` (episode failure stays data — `go` decides what a dead seat means), a second in-flight `turn()` is refused, and a prompted task / a task with its own `load_user` / a harness that can't take injected user turns are refused at the `interact()` call. No `retry=` — half-played interactions can't be re-run. `refused()` now also honors a pre-set stop condition (a stopped trace takes no more turns), which is all `end()` needs. Scripted counterparts stay `vf.User` sims; sessions are for counterparts that are agents. Examples: `chess-v1` (two seats, host-side `chess.Board` as referee, illegal- move feedback turns, forfeit via `session.end()`; verified live — 8 plies of Ruy Lopez theory, adjudicated draw) and `debate-v1` (4 concurrent seats of one agent config: openings, rebuttals, and peer votes gathered N-wide; verified live — all ballots valid, vote-share rewards landed per seat). Each seat's trace is ONE multi-turn trajectory with its counterparts as user turns — the self-play training-sample shape, not one episode per move. Also fixes a live-run discovery in the chat dialect: a reasoning-only assistant turn (null content, no tool calls) is valid output but was rejected on re-send by upstreams ("content is required unless tool_calls"), killing simulated conversations mid-game; `extend` now coerces it to the empty string. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ygiene Final comb over the sessions work: - A `turn()` cancelled mid-flight (a sibling seat in a gather raising) leaves its message with the model; the eventual reply would land as queue residue and a later `turn()` would consume it one turn late. `_pending` now stays set through cancellation, so the next `turn()` refuses loudly — `end()` is the sanctioned way out of a desynced seat. Tested. - Regression test for the chat-dialect `extend` coercion (truncated reasoning turn -> null content -> upstream 422 on re-send), including the tool-call turn staying legitimately content-less. - The chess e2e/docs carried the exact 512-token cap that caused the live truncation forfeit: bumped to reasoning headroom (4096; debate 2048) and the example commands now advertise --sampling.max-tokens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…opology layer Main wins the core: TaskData/Task split (frozen wire half + plain behavior class), declarative Task.tools/Task.user ClassVars with config pairing, Taskset.load(), TraceTask provenance on traces, taskset-scoped shared tool servers (serve_shared), the restored-and-deepened judge plugin tier (per-task config.judges run inside task.score — our earlier deletion is dropped in favor of main's landed direction), traces.jsonl, --push, node timestamps, runtime info on traces. Re-applied on main's shapes, surgically: - Rollout regains `runtime=` (borrowed boxes: provision/interact share a world; a borrowed box is never started or stopped by the run) and `user=` (the programmatic user seat sessions plug into serve_user). - refused() honors a pre-set stop; the chat dialect's extend still coerces reasoning-only null-content turns; the null program's MCP connect stays bounded at 60s. - Trace regains the agent-graph links (agent/parents/trainable). - reward/metric decorators regain the topology `agent=` scope. - env.py: the inline stage-timeout block is re-extracted as resolve_stage_timeouts and validate_pairing factors through a new task-class-level validate_task_pairing — both shared by Environment and topology agents instead of duplicated. The multi-agent layer (Agent/Session/interact, Topology/TopologyRunner/ AgentGraph, llm-judge, agentic-judge, and the five example envs + fixtures + tests) is ported wholesale to the new contract: Data/Task pairs everywhere, task.data.* reads, tools as ClassVars (proposer's submit toolset needed zero changes), RunServices slims to interception pools only, and TopologyRunner.serving serves the seed taskset's shared tools once, mirroring Environment.serving — the lazy registry is gone because shared tools are taskset-scoped and the seed taskset is known up front. Docs follow main's consolidation: v1 GUIDE/README/ARCHITECTURE retire and the topology + sessions guide moves to docs/v1/topologies.md (mint nav). 95 tests pass; llm-judge, chess (full 8-ply game, adjudicated draw), and debate (3 seats, valid ballots) verified live on the merged contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| messages.append({"role": "user", "content": prompt}) | ||
| elif prompt is not None: | ||
| messages.extend(message_to_wire(m) for m in prompt) | ||
| client = AsyncOpenAI(base_url=endpoint, api_key=secret) |
There was a problem hiding this comment.
🟠 High direct/harness.py:51
AsyncOpenAI is created with max_retries left at the SDK default, so a transient transport failure on POST /chat/completions triggers an automatic retry that re-samples the model and appends a duplicate turn to messages. This forks the recorded rollout into an extra trajectory instead of preserving a single one. Set max_retries=0 when constructing the client so the harness controls retry behavior rather than the SDK.
| client = AsyncOpenAI(base_url=endpoint, api_key=secret) | |
| client = AsyncOpenAI(base_url=endpoint, api_key=secret, max_retries=0) |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/direct/harness.py around line 51:
`AsyncOpenAI` is created with `max_retries` left at the SDK default, so a transient transport failure on `POST /chat/completions` triggers an automatic retry that re-samples the model and appends a duplicate turn to `messages`. This forks the recorded rollout into an extra trajectory instead of preserving a single one. Set `max_retries=0` when constructing the client so the harness controls retry behavior rather than the SDK.
| return None | ||
|
|
||
|
|
||
| class JudgeTask(Task[DataT]): |
There was a problem hiding this comment.
🟡 Medium topologies/llm_judge.py:79
JudgeTask.for_attempt renders the judge's <task> section from task.data.prompt_text, but TaskData.prompt can be None for user-simulator tasks where the task framing is only established conversationally. For those tasks the judge prompt contains an empty <task> block, so the judge scores an attempt without ever seeing what problem was being solved — producing incorrect rewards for any user-driven taskset. Consider falling back to the full conversation text (or the trace's messages) when prompt_text is empty, so the judge always sees the task framing.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/topologies/llm_judge.py around line 79:
`JudgeTask.for_attempt` renders the judge's `<task>` section from `task.data.prompt_text`, but `TaskData.prompt` can be `None` for user-simulator tasks where the task framing is only established conversationally. For those tasks the judge prompt contains an empty `<task>` block, so the judge scores an attempt without ever seeing what problem was being solved — producing incorrect rewards for any user-driven taskset. Consider falling back to the full conversation text (or the trace's messages) when `prompt_text` is empty, so the judge always sees the task framing.
There was a problem hiding this comment.
🟡 Medium
verifiers/verifiers/v1/cli/eval/main.py
Line 107 in e2852fa
When a topology eval runs with --push, push_traces(traces, config) derives the environment/dataset name from config.taskset.id or config.id, but EvalConfig.check_topology() rejects both of those fields when config.topology is set. As a result, pushed topology results upload with an empty environment/dataset name instead of config.topology.id, producing incorrect metadata or failing to resolve the environment. Consider deriving the topology upload name from config.topology.id (e.g., by short-circuiting push_traces before the taskset/id lookup), or document the intended push semantics for topology runs if this is by design.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/cli/eval/main.py around line 107:
When a topology eval runs with `--push`, `push_traces(traces, config)` derives the environment/dataset name from `config.taskset.id or config.id`, but `EvalConfig.check_topology()` rejects both of those fields when `config.topology` is set. As a result, pushed topology results upload with an empty environment/dataset name instead of `config.topology.id`, producing incorrect metadata or failing to resolve the environment. Consider deriving the topology upload name from `config.topology.id` (e.g., by short-circuiting `push_traces` before the taskset/id lookup), or document the intended push semantics for topology runs if this is by design.
| ] | ||
| # Opening statements — N suspended episodes generating concurrently. | ||
| statements = list( | ||
| await asyncio.gather( |
There was a problem hiding this comment.
🟡 Medium debate_v1/topology.py:117
A single debater's seat.turn(...) raising SessionEnded (timeout, stop condition, or episode death) aborts the entire debate — asyncio.gather propagates the exception, unwinding the AsyncExitStack and converting the whole topology run into a TopologyError instead of recording that one seat's failure as data. This happens at all three gather sites (lines 117, 130, 149) because gather is called without return_exceptions=True and there is no try/except around the turn calls. Consider passing return_exceptions=True and handling SessionEnded per-seat so a failed debater is recorded in trace.info rather than crashing the debate.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @environments/debate_v1/debate_v1/topology.py around line 117:
A single debater's `seat.turn(...)` raising `SessionEnded` (timeout, stop condition, or episode death) aborts the entire debate — `asyncio.gather` propagates the exception, unwinding the `AsyncExitStack` and converting the whole topology run into a `TopologyError` instead of recording that one seat's failure as data. This happens at all three gather sites (lines 117, 130, 149) because `gather` is called without `return_exceptions=True` and there is no `try`/`except` around the turn calls. Consider passing `return_exceptions=True` and handling `SessionEnded` per-seat so a failed debater is recorded in `trace.info` rather than crashing the debate.
| return cls( | ||
| AgenticJudgeData( | ||
| idx=task.data.idx, | ||
| prompt=prompt.format(path=TRACE_PATH), |
There was a problem hiding this comment.
🟡 Medium topologies/agentic_judge.py:69
for_trace calls prompt.format(path=TRACE_PATH) on the user-configurable --topology.prompt string, so any custom rubric that contains other braces — e.g. a JSON snippet like { "score": 10 } or a placeholder like {answer} — raises KeyError/ValueError and crashes the agentic-judge run before the judge rollout starts. Only the {path} placeholder should be substituted. Consider replacing the literal with a targeted replacement like prompt.replace("{path}", TRACE_PATH).
| prompt=prompt.format(path=TRACE_PATH), | |
| prompt=prompt.replace("{path}", TRACE_PATH), |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/topologies/agentic_judge.py around line 69:
`for_trace` calls `prompt.format(path=TRACE_PATH)` on the user-configurable `--topology.prompt` string, so any custom rubric that contains other braces — e.g. a JSON snippet like `{ "score": 10 }` or a placeholder like `{answer}` — raises `KeyError`/`ValueError` and crashes the `agentic-judge` run before the judge rollout starts. Only the `{path}` placeholder should be substituted. Consider replacing the literal with a targeted replacement like `prompt.replace("{path}", TRACE_PATH)`.
| f.write(data + b"\n") | ||
|
|
||
|
|
||
| def write_graph(results_dir: Path, graph: AgentGraph) -> None: |
There was a problem hiding this comment.
🟡 Medium cli/output.py:83
write_graph() writes AgentGraph records (via graph.to_record()) into traces.jsonl, but read_traces() unconditionally parses every line as a Trace via TypeAdapter(trace_type). When replay reads a topology run's traces.jsonl, the first AgentGraph record fails validation, so replay <topology-output-dir> errors instead of replaying the saved run. The read path needs to handle graph records (e.g., detect or branch on the record type) rather than always deserializing as Trace.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/cli/output.py around line 83:
`write_graph()` writes `AgentGraph` records (via `graph.to_record()`) into `traces.jsonl`, but `read_traces()` unconditionally parses every line as a `Trace` via `TypeAdapter(trace_type)`. When `replay` reads a topology run's `traces.jsonl`, the first `AgentGraph` record fails validation, so `replay <topology-output-dir>` errors instead of replaying the saved run. The read path needs to handle graph records (e.g., detect or branch on the record type) rather than always deserializing as `Trace`.
Main's task module documents `MyTask.from_trace(trace)` as the opt-in constructor for a task derived from a finished rollout — exactly the forward-arrow classmethods our judge topologies already carried as `for_attempt`/`for_trace`, with one redundancy: they took the seed task as a separate argument when `trace.task.data` already carries it (the trace is self-describing; that's what TraceTask provenance is for). Renamed to `from_trace(trace, ...)` and the inputs read off the trace. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The post-merge audit: all example code was already on the new contract (Data/Task pairs, task.data reads, declarative tools, state generics) — the stragglers were prose. Fixed: proposer-solver's docstring still said load_tools/results.jsonl, topology.py's graph docstrings said results.jsonl, and docs/v1/topologies.md's session snippet showed the pre-split SeatTask construction and the old load_user refusal wording. One idiom upgrade: proposer-solver's SolverTask now demonstrates `Task.from_trace` — it was the textbook case (question, ground-truth code, and input all come off the proposer trace's info["submission"]). Writer-editors' critique tasks deliberately stay inline: in round 2+ the upstream trace is a ReviseTask whose data carries no brief, so the trace alone isn't self-describing there — from_trace only where it's honest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
What
Multi-agent environments for v1, in three layers on top of main's task-first core (this branch merges the landed #1948 and builds strictly above it — the diff is near-pure addition):
Agent— the execution primitive: harness × model context × runtime policy, one arrowrun(task) -> Trace, plusprovision()(share a runtime box between agents) andinteract()(hold a live episode open and converse with it).Topology— the config-driven packaging: named agents as typed config fields, orchestration as plain imperative code ingo(), cross-agent rewards declared and deferred, each instance persisted as anAgentGraph.llm-judge,agentic-judge) and five example envs, each emblematic of one contract, all verified live.Core in
agent.py/topology.py; guide atdocs/v1/topologies.md.Built on the task-first contract
A topology's whole premise is main's #1948: a task minted anywhere is first-class (its
TaskDatarow + its behavior class).go()mints tasks mid-run three equal ways — a taskset'sload(), inlineCritiqueTask(vf.TaskData(...)), or theTask.from_trace(trace)convention #1948 documents: a task derived from a finished rollout, reading everything off the self-describing trace (trace.task.data). The judge topologies and proposer-solver implementfrom_traceverbatim.go(): orchestration as plain imperative codeAn agent is pure routing (harness + model/client/sampling +
trainable); tasks arrive per episode. The forward arrow is ordinary host-side construction; loops are rounds,asyncio.gatheris fan-out, awaiting several traces is fan-in:Episode failures never raise — they return as data on the trace, and
godecides what a failed child means.Sessions: agents interacting within each other's episodes
run()composes completed episodes; back-and-forth interaction (chess, debate — each agent effectively the other's user) needs episodes held open.run.agent(name).interact(task)yields aSessionwith exactly three members:turn(message) -> reply,end(),.trace. The episode suspends between turns; N sessions + imperative routing compose into alternation, simultaneous moves (gatheroverturns), moderated rooms, and hidden-information views — no scheduler, no message bus.Mechanically a session is a user simulator without the server: the interception layer already suspends an episode between turns awaiting the user seat's
Respond; a session supplies that callable over an in-process queue handshake (Rollout(user=)). The turn loop, budgets,@stops, and trace building are reused unchanged. The safety contract is loud, not patient:turn()on a dead episode raisesSessionEnded(never hangs), double-driving and cancelled-mid-flight turns are refused, prompted tasks / own-Task.usertasks / non-user-capable harnesses are refused at the call, and there is noretry=— half-played interactions can't be re-run. Each seat's trace is ONE multi-turn trajectory with its counterparts as user turns — the self-play training-sample shape, not one episode per move.Deferred rewards: task-tied or topology-declared
Task rewards run at episode end, inside the rollout, runtime still live (main's contract — including
config.judges). Topology rewards are declared@vf.reward(agent=...)/@vf.metric(agent=...)methods, run at instance end once per matching trace — which is what makes "proposer reward = distance of solver pass-rate from 50%" expressible. Scopes are validated at load; metrics run before rewards; both land in the sametrace.rewardsunder distinct keys (verified live:{'correct': 1.0, 'judge': 1.0}).Seeds, the agent graph, and full config addressability
--topology.taskset.id(same slot and grammar as route 1) XOR aload_tasksoverride — a self-seeding topology passed the flag is refused rather than silently ignoring it. The seed taskset's shared tool servers (Taskset.tools) are served once byTopologyRunner.serving, mirroringEnvironment.serving.agent/parents/trainable; a topology run persists oneAgentGraphpertraces.jsonlline (traces nested, completion order). The links live on the traces, so the DAG is recoverable from a flat dump.AgentConfigfields (field name = agent name) — everything is CLI/TOML-addressable with typed help (--topology.black.model,--topology.judge.harness.runtime.type docker). Pins live onAgentConfigsubclasses (vf.DirectAgentConfig/vf.NullAgentConfigshared); partial overrides deep-merge into a pin, an explicitharness.idswaps it, and a topology can lock a slot (llm-judge's judge) with a refusal that points at the alternative.The instantiations (all verified live,
deepseek/deepseek-v4-flash)llm-judge(built-in)from_trace+ a locked agent slot; judge fixed to the in-processdirectharness (episode ≈ one API call)agentic-judge(built-in)setupuploads the solver's entire serialized trace into the judge's runtime; harness + prompt configurableSCORE: 10; also ran on harbor in a prime sandboxproposer-solver-v1submit_questiontool → typed state →SolverTask.from_trace; asymmetric capability (proposer onnullwith tools, solvers on tool-lessdirect);difficultypeaked at 50% solve ratedifficulty 0.0(too-easy punished); broken-code submissions eatwell_formed 0writer-editors-v1parents=[draft, *edits]); onevf.Judgecall per instance puts the same reward on every traceimprovementacross each instancechess-v1chess.Boardas referee, illegal-move feedback as ordinary turns, forfeit viasession.end(), per-seat model routingdebate-v1shared-runtime-v1provision()+ borrowedruntime=: two agents' rollouts share one world; the run never starts/stops a borrowed boxborrowed=Trueon the same box, handoff verified in-runtimeFramework deltas outside the new layer (small, deliberate)
Rolloutgainsruntime=(borrowed boxes; never started/stopped by the run) anduser=(the session seat, pre-emptingTask.user).RolloutSession.refused()honors a pre-set stop condition ("a stopped trace takes no more turns" — whatSession.end()rides on).Tracegains the agent-graph links:agent,parents,trainable.@reward/@metricgain the topologyagent=scope.env.py: the inline stage-timeout block is extracted asresolve_stage_timeouts, andvalidate_pairingfactors through a task-class-levelvalidate_task_pairing— both shared byEnvironmentand topology agents instead of duplicated.extendcoerces reasoning-only null-content assistant turns to""(upstreams 422 them on re-send — found by a live chess game; regression-tested).harness_timeout.config.judges(in-episode verdicts),vf.Judge(verdicts in your reward code), and the judge topologies (verdicts as their own agent episodes) are three distinct tiers by where the verdict lives, and they compose.Testing
95 unit tests, no keys needed: config narrowing + pin semantics + locked slots + seed XOR + scope validation; instance links, deferred rewards,
go-failure capture, graph record round-trip; session handshake/poison/double-drive/cancel-desync semantics and everyinteract()refusal; scripted-session tests driving chess (fool's mate through the real board) and debate (vote tallying) without a model; borrowed-runtime lifetime. Key-gated e2e runs all seven topologies live. Everything above was additionally smoke-run live on the merged contract.Out of scope (deliberately)
Trainer wire (env-server
run_topology, grouping semantics),--server/--resume/--richunder topologies, agent-as-tool (consultation inside a single turn — composes on sessions later).🤖 Generated with Claude Code