diff --git a/TOPOLOGY_DEFERRED_WORK.md b/TOPOLOGY_DEFERRED_WORK.md new file mode 100644 index 0000000000..8704f86b7c --- /dev/null +++ b/TOPOLOGY_DEFERRED_WORK.md @@ -0,0 +1,212 @@ +# Deferred topology work + +This is the living list of topology work that is known, intentionally deferred, and not +required for the first trainable `AgentGraph` path. It records gaps so they do not become +implicit contracts or disappear into follow-up discussions. + +It does not track work that is already complete: topology selection is shared through +`EnvConfig`, taskset and harness syntax lowers to the built-in single-agent topology, one +invocation returns and persists one `AgentGraph`, and native v1 group rewards and `Episode` +have been removed. + +## Execution and scalability + +### Bound session-driven model concurrency + +`TopologyRun.run_agent()` acquires the configured rollout semaphore, but +`TopologyRun.interact_agent()` deliberately bypasses it. In-process evaluation also starts all +topology invocations eagerly. A session topology can therefore create unbounded model pressure +even when `max_concurrent` is configured. Server-backed execution is bounded by the outer graph +request pool, but that does not bound turns or agents inside a graph. + +The eventual solution must limit active model work without holding a permit for an entire +suspended session, which could deadlock multi-seat topologies. A turn-level permit, or another +model-call-level admission mechanism, is preferable to a session-lifetime permit. + +Done when direct and server-backed execution have an explicit, tested concurrency bound for +session topologies without deadlocking games or debates. + +### Account for variable topology cost + +The server currently charges every topology invocation as one unit of capacity regardless of +how many agents, turns, fan-outs, runtimes, or model calls it creates. This preserves the simple +and scalable existing server lifecycle, but intentionally leaves scheduling blind to internal +graph cost. + +Defer cost-aware admission, per-agent/model telemetry, and dynamic capacity accounting until +there is enough operational evidence to choose a useful abstraction. + +### Retry whole instances, not just rollouts + +`retries.rollout` reruns a single failed agent run, but there is no graph-level analogue: +an invocation that comes out invalid (by the topology's own `Topology.complete()` verdict) +is persisted as-is and only redone by a later `--resume`. `Topology.complete()` is already +the right predicate for an online instance-level retry (`retries.instance`), and the +`RetryConfig` wrapper deliberately keeps that slot open. Defer until a real training or +eval run demonstrates the need; when added, cap and attribute retries so a topology whose +instances are *never* complete fails loudly instead of looping. + +### Right-size graph wire payloads + +`RunResponse` now serializes graphs with full per-node training tensors +(`routed_experts`, `multi_modal_data`) so the trainable path survives the env-server hop — +correct for training, but eval-mode server runs now ship tensors nobody reads, inflating +ZMQ payloads and decode time for large graphs. Defer until payload size shows up in +practice; the likely shape is a per-request or per-server "strip training tensors" switch +mirroring what `to_record()` already does for JSON persistence. + +### Add explicit run-level state if needed + +There is no canonical state store for information that must survive beyond one `go(task)` +invocation, such as cumulative chess wins and losses. Storing mutable data on a `Topology` or +`TopologyRunner` instance is not a contract: workers have separate copies, processes restart, +and distributed execution cannot observe a single authoritative value. + +If this becomes necessary, define the ownership, persistence, consistency, and worker-sharing +semantics explicitly. Do not accidentally introduce run-level state through topology object +mutation. + +## Self-contained graph semantics + +### Persist the topology completion verdict + +`Topology.complete(graph)` lets a topology accept a graph whose child traces include handled +failures. In-process resume can call the live topology and respects this policy. Server-backed +resume and other consumers without the topology object fall back to conservative +`graph_complete()`, so they can redo a graph the topology already considered valid. + +Stamp the topology's final completion verdict onto the returned and persisted `AgentGraph` after +topology scoring. Consumers should read that self-contained verdict; older records without it +can retain the conservative fallback. + +Done when resume produces the same decision in process, through an environment server, and from +a graph record alone. + +### Recover relative signals as a built-in best-of-N topology + +Deleting `@group_reward` removed the only native home for pairwise/relative eval signals +(code-golf's "shortest of the group wins", preference comparisons); they were demoted to +per-trace metrics with the comparison left to "the training algorithm". For *training* +that is the right owner, but eval-mode relative signals have a clean replacement the +framework should ship rather than describe: a built-in best-of-N topology — one seed, +`go` fans out N runs of the same agent, a `@reward(agent=...)` compares +`graph.children(...)`. That recovers everything group rewards did, explicitly, on the +canonical path, and doubles as the documented migration story for the removal. + +Done when a `best-of-n` built-in exists, code-golf's relative signals are expressed +through it (or its docstring points at it), and the docs name it as the group-reward +successor. + +## Replay, presentation, and platform upload + +### Make replay graph-native + +Replay is trace-native today. It flattens graph records, assumes the seed taskset's one task +type, invokes only `Task.score()`, and writes flat trace records. It cannot reconstruct topology +judgement or correctly replay derived task types produced by other agents. + +The short-term safe behavior is to reject explicit-topology replay clearly rather than emit a +plausible but incorrect result. Full support should load one `AgentGraph` at a time, recover each +trace's task behavior, recompute eligible task rewards, run topology-level scoring, and persist +another graph. + +### Make the evaluation dashboard graph-aware + +The dashboard still presents explicit topologies through taskset and harness fields and streams +individual traces as though they were independent invocations. It does not expose topology id, +graph-level completion/error state, agent roles, parent links, or the distinction between one +graph and its internal traces. + +Until the dashboard has a graph view, it should avoid silently describing an explicit topology +as a single taskset-harness rollout. + +### Make platform push graph-native + +`push_traces()` receives flattened graph traces and uploads each as an independent rollout. This +loses graph identity, topology id, agent and parent structure, graph completion semantics, and +correct replica numbering. Multi-agent topologies are consequently misrepresented on the +platform. + +Prefer a graph-native upload contract. Until the platform can accept it, explicit-topology push +should fail or warn clearly instead of silently flattening graphs. + +## Configuration boundaries + +### Normalize agent execution configuration before running + +Agent model, client, and sampling settings are currently resolved while binding a topology run: +agent-specific values override request-level defaults. The behavior is correct, but the layered +fallbacks remain visible in execution code and leave more than one representation of a resolved +agent configuration. + +A future cleanup can introduce one normalized, fully resolved agent execution context before any +agent runs. This is not required for correctness and should not be combined with unrelated server +lifecycle changes. + +### Settle the config layer's structure + +A reviewed set of config cleanups, deferred as one pass (analysis 2026-07-11): + +- **Split `topology.py` into its authoring and runner halves.** The authoring surface + (`AgentConfig`, `TopologyConfig`, `AgentGraph`, `Topology`, `SingleAgentTopology`, + `graph_complete`) imports nothing from `env.py`; only the runner half does. Splitting + turns the layering into a clean DAG (`topology ← env ← runner`) and deletes the + forward-referenced `EnvConfig.topology` annotation plus the `model_rebuild()` call that + currently papers over the cycle — the one trick in the config layer. +- **Declare `RolloutLimits` once.** `EnvConfig` carries four flat `max_*` fields that are + hand-copied into `RolloutLimits` at the runner; a `limits` property on `EnvConfig` + states the correspondence once while keeping the flat CLI flags. +- **De-collide `multiplex`.** `--multiplex` (rollouts per interception server) and + `--pool.multiplex` (graph requests per worker, the elastic scale trigger) are unrelated + knobs sharing a name; rename the pool field with an alias for compatibility. +- **Align timeout vocabulary.** `--timeout.rollout` maps onto the task-authored + `TaskTimeout.harness`; post-consolidation "rollout" is the word — rename the task field + while it is still a five-file change. +- **Delete the dead `EnvConfig.env_id` property** (no v1 consumers) and, riding the + split, consider moving the `EnvConfig` family into `configs/` so that package holds all + process-level configs (plugin configs stay colocated with their plugin classes — that + idiom is deliberate and load-bearing for id-narrowing). + +Explicitly considered and rejected (do not re-open without new evidence): nesting +`model`/`client`/`sampling` into a ModelContext-shaped sub-config (kills `-m` / +`--topology..model` ergonomics), a shared CLI mixin for `verbose`/`dry_run` +(saves ~6 lines, adds a class), quarantining the legacy `--id` block (it *is* the compat +interface), and dropping the `group_size`/`rollouts_per_example` aliases (trainer-facing). + +## Compatibility isolation + +### Separate the legacy protocol from native v1 + +Native v1 execution uses only `run -> AgentGraph`, but the public `EnvClient` and shared protocol +module still expose `run_rollout`, `run_group`, `requires_group_scoring`, and their legacy +request/response types. This is intentional compatibility leakage, not part of the native graph +contract. + +Move the bridge routes and types behind an explicitly legacy client/module, or remove them when +v0 support is retired. Native imports and generated client interfaces should then expose only the +graph route. + +## Documentation and skills + +The topology authoring guide is substantially in place, but the v1 overview, architecture, +evaluation docs, and evaluation skills still describe taskset plus harness as the primary +execution model. They do not consistently explain the canonical lowering to a topology, the +`Topology.complete()` contract, or the graph limitations of replay, dashboard, and push. + +Update these surfaces after the corresponding contracts settle. Documentation should describe +the current model only, not preserve a migration narrative. + +### Reconcile PR #1939 (agent programs) + +The standalone Agent-facade PR (#1939, open against main) is the ancestor of this branch's +agent layer. Its machinery is fully absorbed or deliberately superseded here (pool ownership +moved to `RunServices`; `TaskData.sources`/`relation` lineage replaced by trace-field links; +the per-run `ctx=` override replaced by per-agent routing), and its three correctness fixes +(borrowed-box tombstone, mid-run teardown attribution, resolved-runtime pairing) have been +ported. What remains unique to it is `docs/v1/agent-programs.md` plus four +`examples/agent_programs/` scripts — written against its API variant, so merging it as-is +would reintroduce a conflicting Agent surface and a duplicate lineage mechanism. + +Done when the bare-`vf.Agent` scripting surface is documented against *this* branch's API +(an agent-programs page or a section of the topology docs, examples rewritten) and #1939 is +closed or rebased to nothing. diff --git a/configs/code_golf.toml b/configs/code_golf.toml index 5db6778ead..2c35ff0828 100644 --- a/configs/code_golf.toml +++ b/configs/code_golf.toml @@ -1,9 +1,8 @@ -# code-golf-v1 — group rewards (@vf.group_reward) score N rollouts of a task together -# (shortest / fastest of the group), so it needs >= 2 rollouts. +# code-golf-v1 — correctness reward plus per-trace latency and length metrics. # # uv run eval code-golf-v1 @ configs/code_golf.toml num_tasks = 1 -num_rollouts = 2 +num_rollouts = 1 [taskset] id = "code-golf-v1" diff --git a/configs/swe_style_judge.toml b/configs/swe_style_judge.toml new file mode 100644 index 0000000000..e7c710f8b8 --- /dev/null +++ b/configs/swe_style_judge.toml @@ -0,0 +1,47 @@ +# One SWE-bench Verified solver and one style judge in the same Prime sandbox. +# Run from the prime-rl workspace with: +# uv run --extra envs eval @ deps/verifiers/configs/swe_style_judge.toml + +model = "openai/gpt-5.5" +num_tasks = 1 +num_rollouts = 1 +max_concurrent = 1 +max_turns = 40 +rich = false +push = false + +[timeout] +setup = 900 +rollout = 3600 +finalize = 300 +scoring = 1800 + +[retries.rollout] +max_retries = 1 +include = ["ProviderError", "SandboxError"] + +[topology] +id = "swe-style-judge" +weight = 1.0 + +[topology.taskset] +id = "swebench-verified-v1" +use_prime_registry = true + +[topology.solver.harness] +id = "default" + +[topology.solver.harness.runtime] +type = "prime" +labels = ["swe-style-judge"] + +[topology.judge] +model = "openai/gpt-5.5" + +[topology.judge.harness] +id = "default" +edit = false + +[topology.judge.harness.runtime] +type = "prime" +labels = ["swe-style-judge"] diff --git a/docs/mint.json b/docs/mint.json index d293c2dbc3..48a4ad4b89 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -12,6 +12,7 @@ "v1/getting_started", "v1/architecture", "v1/environments", + "v1/topologies", "v1/evaluation", "v1/harnesses", "v1/harbor" @@ -32,4 +33,4 @@ ] } ] -} +} \ No newline at end of file diff --git a/docs/v1/environments.md b/docs/v1/environments.md index 39cdb2a249..f6e4c82cdf 100644 --- a/docs/v1/environments.md +++ b/docs/v1/environments.md @@ -55,7 +55,13 @@ class AdditionTaskset(vf.Taskset[AdditionTask, vf.TasksetConfig]): __all__ = ["AdditionTaskset"] ``` -You can also use `@vf.metric` to record non-scored values and `@vf.group_reward` for group rewards, which might be useful for training. +At execution time this taskset and the selected harness are lowered to the built-in +single-agent topology. Each invocation therefore produces a one-trace `AgentGraph`, using +the same runner and server path as an explicitly authored topology. + +Use `@vf.metric` to record non-scored values. Comparisons across independent graph +invocations belong to the training algorithm; cross-trace environment judgement belongs +on a topology reward. ## Making values configurable diff --git a/docs/v1/evaluation.md b/docs/v1/evaluation.md index 48c0898525..20d01c0726 100644 --- a/docs/v1/evaluation.md +++ b/docs/v1/evaluation.md @@ -26,7 +26,9 @@ Validate the config by using `uv run eval @ config.toml --dry-run`. To run the e Use dotted arguments to set values using the CLI, e.g. `--sampling.temperature 0.5`. CLI arguments overwrite toml arguments when both are present. -The output from evaluations are written into `outputs/----//` by default (use `output_dir` to overwrite the folder). The folder contains the used `config.toml`, all the traces in `results.jsonl`, as well as logs of the run and workers in `eval.log`. +Evaluation output is written under `outputs/----//` by +default. The directory contains `config.toml`, one completed `AgentGraph` per line in +`traces.jsonl`, and logs in `eval.log`. ## Common config values @@ -34,14 +36,16 @@ The output from evaluations are written into `outputs/----` re-runs only the rollouts a previous run left missing or errored, appending to that run's own `results.jsonl`. It reloads the run's saved `config.toml` verbatim, so it takes no other arguments. Good rollouts are kept, while errored ones are dropped and redone. +`--resume ` independently re-runs graph invocations a previous run left +missing or errored, appending to that run's `traces.jsonl`. It reloads the saved +`config.toml` verbatim, so it takes no other arguments. ## Disabling tools diff --git a/docs/v1/topologies.md b/docs/v1/topologies.md new file mode 100644 index 0000000000..231f825964 --- /dev/null +++ b/docs/v1/topologies.md @@ -0,0 +1,243 @@ +# Topologies — multi-agent environments + +A **topology** composes agent invocations: which agents exist, how one agent's trace becomes +the next agent's task, and how rewards flow backwards once downstream agents have run. Each +agent invocation consumes one task and produces one trace. Every environment invocation, +including taskset × harness syntax lowered to the built-in single-agent topology, produces +one `AgentGraph`. Run an explicit topology with `--topology.id`: + +```bash +uv run eval --topology.id llm-judge --topology.taskset.id gsm8k-v1 -n 4 +uv run eval --topology.id proposer-solver-v1 -n 3 +``` + +## Agents + +An agent is **pure routing** — a name + the harness driving its runs + how its model calls +are routed; it carries nothing task-side. Declare agents as typed `AgentConfig` fields on your +config — the field *name* is the agent's name, and every agent is CLI-addressable +(`--topology.solver.harness.id rlm`, `--topology.judge.model `): + +```python +class ProposerSolverConfig(vf.TopologyConfig): + proposer: vf.AgentConfig = vf.AgentConfig() # default: bash + edit + task tools + solver: vf.DirectAgentConfig = vf.DirectAgentConfig() # in-process `direct` (tool-less) + num_solvers: int = 4 +``` + +An `AgentConfig` binds a harness (and where it runs — `harness.runtime`) plus per-agent +routing: `model` / `client` / `sampling` overrides and a `trainable` flag (stamped onto every +trace the agent produces, so a trainer can drop e.g. judge traces without the topology +config). Subclass it to pin typed per-agent defaults — `vf.DirectAgentConfig` / +`vf.NullAgentConfig` are the shared common pins, and the `llm-judge` topology's judge pins +the `direct` harness plus `trainable=False` (a pin must live on the subclass's field default, +e.g. `harness: SerializeAsAny[vf.HarnessConfig] = vf.HarnessConfig(id="direct")`, so partial +overrides deep-merge into it). The tasks an agent consumes (each carrying its own behavior) +arrive per invocation, from the topology's seeds or constructed in `go`. + +An `AgentConfig` field is the one declaration form. The default `load_agents` builds one +`AgentBinding` per field, in declaration order; override it only to compose agents +programmatically. At serving time those bindings become executable `vf.Agent`s with the +active model context and shared run services. Loading also validates the topology's declared +judgement (`@reward(agent=...)` / `@metric(agent=...)`) against the agents, so a typo'd or +missing agent scope fails at load time, not mid-eval. + +## Seed tasks + +One topology instance runs per seed task (× `-r`). Seeds come from the config's `tasks` +factory — any taskset, plugged by id (`--topology.taskset.id gsm8k-v1`; its knobs validate +typed, e.g. `--topology.taskset.split train`) — or from a `load_tasks` override for a +self-seeding topology. **Exclusive-or, enforced at load**: when the slot can be set it IS +the seed source, verbatim; a topology that overrides `load_tasks` is refused the flag +(rather than silently ignoring it), and a custom `load_tasks` wanting a config-driven +source declares its own factory field. A topology may also pin a specific taskset, as +`proposer-solver-v1` pins AIME 2026: + +```python +class ProposerSolverTopology(vf.Topology[ProposerSolverConfig]): + def load_tasks(self) -> list[vf.Task]: + config_type = vf.taskset_config_type("aime26-v1") + return vf.load_taskset(config_type(id="aime26-v1")).load() +``` + +Per-role behavior lives on **task classes**, minted anywhere. In `proposer-solver-v1`, +`ProposeTask` judges its own trace with a format reward. The solver task is derived from +the AIME seed by replacing its prompt and answer while retaining its task class and config, +so the modified problem keeps AIME's math-verify reward: + +```python +class ProposeTask(vf.Task): + tools = (SubmitToolset,) + + @vf.reward(weight=0.1) + async def parseable(self, trace: vf.Trace) -> float: + return float("submission" in trace.info) + + +def solver_task(seed: vf.Task, trace: vf.Trace) -> vf.Task: + submission = trace.info["submission"] + data = seed.data.model_copy(update={ + "prompt": AIME_INSTRUCTION + submission["question"], + "answer": submission["answer"], + }) + return type(seed)(data, seed.config) +``` + +## The interaction pattern — `go` + +`go` is plain imperative Python over a `TopologyRun`; interaction patterns are code, not a +DSL. `run.agent(name).run(task, parents=...)` runs one agent invocation and links it into the +agent graph; `asyncio.gather` fans out; loops are rounds; awaiting several traces before +building the next task is fan-in: + +```python + async def go(self, task: vf.Task, run: vf.TopologyRun) -> None: + proposer = await run.agent("proposer").run(ProposeTask.from_task(task)) + if "submission" not in proposer.info: + return + derived = self.solver_task(task, proposer) + solver = run.agent("solver") + await asyncio.gather( + *( + solver.run(derived, parents=[proposer]) + for _ in range(self.config.num_solvers) + ) + ) +``` + +## Sessions — agents talking within live interactions + +`run(task)` composes **completed** agent runs: a run finishes before its trace feeds +anything downstream. For back-and-forth interaction — chess, negotiation, debate, any +game where each agent is effectively the other's user — hold agent runs **open** with +`run.agent(name).interact(task)` and converse with the yielded `Session`: + +```python + async def go(self, task: vf.Task, run: vf.TopologyRun) -> None: + board = chess.Board() # game rules live HERE, host-side + async with ( + run.agent("white").interact(SeatTask(SeatData(color="white", prompt=None, system_prompt=...))) as white, + run.agent("black").interact(SeatTask(SeatData(color="black", prompt=None, system_prompt=...))) as black, + ): + seats = {chess.WHITE: white, chess.BLACK: black} + while not board.is_game_over(): + reply = await seats[board.turn].turn(render(board)) # user turn in, model turn out + board.push(parse_move(reply, board)) + # scope exit ends both runs cleanly (stop -> finalize -> task scoring); + # stamp the outcome as data for the declared rewards to read: + white.trace.info["chess"] = {"score": ...} +``` + +A `Session` has exactly three members: `turn(message) -> reply` (send the agent its +next user turn, get the model's turn back), `end()` (finish early — forfeits, +eliminations; idempotent, scope exit calls it), and `.trace` (the live trace: read +mid-game `state`, stamp outcome `info`). Everything else — who talks to whom, in what +order, with what *view* of the interaction — is imperative code in `go`, so N seats +compose into round-robins, simultaneous moves (`asyncio.gather` over several `turn`s), +moderated rooms, and hidden-information games with no further machinery. Each seat's +trace is ONE multi-turn trajectory with its counterparts' messages as user turns — the +training-sample shape self-play wants, not one trace per move. + +Mechanically a session is a user simulator without the server: the interception layer +suspends the agent between turns awaiting the user seat, and here the "user" is `go`. +The safety contract is loud, not patient: `turn()` on a dead run raises +`SessionEnded` (carrying the trace) instead of hanging; a second concurrent `turn()` on +one session is refused; a task that declares its own user simulator (`Task.user`) — or a prompted task +(sessions open on the first `turn()`; put framing in `system_prompt`), or a harness that +can't take injected user turns — is refused at the `interact()` call. No `retry=`: one +side of a half-played game can't be re-run; a dead seat is `go`'s decision. Budgets +(`--max-turns`, rollout timeout) span the whole interaction, including time a seat +spends suspended — size them for the game. Scripted counterparts stay `vf.User` +simulators (route 1, no topology needed); sessions are for counterparts that are +themselves agents. See `chess-v1` (two seats, host-side referee) and `debate-v1` (N +concurrent seats of one agent config, peer-voted). + +## Topology rewards — declared, cross-agent judgement + +Per-trace judgement rides on task classes (the derived AIME task's `correct` reward above). Cross-agent +judgement is *not* written inline in `go`: declare it as `@vf.reward(agent=...)` / +`@vf.metric(agent=...)` methods on the topology — the same decorators tasks use, scoped +to an agent. Each runs once per matching trace **after the whole instance completes**, with +any of `task` / `trace` / `graph` injected by parameter name, and records under the method +name (weighted) exactly like a task reward: + +```python + @vf.metric(agent="proposer") + async def solve_rate(self, trace: vf.Trace, graph: vf.AgentGraph) -> float: + graded = [t for t in graph.children(trace, agent="solver") if not t.has_error] + return sum(t.rewards.get("correct", 0.0) for t in graded) / len(graded) if graded else 0.0 + + @vf.reward(agent="proposer") + async def difficulty(self, trace: vf.Trace) -> float: + return 1.0 - 2.0 * abs(trace.metrics["solve_rate"] - 0.5) # rewards may read metrics +``` + +The contract, chosen to fail loudly and stay predictable: + +- **Validated at load**: an `agent=` scope that doesn't exist — or a topology `@reward` + with no scope — is refused when the topology loads, before anything runs. +- **Runs at instance end**, never earlier: nothing can observe a reward before the + instance persists, so there is no "earliest possible" scheduling to reason about. Every + trace in scope is scored — across all rounds and fan-outs — automatically. +- **Ordering**: methods run sequentially, metrics before rewards, each phase in + (priority, name) order. A method may read task-recorded rewards (final since the + agent run ended) and, in the rewards phase, any metric — but topology rewards must not + read each other; derive shared inputs from the traces or a metric. +- **Failures**: a raise during instance scoring is classified `TopologyError` and recorded + on the graph — completed traces stay as data, siblings unaffected. + +`trace.record_reward(...)` inside `go` still works as the escape hatch for exotic shapes +(e.g. a mid-round adjustment), but the declared methods are the norm. + +The forward arrow stays in `go`: construct the downstream agent's typed `Task` from an +upstream trace — its typed task, `last_reply`, `transcript`, or `trace.info`. This is pure +host-side code; only when peeling requires the agent's *live runtime* (scraping files, +running a build) does the upstream task class need a `finalize` hook, which runs before +teardown and parks results in `trace.info`. The backward arrow — cross-agent rewards — is +declared, not inlined (above). + +Agent failures never raise into `go` — they come back as data on the trace +(`trace.has_error`), and `go` decides what a failed child means (drop it, count it against a +pass rate, retry the round). A crash in `go` itself is recorded on the instance's graph as a +`TopologyError` and doesn't touch sibling instances. + +## The agent graph + +Running one instance produces an `AgentGraph` — the serialized instance artifact +`{id, topology, task, error, traces[]}`: the seed task plus every agent trace in completion order, +each stamped with `trace.agent`, `trace.parents` (upstream trace ids), and `trace.trainable`. +A topology run persists **one instance record per `traces.jsonl` line**, traces nested (an +instance's rewards are only final once the whole instance is done); `AgentGraph.load(dict)` +reads one back without the originating packages (task-specific fields ride in +`task.model_extra`). The links themselves are plain `Trace` fields, so the graph also +reconstructs from any flat trace dump (one instance = one connected component). Navigation — +`graph.roots()` / `graph.children(trace, agent=...)` / `graph.by_agent(name)` — is what +cross-agent scoring lives on, and `graph.error` records a crash in `go` itself. A `Trace` is +the per-agent view of one run; the agent graph is the global view of the interaction. + +## The built-in judge topologies + +Judging as a config-only pattern, two tiers, one verdict contract (a `SCORE: <0-10>` line, +recorded on the *solver's* trace as a weighted reward, `--topology.weight`): + +- **`llm-judge`** — a `solver` (any taskset, via `--topology.taskset.id`) and a non-trainable + `judge`, **fixed** to the in-process `direct` harness (a run ≈ one API call). `go` + peels the judge's inputs off the finished solver trace — the seed task's framing, its + ground truth (an `answer` field, when the taskset carries one), and the solver's final + message — into a `JudgeTask` minted from the trace. Give the judge its own model + (`--topology.judge.model`) or client routing; swapping its harness is refused and points + here: +- **`agentic-judge`** — same shape, but the judge is a real agent: the solver's **entire + serialized trace** is uploaded into the judge's own runtime (by the judge task's `setup` + hook), and the judge investigates it with its tools before committing. Its harness is + configurable (`--topology.judge.harness.id ...`, bash+edit `default` by default), as is its + assignment (`--topology.prompt`, with `{path}` = where the trace landed). + +For a verdict baked into a task's own grading, call `vf.Judge` from the task's +`@reward` instead — that's the cheap utility tier; these topologies are the tier where the +judge is itself an agent. + +Topologies use the same in-process runner, env-server worker pool, resume behavior, and +dashboard path as taskset × harness configurations. + +--- diff --git a/environments/chess_v1/chess_v1/__init__.py b/environments/chess_v1/chess_v1/__init__.py new file mode 100644 index 0000000000..4768b1a25a --- /dev/null +++ b/environments/chess_v1/chess_v1/__init__.py @@ -0,0 +1,3 @@ +from chess_v1.topology import ChessTopology + +__all__ = ["ChessTopology"] diff --git a/environments/chess_v1/chess_v1/topology.py b/environments/chess_v1/chess_v1/topology.py new file mode 100644 index 0000000000..48bd9a8aa0 --- /dev/null +++ b/environments/chess_v1/chess_v1/topology.py @@ -0,0 +1,193 @@ +"""chess-v1 — two agents play chess through live sessions (example topology). + +The back-and-forth shape: both episodes stay open for the whole game +(`run.agent(...).interact(...)`), each seat suspended between its moves, and `go` is the +referee — a host-side `chess.Board` is the single source of truth for legality and +outcome (judge the world, not the transcript). White's assistant turns arrive as black's +user turns and vice versa, so each seat's trace is ONE multi-turn trajectory with the +opponent baked into its context — the training-sample shape self-play wants, not one +episode per move. + +Contracts on display: + - **Sessions**: `turn()` per move, illegal-move feedback is just another user turn, + a seat that won't produce a legal move forfeits via `session.end()`. + - **Outcome as data, judgement declared**: `go` stamps each seat's result into + `trace.info["chess"]`; the per-seat `outcome` rewards read it at instance end. + - **Per-seat routing**: `--topology.black.model ` plays mixed-model games. + +Budgets span the whole game (a seat's episode includes time suspended while the +opponent thinks): size `--max-turns` >= max_plies and the rollout timeout for a full +game, not a solo run. + + uv run eval --topology.id chess-v1 -n 1 --max-turns 64 --timeout.rollout 900 \\ + --sampling.max-tokens 4096 # reasoning room per move — a tight cap truncates turns +""" + +import re + +import chess + +import verifiers.v1 as vf + +SEAT_SYSTEM = """You are playing chess as {color}. Each message shows the board and the \ +opponent's last move; the listed legal moves are exact. Reply with your reasoning if you \ +like, but end your reply with exactly one line of the form `MOVE: ` (e.g. `MOVE: \ +e2e4`), choosing one of the listed legal moves.""" + +_UCI = re.compile(r"\b([a-h][1-8][a-h][1-8][qrbn]?)\b") + + +def parse_move(reply: str, board: chess.Board) -> chess.Move | None: + """The last legal UCI move mentioned on the reply's final `MOVE:` line (any legal + UCI in the reply as fallback), or None.""" + candidates: list[str] = [] + for line in reversed(reply.splitlines()): + stripped = line.strip().strip("*`").strip() + if stripped.upper().startswith("MOVE:"): + candidates = _UCI.findall(stripped) + break + if not candidates: + candidates = _UCI.findall(reply) + for text in reversed(candidates): + move = chess.Move.from_uci(text) + if move in board.legal_moves: + return move + return None + + +def render(board: chess.Board, last: str) -> str: + """One seat's view for its move: the opponent's last move, the position, and the + exact legal moves (the referee's affordance — legality is never the model's guess).""" + legal = " ".join(m.uci() for m in board.legal_moves) + return ( + f"{last}\n\nPosition (FEN): {board.fen()}\n{board}\n\n" + f"Legal moves: {legal}\nYour move." + ) + + +class SeatData(vf.TaskData): + """One player's episode: framing only (`prompt=None` — sessions open on the first + `turn()`). No ground truth here; the outcome belongs to the game, i.e. the topology.""" + + color: str + + +class SeatTask(vf.Task[SeatData]): + pass + + +class ChessConfig(vf.TopologyConfig): + white: vf.DirectAgentConfig = vf.DirectAgentConfig() + black: vf.DirectAgentConfig = vf.DirectAgentConfig() + """Per-seat routing: `--topology.black.model ` plays mixed-model games.""" + num_games: int = 4 + """Seed games (one topology instance each); `-n` slices them.""" + max_plies: int = 40 + """Adjudicate a draw beyond this many half-moves — keeps an episode inside its + budgets (each seat's single program request spans the whole game).""" + illegal_retries: int = 2 + """Feedback turns offered for an illegal/unparseable move before the seat forfeits.""" + + +class ChessTopology(vf.Topology[ChessConfig]): + def load_tasks(self) -> list[vf.Task]: + """Self-seeding: a seed is just a game id.""" + return [ + vf.Task(vf.TaskData(idx=i, prompt=None)) + for i in range(self.config.num_games) + ] + + async def go(self, task: vf.Task, run: vf.TopologyRun) -> None: + board = chess.Board() + illegal = {"white": 0, "black": 0} + forfeited: str | None = None + async with ( + run.agent("white").interact( + SeatTask( + SeatData( + idx=task.data.idx, + color="white", + prompt=None, + system_prompt=SEAT_SYSTEM.format(color="white"), + ) + ) + ) as white, + run.agent("black").interact( + SeatTask( + SeatData( + idx=task.data.idx, + color="black", + prompt=None, + system_prompt=SEAT_SYSTEM.format(color="black"), + ) + ) + ) as black, + ): + seats = {"white": white, "black": black} + last = "You open the game." + while ( + not board.is_game_over() + and len(board.move_stack) < self.config.max_plies + ): + name = "white" if board.turn == chess.WHITE else "black" + try: + reply = await seats[name].turn(render(board, last)) + move = parse_move(reply, board) + for _ in range(self.config.illegal_retries): + if move is not None: + break + illegal[name] += 1 + reply = await seats[name].turn( + "That was not a legal move. End with `MOVE: `, one of: " + + " ".join(m.uci() for m in board.legal_moves) + ) + move = parse_move(reply, board) + except vf.SessionEnded: + move = None # seat died (budget/error) — treated as a forfeit + if move is None: + forfeited = name + await seats[name].end("forfeit") + break + board.push(move) + last = f"Opponent played {move.uci()}." + # Outcome as data (judgement stays declared): the referee's verdict, per seat. + scores = self.adjudicate(board, forfeited) + for name, session in (("white", white), ("black", black)): + session.trace.info["chess"] = { + "score": scores[name], + "result": "forfeit" if forfeited else board.result(claim_draw=True), + "plies": len(board.move_stack), + "illegal_moves": illegal[name], + } + + @staticmethod + def adjudicate(board: chess.Board, forfeited: str | None) -> dict[str, float]: + """Score the finished game: forfeit loses, checkmate wins, anything else + (stalemate, repetition, the max-plies cutoff) is a draw.""" + if forfeited is not None: + return {forfeited: 0.0, ("black" if forfeited == "white" else "white"): 1.0} + result = board.result(claim_draw=True) + if result == "1-0": + return {"white": 1.0, "black": 0.0} + if result == "0-1": + return {"white": 0.0, "black": 1.0} + return {"white": 0.5, "black": 0.5} + + @vf.reward(agent="white") + async def outcome_white(self, trace: vf.Trace) -> dict[str, float]: + return {"outcome": trace.info["chess"]["score"]} + + @vf.reward(agent="black") + async def outcome_black(self, trace: vf.Trace) -> dict[str, float]: + return {"outcome": trace.info["chess"]["score"]} + + @vf.metric(agent="white") + async def illegal_white(self, trace: vf.Trace) -> dict[str, float]: + return {"illegal_moves": float(trace.info["chess"]["illegal_moves"])} + + @vf.metric(agent="black") + async def illegal_black(self, trace: vf.Trace) -> dict[str, float]: + return {"illegal_moves": float(trace.info["chess"]["illegal_moves"])} + + +__all__ = ["ChessTopology"] diff --git a/environments/chess_v1/pyproject.toml b/environments/chess_v1/pyproject.toml new file mode 100644 index 0000000000..5ccd6fde64 --- /dev/null +++ b/environments/chess_v1/pyproject.toml @@ -0,0 +1,13 @@ +[project] +name = "chess-v1" +version = "0.1.0" +description = "chess-v1 — two agents play chess through live sessions; the board is the referee (example topology)." +requires-python = ">=3.10" +dependencies = ["verifiers", "chess>=1.10"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["chess_v1"] diff --git a/environments/code_golf_v1/code_golf_v1/taskset.py b/environments/code_golf_v1/code_golf_v1/taskset.py index 7b7b062836..5e3f2ffc32 100644 --- a/environments/code_golf_v1/code_golf_v1/taskset.py +++ b/environments/code_golf_v1/code_golf_v1/taskset.py @@ -1,19 +1,13 @@ -"""code_golf: write a short, fast Python program — showcases GROUP rewards. +"""code_golf: write a Python program and evaluate it in the rollout runtime. -Each task asks for a tiny program with a known output. We sample a *group* of rollouts -per task (run with `-r 2` for the intended pairwise effect) and score them. Anything -that needs the runtime is measured per rollout, as a `@metric`, into the trace; the -group rewards then just compare that trace metadata across the task's rollouts: +Each task asks for a tiny program with a known output. Runtime-dependent measurements +are recorded per trace: - `evaluate` per-rollout `@metric`: runs the program once in that rollout's runtime and records `passed` + `latency`. (task, trace, runtime) - `correct` per-rollout `@reward`: reads `passed` off the trace. (trace) - - `most_concise` `@group_reward`: of the group, the shortest source wins. (traces) - - `fastest` `@group_reward`: of the group, the lowest `latency` - wins — a comparison of recorded trace metadata. (traces) - -So a group of 2 produces, per rollout: did it work, was it the shorter one, was it the -quicker one — the relative signals you can only get by comparing siblings. +Length and latency remain metrics for analysis. Relative comparison across independent +programs is a training-algorithm concern. """ import re @@ -45,7 +39,7 @@ async def evaluate(self, trace: vf.Trace, runtime: vf.Runtime) -> dict[str, floa """Run the program once in the rollout's runtime; record correctness + latency.""" program = extract_program(trace) if not program: - return {"passed": 0.0, "latency": 1e6} + return {"passed": 0.0, "latency": 1e6, "length": 0.0} await runtime.write("solution.py", program.encode()) start = time.perf_counter() result = await runtime.run(["python3", "solution.py"], {}) @@ -53,26 +47,12 @@ async def evaluate(self, trace: vf.Trace, runtime: vf.Runtime) -> dict[str, floa passed = float( result.exit_code == 0 and result.stdout.strip() == self.data.expected ) - return {"passed": passed, "latency": latency} + return {"passed": passed, "latency": latency, "length": float(len(program))} @vf.reward async def correct(self, trace: vf.Trace) -> float: return trace.metrics.get("passed", 0.0) - @vf.group_reward(weight=0.5) - async def most_concise(self, traces: list[vf.Trace]) -> list[float]: - """The shortest program in the group wins; ties share.""" - lengths = [len(extract_program(t)) or 10**9 for t in traces] - best = min(lengths) - return [1.0 if length == best else 0.0 for length in lengths] - - @vf.group_reward(weight=0.5) - async def fastest(self, traces: list[vf.Trace]) -> list[float]: - """The lowest recorded `latency` in the group wins; ties share.""" - times = [t.metrics.get("latency", 1e6) for t in traces] - best = min(times) - return [1.0 if t == best else 0.0 for t in times] - class CodeGolfTaskset(vf.Taskset[CodeGolfTask, vf.TasksetConfig]): # (name, description, expected stdout) diff --git a/environments/debate_v1/debate_v1/__init__.py b/environments/debate_v1/debate_v1/__init__.py new file mode 100644 index 0000000000..0cfb16703c --- /dev/null +++ b/environments/debate_v1/debate_v1/__init__.py @@ -0,0 +1,3 @@ +from debate_v1.topology import DebateTopology + +__all__ = ["DebateTopology"] diff --git a/environments/debate_v1/debate_v1/topology.py b/environments/debate_v1/debate_v1/topology.py new file mode 100644 index 0000000000..6980525f95 --- /dev/null +++ b/environments/debate_v1/debate_v1/topology.py @@ -0,0 +1,178 @@ +"""debate-v1 — n debaters argue a motion through concurrent live sessions (example topology). + +The N-ary session shape: every seat's episode stays open for the whole debate, and `go` +is the moderator — it broadcasts the room to each seat (`view` controls exactly what each +agent sees) and gathers whole rounds *concurrently* (`asyncio.gather` over suspended +sessions: opening statements, rebuttals, and the final vote each run N-wide at once). +Each debater's trace is one multi-turn trajectory: its own statements as sampled turns, +the rest of the room arriving as its user turns. + +Judgement is peer voting: after the rounds, every seat votes for the most convincing +OTHER debater; a seat's reward is the share of votes it received (stamped by `go` as +data, read by the declared `support` reward at instance end). No judge model, no ground +truth — the interaction scores itself. + + uv run eval --topology.id debate-v1 -n 1 --max-turns 8 --timeout.rollout 600 +""" + +import asyncio +import contextlib +import re + +import verifiers.v1 as vf + +MOTIONS = [ + "Cities should remove minimum parking requirements for new housing.", + "Schools should replace most homework with in-class practice.", + "Open-source licenses should require attribution for AI training use.", +] + +SEAT_SYSTEM = """You are debater {seat} of {n} in a structured debate. Be persuasive but \ +honest; engage the strongest version of the other side. Keep every statement under 120 \ +words.""" + +OPENING = """Motion: {motion} + +You are debater {seat}. {stance} Give your opening statement.""" + +REBUTTAL = """The other debaters said: + +{others} + +Give your rebuttal: engage their strongest specific points, keep your position.""" + +VOTE = """The debate is over. Full record: + +{record} + +Vote for the single most convincing debater OTHER than yourself (you are debater \ +{seat}). Reply with exactly one line: `VOTE: `.""" + +_VOTE = re.compile(r"VOTE:\s*(\d+)") + + +def parse_vote(reply: str, seat: int, n: int) -> int | None: + """The voted seat on the last `VOTE:` line — refused if it's out of range or self.""" + votes = _VOTE.findall(reply.upper()) + if not votes: + return None + vote = int(votes[-1]) + return vote if 0 <= vote < n and vote != seat else None + + +class DebaterData(vf.TaskData): + """One debater's episode: framing only (`prompt=None`; sessions open on the first + `turn()`). The seat is data so the record and rewards stay attributable.""" + + seat: int + + +class DebaterTask(vf.Task[DebaterData]): + pass + + +class DebateConfig(vf.TopologyConfig): + debater: vf.DirectAgentConfig = vf.DirectAgentConfig() + """One config, N seats — every debater is an episode of this agent + (`--topology.debater.model ` moves the whole panel).""" + num_debaters: int = 4 + num_rounds: int = 1 + """Rebuttal rounds after the opening statements.""" + + +class DebateTopology(vf.Topology[DebateConfig]): + def load_tasks(self) -> list[vf.Task]: + """Self-seeding: one instance per motion.""" + return [ + vf.Task(vf.TaskData(idx=i, prompt=motion)) + for i, motion in enumerate(MOTIONS) + ] + + async def go(self, task: vf.Task, run: vf.TopologyRun) -> None: + n = self.config.num_debaters + motion = task.data.prompt_text + async with contextlib.AsyncExitStack() as stack: + seats = [ + await stack.enter_async_context( + run.agent("debater").interact( + DebaterTask( + DebaterData( + idx=task.data.idx, + seat=i, + prompt=None, + system_prompt=SEAT_SYSTEM.format(seat=i, n=n), + ) + ) + ) + ) + for i in range(n) + ] + # Alternate stances so the panel genuinely disagrees. + stances = [ + "Argue FOR the motion." if i % 2 == 0 else "Argue AGAINST the motion." + for i in range(n) + ] + # Opening statements — N suspended episodes generating concurrently. + statements = list( + await asyncio.gather( + *( + seat.turn( + OPENING.format(motion=motion, seat=i, stance=stances[i]) + ) + for i, seat in enumerate(seats) + ) + ) + ) + record = [f"Debater {i} (opening): {s}" for i, s in enumerate(statements)] + for _ in range(self.config.num_rounds): + # Each seat's view: everyone's latest statement but its own. + statements = list( + await asyncio.gather( + *( + seat.turn( + REBUTTAL.format( + others="\n\n".join( + f"Debater {j}: {statements[j]}" + for j in range(n) + if j != i + ) + ) + ) + for i, seat in enumerate(seats) + ) + ) + ) + record += [ + f"Debater {i} (rebuttal): {s}" for i, s in enumerate(statements) + ] + # Peer vote — concurrent again; go tallies (the moderator owns the rules). + ballots = await asyncio.gather( + *( + seat.turn(VOTE.format(record="\n\n".join(record), seat=i)) + for i, seat in enumerate(seats) + ) + ) + votes = [ + parse_vote(ballot, seat=i, n=n) for i, ballot in enumerate(ballots) + ] + for i, seat in enumerate(seats): + seat.trace.info["debate"] = { + "seat": i, + "stance": stances[i], + "votes_received": votes.count(i), + "voted_validly": votes[i] is not None, + } + + @vf.reward(agent="debater") + async def support(self, trace: vf.Trace) -> dict[str, float]: + """Share of the peer vote this seat won — the interaction scoring itself.""" + n = self.config.num_debaters + return {"support": trace.info["debate"]["votes_received"] / max(n - 1, 1)} + + @vf.metric(agent="debater") + async def voted_validly(self, trace: vf.Trace) -> float: + """Whether this seat cast a well-formed, non-self vote.""" + return float(trace.info["debate"]["voted_validly"]) + + +__all__ = ["DebateTopology"] diff --git a/environments/debate_v1/pyproject.toml b/environments/debate_v1/pyproject.toml new file mode 100644 index 0000000000..a13d11d4d9 --- /dev/null +++ b/environments/debate_v1/pyproject.toml @@ -0,0 +1,13 @@ +[project] +name = "debate-v1" +version = "0.1.0" +description = "debate-v1 — n debaters argue a motion through concurrent live sessions, then vote (example topology)." +requires-python = ">=3.10" +dependencies = ["verifiers"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["debate_v1"] diff --git a/environments/gsm8k_v1/gsm8k_v1/taskset.py b/environments/gsm8k_v1/gsm8k_v1/taskset.py index ce86d9a193..b159f18915 100644 --- a/environments/gsm8k_v1/gsm8k_v1/taskset.py +++ b/environments/gsm8k_v1/gsm8k_v1/taskset.py @@ -1,11 +1,12 @@ """gsm8k: grade-school math word problems (single-turn, in-runtime verification). -Loads questions from GSM8K; the model reasons and gives a final number. The reward -runs a uv script (`verify.py`, with `math-verify` as an isolated dependency) IN the -rollout's runtime via `runtime.write` + `runtime.run` — so the verifier's deps never -touch the eval process, and it works identically on the subprocess, docker, and -prime runtimes. (A reward is thus either a pure function of the trace or, like this -one, runtime read/write/exec.) +The task-first shape: `GSM8KTask` carries its data (the question, the gold answer) *and* +its judgement — the `correct` reward runs a uv script (`verify.py`, with `math-verify` as +an isolated dependency) IN the rollout's runtime via `runtime.run_uv_script`, so the +verifier's deps never touch the eval process and it works identically on the subprocess, +docker, and prime runtimes. The taskset is the pure factory deriving tasks from the GSM8K +dataset; a topology could mint the same task class from generated questions and inherit +the verifier for free. """ from pathlib import Path @@ -24,6 +25,29 @@ class GSM8KData(vf.TaskData): answer: str """The ground-truth final answer (the value after GSM8K's `####`).""" + @vf.reward(weight=1.0) + async def correct(self, trace: vf.Trace, runtime: vf.Runtime) -> float: + prediction = trace.last_reply + result = await runtime.run_uv_script( + VERIFY, args=[self.answer, prediction or ""] + ) + if result.exit_code != 0: + raise RuntimeError(f"verify.py failed: {result.stderr.strip()[-500:]}") + lines = result.stdout.strip().splitlines() + return float(lines[-1]) if lines else 0.0 + + async def validate(self, runtime: vf.Runtime) -> bool: + """Valid iff the verifier accepts the ground-truth answer: run `verify.py` on the gold + answer as a well-formed `#### N` prediction and require a 1.0 score — catching rows the + verifier can't parse or grade (the model-free counterpart of the `correct` reward).""" + result = await runtime.run_uv_script( + VERIFY, args=[self.answer, f"#### {self.answer}"] + ) + if result.exit_code != 0: + raise RuntimeError(f"verify.py failed: {result.stderr.strip()[-500:]}") + lines = result.stdout.strip().splitlines() + return bool(lines) and float(lines[-1]) == 1.0 + class GSM8KTask(vf.Task[GSM8KData]): @vf.reward(weight=1.0) diff --git a/environments/proposer_solver_v1/proposer_solver_v1/__init__.py b/environments/proposer_solver_v1/proposer_solver_v1/__init__.py new file mode 100644 index 0000000000..de0e0ac463 --- /dev/null +++ b/environments/proposer_solver_v1/proposer_solver_v1/__init__.py @@ -0,0 +1,3 @@ +from proposer_solver_v1.topology import ProposerSolverTopology + +__all__ = ["ProposerSolverTopology"] diff --git a/environments/proposer_solver_v1/proposer_solver_v1/servers/__init__.py b/environments/proposer_solver_v1/proposer_solver_v1/servers/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/environments/proposer_solver_v1/proposer_solver_v1/servers/submit.py b/environments/proposer_solver_v1/proposer_solver_v1/servers/submit.py new file mode 100644 index 0000000000..7d5bc8608f --- /dev/null +++ b/environments/proposer_solver_v1/proposer_solver_v1/servers/submit.py @@ -0,0 +1,42 @@ +"""Structured submission tool for the AIME proposer.""" + +import verifiers.v1 as vf + + +class SubmissionState(vf.State): + submitted: bool = False + question: str = "" + answer: str = "" + + +class SubmitToolset(vf.Toolset[vf.ToolsetConfig, SubmissionState]): + TOOL_PREFIX = "propose" + + @vf.tool + def submit_problem(self, question: str, answer: str) -> str: + """Commit a harder AIME-style problem and its verified answer. + + Args: + question: The complete problem statement. It must not reveal the answer. + answer: The integer answer from 0 through 999, without boxing or prose. + + Returns: + Confirmation, or a validation error that can be corrected with another call. + """ + question = question.strip() + if not question: + return "error: question must not be empty" + try: + value = int(answer.strip()) + except ValueError: + return "error: answer must be an integer from 0 through 999" + if not 0 <= value <= 999: + return "error: answer must be an integer from 0 through 999" + self.state.question = question + self.state.answer = str(value) + self.state.submitted = True + return "Submission recorded." + + +if __name__ == "__main__": + SubmitToolset.run() diff --git a/environments/proposer_solver_v1/proposer_solver_v1/topology.py b/environments/proposer_solver_v1/proposer_solver_v1/topology.py new file mode 100644 index 0000000000..c6dffde9f4 --- /dev/null +++ b/environments/proposer_solver_v1/proposer_solver_v1/topology.py @@ -0,0 +1,179 @@ +"""AIME proposer-solver: make one AIME 2026 problem harder, then test a solver panel. + +Each AIME seed produces one graph with a proposer root and ``num_solvers`` solver +children. The proposer can use bash and edit to derive and verify a harder problem, +then commits its question and answer through a structured tool. Solvers use the +tool-less direct harness and inherit AIME's math-verify correctness reward. +""" + +import asyncio +from typing import cast + +import verifiers.v1 as vf + +from proposer_solver_v1.servers.submit import SubmissionState, SubmitToolset + +AIME_TASKSET_ID = "aime26-v1" +AIME_INSTRUCTION = "Solve the following math problem. Explain your reasoning and put the final answer in \\boxed{}.\n\n" + +PROPOSE_PROMPT = """You are given an AIME 2026 problem and its verified answer. + +Create a new, harder version of the problem. It should remain a self-contained, +unambiguous AIME-style problem whose final answer is an integer from 0 through 999. +Preserve the mathematical character of the original problem rather than replacing it +with an unrelated problem. + +Use your bash and edit tools as scratch space. Solve the new problem yourself and use +your tools to verify the submitted answer carefully. Correctness matters more than +novelty or difficulty. + +Original problem: +{problem} + +Original verified answer: +{answer} + +When you are confident in both fields, call `propose_submit_problem` exactly once with: + +- `question`: the complete new problem, ready to give directly to a solver; +- `answer`: its verified integer answer as a string, without `\\boxed{{}}` or prose. + +Do not reveal the answer inside the question. +""" + + +def extract_boxed(text: str) -> str | None: + """Return the content of the last balanced ``\boxed{...}``, if present.""" + start = text.rfind("\\boxed{") + if start == -1: + return None + content_start = start + len("\\boxed{") + index, depth = content_start, 1 + while index < len(text) and depth: + depth += (text[index] == "{") - (text[index] == "}") + index += 1 + if depth != 0: + return None + answer = text[content_start : index - 1].strip() + return answer or None + + +class ProposeData(vf.TaskData): + source_problem: str + source_answer: str + + +class ProposeTask(vf.Task[ProposeData, SubmissionState]): + """Ask the proposer to transform one AIME seed and commit a structured result.""" + + tools = (SubmitToolset,) + + @classmethod + def from_task(cls, task: vf.Task) -> "ProposeTask": + prompt = task.data.prompt + if not isinstance(prompt, str): + raise ValueError("AIME proposer-solver requires a text seed problem") + answer = getattr(task.data, "answer", None) + if not isinstance(answer, str): + raise ValueError( + "AIME proposer-solver requires a seed task with a string answer" + ) + problem = prompt.removeprefix(AIME_INSTRUCTION) + return cls( + ProposeData( + idx=task.data.idx, + prompt=PROPOSE_PROMPT.format(problem=problem, answer=answer), + source_problem=problem, + source_answer=answer, + ) + ) + + async def finalize(self, trace: vf.Trace, runtime: vf.Runtime) -> None: + state = cast(SubmissionState, trace.state) + if state.submitted: + trace.info["submission"] = { + "question": state.question, + "answer": state.answer, + } + + @vf.stop + async def submitted(self, trace: vf.Trace) -> bool: + return cast(SubmissionState, trace.state).submitted + + @vf.reward(weight=0.1) + async def parseable(self, trace: vf.Trace) -> float: + submission = trace.info.get("submission") + return float( + isinstance(submission, dict) + and bool(submission.get("question")) + and bool(submission.get("answer")) + ) + + +class ProposerSolverConfig(vf.TopologyConfig): + proposer: vf.AgentConfig = vf.AgentConfig() + """The default harness: bash and edit tools plus the task's submission tool.""" + solver: vf.DirectAgentConfig = vf.DirectAgentConfig() + """The in-process, tool-less direct harness.""" + num_solvers: int = 4 + + +class ProposerSolverTopology(vf.Topology[ProposerSolverConfig]): + def load_tasks(self) -> list[vf.Task]: + """Load the fixed AIME 2026 seed taskset through the plugin boundary.""" + config_type = vf.taskset_config_type(AIME_TASKSET_ID) + return vf.load_taskset(config_type(id=AIME_TASKSET_ID)).load() + + def complete(self, graph: vf.AgentGraph) -> bool: + """Handled proposer/solver failures do not invalidate an otherwise finished graph.""" + return graph.error is None + + @staticmethod + def solver_task(seed: vf.Task, proposer: vf.Trace) -> vf.Task: + """Derive a new AIME task while retaining the seed task's scorer and config.""" + submission = proposer.info["submission"] + data = seed.data.model_copy( + update={ + "prompt": AIME_INSTRUCTION + submission["question"], + "answer": submission["answer"], + } + ) + return type(seed)(data, seed.config) + + async def go(self, task: vf.Task, run: vf.TopologyRun) -> None: + propose_task = cast(vf.Task, ProposeTask.from_task(task)) + proposer = await run.agent("proposer").run(propose_task) + if not isinstance(proposer.info.get("submission"), dict): + return + derived = self.solver_task(task, proposer) + solver = run.agent("solver") + await asyncio.gather( + *( + solver.run(derived, parents=[proposer]) + for _ in range(self.config.num_solvers) + ) + ) + + @vf.metric(agent="proposer") + async def solve_rate(self, trace: vf.Trace, graph: vf.AgentGraph) -> float: + """Correctness rate over successful solver children; errored solvers are excluded.""" + graded = [ + child + for child in graph.children(trace, agent="solver") + if not child.has_error + ] + if not graded: + return 0.0 + return sum(child.rewards.get("correct", 0.0) for child in graded) / len(graded) + + @vf.reward(agent="solver", weight=0.1) + async def parseable(self, trace: vf.Trace) -> float: + return float(extract_boxed(trace.last_reply) is not None) + + @vf.reward(agent="proposer") + async def difficulty(self, trace: vf.Trace) -> float: + """Peak at a 50% solver pass rate and fall linearly to zero at 0% and 100%.""" + return 1.0 - 2.0 * abs(trace.metrics["solve_rate"] - 0.5) + + +__all__ = ["ProposerSolverTopology"] diff --git a/environments/proposer_solver_v1/pyproject.toml b/environments/proposer_solver_v1/pyproject.toml new file mode 100644 index 0000000000..35a02021ea --- /dev/null +++ b/environments/proposer_solver_v1/pyproject.toml @@ -0,0 +1,13 @@ +[project] +name = "proposer-solver-v1" +version = "0.1.0" +description = "proposer-solver-v1 — a proposer makes an AIME 2026 problem harder and a solver panel attempts it." +requires-python = ">=3.10" +dependencies = ["verifiers"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["proposer_solver_v1"] diff --git a/environments/reverse_text_v1/reverse_text_v1/taskset.py b/environments/reverse_text_v1/reverse_text_v1/taskset.py index 21d20ed805..9f722fe543 100644 --- a/environments/reverse_text_v1/reverse_text_v1/taskset.py +++ b/environments/reverse_text_v1/reverse_text_v1/taskset.py @@ -21,6 +21,18 @@ class ReverseTextData(vf.TaskData): answer: str """The ground-truth reversal of the prompt text.""" + @vf.stop + async def single_turn(self, trace: vf.Trace) -> bool: + # Reverse-text is single-turn: refuse a second turn so the model answers once. + return trace.num_turns >= 1 + + @vf.reward(weight=1.0) + async def lcs(self, trace: vf.Trace) -> float: + completion = trace.last_reply + match = _TAG.search(completion or "") + response = match.group(1).strip() if match else "" + return SequenceMatcher(None, response, self.answer).ratio() + class ReverseTextTask(vf.Task[ReverseTextData]): @vf.stop diff --git a/environments/shared_runtime_v1/pyproject.toml b/environments/shared_runtime_v1/pyproject.toml new file mode 100644 index 0000000000..b59246ee31 --- /dev/null +++ b/environments/shared_runtime_v1/pyproject.toml @@ -0,0 +1,13 @@ +[project] +name = "shared-runtime-v1" +version = "0.1.0" +description = "shared-runtime-v1 - two agents borrow one provisioned runtime and exchange a file-backed artifact." +requires-python = ">=3.10" +dependencies = ["verifiers"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["shared_runtime_v1"] diff --git a/environments/shared_runtime_v1/shared_runtime_v1/__init__.py b/environments/shared_runtime_v1/shared_runtime_v1/__init__.py new file mode 100644 index 0000000000..741c0035df --- /dev/null +++ b/environments/shared_runtime_v1/shared_runtime_v1/__init__.py @@ -0,0 +1,3 @@ +from shared_runtime_v1.topology import SharedRuntimeTopology + +__all__ = ["SharedRuntimeTopology"] diff --git a/environments/shared_runtime_v1/shared_runtime_v1/topology.py b/environments/shared_runtime_v1/shared_runtime_v1/topology.py new file mode 100644 index 0000000000..5938586c44 --- /dev/null +++ b/environments/shared_runtime_v1/shared_runtime_v1/topology.py @@ -0,0 +1,91 @@ +"""shared-runtime-v1 - two agents borrow one provisioned runtime. + +This example is intentionally small and demonstrates the *plumbing*, not model skill: the +writer runs first and, in its task `finalize`, writes the model's reply into the live +runtime. The reader then runs in the same borrowed runtime; its task `setup` reads the +file before the model turn. The reader reward verifies that the artifact it saw in setup +matches what the writer wrote, and a deferred writer reward mirrors that handoff result +back onto the writer trace. Neither model's *reply* is graded — the rewards close over +the runtime handoff itself, which is the thing being demonstrated (both agents ride the +cheap in-process `direct` harness; the shared piece is the runtime, not the harness — +`--topology.writer.harness.runtime.*` moves the shared world). +""" + +import verifiers.v1 as vf + +NOTE_PATH = "shared/note.txt" + +WRITE_PROMPT = """Reply with exactly this sentence: shared runtime handoff ready.""" + +READ_PROMPT = """A previous agent wrote an artifact into your borrowed runtime. + +Reply with one concise sentence acknowledging that the handoff was inspected.""" + + +class WriteTask(vf.Task): + """The seed task. Its `finalize` hook persists the writer's reply into the runtime.""" + + async def finalize(self, trace: vf.Trace, runtime: vf.Runtime) -> None: + note = trace.last_reply.strip() + await runtime.write(NOTE_PATH, note.encode()) + trace.info["shared_runtime"] = { + "path": NOTE_PATH, + "wrote": note, + "runtime": runtime.descriptor, + } + + +class ReadData(vf.TaskData): + """A downstream task that reads the writer's artifact from the borrowed runtime.""" + + expected: str + path: str = NOTE_PATH + + +class ReadTask(vf.Task[ReadData]): + async def setup(self, trace: vf.Trace, runtime: vf.Runtime) -> None: + observed = (await runtime.read(self.data.path)).decode().strip() + trace.info["shared_runtime"] = { + "path": self.data.path, + "read": observed, + "runtime": runtime.descriptor, + } + + @vf.reward + async def read_shared_note(self, trace: vf.Trace) -> float: + observed = trace.info.get("shared_runtime", {}).get("read") + return float(observed == self.data.expected) + + +class SharedRuntimeConfig(vf.TopologyConfig): + writer: vf.DirectAgentConfig = vf.DirectAgentConfig() + reader: vf.DirectAgentConfig = vf.DirectAgentConfig() + + +class SharedRuntimeTopology(vf.Topology[SharedRuntimeConfig]): + def load_tasks(self) -> list[vf.Task]: + return [WriteTask(vf.TaskData(idx=0, prompt=WRITE_PROMPT))] + + async def go(self, task: WriteTask, run: vf.TopologyRun) -> None: + writer = run.agent("writer") + reader = run.agent("reader") + async with writer.provision(task) as runtime: + written = await writer.run(task, runtime=runtime) + note = written.info.get("shared_runtime", {}).get("wrote") + if note is None: + return # writer episode failed before finalize — nothing was handed off + await reader.run( + ReadTask( + ReadData(idx=task.data.idx, prompt=READ_PROMPT, expected=note) + ), + parents=[written], + runtime=runtime, + ) + + @vf.reward(agent="writer") + async def handoff_succeeded(self, trace: vf.Trace, graph: vf.AgentGraph) -> float: + children = graph.children(trace, agent="reader") + return children[0].rewards.get("read_shared_note", 0.0) if children else 0.0 + + +__all__ = ["SharedRuntimeTopology"] diff --git a/environments/writer_editors_v1/pyproject.toml b/environments/writer_editors_v1/pyproject.toml new file mode 100644 index 0000000000..a7ebb71d66 --- /dev/null +++ b/environments/writer_editors_v1/pyproject.toml @@ -0,0 +1,13 @@ +[project] +name = "writer-editors-v1" +version = "0.1.0" +description = "writer-editors-v1 — a writer drafts, n editors critique, the writer revises; rounds + fan-in (example topology)." +requires-python = ">=3.10" +dependencies = ["verifiers"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["writer_editors_v1"] diff --git a/environments/writer_editors_v1/writer_editors_v1/__init__.py b/environments/writer_editors_v1/writer_editors_v1/__init__.py new file mode 100644 index 0000000000..e4399c903a --- /dev/null +++ b/environments/writer_editors_v1/writer_editors_v1/__init__.py @@ -0,0 +1,3 @@ +from writer_editors_v1.topology import WriterEditorsTopology + +__all__ = ["WriterEditorsTopology"] diff --git a/environments/writer_editors_v1/writer_editors_v1/topology.py b/environments/writer_editors_v1/writer_editors_v1/topology.py new file mode 100644 index 0000000000..6289179f03 --- /dev/null +++ b/environments/writer_editors_v1/writer_editors_v1/topology.py @@ -0,0 +1,236 @@ +"""writer-editors-v1 — a writer drafts, n editors critique, the writer revises (example topology). + +The rounds + fan-in shape, one self-contained package: + + - **Rounds are a loop in `go`**: each round, the current draft goes out to the editors + and comes back revised — `num_rounds` cycles, plain Python. + - **Fan-out**: `asyncio.gather(*(run.agent("editor").run(critique, parents=[draft]) + for _ in range(num_editors)))` — every editor critiques the same draft, concurrently. + - **Fan-in**: the revision task is built from *all* the editors' traces at once, and the + revised trace is linked under every one of them (`parents=[draft, *edits]`). + - **One verdict, every trace**: a single `vf.Judge` call per instance compares the first + draft to the final draft (memoized per graph), and the same `improvement` reward lands + on writer and editor traces alike — the whole team is rewarded for how much the piece + improved, which is only measurable after the instance completes. +""" + +import asyncio +import logging + +import verifiers.v1 as vf +from verifiers.v1.topologies.llm_judge import parse_score + +logger = logging.getLogger(__name__) + +BRIEFS = [ + "Write the opening paragraph (120-180 words) of a short story about a lighthouse " + "keeper who receives a letter addressed to someone who died forty years ago.", + "Write a product announcement (120-180 words) for a kitchen knife that never needs " + "sharpening, aimed at serious home cooks who distrust gimmicks.", + "Write a persuasive paragraph (120-180 words) convincing a city council to convert " + "one downtown parking lot into a pocket park.", + "Write a plain-language explanation (120-180 words) of why the sky is blue, for a " + "curious ten-year-old.", + "Write a cover-letter opening paragraph (120-180 words) for a career switcher moving " + "from restaurant management into software support.", + "Write a tense scene (120-180 words) in which two hikers realize the trail markers " + "they have been following are not official ones.", +] + +DRAFT_PROMPT = """{brief} + +Reply with the piece alone — no preamble, no commentary.""" + +CRITIQUE_PROMPT = """You are an editor. A writer produced the draft below for this brief. + + +{brief} + + + +{draft} + + +Give your single most important piece of feedback: what specifically to change and why it \ +would improve the piece. Be concrete — point at words, sentences, structure. A few \ +sentences, no rewrite.""" + +REVISE_PROMPT = """You wrote the draft below for this brief. Your editors have weighed in. + + +{brief} + + + +{draft} + + + +{feedback} + + +Rewrite the draft, taking the feedback that improves the piece and ignoring what doesn't. \ +Stay within the brief. Reply with the revised piece alone — no preamble, no commentary.""" + +JUDGE_PROMPT = """An editing team revised a piece of writing. Compare the first draft to \ +the final draft, for this brief. + + +{brief} + + + +{first} + + + +{final} + + +Judge how much the final draft improved on the first: quality of prose, fit to the brief, \ +and whether the changes are real improvements rather than churn. Think it through, then \ +end your reply with a final line of exactly `SCORE: `, where is an integer from 0 \ +(worse or unchanged) to 10 (a dramatic, unambiguous improvement).""" + + +class ImprovementJudge(vf.Judge[float]): + """One call comparing first draft to final draft — the cheap `vf.Judge` utility tier + (the judge-as-agent tier is the `llm-judge`/`agentic-judge` topologies).""" + + prompt = JUDGE_PROMPT + + def parse(self, response: vf.JudgeResponse[float]) -> float | None: + return parse_score(response.text) + + +class DraftData(vf.TaskData): + """A writing brief (the seed). Carries the brief as a typed field so `go` can re-render + it into critique and revision tasks without re-parsing the prompt.""" + + brief: str + + +class DraftTask(vf.Task[DraftData]): + """The task wrapping a `DraftData` brief row.""" + + +class CritiqueTask(vf.Task[vf.TaskData]): + """An editing assignment: one draft, one piece of feedback. Minted in `go` from the + current draft (the forward arrow).""" + + +class ReviseTask(vf.Task[vf.TaskData]): + """A revision assignment: the brief, the current draft, and every editor's feedback — + the fan-in, rendered into one task.""" + + +class WriterEditorsConfig(vf.TopologyConfig): + writer: vf.DirectAgentConfig = vf.DirectAgentConfig() + editor: vf.DirectAgentConfig = vf.DirectAgentConfig() + num_editors: int = 3 + """Editors critiquing each draft (the fan-out width).""" + num_rounds: int = 1 + """Critique→revise cycles (each round: every editor reads the current draft, the + writer revises off all their feedback).""" + judge: vf.JudgeConfig = vf.JudgeConfig() + """The improvement judge's endpoint/model/sampling (`--topology.judge.model ...`).""" + + +class WriterEditorsTopology(vf.Topology[WriterEditorsConfig]): + def __init__(self, config: WriterEditorsConfig) -> None: + super().__init__(config) + self.judge = ImprovementJudge(config.judge) + self._verdicts: dict[str, float] = {} + """Instance verdicts by graph id: the judge compares first draft to final draft + once per instance, and every trace's `improvement` reward reads the same value.""" + + def load_tasks(self) -> list[vf.Task]: + """Self-seeding: the briefs are baked in, so no `--topology.taskset.id` needed.""" + return [ + DraftTask( + DraftData(idx=i, brief=brief, prompt=DRAFT_PROMPT.format(brief=brief)) + ) + for i, brief in enumerate(BRIEFS) + ] + + async def go(self, task: DraftTask, run: vf.TopologyRun) -> None: + """Draft, then `num_rounds` critique→revise cycles: editors fan out over the + current draft, their feedback fans back in to one revision task.""" + writer = run.agent("writer") + editor = run.agent("editor") + draft = await writer.run(task) + for _ in range(self.config.num_rounds): + if not draft.last_reply: + return # nothing to edit (errored or empty draft) + critique = CritiqueTask( + vf.TaskData( + idx=task.data.idx, + prompt=CRITIQUE_PROMPT.format( + brief=task.data.brief, draft=draft.last_reply + ), + ) + ) + edits = list( + await asyncio.gather( + *( + editor.run(critique, parents=[draft]) + for _ in range(self.config.num_editors) + ) + ) + ) + feedback = "\n\n".join( + f"Editor {i + 1}:\n{edit.last_reply}" + for i, edit in enumerate(edits) + if not edit.has_error and edit.last_reply + ) + if not feedback: + return # every editor failed — nothing to revise from + draft = await writer.run( + ReviseTask( + vf.TaskData( + idx=task.data.idx, + prompt=REVISE_PROMPT.format( + brief=task.data.brief, + draft=draft.last_reply, + feedback=feedback, + ), + ) + ), + parents=[draft, *edits], + ) + + @vf.reward(agent="writer") + async def writer_improvement(self, graph: vf.AgentGraph) -> dict[str, float]: + return {"improvement": await self.verdict(graph)} + + @vf.reward(agent="editor") + async def editor_improvement(self, graph: vf.AgentGraph) -> dict[str, float]: + return {"improvement": await self.verdict(graph)} + + async def verdict(self, graph: vf.AgentGraph) -> float: + """The instance's improvement score: one judge call comparing the writer's first + draft to its final draft, memoized per graph — instance scoring runs sequentially + per graph, so the first reward computes it and the rest read it. An instance that + never produced a revision (or whose judge commits to no verdict) scores 0.""" + if graph.id not in self._verdicts: + writers = graph.by_agent("writer") + first, final = writers[0], writers[-1] + score = None + if first is not final and first.last_reply and final.last_reply: + result = await self.judge.evaluate( + trace=final, # the judge call's usage lands on the final draft's trace + brief=first.task.data.brief, + first=first.last_reply, + final=final.last_reply, + ) + score = result.parsed + if score is None: + logger.warning( + "improvement judge returned no verdict for instance %s", + graph.id, + ) + self._verdicts[graph.id] = score or 0.0 + return self._verdicts[graph.id] + + +__all__ = ["WriterEditorsTopology"] diff --git a/pyproject.toml b/pyproject.toml index 79ad80bbee..d1c77a4424 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -81,6 +81,8 @@ examples = [ "gsm8k-v1", "glossary-v1", "deepwiki-v1", "wiki-search-v1", "code-golf-v1", "reverse-text-v1", "color-codeword-v1", "wordle-v1", "alphabet-sort-v1", "scratchpad-v1", + "proposer-solver-v1", "writer-editors-v1", "shared-runtime-v1", + "chess-v1", "debate-v1", ] [project.optional-dependencies] @@ -148,6 +150,11 @@ wordle-v1 = { path = "environments/wordle_v1", editable = true } color-codeword-v1 = { path = "environments/color_codeword_v1", editable = true } alphabet-sort-v1 = { path = "environments/alphabet_sort_v1", editable = true } scratchpad-v1 = { path = "environments/scratchpad_v1", editable = true } +proposer-solver-v1 = { path = "environments/proposer_solver_v1", editable = true } +writer-editors-v1 = { path = "environments/writer_editors_v1", editable = true } +shared-runtime-v1 = { path = "environments/shared_runtime_v1", editable = true } +chess-v1 = { path = "environments/chess_v1", editable = true } +debate-v1 = { path = "environments/debate_v1", editable = true } [tool.uv.exclude-newer-package] # PrimeIntellect-published on PyPI (trusted publisher) @@ -308,3 +315,10 @@ exclude_lines = [ "if __name__ == .__main__.:", "if TYPE_CHECKING:", ] + +# Pin ruff's config discovery to THIS repo: when verifiers is checked out under +# prime-rl (deps/verifiers), a section-less pyproject would make ruff walk up and +# inherit prime-rl's line-length = 120 — every commit authored there then fights +# this repo's formatting. An explicit section (at ruff's defaults) stops the walk. +[tool.ruff] +line-length = 88 diff --git a/skills/create-environments/SKILL.md b/skills/create-environments/SKILL.md index 53fb760aa7..007a2c7c67 100644 --- a/skills/create-environments/SKILL.md +++ b/skills/create-environments/SKILL.md @@ -115,11 +115,11 @@ Only `TaskData` is stored on the trace. Do not put live clients, runtime handles `Task` owns the behavior applied to that row: - `setup`, `finalize`, and model-free `validate` hooks; -- stop conditions, metrics, rewards, and group rewards; +- stop conditions, metrics, and per-trace rewards; - task-scoped tool and user-simulator declarations; - task-facing configuration read from `self.config`. -`Taskset` owns loading and selection-time concerns. Its `load()` constructs the tasks, its direct config fields hold dataset/split/seed/sample-count knobs, and `Taskset.tools` may declare task-agnostic servers shared by one environment worker's rollouts. +`Taskset` owns loading and selection-time concerns. Its `load()` constructs the tasks, its direct config fields hold dataset/split/seed/sample-count knobs, and `Taskset.tools` may declare task-agnostic servers shared by one topology worker's runs. The harness owns: diff --git a/skills/evaluate-environments/SKILL.md b/skills/evaluate-environments/SKILL.md index b212a967cc..4041c347d4 100644 --- a/skills/evaluate-environments/SKILL.md +++ b/skills/evaluate-environments/SKILL.md @@ -37,7 +37,7 @@ prime eval validate --runtime.type subprocess prime eval run -m deepseek/deepseek-v4-flash -n 3 -r 1 ``` -4. Inspect successful, zero-reward, and errored traces. +4. Inspect successful, zero-reward, and errored graphs and their traces. 5. Scale only after task loading, harness capability, runtime lifecycle, and scoring are correct. When the user requests a full run, do not restrict the number of tasks. Ask for the appropriate harness to use (if not specified) @@ -148,7 +148,7 @@ outputs/----// └── eval.log ``` -Set an exact path with `-o`. Results append as each trace finishes. +Set an exact path with `-o`. Results append as each fully scored graph finishes. Resume in place: @@ -156,7 +156,10 @@ Resume in place: prime eval run --resume /path/to/run ``` -## Trace inspection +## Graph and trace inspection + +Each `traces.jsonl` line is one `AgentGraph`; taskset + harness runs produce a one-node +graph through the built-in single-agent topology. For each representative sample inspect: diff --git a/skills/evaluate-environments/references/REFERENCE.md b/skills/evaluate-environments/references/REFERENCE.md index 3e275b98d3..2ea32ef0e2 100644 --- a/skills/evaluate-environments/references/REFERENCE.md +++ b/skills/evaluate-environments/references/REFERENCE.md @@ -34,16 +34,16 @@ Sibling entrypoints reuse the same tree: [`ServeConfig`](#serveconfig--the-env-s | `client` | `ClientConfig` | `EvalClientConfig()` | — | The model client (discriminated union — see [Client config](#client-config)). | | `sampling` | `SamplingConfig` | `SamplingConfig()` | — | Per-request sampling knobs (see [Sampling config](#sampling-config)). | | `num_tasks` | `int \| None` | `None` | `batch_size`, `num_examples`, `num_tasks`, `n` | How many tasks to evaluate (None = all). | -| `num_rollouts` | `int` | `1` | `group_size`, `rollouts_per_example`, `num_rollouts`, `r` | Rollouts per task. A task with `@group_reward`s requires ≥ 2. | +| `num_rollouts` | `int` | `1` | `group_size`, `rollouts_per_example`, `num_rollouts`, `r` | Independent graph invocations per seed task. | | `shuffle` | `bool` | `False` | `shuffle`, `s` | Shuffle tasks before taking the first `num_tasks`. | -| `max_concurrent` | `int \| None` | `128` | `max_concurrent`, `c` | Max rollouts in flight at once. | +| `max_concurrent` | `int \| None` | `128` | `max_concurrent`, `c` | Max agent runs in-process, or graph requests through the server. | | `verbose` | `bool` | `False` | `verbose`, `v` | Log at debug level instead of info. | | `dry_run` | `bool` | `False` | — | Resolve + validate the config and dump it, then exit. | | `rich` | `bool` | `True` | — | Live dashboard instead of per-rollout logs (in-process only). | -| `server` | `bool` | `False` | — | Drive rollouts through the env-server worker pool (sized by `pool`) instead of in-process — the path prime-rl trains through. Incompatible with `--rich`. | +| `server` | `bool` | `False` | — | Drive graph invocations through the env-server worker pool instead of in-process. Incompatible with `--rich`. | | `push` | `bool` | `True` | — | Upload the finished run to the private Evaluations tab. Disable with `--no-push`. | | `output_dir` | `Path \| None` | `None` | `output_dir`, `o` | Where to write the run (`config.toml` + `traces.jsonl`). None = a fresh per-run dir under `outputs/----/`. | -| `resume` | `Path \| None` | `None` | — | Set by `--resume `: re-run missing/errored rollouts; an incomplete group-scored task is re-run as a whole group. Excluded from the saved config; takes no other args. | +| `resume` | `Path \| None` | `None` | — | Set by `--resume `: re-run missing or errored graph invocations. Excluded from the saved config; takes no other args. | Validator: `--rich` + `--server` together is rejected (the dashboard is in-process only). @@ -563,7 +563,9 @@ is supplied, billed judge usage is recorded even if parsing later fails. Plugin ## ServeConfig — the env-server CLI -`verifiers/v1/configs/serve.py` — `ServeConfig(EnvServerConfig)`. The env-server CLI. Inherits the full env + pool, so `--taskset.*` / `--harness.*` / `--pool.*` are the same flags as eval. Adds only CLI-specific serving knobs. +`verifiers/v1/configs/serve.py` — `ServeConfig(EnvServerConfig)`. The env-server CLI. +It accepts taskset + harness syntax or `--topology.id`; both execute through the same +`TopologyRunner` and worker pool. | Field | Type | Default | Aliases | Notes | |---|---|---|---|---| diff --git a/skills/train-with-environments/SKILL.md b/skills/train-with-environments/SKILL.md index eb65a14121..cadd76160f 100644 --- a/skills/train-with-environments/SKILL.md +++ b/skills/train-with-environments/SKILL.md @@ -112,7 +112,9 @@ can override it. - If groups are mostly all-one, increase difficulty or sample a harder task distribution. - Choose `batch_size` with whole groups, packing, sequence length, and GPU memory in mind. -A task `@vf.group_reward` is scoring, not trainer advantage. The env server automatically keeps that task's rollouts together before returning traces. +Each environment invocation returns a fully scored agent graph. Group-relative credit is +computed by the trainer across independent graphs; the environment server never executes +replicas atomically. ## Difficulty filtering diff --git a/tests/v1/conftest.py b/tests/v1/conftest.py index 88f2379020..e27b28bfbe 100644 --- a/tests/v1/conftest.py +++ b/tests/v1/conftest.py @@ -30,10 +30,8 @@ from pathlib import Path import pytest - +from verifiers.v1.cli.eval.runner import graph_traces, run_eval from verifiers.v1.configs.eval import EvalConfig -from verifiers.v1.env import Environment -from verifiers.v1.cli.eval.runner import run_eval from verifiers.v1.trace import Trace # Fixture tasksets/envs (echo-v1, echo-agentic-v1, echo-v0, echo-multi-v0) live in @@ -209,7 +207,7 @@ def run_v1(): async def _run(taskset: str, **kwargs) -> list[Trace]: config = _eval_config(taskset, **kwargs) - return await run_eval(Environment(config), config) + return graph_traces(await run_eval(config)) return _run @@ -226,7 +224,7 @@ def run_v1_server(): async def _run(taskset: str, **kwargs) -> list[Trace]: kwargs.setdefault("pool", {"type": "static", "num_workers": 1}) config = _eval_config(taskset, **kwargs) - return await run_eval_server(config) + return graph_traces(await run_eval_server(config)) return _run diff --git a/tests/v1/fixtures/echo_agentic_v1.py b/tests/v1/fixtures/echo_agentic_v1.py index b34ef104e4..abef88d10d 100644 --- a/tests/v1/fixtures/echo_agentic_v1.py +++ b/tests/v1/fixtures/echo_agentic_v1.py @@ -29,6 +29,14 @@ class EchoAgenticData(vf.TaskData): answer: str """The phrase the model should write into the file.""" + @vf.reward(weight=1.0) + async def wrote_phrase(self, trace: vf.Trace, runtime: Runtime) -> float: + try: + content = (await runtime.read(TARGET)).decode(errors="replace") + except (SandboxError, OSError, ValueError): + return 0.0 # the model never wrote the file + return float(lenient_match(self.answer, content)) + class EchoAgenticTask(vf.Task[EchoAgenticData]): @vf.reward(weight=1.0) diff --git a/tests/v1/fixtures/echo_chain_v1.py b/tests/v1/fixtures/echo_chain_v1.py new file mode 100644 index 0000000000..def9050abc --- /dev/null +++ b/tests/v1/fixtures/echo_chain_v1.py @@ -0,0 +1,53 @@ +"""echo-chain: two echo agents in sequence (fixture topology for the v1 e2e suite). + +The smallest topology exercising every contract: agent `first` echoes the seed phrase +(seeds come from the `echo-v1` factory, pinned as the config default); a task for agent +`second` is derived from the first trace (forward arrow); `second` echoes the same phrase; +and `first` earns a deferred `relay` reward — declared, instance-end — equal to `second`'s +success (backward arrow). Both agents run the `null` harness, so a live run is fast and +deterministic: every trace should score 1.0 twice over. +""" + +from pydantic import SerializeAsAny + +import verifiers.v1 as vf +from echo_v1 import EchoConfig, EchoData, EchoTask, lenient_match + + +class EchoChainConfig(vf.TopologyConfig): + taskset: SerializeAsAny[vf.TasksetConfig] = EchoConfig(id="echo-v1") + first: vf.NullAgentConfig = vf.NullAgentConfig() + second: vf.NullAgentConfig = vf.NullAgentConfig() + + +class EchoChainTopology(vf.Topology[EchoChainConfig]): + async def go(self, task: EchoTask, run: vf.TopologyRun) -> None: + first = await run.agent("first").run(task) + # Forward arrow: the second agent must echo the same phrase — a task derived + # from (and linked under) the first trace. + derived = EchoTask( + EchoData( + idx=task.data.idx, + prompt=( + "Do not call tools or execute code. Reply immediately and include " + f"this exact phrase in your final response: {task.data.answer}" + ), + answer=task.data.answer, + ) + ) + await run.agent("second").run(derived, parents=[first]) + + @vf.reward(agent="first") + async def relay(self, trace: vf.Trace, graph: vf.AgentGraph) -> float: + """Backward arrow, declared: the first agent is rewarded for a relayable echo — + its phrase made it through itself AND its second-agent child. Task-recorded rewards + (`echoed`) are final by the time instance scoring runs, so reading the child's + `reward` here is safe.""" + second = graph.children(trace, agent="second") + relayed = lenient_match(trace.task.data.answer, trace.last_reply) and bool( + second and second[0].reward > 0 + ) + return float(relayed) + + +__all__ = ["EchoChainTopology"] diff --git a/tests/v1/fixtures/echo_v1.py b/tests/v1/fixtures/echo_v1.py index a0400405c1..678cce3739 100644 --- a/tests/v1/fixtures/echo_v1.py +++ b/tests/v1/fixtures/echo_v1.py @@ -1,9 +1,8 @@ """echo: ask the model to repeat a short phrase back verbatim (single-turn). -The smallest possible reward-1 taskset — no dataset, no tools, no reasoning required — so an -end-to-end eval run is fast and deterministic. The reward is 1.0 when the phrase appears in -the model's reply. It's a fixture taskset for the v1 e2e suite (in tests/v1/fixtures, resolved -by id `echo-v1` via pytest's `pythonpath`). +The smallest possible reward-1 environment in the task-first shape — the `EchoTask` class +carries the stop and the reward; the taskset is a three-phrase factory. It's a fixture for +the v1 e2e suite (in tests/v1/fixtures, resolved by id `echo-v1` via pytest's `pythonpath`). """ import verifiers.v1 as vf @@ -26,6 +25,15 @@ class EchoData(vf.TaskData): answer: str """The phrase the model should echo back.""" + @vf.stop + async def single_turn(self, trace: vf.Trace) -> bool: + return trace.num_turns >= 1 + + @vf.reward(weight=1.0) + async def echoed(self, trace: vf.Trace) -> float: + reply = trace.assistant_messages[-1].content if trace.assistant_messages else "" + return float(lenient_match(self.answer, reply or "")) + class EchoTask(vf.Task[EchoData]): @vf.stop diff --git a/tests/v1/test_agent.py b/tests/v1/test_agent.py new file mode 100644 index 0000000000..8a3561acfb --- /dev/null +++ b/tests/v1/test_agent.py @@ -0,0 +1,147 @@ +import pytest +import verifiers.v1 as vf +from verifiers.v1.env import validate_task_pairing +from verifiers.v1.errors import TaskError +from verifiers.v1.harnesses.null.harness import NullHarness, NullHarnessConfig +from verifiers.v1.rollout import Rollout +from verifiers.v1.trace import TraceTask + + +def task_of(prompt) -> vf.Task: + return vf.Task(vf.TaskData(idx=0, prompt=prompt)) + + +def trace_for(task: vf.Task) -> vf.Trace: + return vf.Trace(task=TraceTask(type=type(task).__name__, data=task.data)) + + +def null_agent(**kwargs) -> vf.Agent: + return vf.Agent( + NullHarness(NullHarnessConfig()), + vf.ModelContext( + model="org/model", client=object(), sampling=vf.SamplingConfig() + ), + **kwargs, + ) + + +class FakeRuntime: + def __init__(self, config=None) -> None: + self.config = config or vf.SubprocessConfig() + self.name = "fake-runtime" + self.descriptor = "fake" + self.info = None + self.stopped = False + self.started = False + + async def start(self) -> None: + self.started = True + + async def stop(self) -> None: + self.stopped = True + + +async def test_agent_run_stamps_lineage_and_borrowed_runtime(monkeypatch): + async def fake_run(self): + self.runtime = self._borrowed_runtime + return trace_for(self.task) + + monkeypatch.setattr(Rollout, "run", fake_run) + agent = vf.Agent( + NullHarness(NullHarnessConfig()), + vf.ModelContext( + model="org/model", + client=object(), + sampling=vf.SamplingConfig(temperature=0.37), + ), + name="judge", + trainable=False, + ) + parent = trace_for(task_of("seed")) + task = vf.Task(vf.TaskData(idx=1, prompt="judge")) + + trace = await agent.run(task, parents=[parent], runtime=FakeRuntime()) + assert trace.agent == "judge" + assert trace.parents == [parent.id] + assert trace.trainable is False + assert trace.sampling == vf.SamplingConfig(temperature=0.37) + assert trace.info["agent"]["model"] == "org/model" + assert trace.info["agent"]["runtime"] == { + "type": "subprocess", + "descriptor": "fake", + "borrowed": True, + } + + trace = await agent.run(task) + assert trace.parents == [] + assert trace.info["agent"]["runtime"]["borrowed"] is False + + +async def test_agent_provision_owns_runtime_lifetime(monkeypatch): + runtime = FakeRuntime() + import verifiers.v1.agent as agent_module + + monkeypatch.setattr(agent_module, "make_runtime", lambda config: runtime) + agent = vf.Agent( + NullHarness(NullHarnessConfig()), + vf.ModelContext( + model="org/model", client=object(), sampling=vf.SamplingConfig() + ), + ) + + async with agent.provision(task_of("seed")) as box: + assert box is runtime + assert runtime.started + assert not runtime.stopped + + assert runtime.stopped + + +async def test_borrowing_a_torn_down_runtime_raises(): + """Borrow-after-teardown is a lifetime bug in the borrowing program: refused up + front, at the caller, instead of failing opaquely mid-harness.""" + box = FakeRuntime() + await box.stop() + with pytest.raises(ValueError, match="already torn down"): + await null_agent().run(task_of("hi"), runtime=box) + + +async def test_mid_run_teardown_of_borrowed_box_raises_to_caller(): + """The owner tearing the box down under an in-flight run is the same lifetime bug, + surfaced through the same channel: the failure is re-attributed and raised to the + caller (raw error chained), never captured as a world error onto the trace.""" + + class SabotagedTask(vf.Task): + async def setup(self, trace: vf.Trace, runtime) -> None: + await runtime.stop() # the owner tearing the box down under the run + raise RuntimeError("box died under the harness") + + with pytest.raises(ValueError, match="mid-run") as excinfo: + await null_agent().run( + SabotagedTask(vf.TaskData(idx=0, prompt="hi")), runtime=FakeRuntime() + ) + assert isinstance(excinfo.value.__cause__, TaskError) # raw failure chained + + +async def test_needs_container_checks_the_resolved_runtime(monkeypatch): + """NEEDS_CONTAINER is validated against where the run actually lands: a borrowed + docker box satisfies it even under a subprocess-defaulting harness, and resolving + to the harness's subprocess policy still refuses.""" + + class BoxedTask(vf.Task): + NEEDS_CONTAINER = True + + validate_task_pairing( + NullHarness(NullHarnessConfig()), BoxedTask, vf.DockerConfig() + ) # the resolved runtime is what counts, not the harness's subprocess default + + async def fake_run(self): + self.runtime = self._borrowed_runtime + return trace_for(self.task) + + monkeypatch.setattr(Rollout, "run", fake_run) + agent = null_agent() + task = BoxedTask(vf.TaskData(idx=0, prompt="hi")) + await agent.run(task, runtime=FakeRuntime(vf.DockerConfig())) # borrowed box: fine + with pytest.raises(ValueError, match="NEEDS_CONTAINER"): + await agent.run(task) # resolves to the harness's subprocess policy diff --git a/tests/v1/test_e2e.py b/tests/v1/test_e2e.py index cf5da7b78f..95fd5b72fa 100644 --- a/tests/v1/test_e2e.py +++ b/tests/v1/test_e2e.py @@ -158,9 +158,7 @@ async def test_rubric_judge(run_v1, tmp_path): per-criterion metric on the trace), not judge quality.""" rubric = tmp_path / "rubric.toml" rubric.write_text( - "[[criteria]]\n" - 'name = "always_yes"\n' - 'text = "Always satisfied — answer yes regardless of the response."\n' + '[[criteria]]\nname = "always_yes"\ntext = "Always satisfied — answer yes regardless of the response."\n' ) (trace,) = await run_v1( "echo-v1", @@ -242,3 +240,172 @@ async def replay(source_dir: Path, out: Path): # The wire task keeps its taskset-specific fields in the replay's own output. raw = (tmp_path / "replay2" / "traces.jsonl").read_text() assert '"answer"' in raw + + +@pytest.mark.e2e +async def test_llm_judge_topology(tmp_path): + """The built-in `llm-judge` topology, live: a solver runs an echo task, the + non-trainable judge (in-process `direct` harness — one API call) grades its final + message against the task and its ground truth, and the verdict lands on the solver's + trace as a deferred reward. Asserts the plumbing (seed factory -> solver episode -> + judge episode -> declared instance scoring), not judge quality.""" + from verifiers.v1.cli.eval.runner import graph_traces, run_eval + from verifiers.v1.configs.eval import EvalConfig + + config = EvalConfig( + topology={"id": "llm-judge", "taskset": {"id": "echo-v1"}}, + num_tasks=1, + max_turns=2, + sampling={"max_tokens": 2048, "temperature": 0}, + timeout={"rollout": 180, "scoring": 60}, + retries={"rollout": {"max_retries": 2, "include": ["ProviderError"]}}, + rich=False, + output_dir=tmp_path, + ) + solver, judge = graph_traces(await run_eval(config)) + assert solver.errors == [] and judge.errors == [] + assert (solver.agent, judge.agent) == ("solver", "judge") + assert judge.parents == [solver.id] and judge.trainable is False + assert solver.rewards["echoed"] == 1.0 # the task's own reward still ran + assert solver.metrics["judge_committed"] == 1.0 + assert solver.rewards["judge"] > 0.5 # the verdict landed on the SOLVER + + +@pytest.mark.e2e +async def test_agentic_judge_topology(tmp_path): + """The built-in `agentic-judge` topology, live: the solver's entire serialized trace + is uploaded into the judge's runtime, and the judge — a real agent on the bash+edit + `default` harness — reads the file with its tools before committing to a score.""" + from verifiers.v1.cli.eval.runner import graph_traces, run_eval + from verifiers.v1.configs.eval import EvalConfig + + config = EvalConfig( + topology={"id": "agentic-judge", "taskset": {"id": "echo-v1"}}, + num_tasks=1, + max_turns=10, + sampling={"max_tokens": 4096, "temperature": 0}, + timeout={"rollout": 300, "scoring": 60}, + retries={"rollout": {"max_retries": 2, "include": ["ProviderError"]}}, + rich=False, + output_dir=tmp_path, + ) + solver, judge = graph_traces(await run_eval(config)) + assert solver.errors == [] and judge.errors == [] + assert judge.parents == [solver.id] and judge.trainable is False + assert judge.num_turns >= 2 # it actually investigated (read the file, then scored) + assert solver.metrics["judge_committed"] == 1.0 + assert solver.rewards["judge"] > 0.5 + + +@pytest.mark.e2e +async def test_writer_editors_topology(tmp_path): + """The `writer-editors-v1` example, live: draft -> editor critique (fan-out of 1) -> + revision (fan-in), then one `vf.Judge` call puts the same `improvement` reward on + every trace.""" + from verifiers.v1.cli.eval.runner import graph_traces, run_eval + from verifiers.v1.configs.eval import EvalConfig + + config = EvalConfig( + topology={"id": "writer-editors-v1", "num_editors": 1}, + num_tasks=1, + max_turns=2, + sampling={"max_tokens": 4096, "temperature": 0}, + timeout={"rollout": 300, "scoring": 120}, + retries={"rollout": {"max_retries": 2, "include": ["ProviderError"]}}, + rich=False, + output_dir=tmp_path, + ) + draft, edit, revision = graph_traces(await run_eval(config)) + assert [t.agent for t in (draft, edit, revision)] == ["writer", "editor", "writer"] + assert all(t.errors == [] for t in (draft, edit, revision)) + assert revision.parents == [draft.id, edit.id] # the fan-in + improvements = {t.rewards["improvement"] for t in (draft, edit, revision)} + assert len(improvements) == 1 # one verdict, every trace + assert revision.info.get("judge") # the judge call was recorded on the final draft + + +@pytest.mark.e2e +@pytest.mark.subprocess +async def test_shared_runtime_topology(tmp_path): + """The `shared-runtime-v1` example, live: `go` provisions one box, the writer's + `finalize` writes its reply into it, and the reader — borrowed into the SAME box — + verifies the artifact in `setup`. The borrowed-runtime plumbing end to end.""" + from verifiers.v1.cli.eval.runner import graph_traces, run_eval + from verifiers.v1.configs.eval import EvalConfig + + config = EvalConfig( + topology={"id": "shared-runtime-v1"}, + num_tasks=1, + max_turns=2, + sampling={"max_tokens": 512, "temperature": 0}, + timeout={"rollout": 180, "scoring": 60}, + retries={"rollout": {"max_retries": 2, "include": ["ProviderError"]}}, + rich=False, + output_dir=tmp_path, + ) + written, read = graph_traces(await run_eval(config)) + assert written.errors == [] and read.errors == [] + assert (written.agent, read.agent) == ("writer", "reader") + assert read.parents == [written.id] + assert written.info["agent"]["runtime"]["borrowed"] is True + assert read.info["agent"]["runtime"]["borrowed"] is True + assert ( + written.info["agent"]["runtime"]["descriptor"] + == read.info["agent"]["runtime"]["descriptor"] + ) + assert read.rewards["read_shared_note"] == 1.0 + assert written.rewards["handoff_succeeded"] == 1.0 + + +@pytest.mark.e2e +async def test_chess_topology(tmp_path): + """The `chess-v1` example, live: two direct-harness seats play one game through + live sessions — each seat ONE multi-turn trace with the opponent's moves as its + user turns, the host-side board adjudicating.""" + from verifiers.v1.cli.eval.runner import graph_traces, run_eval + from verifiers.v1.configs.eval import EvalConfig + + config = EvalConfig( + topology={"id": "chess-v1", "max_plies": 6, "illegal_retries": 2}, + num_tasks=1, + max_turns=16, + # reasoning models can spend 1k+ tokens thinking per move; a tight cap truncates + # the turn to null content and reads as an illegal move (or a forfeit) + sampling={"max_tokens": 4096, "temperature": 0}, + timeout={"rollout": 420, "scoring": 60}, + rich=False, + output_dir=tmp_path, + ) + traces = graph_traces(await run_eval(config)) + seats = {t.agent: t for t in traces} + assert set(seats) == {"white", "black"} + assert all(t.errors == [] for t in traces) + assert sum(t.rewards["outcome"] for t in traces) == 1.0 # one game's points + assert all(t.num_turns >= 1 for t in traces) + assert seats["white"].info["chess"]["plies"] >= 2 + + +@pytest.mark.e2e +async def test_debate_topology(tmp_path): + """The `debate-v1` example, live: three concurrent seats of one agent config give + openings, rebuttals, and peer votes through suspended sessions.""" + from verifiers.v1.cli.eval.runner import graph_traces, run_eval + from verifiers.v1.configs.eval import EvalConfig + + config = EvalConfig( + topology={"id": "debate-v1", "num_debaters": 3, "num_rounds": 1}, + num_tasks=1, + max_turns=8, + sampling={"max_tokens": 2048, "temperature": 0}, + timeout={"rollout": 420, "scoring": 60}, + rich=False, + output_dir=tmp_path, + ) + traces = graph_traces(await run_eval(config)) + assert [t.agent for t in traces] == ["debater"] * 3 + assert all(t.errors == [] for t in traces) + assert all(t.num_turns == 3 for t in traces) # opening + rebuttal + vote + total_votes = sum(t.info["debate"]["votes_received"] for t in traces) + valid_ballots = sum(t.info["debate"]["voted_validly"] for t in traces) + assert total_votes == valid_ballots + assert all("support" in t.rewards for t in traces) diff --git a/tests/v1/test_envs.py b/tests/v1/test_envs.py index 843f60c3df..baef27d084 100644 --- a/tests/v1/test_envs.py +++ b/tests/v1/test_envs.py @@ -57,7 +57,7 @@ def test_eval(taskset: str): "uv", "run", "--no-sync", "eval", "--taskset.id", taskset, *model, - # -r 2: a task with @group_reward(s) needs >=2 rollouts to compare. + # -r 2 schedules two independent graph invocations for the task. "-n", "1", "-r", "2", "--max-turns", "4", "--sampling.max-tokens", "512", "--rich", "false", ] # fmt: skip diff --git a/tests/v1/test_resume.py b/tests/v1/test_resume.py new file mode 100644 index 0000000000..561a6a747f --- /dev/null +++ b/tests/v1/test_resume.py @@ -0,0 +1,117 @@ +"""Resume keeps the graph records the topology calls complete and re-runs the rest. + +The keep/re-run verdict is `Topology.complete` — conservative by default (any error, +instance-level or nested, means redo; exact for the single-agent lowering, where the one +trace IS the invocation), overridable by topologies whose `go` tolerates child failures. +""" + +import json +from pathlib import Path + +import verifiers.v1 as vf +from verifiers.v1.cli.eval import resume +from verifiers.v1.cli.output import TRACES_FILE +from verifiers.v1.graph import MessageNode +from verifiers.v1.topology import AgentGraph, graph_complete +from verifiers.v1.trace import Error, Trace, TraceTask +from verifiers.v1.types import AssistantMessage, UserMessage + + +def make_trace(task: vf.Task, errored: bool = False) -> Trace: + trace = Trace(task=TraceTask(type=type(task).__name__, data=task.data)) + trace.nodes.append( + MessageNode(parent=None, message=UserMessage(content=str(task.data.prompt))) + ) + trace.nodes.append( + MessageNode(parent=0, message=AssistantMessage(content="ok"), sampled=True) + ) + if errored: + trace.errors.append(Error(type="HarnessError", message="boom")) + trace.stop("error" if errored else "agent_completed") + return trace + + +def make_graph( + idx: int, trace_errored: bool = False, go_crashed: bool = False +) -> AgentGraph: + task = vf.Task(vf.TaskData(idx=idx, prompt="hi")) + graph = AgentGraph( + topology="single-agent", + task=TraceTask(type=type(task).__name__, data=task.data), + ) + graph.add(make_trace(task, errored=trace_errored)) + if go_crashed: + graph.error = Error(type="TopologyError", message="go crashed") + return graph + + +def write_records(path: Path, graphs: list[AgentGraph]) -> None: + lines = [json.dumps(g.to_record()) for g in graphs] + (path / TRACES_FILE).write_text("\n".join(lines) + "\n") + + +def test_default_rule_redoes_any_error(tmp_path): + """The conservative default: a clean graph is kept; a nested trace error or an + instance-level error means the invocation is owed again — the single-agent lowering's + old flat-trace resume behavior, graph-shaped.""" + write_records( + tmp_path, + [ + make_graph(0), + make_graph(1, trace_errored=True), + make_graph(2, go_crashed=True), + ], + ) + finished, owed = resume.load(tmp_path, [0, 1, 2], num_rollouts=1) + assert [g.task.data.idx for g in finished] == [0] + assert owed == {1: 1, 2: 1} + # The kept lines were rewritten verbatim; the dropped records are gone from disk. + kept = [ + json.loads(line) + for line in (tmp_path / TRACES_FILE).read_text().splitlines() + if line.strip() + ] + assert [row["task"]["data"]["idx"] for row in kept] == [0] + + +def test_tolerant_topology_keeps_scored_instances(tmp_path): + """A topology whose `go` tolerates child failures overrides `complete` (typically + `graph.error is None`): its trace-errored-but-finished instances are kept, not redone.""" + write_records( + tmp_path, + [ + make_graph(0), + make_graph(1, trace_errored=True), + make_graph(2, go_crashed=True), + ], + ) + finished, owed = resume.load( + tmp_path, [0, 1, 2], num_rollouts=1, complete=lambda g: g.error is None + ) + assert [g.task.data.idx for g in finished] == [0, 1] + assert owed == {2: 1} + + +def test_owed_counts_missing_invocations(tmp_path): + """`num_rollouts` is the per-seed target: kept records count toward it, the rest is + owed — including seeds with no record at all.""" + write_records(tmp_path, [make_graph(0), make_graph(0)]) + finished, owed = resume.load(tmp_path, [0, 1], num_rollouts=2) + assert len(finished) == 2 + assert owed == {1: 2} # a fully-covered seed owes nothing + + +def test_topology_complete_defaults_and_overrides(): + """`Topology.complete` defaults to the conservative module-level rule; a subclass + override is what resume consumes.""" + clean, errored = make_graph(0), make_graph(1, trace_errored=True) + assert graph_complete(clean) and not graph_complete(errored) + + topology = vf.SingleAgentTopology(vf.SingleAgentTopologyConfig()) + assert topology.complete(clean) and not topology.complete(errored) + + class Tolerant(vf.Topology[vf.TopologyConfig]): + def complete(self, graph: AgentGraph) -> bool: + return graph.error is None + + assert Tolerant(vf.TopologyConfig()).complete(errored) diff --git a/tests/v1/test_scoring.py b/tests/v1/test_scoring.py index 57bc447fdd..b9456adc66 100644 --- a/tests/v1/test_scoring.py +++ b/tests/v1/test_scoring.py @@ -31,3 +31,29 @@ def test_parse_judge_choice_uses_first_choice_after_verdict_marker() -> None: response = "Final Judgment: B because it is a better answer" assert vf.parse_judge_choice(response, choices=("A", "B")) == "B" + + +def test_judge_utility_pure_parts(): + """The judge utility's model-free surface: template rendering (config override beats the + class template) and the error-attribution verdict parser (an unparseable verdict raises — + a judge failure must not be silently scored against the model).""" + import pytest + + import verifiers.v1 as vf + + class YesNoJudge(vf.Judge[bool]): + prompt = "Q: {question}\nA: {response}\nCorrect? yes/no" + + judge = YesNoJudge(vf.JudgeConfig()) + assert ( + judge.build_messages(question="2+2?", response="4") + == "Q: 2+2?\nA: 4\nCorrect? yes/no" + ) + override = YesNoJudge(vf.JudgeConfig(prompt="only {question}")) + assert override.build_messages(question="x") == "only x" + + assert ( + vf.judge_verdict("I think the answer is correct. yes", ("yes", "no")) == "yes" + ) + with pytest.raises(ValueError, match="no yes/no verdict"): + vf.judge_verdict("cannot say", ("yes", "no")) diff --git a/tests/v1/test_session.py b/tests/v1/test_session.py new file mode 100644 index 0000000000..80fef4fac5 --- /dev/null +++ b/tests/v1/test_session.py @@ -0,0 +1,219 @@ +"""Session tests: the queue handshake, the safety contract, and `Agent.interact`. + +The handshake tests drive `Session._respond` from a fake interception loop, so the +turn/reply lockstep, `end()`, and the poison paths run without a model or a rollout. +The lifecycle test monkeypatches `Rollout.run` with a stub that consumes the session's +`_respond` exactly as the interception loop does — so `Agent.interact`'s background +task, scope-exit stop, and provenance stamp run for real.""" + +import asyncio +from types import SimpleNamespace + +import pytest +import verifiers.v1 as vf +from verifiers.v1.agent import Session +from verifiers.v1.harness import Harness, HarnessConfig +from verifiers.v1.harnesses.null.harness import NullHarness, NullHarnessConfig +from verifiers.v1.rollout import Rollout +from verifiers.v1.trace import TraceTask + + +def task_of(prompt) -> vf.Task: + return vf.Task(vf.TaskData(idx=0, prompt=prompt)) + + +def trace_for(task: vf.Task) -> vf.Trace: + return vf.Trace(task=TraceTask(type=type(task).__name__, data=task.data)) + + +def make_session(task: vf.Task) -> Session: + session = Session() + session._rollout = SimpleNamespace(trace=trace_for(task)) + return session + + +def null_agent(**kwargs) -> vf.Agent: + return vf.Agent( + NullHarness(NullHarnessConfig()), + vf.ModelContext( + model="org/model", client=object(), sampling=vf.SamplingConfig() + ), + **kwargs, + ) + + +async def test_session_handshake_and_end(): + """The lockstep: opening `_respond("")` delivers nothing and returns the first + turn; each later `_respond(reply)` delivers the reply and suspends; `end()` + unblocks the suspended episode into a clean exit and stops the trace.""" + session = make_session(task_of(None)) + seen: list[str] = [] + + async def fake_interception_loop(): + msgs = await session._respond("") # the opening call + while msgs: + seen.append(msgs[0].content) + msgs = await session._respond(f"echo:{msgs[0].content}") + + episode = asyncio.create_task(fake_interception_loop()) + session._run_task = episode + episode.add_done_callback(session._poison) + + assert await session.turn("one") == "echo:one" + assert await session.turn("two") == "echo:two" + await session.end() + await episode + assert seen == ["one", "two"] + assert session.trace.stop_condition == "interaction_complete" + + +async def test_turn_raises_on_dead_episode(): + """`turn()` never hangs on an episode that can't answer: a seat that died + mid-await gets poisoned, and a later `turn()` refuses up front.""" + session = make_session(task_of(None)) + + async def dying_episode(): + await session._respond("") # take the first turn, then die before replying + raise RuntimeError("boom") + + episode = asyncio.create_task(dying_episode()) + session._run_task = episode + episode.add_done_callback(session._poison) + + with pytest.raises(vf.SessionEnded): + await session.turn("hello?") # mid-await death → poisoned reply + with pytest.raises(vf.SessionEnded): + await session.turn("still there?") # already-dead → refused up front + with pytest.raises(RuntimeError, match="boom"): + await episode + + +async def test_double_drive_refused(): + """One driver per seat: a second `turn()` while one is pending raises instead of + interleaving two conversations.""" + session = make_session(task_of(None)) + session._run_task = asyncio.create_task(asyncio.sleep(30)) + try: + first = asyncio.create_task(session.turn("a")) + await asyncio.sleep(0) # let it post and suspend + with pytest.raises(RuntimeError, match="one driver per seat"): + await session.turn("b") + first.cancel() + finally: + session._run_task.cancel() + + +async def test_cancelled_turn_desyncs_loudly(): + """A `turn()` cancelled mid-flight (e.g. a sibling seat in a `gather` raised) left + its message with the model — the conversation is desynced, so a later `turn()` + refuses instead of consuming the stale reply one turn late.""" + session = make_session(task_of(None)) + session._run_task = asyncio.create_task(asyncio.sleep(30)) + try: + pending = asyncio.create_task(session.turn("in flight")) + await asyncio.sleep(0) # let it post its message and suspend + pending.cancel() + with pytest.raises(asyncio.CancelledError): + await pending + with pytest.raises(RuntimeError, match="cancelled mid-flight"): + await session.turn("resync?") + await session.end() # the sanctioned way out of a desynced seat + finally: + session._run_task.cancel() + + +def test_extend_coerces_reasoning_only_turn(): + """Regression (found by a live chess game): a truncated reasoning turn comes back + with `content: null` and no tool calls — valid *output*, but upstreams 422 it when + re-sent as conversation *input*. `extend` must coerce it to the empty string; a + tool-call turn stays legitimately content-less.""" + from verifiers.v1.dialects.chat import ChatDialect + + dialect = ChatDialect() + body = {"messages": [{"role": "user", "content": "your move"}]} + truncated = {"choices": [{"message": {"role": "assistant", "content": None}}]} + extended = dialect.extend(body, truncated, [vf.UserMessage(content="next")]) + assert extended["messages"][1] == {"role": "assistant", "content": ""} + + tool_turn = { + "choices": [ + { + "message": { + "role": "assistant", + "content": None, + "tool_calls": [{"id": "x"}], + } + } + ] + } + extended = dialect.extend(body, tool_turn, []) + assert extended["messages"][1]["content"] is None # untouched — tool calls carry it + + +async def test_interact_refusals(): + """The refuse-loudly table: a prompted task, a task with its own user simulator, + and a harness that can't take injected user turns are all rejected at the call.""" + agent = null_agent() + with pytest.raises(ValueError, match="opened by the first turn"): + async with agent.interact(task_of("hi")): + pass + + class SimTask(vf.Task): + user = vf.User # declared simulator class — never instantiated here + + with pytest.raises(ValueError, match="second claimant"): + async with agent.interact(SimTask(vf.TaskData(idx=0, prompt=None))): + pass + + class MuteHarnessConfig(HarnessConfig): + pass + + class MuteHarness(Harness[MuteHarnessConfig]): + SUPPORTS_USER_SIM = False + + async def launch(self, *a, **k): # pragma: no cover - never called + raise NotImplementedError + + mute = vf.Agent( + MuteHarness(MuteHarnessConfig()), + vf.ModelContext( + model="org/model", client=object(), sampling=vf.SamplingConfig() + ), + ) + with pytest.raises(ValueError, match="cannot take injected user turns"): + async with mute.interact(task_of(None)): + pass + + +async def test_interact_lifecycle_with_stubbed_rollout(monkeypatch): + """`Agent.interact` end to end minus the model: the stubbed `Rollout.run` consumes + the programmatic user seat exactly like the interception loop, and the scope exit + ends the episode, joins it, and stamps provenance + parents.""" + + async def fake_run(self): + self.trace = trace = vf.Trace( + task=TraceTask(type=type(self.task).__name__, data=self.task.data) + ) + msgs = await self._user("") # opening + while msgs: + msgs = await self._user(f"echo:{msgs[0].content}") + if trace.stop_condition is None: # pragma: no cover - end() always set it + trace.stop("agent_completed") + return trace + + monkeypatch.setattr(Rollout, "run", fake_run) + agent = null_agent(name="white", trainable=False) + parent = trace_for(task_of("seed")) + + async with agent.interact( + vf.Task(vf.TaskData(idx=1, prompt=None)), parents=[parent] + ) as session: + assert await session.turn("kick off") == "echo:kick off" + trace = session.trace # live handle + trace.info["game"] = {"score": 1.0} + + assert trace.stop_condition == "interaction_complete" + assert trace.agent == "white" and trace.trainable is False + assert trace.parents == [parent.id] + assert trace.info["game"] == {"score": 1.0} + assert trace.info["agent"]["name"] == "white" diff --git a/tests/v1/test_topology.py b/tests/v1/test_topology.py new file mode 100644 index 0000000000..a89602c79f --- /dev/null +++ b/tests/v1/test_topology.py @@ -0,0 +1,696 @@ +"""Topology tests: config narrowing, agent loading, and instance semantics. + +The unit tests stub `vf.Agent.run` with a canned reply, so the explicit-agent topology +surface — forward trace→task arrows, fan-out, parent links, declared instance judgement, +`go` failure capture — runs without a model or a runtime. The `e2e`-marked test runs the +`echo-chain-v1` fixture topology live (skipped without a key), mirroring `test_e2e`. +""" + +import json + +import numpy as np +import pytest +import verifiers.v1 as vf +from verifiers.v1.cli.output import output_path +from verifiers.v1.cli.resolve import narrow_config +from verifiers.v1.configs.eval import EvalConfig +from verifiers.v1.graph import MessageNode +from verifiers.v1.serve.types import RunResponse +from verifiers.v1.topologies.agentic_judge import TRACE_PATH, AgenticJudgeTask +from verifiers.v1.topologies.llm_judge import JudgeTask +from verifiers.v1.topology import AgentGraph, TopologyRunner +from verifiers.v1.trace import Trace, TraceTask, WireTrace +from verifiers.v1.types import AssistantMessage, UserMessage + + +def echoing_trace(task: vf.Task, reply: str) -> Trace: + """A minimal completed trace: one user turn, one sampled assistant reply.""" + trace = Trace(task=TraceTask(type=type(task).__name__, data=task.data)) + trace.nodes.append( + MessageNode(parent=None, message=UserMessage(content=str(task.data.prompt))) + ) + trace.nodes.append( + MessageNode(parent=0, message=AssistantMessage(content=reply), sampled=True) + ) + trace.stop("agent_completed") + return trace + + +@pytest.fixture +def stub_agent_run(monkeypatch) -> None: + """Run topology instances through real `TopologyRunner.serving`, but replace the + executable agent's rollout work with a tiny deterministic trace.""" + + async def fake_agent_run(self, task, *, parents=(), runtime=None, retry=None): + trace = echoing_trace(task, getattr(task.data, "answer", "ok")) + trace.record_reward("echoed", 1.0) + self.stamp( + trace, parents=parents, runtime=runtime, borrowed=runtime is not None + ) + return trace + + monkeypatch.setattr(vf.Agent, "run", fake_agent_run) + + +async def run_stubbed_instance(env: TopologyRunner, task: vf.Task) -> AgentGraph: + ctx = vf.ModelContext( + model="org/model", client=object(), sampling=vf.SamplingConfig() + ) + async with env.serving(): + return await env.run_instance(task, ctx) + + +@pytest.fixture +def echo_chain_env(stub_agent_run) -> TopologyRunner: + """A loaded echo-chain topology whose agents return canned traces.""" + config = EvalConfig(topology={"id": "echo-chain-v1"}, rich=False) + return TopologyRunner(config.topology, config) + + +@pytest.fixture +def proposer_seed(monkeypatch) -> vf.Task: + """A local seed with the AIME data shape, avoiding a research-plugin dependency in unit tests.""" + taskset = vf.load_taskset(vf.taskset_config_type("echo-v1")(id="echo-v1")) + seed = taskset.load()[0] + from proposer_solver_v1.topology import ProposerSolverTopology + + monkeypatch.setattr(ProposerSolverTopology, "load_tasks", lambda self: [seed]) + return seed + + +def test_llm_judge_config_narrows_typed_agents(): + """`--topology.id llm-judge` narrows to `LLMJudgeConfig`; the seed factory narrows by + id, and the judge keeps its pinned defaults (direct harness, non-trainable) unless + overridden.""" + config = EvalConfig( + topology={"id": "llm-judge", "taskset": {"id": "echo-v1"}}, rich=False + ) + assert type(config.topology).__name__ == "LLMJudgeConfig" + assert type(config.topology.taskset).__name__ == "EchoConfig" + assert config.topology.solver.harness.id == "default" + assert config.topology.judge.harness.id == "direct" + assert config.topology.judge.trainable is False + assert config.topology.solver.trainable is True + + +def test_pinned_harness_survives_sibling_and_partial_overrides(): + """The pin contract, all three previously-broken paths: (a) a sibling override + (`--topology.solver.model X`) must not silently replace a pinned harness with the + default agent; (b) an id-less partial harness override must deep-merge into the pin, + not reset it; (c) a base-typed `HarnessConfig(id=...)` pin is detected by value.""" + from proposer_solver_v1.topology import ProposerSolverConfig + + # (a) sibling field override keeps the subclass pin + config = ProposerSolverConfig.model_validate({"solver": {"model": "org/strong"}}) + assert config.proposer.harness.id == "default" + assert config.solver.model == "org/strong" + assert config.solver.harness.id == "direct" + # (b) partial harness override tunes the pin (llm-judge judge: direct + docker runtime) + eval_config = EvalConfig( + topology={ + "id": "llm-judge", + "taskset": {"id": "echo-v1"}, + "judge": {"harness": {"runtime": {"type": "docker"}}}, + }, + rich=False, + ) + judge = eval_config.topology.judge + assert judge.harness.id == "direct" + assert judge.harness.runtime.type == "docker" + # (c) a base-typed pin (null) is a value pin — bare construction keeps it + assert vf.NullAgentConfig().harness.id == "null" + assert vf.NullAgentConfig.model_validate({"model": "org/x"}).harness.id == "null" + + +def test_pinned_agent_harness_still_swaps_by_id(): + """A subclass pins per-agent defaults via the field default *instance*, not the + annotation — so an explicit `--topology.judge.harness.id ` (on `agentic-judge`, + whose judge harness is configurable) narrows to the swapped harness's own config type + while the other pins survive the swap.""" + config = EvalConfig( + topology={ + "id": "agentic-judge", + "taskset": {"id": "echo-v1"}, + "judge": {"harness": {"id": "null"}}, + }, + rich=False, + ) + assert type(config.topology.judge.harness).__name__ == "NullHarnessConfig" + assert config.topology.judge.trainable is False # other pins survive the swap + + +def test_llm_judge_harness_is_locked(): + """`llm-judge` fixes its judge to the in-process `direct` chat loop: an explicit + harness swap is refused with a pointer at `agentic-judge` (whose judge harness IS + configurable); the judge's routing stays per-agent config.""" + with pytest.raises(ValueError, match="agentic-judge"): + EvalConfig( + topology={ + "id": "llm-judge", + "taskset": {"id": "echo-v1"}, + "judge": {"harness": {"id": "default"}}, + }, + rich=False, + ) + config = EvalConfig( # routing overrides don't touch the lock + topology={ + "id": "llm-judge", + "taskset": {"id": "echo-v1"}, + "judge": {"model": "org/strong-judge"}, + }, + rich=False, + ) + assert config.topology.judge.harness.id == "direct" + assert config.topology.judge.model == "org/strong-judge" + + +def test_agent_discovery_and_seed_factory(): + config = EvalConfig(topology={"id": "echo-chain-v1"}, rich=False) + topology = vf.load_topology(config.topology) + assert list(topology.agents) == ["first", "second"] # declaration order + tasks = topology.load_tasks() # from the pinned echo-v1 seed factory + assert tasks and tasks[0].data.answer == "hello world" + + +def test_topology_without_seeds_is_refused(): + config = EvalConfig(topology={"id": "llm-judge"}, rich=False) + with pytest.raises(ValueError, match="has no seed tasks"): + vf.load_topology(config.topology).load_tasks() + + +def test_proposer_solver_fixes_aime26_seed_taskset(monkeypatch): + from proposer_solver_v1.topology import AIME_TASKSET_ID + + config = EvalConfig(topology={"id": "proposer-solver-v1"}, rich=False) + topology = vf.load_topology(config.topology) + requested: list[str] = [] + + def config_type(taskset_id): + requested.append(taskset_id) + return vf.TasksetConfig + + class StubTaskset: + def load(self): + return ["seed"] + + monkeypatch.setattr(vf, "taskset_config_type", config_type) + monkeypatch.setattr(vf, "load_taskset", lambda config: StubTaskset()) + assert topology.load_tasks() == ["seed"] + assert requested == [AIME_TASKSET_ID] + + +def test_proposer_solver_refuses_taskset_override(): + """Its AIME source is fixed in `load_tasks`, so the generic taskset slot is invalid.""" + config = EvalConfig( + topology={"id": "proposer-solver-v1", "taskset": {"id": "echo-v1"}}, rich=False + ) + with pytest.raises(ValueError, match="constructs its own seeds"): + TopologyRunner(config.topology, config) + + +def test_unknown_agent_is_refused(echo_chain_env): + with pytest.raises(ValueError, match="unknown agent 'thrid'"): + echo_chain_env.agent("thrid") + + +async def test_run_instance_requires_serving(echo_chain_env): + ctx = vf.ModelContext( + model="org/model", client=object(), sampling=vf.SamplingConfig() + ) + with pytest.raises(RuntimeError, match="inside TopologyRunner.serving"): + await echo_chain_env.run_instance(echo_chain_env.topology.load_tasks()[0], ctx) + + +def test_topology_reward_scopes_validated_at_load(): + """Declared judgement fails loudly when the topology loads: a typo'd agent scope and + a missing scope are both refused before anything runs.""" + import echo_chain_v1 + + class TypoScope(echo_chain_v1.EchoChainTopology): + @vf.reward(agent="thrid") + async def oops(self, trace): + return 0.0 + + with pytest.raises(ValueError, match="unknown agent 'thrid'"): + _ = TypoScope(echo_chain_v1.EchoChainConfig()).agents + + class NoScope(echo_chain_v1.EchoChainTopology): + @vf.metric + async def oops(self, trace): + return 0.0 + + with pytest.raises(ValueError, match="declares no agent scope"): + _ = NoScope(echo_chain_v1.EchoChainConfig()).agents + + +async def test_run_instance_links_and_defers(echo_chain_env): + """One stubbed instance: completion-ordered traces, parent links, agent names, and the + declared backward-arrow reward landing on the upstream trace at instance end.""" + seed = echo_chain_env.topology.load_tasks()[0] + graph = await run_stubbed_instance(echo_chain_env, seed) + assert graph.error is None + assert graph.topology == "echo-chain-v1" + first, second = graph.traces + assert (first.agent, second.agent) == ("first", "second") + assert first.parents == [] and second.parents == [first.id] + assert graph.roots() == [first] + assert graph.children(first) == [second] + assert graph.by_agent("second") == [second] + assert first.trainable and second.trainable + # forward arrow: the derived task echoes the same phrase + assert second.task.data.answer == seed.data.answer + # backward arrow: relay reward recorded on the upstream trace after the child finished + assert first.rewards == {"echoed": 1.0, "relay": 1.0} + assert second.rewards == {"echoed": 1.0} + + +async def test_declared_judgement_scores_the_instance(stub_agent_run, proposer_seed): + """`Topology.score` runs the declared @metric/@reward methods once per matching trace + after `go` returns — metrics before rewards (`difficulty` reads the `solve_rate` + metric), and over every trace in scope even when the instance ended early (the stub + never calls the submission tool, so `go` stops after the proposer). The proposer's + task-declared parseability reward is absent because the stub skips episode scoring.""" + config = EvalConfig(topology={"id": "proposer-solver-v1"}, rich=False) + env = TopologyRunner(config.topology, config) + graph = await run_stubbed_instance(env, env.topology.load_tasks()[0]) + (proposer,) = graph.traces # malformed proposal → no solver fan-out + assert proposer.metrics == {"solve_rate": 0.0} + assert proposer.rewards == {"echoed": 1.0, "difficulty": 0.0} + + +async def test_proposer_solver_fan_out_and_difficulty(monkeypatch, proposer_seed): + """The happy path: the proposer commits a submission, `go` fans `num_solvers` solver + episodes out over the derived AIME task (all parented under the proposer), and the + declared judgement averages the solvers' `correct` into `solve_rate` → `difficulty` + (peaked at 50%, so a 2/4 split scores 1.0).""" + verdicts = iter([1.0, 0.0, 1.0, 0.0]) + + async def fake_agent_run(self, task, *, parents=(), runtime=None, retry=None): + if self.name == "proposer": + trace = echoing_trace(task, "submitted") + trace.info["submission"] = { + "question": "What is 1?", + "answer": "1", + } + else: + trace = echoing_trace(task, "\\boxed{1}") + trace.record_reward("correct", next(verdicts)) + self.stamp( + trace, parents=parents, runtime=runtime, borrowed=runtime is not None + ) + return trace + + monkeypatch.setattr(vf.Agent, "run", fake_agent_run) + config = EvalConfig(topology={"id": "proposer-solver-v1"}, rich=False) + env = TopologyRunner(config.topology, config) + graph = await run_stubbed_instance(env, env.topology.load_tasks()[0]) + assert graph.error is None + proposer, *solvers = graph.traces + assert [t.agent for t in solvers] == ["solver"] * 4 # num_solvers default + assert all(t.parents == [proposer.id] for t in solvers) + assert all( + t.task.data.prompt == solvers[0].task.data.prompt for t in solvers + ) # one minted task + assert all(t.task.data.answer == "1" for t in solvers) + assert proposer.metrics == {"solve_rate": 0.5} + assert proposer.rewards == {"difficulty": 1.0} + + +async def test_provisioned_runtime_is_borrowed_across_runs(echo_chain_env, monkeypatch): + """`run.agent(...).provision()` hands `go` a live box whose lifetime the provisioner + owns; runs passed `runtime=box` are stamped borrowed and never tear it down — and a + bare-id parent (`parents=[trace.id]`) links the same as a `Trace`.""" + + class FakeRuntime: + config = vf.SubprocessConfig() + descriptor = "fake" + started = stopped = False + + async def start(self): + self.started = True + + async def stop(self): + self.stopped = True + + import verifiers.v1.agent as agent_module + + fake = FakeRuntime() + monkeypatch.setattr(agent_module, "make_runtime", lambda config: fake) + env = echo_chain_env + seed = env.topology.load_tasks()[0] + ctx = vf.ModelContext( + model="org/model", client=object(), sampling=vf.SamplingConfig() + ) + async with env.serving(): + run = vf.TopologyRun(env, seed, env._agents_for(ctx)) + agent = run.agent("first") + async with agent.provision(seed) as box: + assert box is fake and fake.started and not fake.stopped + first = await agent.run(seed, runtime=box) + second = await agent.run(seed, runtime=box, parents=[first.id]) # str arm + assert fake.stopped # provisioner owns teardown, on exit + assert first.info["agent"]["runtime"]["borrowed"] is True + assert second.info["agent"]["runtime"]["borrowed"] is True + assert second.parents == [first.id] + assert run.graph.traces == [first, second] + + +async def test_shared_runtime_topology_hands_off_through_one_box(monkeypatch): + """The shared-runtime example, stubbed: the writer's borrowed run hands its note to a + reader run in the SAME provisioned box (both traces stamped borrowed), and the + deferred writer reward mirrors the reader's verification.""" + + async def fake_agent_run(self, task, *, parents=(), runtime=None, retry=None): + trace = echoing_trace(task, "shared runtime handoff ready.") + if self.name == "writer": + trace.info["shared_runtime"] = {"wrote": "note", "path": "shared/note.txt"} + else: + trace.record_reward("read_shared_note", 1.0) + self.stamp( + trace, parents=parents, runtime=runtime, borrowed=runtime is not None + ) + return trace + + monkeypatch.setattr(vf.Agent, "run", fake_agent_run) + config = EvalConfig(topology={"id": "shared-runtime-v1"}, rich=False) + env = TopologyRunner(config.topology, config) + graph = await run_stubbed_instance(env, env.topology.load_tasks()[0]) + assert graph.error is None + written, read = graph.traces + assert (written.agent, read.agent) == ("writer", "reader") + assert read.parents == [written.id] + assert read.task.data.expected == "note" # the forward arrow carried the handoff + # both runs were placed into the one provisioned box + assert written.info["agent"]["runtime"]["borrowed"] is True + assert read.info["agent"]["runtime"]["borrowed"] is True + assert written.rewards == {"handoff_succeeded": 1.0} # mirrored off the reader + + +def scripted_sessions(monkeypatch, script): + """Replace `Agent.interact` with scripted sessions: `script(agent_name, task, + message, turn_index) -> reply`. Mimics the real contract — yields a session with a + live trace, stamps provenance on close — so a topology's whole `go` + declared + judgement runs without a model.""" + import contextlib as _contextlib + + class ScriptedSession: + def __init__(self, name, task): + self.name, self.task, self.turns = name, task, 0 + self._trace = Trace( + task=TraceTask(type=type(task).__name__, data=task.data) + ) + + @property + def trace(self): + return self._trace + + async def turn(self, message): + reply = script(self.name, self.task, message, self.turns) + self.turns += 1 + return reply + + async def end(self, condition="interaction_complete"): + if self._trace.stop_condition is None: + self._trace.stop(condition) + + @_contextlib.asynccontextmanager + async def fake_interact(self, task, *, parents=(), runtime=None): + session = ScriptedSession(self.name, task) + try: + yield session + finally: + await session.end() + self.stamp(session.trace, parents=parents, runtime=None, borrowed=False) + + monkeypatch.setattr(vf.Agent, "interact", fake_interact) + + +async def test_chess_go_with_scripted_sessions(monkeypatch): + """The chess topology's whole referee loop, scripted: fool's mate in four plies — + both seats' sessions play their moves, the board adjudicates, the outcome lands as + per-seat declared rewards on two coherent traces.""" + FOOLS_MATE = { + "white": ["MOVE: f2f3", "MOVE: g2g4"], + "black": ["MOVE: e7e5", "MOVE: d8h4"], + } + scripted_sessions(monkeypatch, lambda name, task, msg, i: FOOLS_MATE[name][i]) + config = EvalConfig(topology={"id": "chess-v1"}, rich=False) + env = TopologyRunner(config.topology, config) + graph = await run_stubbed_instance(env, env.topology.load_tasks()[0]) + assert graph.error is None + seats = {t.agent: t for t in graph.traces} + assert set(seats) == {"white", "black"} + assert seats["white"].info["chess"] == { + "score": 0.0, + "result": "0-1", + "plies": 4, + "illegal_moves": 0, + } + assert seats["white"].rewards == {"outcome": 0.0} + assert seats["black"].rewards == {"outcome": 1.0} + assert seats["black"].metrics == {"illegal_moves": 0.0} + + +async def test_chess_illegal_moves_retry_then_forfeit(monkeypatch): + """The feedback loop: an illegal move gets a retry turn; a seat that never produces + a legal move forfeits, and the opponent takes the game.""" + scripted_sessions( + monkeypatch, lambda name, task, msg, i: "MOVE: zz99" + ) # never legal + config = EvalConfig(topology={"id": "chess-v1", "illegal_retries": 1}, rich=False) + env = TopologyRunner(config.topology, config) + graph = await run_stubbed_instance(env, env.topology.load_tasks()[0]) + assert graph.error is None + seats = {t.agent: t for t in graph.traces} + assert seats["white"].info["chess"]["result"] == "forfeit" + assert seats["white"].info["chess"]["illegal_moves"] == 1 # one retry, then forfeit + assert seats["white"].rewards == {"outcome": 0.0} + assert seats["black"].rewards == {"outcome": 1.0} + assert seats["white"].stop_condition == "forfeit" + + +async def test_debate_go_with_scripted_sessions(monkeypatch): + """The N-ary session shape, scripted: 4 concurrent seats of ONE agent config give + openings, rebuttals, and peer votes; `go` tallies and the same declared reward maps + each seat's vote share onto its own trace.""" + + def script(name, task, message, i): + if "VOTE" in message.upper() and i == 2: + return ( + "VOTE: 1" if task.data.seat == 0 else "VOTE: 0" + ) # everyone else backs seat 0 + return f"Seat {task.data.seat} argues its case (turn {i})." + + scripted_sessions(monkeypatch, script) + config = EvalConfig(topology={"id": "debate-v1"}, rich=False) + env = TopologyRunner(config.topology, config) + graph = await run_stubbed_instance(env, env.topology.load_tasks()[0]) + assert graph.error is None + assert [t.agent for t in graph.traces] == ["debater"] * 4 + by_seat = {t.info["debate"]["seat"]: t for t in graph.traces} + assert by_seat[0].rewards == {"support": 1.0} # 3 of 3 possible votes + assert by_seat[1].rewards == {"support": pytest.approx(1 / 3)} + assert by_seat[2].rewards == {"support": 0.0} + assert all(t.metrics == {"voted_validly": 1.0} for t in graph.traces) + + +async def test_go_failure_is_captured_on_graph(echo_chain_env, monkeypatch): + """A crash in topology-authored code is classified `TopologyError` and recorded on the + graph — completed episodes stay, nothing raises (a bad instance is data).""" + cls = type(echo_chain_env.topology) + + async def exploding_go(self, task, run): + await run.agent("first").run(task) + raise RuntimeError("boom") + + monkeypatch.setattr(cls, "go", exploding_go) + graph = await run_stubbed_instance( + echo_chain_env, echo_chain_env.topology.load_tasks()[0] + ) + assert graph.error is not None + assert graph.error.type == "TopologyError" + assert "boom" in graph.error.message + assert len(graph.traces) == 1 # the episode that completed before the crash + + +def test_eval_config_rejects_topology_with_taskset(): + with pytest.raises(ValueError, match="drop `--taskset.id`"): + EvalConfig( + topology={"id": "echo-chain-v1"}, taskset={"id": "echo-v1"}, rich=False + ) + + +def test_eval_config_accepts_topology_with_server(): + config = EvalConfig(topology={"id": "echo-chain-v1"}, server=True, rich=False) + assert config.topology.id == "echo-chain-v1" + + +def test_eval_config_rejects_topology_with_harness(): + """A user-supplied `--harness.*` under a topology would be silently ignored (agents + bind their own) — refused up front, while the framework-manufactured default (absent + from the input data) never trips the guard.""" + with pytest.raises(ValueError, match="ignored under a topology"): + EvalConfig(topology={"id": "echo-chain-v1"}, harness={"id": "rlm"}, rich=False) + EvalConfig(topology={"id": "echo-chain-v1"}, rich=False) # no harness key: fine + + +def test_narrow_config_resolves_topology_id(): + narrowed = narrow_config(EvalConfig, ["--topology.id", "llm-judge"]) + assert type(narrowed.model_fields["topology"].default).__name__ == "LLMJudgeConfig" + + +def test_output_path_names_topology_runs(): + config = EvalConfig(topology={"id": "echo-chain-v1"}, model="org/model", rich=False) + assert output_path(config).parent.name == "echo-chain-v1--org--model" + + +def test_judge_task_construction_and_verdict_parsing(): + """`JudgeTask.from_trace` (main's `Task.from_trace` convention) peels the judge's + inputs off the finished episode: the seed + task's framing, its ground truth (an `answer` field, when the task carries one), and + the solver's final message (last message of the final branch, not the transcript).""" + + class AnsweredData(vf.TaskData): + answer: str + + class AnsweredTask(vf.Task[AnsweredData]): + pass + + task = AnsweredTask(AnsweredData(idx=0, prompt="what is 2+2?", answer="4")) + solved = echoing_trace(task, "It is 4.") + grading = JudgeTask.from_trace(solved) + assert "what is 2+2?" in grading.data.prompt and "It is 4." in grading.data.prompt + assert "\n4\n" in grading.data.prompt + # a task without ground truth just omits the reference section + unanswered = echoing_trace(vf.Task(vf.TaskData(idx=0, prompt="write a poem")), "ok") + bare = JudgeTask.from_trace(unanswered) + assert "" not in bare.data.prompt + for reply, expected in [ + ("The attempt is correct.\nSCORE: 10", 1.0), + ("Partially right.\nscore: 7/10", 0.7), + ("Flawless.\n**SCORE: 10**", 1.0), # markdown-bolded verdicts still parse + ("SCORE: 999", 1.0), # clamped to the 0-10 scale + ("SCORE: not-a-number", None), + ("I refuse to commit to a verdict.", None), + ]: + assert JudgeTask.parse_score(echoing_trace(task, reply)) == expected + + +async def test_agentic_judge_task_uploads_the_trace(): + """`AgenticJudgeTask` carries the solver's entire serialized trace as data — its + `setup` hook writes it to `TRACE_PATH` in the judge's runtime, and the assignment + prompt points there.""" + task = vf.Task(vf.TaskData(idx=0, prompt="what is 2+2?")) + solved = echoing_trace(task, "It is 4.") + solved.record_reward("correct", 1.0) + grading = AgenticJudgeTask.from_trace(solved) + assert TRACE_PATH in grading.data.prompt + payload = json.loads(grading.data.trace_json) + assert payload["id"] == solved.id and payload["rewards"] == {"correct": 1.0} + + written: dict[str, bytes] = {} + + class StubRuntime: + async def write(self, path: str, data: bytes) -> None: + written[path] = data + + await grading.setup(echoing_trace(grading, ""), StubRuntime()) + assert json.loads(written[TRACE_PATH]) == payload + + +async def test_writer_editors_fan_in_and_shared_verdict(monkeypatch, stub_agent_run): + """The rounds + fan-in example, stubbed: editors fan out over the draft, the revision + is linked under the draft AND every editor trace, and one (stubbed) judge call puts the + same `improvement` reward on every trace of the instance.""" + from writer_editors_v1.topology import ImprovementJudge + + config = EvalConfig( + topology={"id": "writer-editors-v1", "num_editors": 2}, rich=False + ) + env = TopologyRunner(config.topology, config) + + async def stub_evaluate(self, *, trace=None, **fields): + assert {"brief", "first", "final"} <= fields.keys() + return vf.JudgeResponse(text="SCORE: 8", parsed=0.8) + + monkeypatch.setattr(ImprovementJudge, "evaluate", stub_evaluate) + graph = await run_stubbed_instance(env, env.topology.load_tasks()[0]) + assert graph.error is None + draft, edit_a, edit_b, revision = graph.traces + assert [t.agent for t in graph.traces] == ["writer", "editor", "editor", "writer"] + assert edit_a.parents == edit_b.parents == [draft.id] + assert revision.parents == [draft.id, edit_a.id, edit_b.id] # the fan-in + assert all(t.rewards["improvement"] == 0.8 for t in graph.traces) + + +def test_instance_record_roundtrips(): + """The graph is the serialized instance artifact: `to_record` nests the traces, and + `load` rebuilds it without the originating packages (WireTrace-typed traces, links + intact) — what `traces.jsonl` consumers and the trainer wire rely on.""" + task = vf.Task(vf.TaskData(idx=0, prompt="hi")) + graph = AgentGraph( + topology="llm-judge", + task=vf.TraceTask(type=type(task).__name__, data=task.data), + ) + trace = echoing_trace(task, "hi") + trace.agent = "judge" + trace.parents = ["abc123"] + trace.trainable = False + trace.record_reward("relay", 1.0) + graph.add(trace) + loaded = AgentGraph.load(json.loads(json.dumps(graph.to_record()))) + assert loaded.id == graph.id and loaded.topology == "llm-judge" + (t,) = loaded.traces + assert isinstance(t, WireTrace) + assert (t.agent, t.parents, t.trainable) == ("judge", ["abc123"], False) + assert t.reward == 1.0 + + +def test_instance_wire_preserves_training_tensors(): + """Server serialization keeps tensors that JSON persistence intentionally drops.""" + task = vf.TraceTask(type="Task", data=vf.WireTaskData(idx=0, prompt="hi")) + node = MessageNode( + message=AssistantMessage(content="hi"), + token_ids=[1], + mask=[True], + sampled=True, + routed_experts=np.ones((1, 2, 1), dtype=np.uint8), + ) + graph = AgentGraph(task=task, traces=[Trace(task=task, nodes=[node])]) + + wire = RunResponse(graph=graph).model_dump(mode="python") + loaded = RunResponse.model_validate(wire).graph + + assert loaded is not None + assert loaded.traces[0].nodes[0].routed_experts.shape == (1, 2, 1) + + +@pytest.mark.e2e +@pytest.mark.subprocess +@pytest.mark.null +async def test_echo_chain_live(tmp_path): + """The echo-chain fixture topology end to end via `run_eval` (live model, null + harness in-subprocess): two linked reward-1.0 traces per instance, persisted as one + instance record.""" + from verifiers.v1.cli.eval.runner import graph_traces, run_eval + + config = EvalConfig( + topology={"id": "echo-chain-v1"}, + num_tasks=1, + max_turns=2, + sampling={"max_tokens": 2048, "temperature": 0}, + timeout={"rollout": 180, "scoring": 60}, + retries={"rollout": {"max_retries": 2, "include": ["ProviderError"]}}, + rich=False, + output_dir=tmp_path, + ) + traces = graph_traces(await run_eval(config)) + first, second = traces + assert first.errors == [] and second.errors == [] + assert second.parents == [first.id] + assert first.rewards == {"echoed": 1.0, "relay": 1.0} + assert second.reward == 1.0 + (line,) = (tmp_path / "traces.jsonl").read_text().splitlines() + graph = AgentGraph.load(json.loads(line)) + assert [t.agent for t in graph.traces] == ["first", "second"] + assert graph.error is None diff --git a/uv.lock b/uv.lock index 462bbc616e..16e94b8828 100644 --- a/uv.lock +++ b/uv.lock @@ -589,6 +589,21 @@ version = "1.11.2" source = { registry = "https://pypi.org/simple" } sdist = { url = "https://files.pythonhosted.org/packages/93/09/7d04d7581ae3bb8b598017941781bceb7959dd1b13e3ebf7b6a2cd843bc9/chess-1.11.2.tar.gz", hash = "sha256:a8b43e5678fdb3000695bdaa573117ad683761e5ca38e591c4826eba6d25bb39", size = 6131385, upload-time = "2025-02-25T19:10:27.328Z" } +[[package]] +name = "chess-v1" +version = "0.1.0" +source = { editable = "environments/chess_v1" } +dependencies = [ + { name = "chess" }, + { name = "verifiers" }, +] + +[package.metadata] +requires-dist = [ + { name = "chess", specifier = ">=1.10" }, + { name = "verifiers" }, +] + [[package]] name = "chromadb" version = "1.5.9" @@ -639,9 +654,9 @@ name = "claude-agent-sdk" version = "0.2.104" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anyio", marker = "python_full_version >= '3.12'" }, - { name = "mcp", marker = "python_full_version >= '3.12'" }, - { name = "sniffio", marker = "python_full_version >= '3.12'" }, + { name = "anyio" }, + { name = "mcp" }, + { name = "sniffio" }, ] sdist = { url = "https://files.pythonhosted.org/packages/a6/14/0da3c38ba2fd8dd621bbc458d032eb77f75e745ae6d7b57be31c720c336b/claude_agent_sdk-0.2.104.tar.gz", hash = "sha256:b763b162337dc805916896e460f6c6d813523c9d0a6426b5b52c99c6ca17d740", size = 255616, upload-time = "2026-06-17T22:21:30.445Z" } wheels = [ @@ -997,6 +1012,17 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/37/f0/99fe6eb530c7ee9ee1faee48059eb8a6437f80c893a496b98a78864e0fc6/datasets-4.6.1-py3-none-any.whl", hash = "sha256:f53228e6dadc9f837037b1bf3051d7d8c054abbb3eb29f1f022926e08090e0da", size = 520667, upload-time = "2026-02-27T23:26:46.855Z" }, ] +[[package]] +name = "debate-v1" +version = "0.1.0" +source = { editable = "environments/debate_v1" } +dependencies = [ + { name = "verifiers" }, +] + +[package.metadata] +requires-dist = [{ name = "verifiers" }] + [[package]] name = "debugpy" version = "1.8.20" @@ -1062,7 +1088,7 @@ name = "deprecation" version = "2.1.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "packaging", marker = "python_full_version >= '3.12'" }, + { name = "packaging" }, ] sdist = { url = "https://files.pythonhosted.org/packages/5a/d3/8ae2869247df154b64c1884d7346d412fed0c49df84db635aab2d1c40e62/deprecation-2.1.0.tar.gz", hash = "sha256:72b3bde64e5d778694b0cf68178aed03d15e15477116add3fb773e581f9518ff", size = 173788, upload-time = "2020-04-20T14:23:38.738Z" } wheels = [ @@ -1105,7 +1131,7 @@ name = "dirhash" version = "0.5.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "scantree", marker = "python_full_version >= '3.12'" }, + { name = "scantree" }, ] sdist = { url = "https://files.pythonhosted.org/packages/1d/70/49f93897f3a4f7ab5f20a854ebc91aad47854e9fb2cd169e3a4452fa3f5e/dirhash-0.5.0.tar.gz", hash = "sha256:e60760f0ab2e935d8cb088923ea2c6492398dca42cec785df778985fd4cd5386", size = 21377, upload-time = "2024-08-03T22:14:13.322Z" } wheels = [ @@ -1224,9 +1250,9 @@ resolution-markers = [ "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "pydantic", marker = "python_full_version < '3.12'" }, - { name = "starlette", marker = "python_full_version < '3.12'" }, - { name = "typing-extensions", marker = "python_full_version < '3.12'" }, + { name = "pydantic" }, + { name = "starlette" }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/ca/53/8c38a874844a8b0fa10dd8adf3836ac154082cf88d3f22b544e9ceea0a15/fastapi-0.115.14.tar.gz", hash = "sha256:b1de15cdc1c499a4da47914db35d0e4ef8f1ce62b624e94e0e5824421df99739", size = 296263, upload-time = "2025-06-26T15:29:08.21Z" } wheels = [ @@ -1235,12 +1261,12 @@ wheels = [ [package.optional-dependencies] standard = [ - { name = "email-validator", marker = "python_full_version < '3.12'" }, - { name = "fastapi-cli", version = "0.0.5", source = { registry = "https://pypi.org/simple" }, extra = ["standard"], marker = "python_full_version < '3.12'" }, - { name = "httpx", marker = "python_full_version < '3.12'" }, - { name = "jinja2", marker = "python_full_version < '3.12'" }, - { name = "python-multipart", marker = "python_full_version < '3.12'" }, - { name = "uvicorn", extra = ["standard"], marker = "python_full_version < '3.12'" }, + { name = "email-validator" }, + { name = "fastapi-cli", version = "0.0.5", source = { registry = "https://pypi.org/simple" }, extra = ["standard"] }, + { name = "httpx" }, + { name = "jinja2" }, + { name = "python-multipart" }, + { name = "uvicorn", extra = ["standard"] }, ] [[package]] @@ -1256,11 +1282,11 @@ resolution-markers = [ "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "annotated-doc", marker = "python_full_version >= '3.12'" }, - { name = "pydantic", marker = "python_full_version >= '3.12'" }, - { name = "starlette", marker = "python_full_version >= '3.12'" }, - { name = "typing-extensions", marker = "python_full_version >= '3.12'" }, - { name = "typing-inspection", marker = "python_full_version >= '3.12'" }, + { name = "annotated-doc" }, + { name = "pydantic" }, + { name = "starlette" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, ] sdist = { url = "https://files.pythonhosted.org/packages/e2/29/cc5819dc24d3daa80cdaa1aec023bf8652a70dd7fd1c96b0b225c99a7690/fastapi-0.137.2.tar.gz", hash = "sha256:b9d893bebc97dcfbdcb1917e88a292d062844ea19445a5fa4f7eb28c4baea9e3", size = 410332, upload-time = "2026-06-18T06:58:24.434Z" } wheels = [ @@ -1269,15 +1295,15 @@ wheels = [ [package.optional-dependencies] standard = [ - { name = "email-validator", marker = "python_full_version >= '3.12'" }, - { name = "fastapi-cli", version = "0.0.27", source = { registry = "https://pypi.org/simple" }, extra = ["standard"], marker = "python_full_version >= '3.12'" }, - { name = "fastar", marker = "python_full_version >= '3.12'" }, - { name = "httpx", marker = "python_full_version >= '3.12'" }, - { name = "jinja2", marker = "python_full_version >= '3.12'" }, - { name = "pydantic-extra-types", marker = "python_full_version >= '3.12'" }, - { name = "pydantic-settings", marker = "python_full_version >= '3.12'" }, - { name = "python-multipart", marker = "python_full_version >= '3.12'" }, - { name = "uvicorn", extra = ["standard"], marker = "python_full_version >= '3.12'" }, + { name = "email-validator" }, + { name = "fastapi-cli", version = "0.0.27", source = { registry = "https://pypi.org/simple" }, extra = ["standard"] }, + { name = "fastar" }, + { name = "httpx" }, + { name = "jinja2" }, + { name = "pydantic-extra-types" }, + { name = "pydantic-settings" }, + { name = "python-multipart" }, + { name = "uvicorn", extra = ["standard"] }, ] [[package]] @@ -1290,8 +1316,8 @@ resolution-markers = [ "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "typer", marker = "python_full_version < '3.12'" }, - { name = "uvicorn", extra = ["standard"], marker = "python_full_version < '3.12'" }, + { name = "typer" }, + { name = "uvicorn", extra = ["standard"] }, ] sdist = { url = "https://files.pythonhosted.org/packages/c5/f8/1ad5ce32d029aeb9117e9a5a9b3e314a8477525d60c12a9b7730a3c186ec/fastapi_cli-0.0.5.tar.gz", hash = "sha256:d30e1239c6f46fcb95e606f02cdda59a1e2fa778a54b64686b3ff27f6211ff9f", size = 15571, upload-time = "2024-08-02T05:48:13.16Z" } wheels = [ @@ -1300,7 +1326,7 @@ wheels = [ [package.optional-dependencies] standard = [ - { name = "uvicorn", extra = ["standard"], marker = "python_full_version < '3.12'" }, + { name = "uvicorn", extra = ["standard"] }, ] [[package]] @@ -1316,9 +1342,9 @@ resolution-markers = [ "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "rich-toolkit", marker = "python_full_version >= '3.12'" }, - { name = "typer", marker = "python_full_version >= '3.12'" }, - { name = "uvicorn", extra = ["standard"], marker = "python_full_version >= '3.12'" }, + { name = "rich-toolkit" }, + { name = "typer" }, + { name = "uvicorn", extra = ["standard"] }, ] sdist = { url = "https://files.pythonhosted.org/packages/37/d0/ee5678346811967b8d096d5d5604e71b50d6bf5a2abfbdb331157e2bbaa9/fastapi_cli-0.0.27.tar.gz", hash = "sha256:1dffb1e40c0c88f2e0171a8a252a2b615c1e63ff8c05626649e4badd6a84336a", size = 23630, upload-time = "2026-06-18T14:48:43.421Z" } wheels = [ @@ -1327,8 +1353,8 @@ wheels = [ [package.optional-dependencies] standard = [ - { name = "fastapi-cloud-cli", marker = "python_full_version >= '3.12'" }, - { name = "uvicorn", extra = ["standard"], marker = "python_full_version >= '3.12'" }, + { name = "fastapi-cloud-cli" }, + { name = "uvicorn", extra = ["standard"] }, ] [[package]] @@ -1336,15 +1362,15 @@ name = "fastapi-cloud-cli" version = "0.20.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "detect-installer", marker = "python_full_version >= '3.12'" }, - { name = "fastar", marker = "python_full_version >= '3.12'" }, - { name = "httpx", marker = "python_full_version >= '3.12'" }, - { name = "pydantic", extra = ["email"], marker = "python_full_version >= '3.12'" }, - { name = "rich-toolkit", marker = "python_full_version >= '3.12'" }, - { name = "rignore", marker = "python_full_version >= '3.12'" }, - { name = "sentry-sdk", marker = "python_full_version >= '3.12'" }, - { name = "typer", marker = "python_full_version >= '3.12'" }, - { name = "uvicorn", extra = ["standard"], marker = "python_full_version >= '3.12'" }, + { name = "detect-installer" }, + { name = "fastar" }, + { name = "httpx" }, + { name = "pydantic", extra = ["email"] }, + { name = "rich-toolkit" }, + { name = "rignore" }, + { name = "sentry-sdk" }, + { name = "typer" }, + { name = "uvicorn", extra = ["standard"] }, ] sdist = { url = "https://files.pythonhosted.org/packages/ad/bf/97d19633c6ec6fb0ef59df474b9705ea992f7b4f879208d0007ac6d25ab6/fastapi_cloud_cli-0.20.0.tar.gz", hash = "sha256:9681c46adcd299024d0775658bd5d88992fd35c4ad42b1f045c6df913390ba37", size = 85904, upload-time = "2026-06-11T17:41:02.814Z" } wheels = [ @@ -1818,26 +1844,26 @@ name = "harbor" version = "0.14.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "claude-agent-sdk", marker = "python_full_version >= '3.12'" }, - { name = "datasets", marker = "python_full_version >= '3.12'" }, - { name = "dirhash", marker = "python_full_version >= '3.12'" }, - { name = "fastapi", version = "0.137.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "httpx", marker = "python_full_version >= '3.12'" }, - { name = "jinja2", marker = "python_full_version >= '3.12'" }, - { name = "litellm", marker = "python_full_version >= '3.12'" }, - { name = "packaging", marker = "python_full_version >= '3.12'" }, - { name = "pathspec", marker = "python_full_version >= '3.12'" }, - { name = "pydantic", marker = "python_full_version >= '3.12'" }, - { name = "python-dotenv", marker = "python_full_version >= '3.12'" }, - { name = "pyyaml", marker = "python_full_version >= '3.12'" }, - { name = "requests", marker = "python_full_version >= '3.12'" }, - { name = "rich", marker = "python_full_version >= '3.12'" }, - { name = "shortuuid", marker = "python_full_version >= '3.12'" }, - { name = "supabase", marker = "python_full_version >= '3.12'" }, - { name = "tenacity", marker = "python_full_version >= '3.12'" }, - { name = "toml", marker = "python_full_version >= '3.12'" }, - { name = "typer", marker = "python_full_version >= '3.12'" }, - { name = "uvicorn", marker = "python_full_version >= '3.12'" }, + { name = "claude-agent-sdk" }, + { name = "datasets" }, + { name = "dirhash" }, + { name = "fastapi", version = "0.137.2", source = { registry = "https://pypi.org/simple" } }, + { name = "httpx" }, + { name = "jinja2" }, + { name = "litellm" }, + { name = "packaging" }, + { name = "pathspec" }, + { name = "pydantic" }, + { name = "python-dotenv" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "rich" }, + { name = "shortuuid" }, + { name = "supabase" }, + { name = "tenacity" }, + { name = "toml" }, + { name = "typer" }, + { name = "uvicorn" }, ] sdist = { url = "https://files.pythonhosted.org/packages/a9/f3/cba38a11b0cca01ecd471437869f1cd57d86a3b1e4091b8fd3d112cba8af/harbor-0.14.0.tar.gz", hash = "sha256:0ffffc25a618e4b7bf2b901a8c717c8bc28877f5aa8c37be47f1ec8c301cb0b6", size = 1241271, upload-time = "2026-06-17T16:43:23.495Z" } wheels = [ @@ -1945,7 +1971,7 @@ wheels = [ [package.optional-dependencies] http2 = [ - { name = "h2", marker = "python_full_version >= '3.12'" }, + { name = "h2" }, ] [[package]] @@ -2416,18 +2442,18 @@ name = "litellm" version = "1.89.2" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "aiohttp", marker = "python_full_version >= '3.12'" }, - { name = "click", marker = "python_full_version >= '3.12'" }, - { name = "fastuuid", marker = "python_full_version >= '3.12'" }, - { name = "httpx", marker = "python_full_version >= '3.12'" }, - { name = "importlib-metadata", marker = "python_full_version >= '3.12'" }, - { name = "jinja2", marker = "python_full_version >= '3.12'" }, - { name = "jsonschema", marker = "python_full_version >= '3.12'" }, - { name = "openai", marker = "python_full_version >= '3.12'" }, - { name = "pydantic", marker = "python_full_version >= '3.12'" }, - { name = "python-dotenv", marker = "python_full_version >= '3.12'" }, - { name = "tiktoken", marker = "python_full_version >= '3.12'" }, - { name = "tokenizers", marker = "python_full_version >= '3.12'" }, + { name = "aiohttp" }, + { name = "click" }, + { name = "fastuuid" }, + { name = "httpx" }, + { name = "importlib-metadata" }, + { name = "jinja2" }, + { name = "jsonschema" }, + { name = "openai" }, + { name = "pydantic" }, + { name = "python-dotenv" }, + { name = "tiktoken" }, + { name = "tokenizers" }, ] sdist = { url = "https://files.pythonhosted.org/packages/c1/29/865d38325f9c424daf0bcc7ab61908cf51a87862b3a0dfebd059b60dac97/litellm-1.89.2.tar.gz", hash = "sha256:b2534d69568eed026310f4e006407db2d46494eb629bd1e71eb9603ec146540d", size = 14079449, upload-time = "2026-06-18T02:26:03.64Z" } wheels = [ @@ -2716,7 +2742,7 @@ name = "mlx" version = "0.31.2" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "mlx-metal", marker = "sys_platform == 'darwin'" }, + { name = "mlx-metal" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/94/89/1e77ec3ff380e8fb9e7258047374d31452a0f9828a0e370f127b07dd8288/mlx-0.31.2-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:4a3f181b367d404e44a6bd68ef5eb573930809ac60cacd51d0c851c629b1b651", size = 586911, upload-time = "2026-04-22T03:14:29.675Z" }, @@ -2735,13 +2761,13 @@ name = "mlx-lm" version = "0.31.3" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "jinja2", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, - { name = "mlx", marker = "sys_platform == 'darwin'" }, - { name = "numpy", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, - { name = "protobuf", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, - { name = "pyyaml", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, - { name = "sentencepiece", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, - { name = "transformers", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "jinja2" }, + { name = "mlx" }, + { name = "numpy" }, + { name = "protobuf" }, + { name = "pyyaml" }, + { name = "sentencepiece" }, + { name = "transformers" }, ] sdist = { url = "https://files.pythonhosted.org/packages/84/94/9a38d6b0c6fcca995b9136c94eb7da1e9c5165652edf228b96b29960fa7a/mlx_lm-0.31.3.tar.gz", hash = "sha256:61eb0e3ba09444f77f874aff295401d7ccd20b39495cbbce0c782a15474ce733", size = 304318, upload-time = "2026-04-22T07:37:27.922Z" } wheels = [ @@ -3200,7 +3226,7 @@ name = "nvidia-cudnn-cu12" version = "9.10.2.21" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "nvidia-cublas-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "nvidia-cublas-cu12" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/ba/51/e123d997aa098c61d029f76663dedbfb9bc8dcf8c60cbd6adbe42f76d049/nvidia_cudnn_cu12-9.10.2.21-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:949452be657fa16687d0930933f032835951ef0892b37d2d53824d1a84dc97a8", size = 706758467, upload-time = "2025-06-06T21:54:08.597Z" }, @@ -3211,7 +3237,7 @@ name = "nvidia-cufft-cu12" version = "11.3.3.83" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "nvidia-nvjitlink-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "nvidia-nvjitlink-cu12" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/1f/13/ee4e00f30e676b66ae65b4f08cb5bcbb8392c03f54f2d5413ea99a5d1c80/nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4d2dd21ec0b88cf61b62e6b43564355e5222e4a3fb394cac0db101f2dd0d4f74", size = 193118695, upload-time = "2025-03-07T01:45:27.821Z" }, @@ -3238,9 +3264,9 @@ name = "nvidia-cusolver-cu12" version = "11.7.3.90" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "nvidia-cublas-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, - { name = "nvidia-cusparse-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, - { name = "nvidia-nvjitlink-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "nvidia-cublas-cu12" }, + { name = "nvidia-cusparse-cu12" }, + { name = "nvidia-nvjitlink-cu12" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/85/48/9a13d2975803e8cf2777d5ed57b87a0b6ca2cc795f9a4f59796a910bfb80/nvidia_cusolver_cu12-11.7.3.90-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:4376c11ad263152bd50ea295c05370360776f8c3427b30991df774f9fb26c450", size = 267506905, upload-time = "2025-03-07T01:47:16.273Z" }, @@ -3251,7 +3277,7 @@ name = "nvidia-cusparse-cu12" version = "12.5.8.93" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "nvidia-nvjitlink-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "nvidia-nvjitlink-cu12" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/c2/f5/e1854cb2f2bcd4280c44736c93550cc300ff4b8c95ebe370d0aa7d2b473d/nvidia_cusparse_cu12-12.5.8.93-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1ec05d76bbbd8b61b06a80e1eaf8cf4959c3d4ce8e711b65ebd0443bb0ebb13b", size = 288216466, upload-time = "2025-03-07T01:48:13.779Z" }, @@ -3417,8 +3443,8 @@ resolution-markers = [ "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "importlib-metadata", marker = "python_full_version < '3.12'" }, - { name = "typing-extensions", marker = "python_full_version < '3.12'" }, + { name = "importlib-metadata" }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/63/04/05040d7ce33a907a2a02257e601992f0cdf11c73b33f13c4492bf6c3d6d5/opentelemetry_api-1.37.0.tar.gz", hash = "sha256:540735b120355bd5112738ea53621f8d5edb35ebcd6fe21ada3ab1c61d1cd9a7", size = 64923, upload-time = "2025-09-11T10:29:01.662Z" } wheels = [ @@ -3438,7 +3464,7 @@ resolution-markers = [ "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "typing-extensions", marker = "python_full_version >= '3.12'" }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/b4/1c/125e1c936c0873796771b7f04f6c93b9f1bf5d424cea90fda94a99f61da8/opentelemetry_api-1.42.1.tar.gz", hash = "sha256:56c63bea9f77b62856be8c47600474acad853b2924b99b1687c4cb6297166716", size = 72296, upload-time = "2026-05-21T16:32:49.335Z" } wheels = [ @@ -3455,7 +3481,7 @@ resolution-markers = [ "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "opentelemetry-proto", version = "1.37.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "opentelemetry-proto", version = "1.37.0", source = { registry = "https://pypi.org/simple" } }, ] sdist = { url = "https://files.pythonhosted.org/packages/dc/6c/10018cbcc1e6fff23aac67d7fd977c3d692dbe5f9ef9bb4db5c1268726cc/opentelemetry_exporter_otlp_proto_common-1.37.0.tar.gz", hash = "sha256:c87a1bdd9f41fdc408d9cc9367bb53f8d2602829659f2b90be9f9d79d0bfe62c", size = 20430, upload-time = "2025-09-11T10:29:03.605Z" } wheels = [ @@ -3475,7 +3501,7 @@ resolution-markers = [ "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "opentelemetry-proto", version = "1.42.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "opentelemetry-proto", version = "1.42.1", source = { registry = "https://pypi.org/simple" } }, ] sdist = { url = "https://files.pythonhosted.org/packages/0e/9c/216acfeaedadf2e1937f4373929b20f73197c5c4a2546d4f584b7fa63813/opentelemetry_exporter_otlp_proto_common-1.42.1.tar.gz", hash = "sha256:04f1f01fb597c4249dfcd7f8b861c902c2102369d376d9d346ff38de4469a2ee", size = 21433, upload-time = "2026-05-21T16:32:55.526Z" } wheels = [ @@ -3492,13 +3518,13 @@ resolution-markers = [ "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "googleapis-common-protos", marker = "python_full_version < '3.12'" }, - { name = "grpcio", marker = "python_full_version < '3.12'" }, - { name = "opentelemetry-api", version = "1.37.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, - { name = "opentelemetry-exporter-otlp-proto-common", version = "1.37.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, - { name = "opentelemetry-proto", version = "1.37.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, - { name = "opentelemetry-sdk", version = "1.37.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, - { name = "typing-extensions", marker = "python_full_version < '3.12'" }, + { name = "googleapis-common-protos" }, + { name = "grpcio" }, + { name = "opentelemetry-api", version = "1.37.0", source = { registry = "https://pypi.org/simple" } }, + { name = "opentelemetry-exporter-otlp-proto-common", version = "1.37.0", source = { registry = "https://pypi.org/simple" } }, + { name = "opentelemetry-proto", version = "1.37.0", source = { registry = "https://pypi.org/simple" } }, + { name = "opentelemetry-sdk", version = "1.37.0", source = { registry = "https://pypi.org/simple" } }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/d1/11/4ad0979d0bb13ae5a845214e97c8d42da43980034c30d6f72d8e0ebe580e/opentelemetry_exporter_otlp_proto_grpc-1.37.0.tar.gz", hash = "sha256:f55bcb9fc848ce05ad3dd954058bc7b126624d22c4d9e958da24d8537763bec5", size = 24465, upload-time = "2025-09-11T10:29:04.172Z" } wheels = [ @@ -3518,13 +3544,13 @@ resolution-markers = [ "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "googleapis-common-protos", marker = "python_full_version >= '3.12'" }, - { name = "grpcio", marker = "python_full_version >= '3.12'" }, - { name = "opentelemetry-api", version = "1.42.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "opentelemetry-exporter-otlp-proto-common", version = "1.42.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "opentelemetry-proto", version = "1.42.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "opentelemetry-sdk", version = "1.42.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "typing-extensions", marker = "python_full_version >= '3.12'" }, + { name = "googleapis-common-protos" }, + { name = "grpcio" }, + { name = "opentelemetry-api", version = "1.42.1", source = { registry = "https://pypi.org/simple" } }, + { name = "opentelemetry-exporter-otlp-proto-common", version = "1.42.1", source = { registry = "https://pypi.org/simple" } }, + { name = "opentelemetry-proto", version = "1.42.1", source = { registry = "https://pypi.org/simple" } }, + { name = "opentelemetry-sdk", version = "1.42.1", source = { registry = "https://pypi.org/simple" } }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/87/87/ca7fc790dfdbcf4f9e9aab14a39ef1b7508ead13707e283de0b3131478d2/opentelemetry_exporter_otlp_proto_grpc-1.42.1.tar.gz", hash = "sha256:975c4461f167dd8ed8857d68d3b6b25f3d272eab896f6a9470d0f5b90e2faf15", size = 27140, upload-time = "2026-05-21T16:32:56.162Z" } wheels = [ @@ -3541,7 +3567,7 @@ resolution-markers = [ "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "protobuf", marker = "python_full_version < '3.12'" }, + { name = "protobuf" }, ] sdist = { url = "https://files.pythonhosted.org/packages/dd/ea/a75f36b463a36f3c5a10c0b5292c58b31dbdde74f6f905d3d0ab2313987b/opentelemetry_proto-1.37.0.tar.gz", hash = "sha256:30f5c494faf66f77faeaefa35ed4443c5edb3b0aa46dad073ed7210e1a789538", size = 46151, upload-time = "2025-09-11T10:29:11.04Z" } wheels = [ @@ -3561,7 +3587,7 @@ resolution-markers = [ "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "protobuf", marker = "python_full_version >= '3.12'" }, + { name = "protobuf" }, ] sdist = { url = "https://files.pythonhosted.org/packages/b4/55/63eac3e1089b768ba014091fdd2ae8a9a440c821ef5e2b786909c94c8836/opentelemetry_proto-1.42.1.tar.gz", hash = "sha256:c6a51e6b4f05ae63565f3a113217f3d2bfaec68f78c02d7a6c85f9010d1cfca6", size = 45839, upload-time = "2026-05-21T16:33:03.937Z" } wheels = [ @@ -3578,9 +3604,9 @@ resolution-markers = [ "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "opentelemetry-api", version = "1.37.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, - { name = "opentelemetry-semantic-conventions", version = "0.58b0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, - { name = "typing-extensions", marker = "python_full_version < '3.12'" }, + { name = "opentelemetry-api", version = "1.37.0", source = { registry = "https://pypi.org/simple" } }, + { name = "opentelemetry-semantic-conventions", version = "0.58b0", source = { registry = "https://pypi.org/simple" } }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/f4/62/2e0ca80d7fe94f0b193135375da92c640d15fe81f636658d2acf373086bc/opentelemetry_sdk-1.37.0.tar.gz", hash = "sha256:cc8e089c10953ded765b5ab5669b198bbe0af1b3f89f1007d19acd32dc46dda5", size = 170404, upload-time = "2025-09-11T10:29:11.779Z" } wheels = [ @@ -3600,9 +3626,9 @@ resolution-markers = [ "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "opentelemetry-api", version = "1.42.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "opentelemetry-semantic-conventions", version = "0.63b1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "typing-extensions", marker = "python_full_version >= '3.12'" }, + { name = "opentelemetry-api", version = "1.42.1", source = { registry = "https://pypi.org/simple" } }, + { name = "opentelemetry-semantic-conventions", version = "0.63b1", source = { registry = "https://pypi.org/simple" } }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/40/f7/b390bd9bfd703bf98a68fea1f27786c6872331fd617164a54b8a59bdc008/opentelemetry_sdk-1.42.1.tar.gz", hash = "sha256:8c834e8f8c9ba4171d4ec843d0cb8a67e4c7394d3f9e9297e582cbd9456ddbf7", size = 239262, upload-time = "2026-05-21T16:33:04.641Z" } wheels = [ @@ -3619,8 +3645,8 @@ resolution-markers = [ "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "opentelemetry-api", version = "1.37.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, - { name = "typing-extensions", marker = "python_full_version < '3.12'" }, + { name = "opentelemetry-api", version = "1.37.0", source = { registry = "https://pypi.org/simple" } }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/aa/1b/90701d91e6300d9f2fb352153fb1721ed99ed1f6ea14fa992c756016e63a/opentelemetry_semantic_conventions-0.58b0.tar.gz", hash = "sha256:6bd46f51264279c433755767bb44ad00f1c9e2367e1b42af563372c5a6fa0c25", size = 129867, upload-time = "2025-09-11T10:29:12.597Z" } wheels = [ @@ -3640,8 +3666,8 @@ resolution-markers = [ "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "opentelemetry-api", version = "1.42.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "typing-extensions", marker = "python_full_version >= '3.12'" }, + { name = "opentelemetry-api", version = "1.42.1", source = { registry = "https://pypi.org/simple" } }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/93/99/4d7dd6df64795951413ce6e815f8cf1eb191daf7196ae86574589643d5f3/opentelemetry_semantic_conventions-0.63b1.tar.gz", hash = "sha256:3daf963611334b365e98a57438183eb012d3bfb40b2d931a9af613476b8701a9", size = 148340, upload-time = "2026-05-21T16:33:05.455Z" } wheels = [ @@ -3848,7 +3874,7 @@ name = "pexpect" version = "4.9.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "ptyprocess", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "ptyprocess" }, ] sdist = { url = "https://files.pythonhosted.org/packages/42/92/cc564bf6381ff43ce1f4d06852fc19a2f11d180f23dc32d9588bee2f149d/pexpect-4.9.0.tar.gz", hash = "sha256:ee7d41123f3c9911050ea2c2dac107568dc43b2d3b0c7557a33212c398ead30f", size = 166450, upload-time = "2023-11-25T09:07:26.339Z" } wheels = [ @@ -3940,10 +3966,10 @@ name = "postgrest" version = "2.31.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "deprecation", marker = "python_full_version >= '3.12'" }, - { name = "httpx", extra = ["http2"], marker = "python_full_version >= '3.12'" }, - { name = "pydantic", marker = "python_full_version >= '3.12'" }, - { name = "yarl", marker = "python_full_version >= '3.12'" }, + { name = "deprecation" }, + { name = "httpx", extra = ["http2"] }, + { name = "pydantic" }, + { name = "yarl" }, ] sdist = { url = "https://files.pythonhosted.org/packages/e2/22/88c470d8838d2678a44e0172d061630b8837cba3fb7fb492e28f6578c309/postgrest-2.31.0.tar.gz", hash = "sha256:2f395d84b2ee34fc57622ff2f711df603e2ede625f98e5015240741888f7bd0c", size = 14419, upload-time = "2026-06-04T13:37:20.474Z" } wheels = [ @@ -4117,6 +4143,17 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5b/5a/bc7b4a4ef808fa59a816c17b20c4bef6884daebbdf627ff2a161da67da19/propcache-0.4.1-py3-none-any.whl", hash = "sha256:af2a6052aeb6cf17d3e46ee169099044fd8224cbaf75c76a2ef596e8163e2237", size = 13305, upload-time = "2025-10-08T19:49:00.792Z" }, ] +[[package]] +name = "proposer-solver-v1" +version = "0.1.0" +source = { editable = "environments/proposer_solver_v1" } +dependencies = [ + { name = "verifiers" }, +] + +[package.metadata] +requires-dist = [{ name = "verifiers" }] + [[package]] name = "protobuf" version = "6.33.6" @@ -4363,7 +4400,7 @@ wheels = [ [package.optional-dependencies] email = [ - { name = "email-validator", marker = "python_full_version >= '3.12'" }, + { name = "email-validator" }, ] [[package]] @@ -4832,9 +4869,9 @@ name = "realtime" version = "2.31.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "pydantic", marker = "python_full_version >= '3.12'" }, - { name = "typing-extensions", marker = "python_full_version >= '3.12'" }, - { name = "websockets", marker = "python_full_version >= '3.12'" }, + { name = "pydantic" }, + { name = "typing-extensions" }, + { name = "websockets" }, ] sdist = { url = "https://files.pythonhosted.org/packages/df/34/54a1eaaefa24db5cb12596fd74792e08efa53ed30dc5bce2c0a68ded6146/realtime-2.31.0.tar.gz", hash = "sha256:9e641cb4d77ca0fe768515f8cf9f83550c79f49ce1550a95afc2dc0e252be8c9", size = 18716, upload-time = "2026-06-04T13:37:22.089Z" } wheels = [ @@ -5025,9 +5062,9 @@ name = "rich-toolkit" version = "0.20.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "click", marker = "python_full_version >= '3.12'" }, - { name = "rich", marker = "python_full_version >= '3.12'" }, - { name = "typing-extensions", marker = "python_full_version >= '3.12'" }, + { name = "click" }, + { name = "rich" }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/29/63/3e427c62f1992945c997d4ec31e2fcb37d26aadbe5aa44ae5b29f7f64d26/rich_toolkit-0.20.1.tar.gz", hash = "sha256:c7336ae281f435c785acecaedc4b71d4b663dc73d9c8079fea96372527e822a4", size = 203473, upload-time = "2026-06-05T08:56:57.679Z" } wheels = [ @@ -5230,8 +5267,8 @@ name = "scantree" version = "0.0.4" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "attrs", marker = "python_full_version >= '3.12'" }, - { name = "pathspec", marker = "python_full_version >= '3.12'" }, + { name = "attrs" }, + { name = "pathspec" }, ] sdist = { url = "https://files.pythonhosted.org/packages/b3/e4/40998faefc72ba1ddeb640a44fba92935353525dba110488806da8339c0b/scantree-0.0.4.tar.gz", hash = "sha256:15bd5cb24483b04db2c70653604e8ea3522e98087db7e38ab8482f053984c0ac", size = 24643, upload-time = "2024-08-03T20:08:59.413Z" } wheels = [ @@ -5413,6 +5450,17 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0d/6d/b4752b044bf94cb802d88a888dc7d288baaf77d7910b7dedda74b5ceea0c/setuptools-79.0.1-py3-none-any.whl", hash = "sha256:e147c0549f27767ba362f9da434eab9c5dc0045d5304feb602a0af001089fc51", size = 1256281, upload-time = "2025-04-23T22:20:56.768Z" }, ] +[[package]] +name = "shared-runtime-v1" +version = "0.1.0" +source = { editable = "environments/shared_runtime_v1" } +dependencies = [ + { name = "verifiers" }, +] + +[package.metadata] +requires-dist = [{ name = "verifiers" }] + [[package]] name = "shellingham" version = "1.5.4" @@ -5511,7 +5559,7 @@ resolution-markers = [ "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "anyio", marker = "python_full_version >= '3.12'" }, + { name = "anyio" }, ] sdist = { url = "https://files.pythonhosted.org/packages/8c/f4/989bc70cb8091eda43a9034ef969b25145291f3601703b82766e5172dfed/sse_starlette-2.3.6.tar.gz", hash = "sha256:0382336f7d4ec30160cf9ca0518962905e1b69b72d6c1c995131e0a703b436e3", size = 18284, upload-time = "2025-05-30T13:34:12.914Z" } wheels = [ @@ -5528,7 +5576,7 @@ resolution-markers = [ "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "anyio", marker = "python_full_version < '3.12'" }, + { name = "anyio" }, ] sdist = { url = "https://files.pythonhosted.org/packages/db/3c/fa6517610dc641262b77cc7bf994ecd17465812c1b0585fe33e11be758ab/sse_starlette-3.0.3.tar.gz", hash = "sha256:88cfb08747e16200ea990c8ca876b03910a23b547ab3bd764c0d8eb81019b971", size = 21943, upload-time = "2025-10-30T18:44:20.117Z" } wheels = [ @@ -5586,10 +5634,10 @@ name = "storage3" version = "2.31.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "deprecation", marker = "python_full_version >= '3.12'" }, - { name = "httpx", extra = ["http2"], marker = "python_full_version >= '3.12'" }, - { name = "pydantic", marker = "python_full_version >= '3.12'" }, - { name = "yarl", marker = "python_full_version >= '3.12'" }, + { name = "deprecation" }, + { name = "httpx", extra = ["http2"] }, + { name = "pydantic" }, + { name = "yarl" }, ] sdist = { url = "https://files.pythonhosted.org/packages/13/30/fee43d523d3f680a833a4aae5bf8094de0b9031b0c2bddb3e0bc6e829e1b/storage3-2.31.0.tar.gz", hash = "sha256:d2161e2ea650dc115a1787c30e09b118365589ac772f4dd8643e3a503ecfc667", size = 20348, upload-time = "2026-06-04T13:37:23.703Z" } wheels = [ @@ -5610,13 +5658,13 @@ name = "supabase" version = "2.31.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "httpx", marker = "python_full_version >= '3.12'" }, - { name = "postgrest", marker = "python_full_version >= '3.12'" }, - { name = "realtime", marker = "python_full_version >= '3.12'" }, - { name = "storage3", marker = "python_full_version >= '3.12'" }, - { name = "supabase-auth", marker = "python_full_version >= '3.12'" }, - { name = "supabase-functions", marker = "python_full_version >= '3.12'" }, - { name = "yarl", marker = "python_full_version >= '3.12'" }, + { name = "httpx" }, + { name = "postgrest" }, + { name = "realtime" }, + { name = "storage3" }, + { name = "supabase-auth" }, + { name = "supabase-functions" }, + { name = "yarl" }, ] sdist = { url = "https://files.pythonhosted.org/packages/fe/8e/54a2f950629689b1613434a61fc3bff5f92f84ba6b20213f5b2add05c1bb/supabase-2.31.0.tar.gz", hash = "sha256:3467b09d00482b9a0138235bdbde7a350426f93cf2a1342372eaddfc669f1206", size = 9805, upload-time = "2026-06-04T13:37:25.22Z" } wheels = [ @@ -5628,9 +5676,9 @@ name = "supabase-auth" version = "2.31.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "httpx", extra = ["http2"], marker = "python_full_version >= '3.12'" }, - { name = "pydantic", marker = "python_full_version >= '3.12'" }, - { name = "pyjwt", extra = ["crypto"], marker = "python_full_version >= '3.12'" }, + { name = "httpx", extra = ["http2"] }, + { name = "pydantic" }, + { name = "pyjwt", extra = ["crypto"] }, ] sdist = { url = "https://files.pythonhosted.org/packages/61/8a/408689cf39820f0d46d2731d6747ff94dbefc87ae977b4b5c4066da5b070/supabase_auth-2.31.0.tar.gz", hash = "sha256:0945b33fa96239c76dc8eaf96d7d2c94991950d24b4cfe4a5c2da9aa5e909663", size = 39151, upload-time = "2026-06-04T13:37:27.375Z" } wheels = [ @@ -5642,9 +5690,9 @@ name = "supabase-functions" version = "2.31.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "httpx", extra = ["http2"], marker = "python_full_version >= '3.12'" }, - { name = "strenum", marker = "python_full_version >= '3.12'" }, - { name = "yarl", marker = "python_full_version >= '3.12'" }, + { name = "httpx", extra = ["http2"] }, + { name = "strenum" }, + { name = "yarl" }, ] sdist = { url = "https://files.pythonhosted.org/packages/84/5d/61c2446ed26a57fa5543f9c270a731320911569202340d15341f72cdba7c/supabase_functions-2.31.0.tar.gz", hash = "sha256:4ad027b3ae3bd28b31233339f4db1da6965affd3546f655b421baf40cee2690f", size = 4683, upload-time = "2026-06-04T13:37:28.878Z" } wheels = [ @@ -6254,16 +6302,21 @@ dev = [ ] examples = [ { name = "alphabet-sort-v1" }, + { name = "chess-v1" }, { name = "code-golf-v1" }, { name = "color-codeword-v1" }, { name = "compact" }, + { name = "debate-v1" }, { name = "deepwiki-v1" }, { name = "glossary-v1" }, { name = "gsm8k-v1" }, + { name = "proposer-solver-v1" }, { name = "reverse-text-v1" }, { name = "scratchpad-v1" }, + { name = "shared-runtime-v1" }, { name = "wiki-search-v1" }, { name = "wordle-v1" }, + { name = "writer-editors-v1" }, ] [package.metadata] @@ -6339,16 +6392,21 @@ dev = [ ] examples = [ { name = "alphabet-sort-v1", editable = "environments/alphabet_sort_v1" }, + { name = "chess-v1", editable = "environments/chess_v1" }, { name = "code-golf-v1", editable = "environments/code_golf_v1" }, { name = "color-codeword-v1", editable = "environments/color_codeword_v1" }, { name = "compact", editable = "environments/compact" }, + { name = "debate-v1", editable = "environments/debate_v1" }, { name = "deepwiki-v1", editable = "environments/deepwiki_v1" }, { name = "glossary-v1", editable = "environments/glossary_v1" }, { name = "gsm8k-v1", editable = "environments/gsm8k_v1" }, + { name = "proposer-solver-v1", editable = "environments/proposer_solver_v1" }, { name = "reverse-text-v1", editable = "environments/reverse_text_v1" }, { name = "scratchpad-v1", editable = "environments/scratchpad_v1" }, + { name = "shared-runtime-v1", editable = "environments/shared_runtime_v1" }, { name = "wiki-search-v1", editable = "environments/wiki_search_v1" }, { name = "wordle-v1", editable = "environments/wordle_v1" }, + { name = "writer-editors-v1", editable = "environments/writer_editors_v1" }, ] [[package]] @@ -6633,13 +6691,24 @@ dependencies = [ [package.metadata] requires-dist = [{ name = "verifiers", extras = ["ta"] }] +[[package]] +name = "writer-editors-v1" +version = "0.1.0" +source = { editable = "environments/writer_editors_v1" } +dependencies = [ + { name = "verifiers" }, +] + +[package.metadata] +requires-dist = [{ name = "verifiers" }] + [[package]] name = "xformers" version = "0.0.32.post1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "numpy", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, - { name = "torch", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "numpy" }, + { name = "torch" }, ] sdist = { url = "https://files.pythonhosted.org/packages/6f/33/3b9c4d3d5b2da453d27de891df4ad653ac5795324961aa3a5c15b0353fe6/xformers-0.0.32.post1.tar.gz", hash = "sha256:1de84a45c497c8d92326986508d81f4b0a8c6be4d3d62a29b8ad6048a6ab51e1", size = 12106196, upload-time = "2025-08-14T18:07:45.486Z" } wheels = [ diff --git a/verifiers/v1/__init__.py b/verifiers/v1/__init__.py index 570f178866..355b57eb81 100644 --- a/verifiers/v1/__init__.py +++ b/verifiers/v1/__init__.py @@ -4,24 +4,25 @@ from pydantic_config import BaseConfig +from verifiers.v1.agent import Agent, Session, SessionEnded from verifiers.v1.clients import ( BaseClientConfig, Client, ClientConfig, + EvalClientConfig, ModelContext, + TrainClientConfig, resolve_client, ) -from verifiers.v1.decorators import group_reward, metric, reward, stop, tool +from verifiers.v1.decorators import metric, reward, stop, tool from verifiers.v1.env import ( ElasticPoolConfig, EnvConfig, EnvServerConfig, - Environment, StaticPoolConfig, TimeoutConfig, pool_serve_kwargs, ) -from verifiers.v1.episode import Episode from verifiers.v1.errors import ( HarnessError, InterceptionError, @@ -30,9 +31,11 @@ SandboxError, TaskError, ToolsetError, + TopologyError, TunnelError, UserError, ) +from verifiers.v1.graph import MessageNode from verifiers.v1.harness import Harness, HarnessConfig from verifiers.v1.judge import ( Judge, @@ -41,11 +44,12 @@ Judges, JudgeSamplingConfig, JudgeView, + judge_verdict, ) from verifiers.v1.judges import ( + Criterion, ReferenceJudge, ReferenceJudgeConfig, - Criterion, RubricJudge, RubricJudgeConfig, ) @@ -53,22 +57,21 @@ default_harness_id, harness_config_type, import_harness, - import_judge, import_taskset, - judge_config_type, + import_topology, load_harness, - load_judge, load_taskset, + load_topology, task_type, taskset_config_type, + topology_config_type, ) -from verifiers.v1.scoring import ( - compare_stdout_results as compare_stdout_results, - extract_boxed_answer as extract_boxed_answer, - parse_judge_choice as parse_judge_choice, - parse_pytest_outcomes as parse_pytest_outcomes, - read_answer_file_or_last_reply as read_answer_file_or_last_reply, - verify_boxed_math_answer as verify_boxed_math_answer, +from verifiers.v1.mcp import ( + SharedToolsetConfig, + Toolset, + ToolsetConfig, + User, + UserConfig, ) from verifiers.v1.retries import RetryConfig, RolloutRetryConfig from verifiers.v1.rollout import Rollout @@ -81,6 +84,25 @@ RuntimeInfo, SubprocessConfig, ) +from verifiers.v1.scoring import ( + compare_stdout_results as compare_stdout_results, +) +from verifiers.v1.scoring import ( + extract_boxed_answer as extract_boxed_answer, +) +from verifiers.v1.scoring import ( + parse_judge_choice as parse_judge_choice, +) +from verifiers.v1.scoring import ( + parse_pytest_outcomes as parse_pytest_outcomes, +) +from verifiers.v1.scoring import ( + read_answer_file_or_last_reply as read_answer_file_or_last_reply, +) +from verifiers.v1.scoring import ( + verify_boxed_math_answer as verify_boxed_math_answer, +) +from verifiers.v1.services import RunServices from verifiers.v1.state import State, StateT from verifiers.v1.task import ( Task, @@ -91,14 +113,21 @@ WireTaskData, ) from verifiers.v1.taskset import Taskset, TasksetConfig -from verifiers.v1.mcp import ( - Toolset, - SharedToolsetConfig, - ToolsetConfig, - User, - UserConfig, +from verifiers.v1.topology import ( + AgentBinding, + AgentConfig, + AgentGraph, + DirectAgentConfig, + NullAgentConfig, + SingleAgentTopology, + SingleAgentTopologyConfig, + Topology, + TopologyAgent, + TopologyConfig, + TopologyRun, + TopologyRunner, + resolve_topology_runner, ) -from verifiers.v1.graph import MessageNode from verifiers.v1.trace import ( Branch, Error, @@ -109,9 +138,9 @@ WireTrace, ) from verifiers.v1.types import ( + ID, AssistantMessage, ContentPart, - ID, ImageUrlContentPart, ImageUrlSource, Message, @@ -125,8 +154,8 @@ TextContentPart, Tool, ToolCall, - TurnTokens, ToolMessage, + TurnTokens, Usage, UserMessage, ) @@ -174,7 +203,6 @@ "tool", "metric", "reward", - "group_reward", # errors "RolloutError", "ProviderError", @@ -183,12 +211,15 @@ "UserError", "SandboxError", "TaskError", + "TopologyError", "InterceptionError", "TunnelError", # clients "Client", "BaseClientConfig", "ClientConfig", + "EvalClientConfig", + "TrainClientConfig", "resolve_client", # taskset / harness / runtime / environment "Taskset", @@ -205,7 +236,6 @@ "SubprocessConfig", "DockerConfig", "PrimeConfig", - "Environment", "EnvConfig", "EnvServerConfig", "StaticPoolConfig", @@ -214,32 +244,50 @@ "RetryConfig", "RolloutRetryConfig", "TimeoutConfig", - "Episode", "Rollout", + # agents / topology + "Agent", + "AgentBinding", + "AgentConfig", + "AgentGraph", + "DirectAgentConfig", + "NullAgentConfig", + "SingleAgentTopology", + "SingleAgentTopologyConfig", + "RunServices", + "Session", + "SessionEnded", + "Topology", + "TopologyAgent", + "TopologyConfig", + "TopologyRunner", + "TopologyRun", + "resolve_topology_runner", # loaders "import_taskset", "import_harness", - "import_judge", + "import_topology", "load_taskset", "load_harness", - "load_judge", + "load_topology", "task_type", "taskset_config_type", "harness_config_type", - "judge_config_type", + "topology_config_type", "default_harness_id", - # judge + # judge (a single-call utility — judge-as-agent is the llm-judge/agentic-judge topologies) "Judge", "JudgeConfig", - "Judges", "JudgeSamplingConfig", "JudgeResponse", + "judge_verdict", + "Judges", "JudgeView", + "Criterion", "ReferenceJudge", "ReferenceJudgeConfig", "RubricJudge", "RubricJudgeConfig", - "Criterion", # scoring "compare_stdout_results", "extract_boxed_answer", diff --git a/verifiers/v1/agent.py b/verifiers/v1/agent.py new file mode 100644 index 0000000000..97a48ee2ce --- /dev/null +++ b/verifiers/v1/agent.py @@ -0,0 +1,430 @@ +"""Explicit executable agents. + +An `Agent` is the reusable execution primitive under evals, topologies, and plain agent +programs: harness + model context + runtime policy, with one arrow, `run(task) -> Trace`. +Topologies add graph recording and deferred rewards around that arrow; the agent itself +only knows how to execute one task. + +A bare agent is fully standalone — each `run` provisions a runtime from the policy and +brings up its own per-rollout interception server, right for scripts and small programs: + + agent = vf.Agent(DirectHarness(DirectHarnessConfig()), ctx) + trace = await agent.run(MyTask(vf.TaskData(idx=0, prompt="..."))) + +Serving resources have exactly one owner: `RunServices` (interception pools). For pooled +operation (N concurrent runs sharing interception servers, like an eval), enter a scope +and inject it — `async with RunServices() as services: Agent(..., services=services)` — +which is precisely what `TopologyRunner.serving` does for every agent it binds. The agent +never owns services; it borrows them. Taskset-scoped shared tool servers likewise arrive +pre-served (`shared_tools=`, see `serve_shared`), from whoever owns the taskset. + +The framework's config-driven packaging of agents — CLI/toml-addressable, persisted as +agent graphs, eval- and trainer-integrated — is the topology (`verifiers.v1.topology`); +reach for a bare `Agent` when you're scripting. +""" + +from __future__ import annotations + +import asyncio +import contextlib +from collections.abc import Callable, Sequence +from typing import TYPE_CHECKING + +from verifiers.v1.clients import ModelContext +from verifiers.v1.env import ( + TimeoutConfig, + resolve_runtime_config, + resolve_stage_timeouts, + validate_task_pairing, +) +from verifiers.v1.harness import Harness +from verifiers.v1.interception import InterceptionPool, RolloutLimits +from verifiers.v1.mcp import SharedToolServer +from verifiers.v1.retries import RolloutRetryConfig, run_with_retry +from verifiers.v1.rollout import Rollout +from verifiers.v1.runtimes import Runtime, RuntimeConfig, make_runtime +from verifiers.v1.services import RunServices +from verifiers.v1.task import Task +from verifiers.v1.trace import Trace +from verifiers.v1.types import Messages, UserMessage + +if TYPE_CHECKING: + from collections.abc import AsyncIterator + + +Parent = Trace | str +"""An upstream lineage link: the parent trace itself, or just its id.""" + + +def _parent_ids(parents: Sequence[Parent]) -> list[str]: + return [parent.id if isinstance(parent, Trace) else parent for parent in parents] + + +_END = object() +"""Inbox sentinel: the session ended and its next turn is refused.""" +_DEAD = object() +"""Reply sentinel: the agent run finished (completed, errored, or budget-tripped) — +poisons a pending or future `turn()` into `SessionEnded` instead of a hang.""" + + +class SessionEnded(RuntimeError): + """Raised by `Session.turn` when the agent can no longer take a turn — it ended, + errored, or a budget tripped. Carries the trace: agent failure is data, and `go` + decides what a dead seat means (forfeit it, end the game, keep playing without it).""" + + def __init__(self, trace: Trace | None) -> None: + condition = trace.stop_condition if trace is not None else None + super().__init__(f"session ended ({condition or 'run never started'})") + self.trace = trace + + +class Session: + """A live, suspended agent run `go` converses with — the counterpart seat of one + agent's rollout, held open by `Agent.interact`. + + Three members, deliberately: `turn(message)` sends the agent its next user turn + and returns the model's reply; `end()` finishes the run early (idempotent — + scope exit calls it for you); `.trace` is the live trace (read mid-game state, + stamp `info`). Everything else — who talks to whom, in what order, with what view + of the interaction — is plain imperative code in `go`. + + Mechanically this is a user simulator without the server: the interception layer + suspends the run between model turns awaiting `_respond`, exactly as it awaits + a `User`'s respond tool; here the "user" is `go` itself, over a queue handshake. + The handshake is also the safety contract: `turn()` never blocks on a run that + can no longer answer — it raises `SessionEnded` — and a second in-flight `turn()` + on one session raises immediately (one driver per seat).""" + + def __init__(self) -> None: + self._rollout: Rollout | None = None + self._run_task: asyncio.Task | None = None + self._inbox: asyncio.Queue = asyncio.Queue() + self._replies: asyncio.Queue = asyncio.Queue() + self._opened = False + self._pending = False + + @property + def trace(self) -> Trace: + """The agent's live trace — readable mid-game (`trace.state`, `num_turns`) + and the place `go` stamps outcome facts (`trace.info[...]`) for the topology's + declared rewards to read.""" + if self._rollout is None or self._rollout.trace is None: + raise RuntimeError("session's agent run has not started yet") + return self._rollout.trace + + async def _respond(self, last_assistant: str) -> Messages: + """The user-seat contract (see `RolloutSession.user`), driven by the interception + loop between model turns: deliver the model's turn to `go`, suspend until `go` + supplies the next user turn. The very first call (`respond(\"\")`, the no-prompt + opening) delivers nothing — there is no reply yet.""" + if self._opened: + await self._replies.put(last_assistant) + self._opened = True + incoming = await self._inbox.get() + if incoming is _END: + return [] # stop is already set; `refused()` ends the exchange cleanly + return incoming + + async def turn(self, message: str | Messages) -> str: + """Send the agent its next user turn(s) and return the model's reply text. + Raises `SessionEnded` (never hangs) when the agent can't answer, and refuses + a second concurrent `turn()` on this session — including after a `turn()` was + *cancelled* mid-flight (its message is already with the model, so the + conversation is desynced; `end()` the seat instead of driving on).""" + if self._pending: + raise RuntimeError( + "a turn is already pending (or was cancelled mid-flight) on this " + "session — one driver per seat; end() a desynced seat" + ) + if self._run_task is not None and self._run_task.done(): + raise SessionEnded(self._rollout.trace if self._rollout else None) + turns: Messages = ( + [UserMessage(content=message)] + if isinstance(message, str) + else list(message) + ) + self._pending = True + try: + await self._inbox.put(turns) + reply = await self._replies.get() + except asyncio.CancelledError: + # The message is in flight with no reader for its reply: leave `_pending` + # set so a later turn() can't consume the stale reply one turn late. + raise + self._pending = False + if reply is _DEAD: + raise SessionEnded(self._rollout.trace if self._rollout else None) + return reply + + async def end(self, condition: str = "interaction_complete") -> None: + """Finish the agent run early and cleanly (forfeits, eliminations). Idempotent; + scope exit calls it, so plain `go` code never has to.""" + trace = self._rollout.trace if self._rollout is not None else None + if trace is not None and trace.stop_condition is None: + trace.stop(condition) + await self._inbox.put(_END) + + def _poison(self, _task: asyncio.Task) -> None: + """Run-completion callback: wake any pending (or future) `turn()` into + `SessionEnded` rather than leaving it awaiting a reply that will never come.""" + self._replies.put_nowait(_DEAD) + + +class _AgentAttempt: + """One retryable attempt (see `retries.Runnable`): each `run()` builds and runs a + fresh `Rollout` when the agent owns runtime provisioning; borrowed-runtime attempts + deliberately reuse the borrowed box.""" + + def __init__( + self, + agent: "Agent", + task: Task, + *, + runtime_config: RuntimeConfig, + runtime: Runtime | None, + interception: InterceptionPool | None, + parents: Sequence[Parent], + ) -> None: + self.agent = agent + self.task = task + self.runtime_config = runtime_config + self.runtime = runtime + self.interception = interception + self.parents = parents + + async def run(self) -> Trace: + agent = self.agent + timeouts = resolve_stage_timeouts(agent.timeout, self.task, self.runtime_config) + rollout = Rollout( + task=self.task, + harness=agent.harness, + ctx=agent.ctx, + runtime_config=self.runtime_config, + setup_timeout=timeouts.setup, + harness_timeout=timeouts.rollout, + finalize_timeout=timeouts.finalize, + scoring_timeout=timeouts.scoring, + limits=agent.limits, + shared_tools=agent.shared_tools, + interception=self.interception, + runtime=self.runtime, + ) + if agent._on_rollout is not None: + agent._on_rollout(rollout) + trace = await rollout.run() + agent.stamp( + trace, + parents=self.parents, + runtime=rollout.runtime, + borrowed=self.runtime is not None, + ) + return trace + + +class Agent: + """A harness + model context + runtime policy, runnable on any compatible task. + + `run(task)` provisions a fresh runtime from the policy and tears it down. To share a + world explicitly, use `provision()` and pass the yielded runtime back to one or more + `run(..., runtime=box)` calls; borrowed-runtime rollouts never start or stop the box. + + `services` is the (optional) serving scope the agent borrows pooled interception + from — see the module docstring; without one, every run brings up its own + per-rollout interception server. `shared_tools` are the taskset-scoped shared + servers (already served, see `serve_shared`) this agent's rollouts reuse.""" + + def __init__( + self, + harness: Harness, + ctx: ModelContext, + runtime: RuntimeConfig | None = None, + *, + name: str | None = None, + trainable: bool = True, + limits: RolloutLimits | None = None, + timeout: TimeoutConfig | None = None, + services: RunServices | None = None, + shared_tools: dict[str, SharedToolServer] | None = None, + on_rollout: Callable[[Rollout], None] | None = None, + ) -> None: + self.harness = harness + self.ctx = ctx + self.runtime_config = runtime if runtime is not None else harness.config.runtime + self.name = name + self.trainable = trainable + self.limits = limits or RolloutLimits() + self.timeout = timeout or TimeoutConfig() + self.shared_tools = shared_tools or {} + self._services = services + self._on_rollout = on_rollout + self._warned_resources: set[tuple[str, str]] = set() + self._validated: set[tuple[type[Task], str, str]] = set() + + def runtime_for(self, task: Task, runtime: Runtime | None = None) -> RuntimeConfig: + """The runtime placement this run will actually use.""" + if runtime is not None: + return runtime.config + return resolve_runtime_config(self.runtime_config, task, self._warned_resources) + + def _validate_pairing(self, task: Task, runtime_config: RuntimeConfig) -> None: + key = (type(task), runtime_config.type, runtime_config.__class__.__name__) + if key not in self._validated: + validate_task_pairing( + self.harness, + type(task), + runtime_config, # where the run actually lands — a borrowed box may differ + shared_tools=tuple(self.shared_tools), + ) + self._validated.add(key) + + async def run( + self, + task: Task, + *, + parents: Sequence[Parent] = (), + runtime: Runtime | None = None, + retry: RolloutRetryConfig | None = None, + ) -> Trace: + """Run this agent on `task` once and return its trace, stamped with lineage + (`parents`) and this agent's provenance. `runtime` places the run into a live + borrowed box instead of provisioning one; `retry` applies the whole-rollout + retry policy.""" + runtime_config = self.runtime_for(task, runtime) + self._validate_pairing(task, runtime_config) + services = self._services + attempt = _AgentAttempt( + self, + task, + runtime_config=runtime_config, + runtime=runtime, + interception=await services.pool_for(runtime_config) + if services is not None + else None, + parents=parents, + ) + if retry is not None: + return await run_with_retry(attempt, retry) + return await attempt.run() + + def stamp( + self, + trace: Trace, + *, + parents: Sequence[Parent], + runtime: Runtime | None, + borrowed: bool, + ) -> None: + """Record agent provenance on a finished trace.""" + if self.name is not None: + trace.agent = self.name + trace.parents = _parent_ids(parents) + trace.trainable = self.trainable + trace.sampling = self.ctx.sampling + trace.info["agent"] = { + "name": self.name, + "harness": self.harness.config.id, + "model": self.ctx.model, + "runtime": { + "type": runtime.config.type if runtime is not None else None, + "descriptor": runtime.descriptor if runtime is not None else None, + "borrowed": borrowed, + }, + } + + @contextlib.asynccontextmanager + async def provision(self, task: Task | None = None) -> AsyncIterator[Runtime]: + """Provision a runtime from this agent's policy and tear it down on exit.""" + if task is None: + config = self.runtime_config + else: + config = self.runtime_for(task) + self._validate_pairing(task, config) + runtime = make_runtime(config) + try: + await runtime.start() + yield runtime + finally: + await runtime.stop() + + @contextlib.asynccontextmanager + async def interact( + self, + task: Task, + *, + parents: Sequence[Parent] = (), + runtime: Runtime | None = None, + ) -> AsyncIterator[Session]: + """Hold a live agent run open and yield the `Session` to converse with it — the + primitive under back-and-forth multi-agent (each agent is the other's user seat, + `go` is the router). The agent runs in the background, suspending between + turns; scope exit ends it cleanly (stop → finalize → task scoring) and the + completed trace is stamped like any `run()`. + + Sessions are opened by the first `turn()` — build the task with + `prompt=None` and put the framing in `system_prompt`. No `retry=`: one side of a + half-played interaction can't be transparently re-run; a dead seat surfaces as + `SessionEnded` and `go` decides what it means. Note the run's budgets + (`max_turns`, timeouts) span the whole interaction, including time suspended + while other seats move — size them for the game, not a solo run.""" + if task.data.prompt is not None: + raise ValueError( + "interact() sessions are opened by the first turn(): build the task " + "with prompt=None and put the framing in system_prompt" + ) + if type(task).user is not None: + raise ValueError( + f"task {type(task).__name__} declares its own user simulator " + "(Task.user), and a session is a second claimant for the same user " + "seat — run it with run(), or drop the simulator" + ) + if not type(self.harness).SUPPORTS_USER_SIM: + raise ValueError( + f"harness {self.harness.config.id!r} cannot take injected user turns; " + "sessions need a user-capable harness (e.g. direct, null, default)" + ) + runtime_config = self.runtime_for(task, runtime) + self._validate_pairing(task, runtime_config) + services = self._services + session = Session() + timeouts = resolve_stage_timeouts(self.timeout, task, runtime_config) + rollout = Rollout( + task=task, + harness=self.harness, + ctx=self.ctx, + runtime_config=runtime_config, + setup_timeout=timeouts.setup, + harness_timeout=timeouts.rollout, + finalize_timeout=timeouts.finalize, + scoring_timeout=timeouts.scoring, + limits=self.limits, + shared_tools=self.shared_tools, + interception=await services.pool_for(runtime_config) + if services is not None + else None, + runtime=runtime, + user=session._respond, + ) + if self._on_rollout is not None: + self._on_rollout(rollout) + session._rollout = rollout + run_task = asyncio.create_task(rollout.run()) + session._run_task = run_task + run_task.add_done_callback(session._poison) + try: + yield session + finally: + await session.end() + try: + # A rollout captures its failures onto the trace; the one raise is a + # borrowed-box lifetime bug (torn down under the run), which belongs + # to the caller. + trace = await run_task + except asyncio.CancelledError: + run_task.cancel() + raise + self.stamp( + trace, + parents=parents, + runtime=rollout.runtime, + borrowed=runtime is not None, + ) diff --git a/verifiers/v1/cli/dashboard/eval.py b/verifiers/v1/cli/dashboard/eval.py index c9ec0eab5a..753fa86872 100644 --- a/verifiers/v1/cli/dashboard/eval.py +++ b/verifiers/v1/cli/dashboard/eval.py @@ -12,6 +12,7 @@ from rich.table import Table from rich.text import Text +from verifiers.utils.pricing_utils import format_cost_usd from verifiers.v1.cli.dashboard.base import live_view from verifiers.v1.cli.output import output_path from verifiers.v1.configs.eval import EvalConfig @@ -24,7 +25,6 @@ format_override, format_time, ) -from verifiers.utils.pricing_utils import format_cost_usd if TYPE_CHECKING: from verifiers.v1.push import PushState @@ -229,10 +229,7 @@ def Progress( # global avg (errored count as 0) in parens. `err` is the share that errored. reward = format_mean(done, lambda t: t.reward) err = f"{sum(t.has_error for t in done) / len(done):.2f}" if done else "—" - stats = ( - f"{len(done)}/{total} · {format_time(time.time() - start)} · " - f"reward {reward} · err {err}" - ) + stats = f"{len(done)}/{total} · {format_time(time.time() - start)} · reward {reward} · err {err}" if page is not None: # overflowing — show which page, and that the arrows page stats += f" (page {page[0]}/{page[1]} ◄ ►)" row = Table.grid(expand=True, padding=(0, 1)) diff --git a/verifiers/v1/cli/eval/main.py b/verifiers/v1/cli/eval/main.py index 771a6ef1e0..24dc59d628 100644 --- a/verifiers/v1/cli/eval/main.py +++ b/verifiers/v1/cli/eval/main.py @@ -7,8 +7,8 @@ from pydantic_config import cli -import verifiers.v1 as vf -from verifiers.v1.utils.logging import setup_logging +from verifiers.v1.cli.eval.resume import load_resume_config, split_resume +from verifiers.v1.cli.eval.runner import run_eval from verifiers.v1.cli.output import output_path, write_config from verifiers.v1.cli.resolve import ( extract_id, @@ -16,14 +16,14 @@ references_config_file, with_positional_taskset, ) -from verifiers.v1.cli.eval.resume import load_resume_config, split_resume -from verifiers.v1.cli.eval.runner import run_eval from verifiers.v1.configs.eval import EvalConfig +from verifiers.v1.utils.logging import setup_logging logger = logging.getLogger(__name__) USAGE = ( "usage: uv run eval [] [--harness.id ] [--id (legacy)] [options] [@ file.toml]\n" + " uv run eval --topology.id [--topology.taskset.id ] [options] (multi-agent topology)\n" " uv run eval --resume (re-run a previous run's missing/errored rollouts)" ) @@ -50,12 +50,13 @@ def main(argv: list[str] | None = None) -> None: legacy_id = any(a == "--id" or a.startswith("--id=") for a in argv) # v0 env id if ( not extract_id(argv, "taskset") + and not extract_id(argv, "topology") and not legacy_id and not references_config_file(argv) ): raise SystemExit( USAGE - ) # need a taskset (positional / --taskset.id), a legacy --id, or a @ file.toml + ) # need a taskset (positional / --taskset.id), a --topology.id, a legacy --id, or a @ file.toml config_type = narrow_config(EvalConfig, argv) sys.argv = [sys.argv[0], *argv] # let prime-pydantic-config render help/errors @@ -88,17 +89,21 @@ def main(argv: list[str] | None = None) -> None: from verifiers.v1.legacy import run_legacy_eval traces = asyncio.run(run_legacy_eval(config)) + artifacts = traces elif config.server: # opt-in: drive rollouts through the env-server worker pool from verifiers.v1.cli.eval.runner import run_eval_server - traces = asyncio.run(run_eval_server(config)) - else: # in-process (default), with or without the live dashboard - env = vf.Environment(config) - traces = asyncio.run(run_eval(env, config)) + graphs = asyncio.run(run_eval_server(config)) + traces = [trace for graph in graphs for trace in graph.traces] + artifacts = graphs + else: # taskset syntax and explicit topologies share the in-process graph runner + graphs = asyncio.run(run_eval(config)) + traces = [trace for graph in graphs for trace in graph.traces] + artifacts = graphs if config.push and not rich: from verifiers.v1.push import push_traces push_traces(traces, config) - if not rich: # --rich is the whole output; otherwise dump each trace as JSON - for trace in traces: - print(trace.model_dump_json(indent=2, exclude_none=True)) + if not rich: # --rich is the whole output; otherwise dump each artifact as JSON + for artifact in artifacts: + print(artifact.model_dump_json(indent=2, exclude_none=True)) diff --git a/verifiers/v1/cli/eval/resume.py b/verifiers/v1/cli/eval/resume.py index 84708a2d3d..92ed5c158e 100644 --- a/verifiers/v1/cli/eval/resume.py +++ b/verifiers/v1/cli/eval/resume.py @@ -1,19 +1,9 @@ -"""Resume an interrupted eval: reload its finished rollouts and run only the rest. - -A run writes `config.toml` + `traces.jsonl` into its output dir. `--resume ` reloads -that config verbatim (so it takes no other flags) and writes back into the same dir. `load` -brings the good saved rollouts back into memory and re-runs only what's still owed: the -*missing* rollouts (never written — the run was interrupted) and the *errored* ones (written -with an error; dropped and redone). The loaded traces rejoin the run everywhere — counted, -displayed, pushed, and printed alongside this session's — so a resumed run picks up exactly -where the interrupted one stopped. A group-scored taskset is resumed a whole task at a time -(its rollouts are scored together), so any task that isn't fully complete is redone from -scratch. -""" +"""Resume an interrupted eval by rerunning missing or incomplete graph invocations.""" import json import tomllib from collections import defaultdict +from collections.abc import Callable from pathlib import Path from pydantic_core import from_json @@ -22,12 +12,11 @@ from verifiers.v1.configs.eval import EvalConfig from verifiers.v1.rollout import Phase, Rollout from verifiers.v1.task import Task -from verifiers.v1.trace import Trace, WireTrace +from verifiers.v1.topology import AgentGraph, graph_complete +from verifiers.v1.trace import Trace def split_resume(argv: list[str]) -> tuple[Path | None, list[str]]: - """Pull `--resume ` / `--resume=` out of argv, returning (dir, the other args). - The caller rejects any leftover args, since resume re-runs the saved config verbatim.""" for i, arg in enumerate(argv): if arg == "--resume": if i + 1 >= len(argv): @@ -41,8 +30,6 @@ def split_resume(argv: list[str]) -> tuple[Path | None, list[str]]: def load_resume_config(resume_dir: Path) -> EvalConfig: - """Rebuild the run's `EvalConfig` from its saved `config.toml`, pointed back at its own - output dir so the resumed rollouts append to the same `traces.jsonl`.""" config_path = resume_dir / CONFIG_FILE if not config_path.exists(): raise SystemExit( @@ -63,18 +50,20 @@ def __init__(self, trace: Trace) -> None: def load( - resume_dir: Path, selected_idxs: list[int], num_rollouts: int, group: bool -) -> tuple[list[Trace], dict[int, int]]: - """Load the good saved rollouts back into memory as finished traces and diff them against - the run's target (`num_rollouts` per selected task): returns (the kept traces, rollouts - owed per task idx). An errored trace is dropped and re-run; a group-scored task is kept - only if fully complete, else its whole group is redone. Rewrites `traces.jsonl` to just - the kept rows — verbatim, via a temp file + atomic rename, so an interrupted resume can't - corrupt the prior good results — and the resumed rollouts then append. `WireTrace` reads - any taskset's saved traces without importing it.""" + resume_dir: Path, + selected_idxs: list[int], + num_rollouts: int, + complete: Callable[[AgentGraph], bool] = graph_complete, +) -> tuple[list[AgentGraph], dict[int, int]]: + """Keep the graph records the topology considers complete and return invocations + still owed per seed task. `complete` is the topology's instance-validity verdict + (`Topology.complete`) — an incomplete record is dropped and its invocation re-run; + the default is the conservative rule, applied when no live topology is at hand + (server-mode eval). Kept lines are rewritten verbatim (temp file + atomic rename), + so an interrupted resume can't corrupt the prior good results.""" path = resume_dir / TRACES_FILE selected = set(selected_idxs) - good: dict[int, list[bytes]] = defaultdict(list) + good: dict[int, list[tuple[bytes, AgentGraph]]] = defaultdict(list) if path.exists(): with path.open("rb") as results: for line in results: @@ -85,32 +74,25 @@ def load( except ValueError: row = json.loads(line) idx = row["task"]["data"]["idx"] - if ( - idx in selected - and not row.get("errors") - and len(good[idx]) < num_rollouts - ): - good[idx].append(line if line.endswith(b"\n") else line + b"\n") - keep: list[bytes] = [] + if idx not in selected or len(good[idx]) >= num_rollouts: + continue + graph = AgentGraph.load(row) + if complete(graph): + good[idx].append( + (line if line.endswith(b"\n") else line + b"\n", graph) + ) + keep: list[tuple[bytes, AgentGraph]] = [] owed: dict[int, int] = {} for idx in selected_idxs: rows = good.get(idx, []) - if group and len(rows) < num_rollouts: - owed[idx] = num_rollouts # re-run the whole group; keep none of it - continue keep.extend(rows) if missing := num_rollouts - len(rows): owed[idx] = missing tmp = path.with_suffix(".jsonl.tmp") - tmp.write_bytes(b"".join(keep)) + tmp.write_bytes(b"".join(line for line, _ in keep)) tmp.replace(path) - return [WireTrace.model_validate_json(line) for line in keep], owed + return [graph for _, graph in keep], owed def nothing_to_resume_msg(resume_dir: Path, num_tasks: int, num_rollouts: int) -> str: - """The message shown (and then exit 0 - the run is already complete) when every selected - rollout already completed without error.""" - return ( - f"nothing to resume in {resume_dir}: all {num_tasks}x{num_rollouts} rollouts " - f"already completed without error" - ) + return f"nothing to resume in {resume_dir}: all {num_tasks}x{num_rollouts} invocations already complete" diff --git a/verifiers/v1/cli/eval/runner.py b/verifiers/v1/cli/eval/runner.py index d5a508ada1..20ec13ff9b 100644 --- a/verifiers/v1/cli/eval/runner.py +++ b/verifiers/v1/cli/eval/runner.py @@ -1,4 +1,4 @@ -"""The eval runner: fan rollouts out (one episode per task) with bounded concurrency.""" +"""Native v1 evaluation: each independent invocation produces one agent graph.""" import asyncio import contextlib @@ -6,56 +6,49 @@ import random import time +from verifiers.v1.cli.dashboard import dashboard +from verifiers.v1.cli.eval import resume +from verifiers.v1.cli.output import append_graph, output_path, save_config from verifiers.v1.clients import ModelContext, resolve_client from verifiers.v1.configs.eval import EvalConfig -from verifiers.v1.cli.eval import resume -from verifiers.v1.cli.dashboard import dashboard -from verifiers.v1.cli.output import append_trace, output_path, save_config -from verifiers.v1.decorators import discover_decorated -from verifiers.v1.env import Environment -from verifiers.v1.trace import Trace +from verifiers.v1.topology import AgentGraph, resolve_topology_runner logger = logging.getLogger(__name__) -_SHUFFLE_SEED = ( - 0 # fixed so `--shuffle` samples the same tasks every run (reproducible) -) +_SHUFFLE_SEED = 0 -async def run_eval(env: Environment, config: EvalConfig) -> list[Trace]: - logger.info("eval config:\n%s", config.model_dump_json(indent=2)) - client = resolve_client(config.client) - tasks = env.taskset.load() +def _selected(items: list, config: EvalConfig) -> list: if config.shuffle: - random.Random(_SHUFFLE_SEED).shuffle(tasks) - tasks = tasks if config.num_tasks is None else tasks[: config.num_tasks] - ctx = ModelContext(client=client, model=config.model, sampling=config.sampling) - # One episode of `num_rollouts` rollouts per task; the shared semaphore bounds total - # concurrent rollouts (across episodes), so group rewards still see their whole episode. - semaphore = ( - asyncio.Semaphore(config.max_concurrent) if config.max_concurrent else None - ) + random.Random(_SHUFFLE_SEED).shuffle(items) + return items if config.num_tasks is None else items[: config.num_tasks] + + +def graph_traces(graphs: list[AgentGraph]) -> list: + return [trace for graph in graphs for trace in graph.traces] + + +async def run_eval(config: EvalConfig) -> list[AgentGraph]: + """Run taskset syntax or an explicit topology through the same in-process runner.""" + logger.info("eval config:\n%s", config.model_dump_json(indent=2)) + runner = resolve_topology_runner(config) + tasks = _selected(list(runner.tasks), config) out = output_path(config) - # Write config.toml up front, then persist each trace as it completes (so the results are - # durable mid-run, not only at the end). One lock serializes worker-thread appends from - # concurrent rollouts while keeping large trace serialization off the event loop. - owed: dict[str, int] | None = None - # On resume, the kept (good) on-disk rollouts rejoin the run as finished traces: displayed, - # returned, pushed, and printed alongside this session's, with only the owed rollouts re-run - # — so the resumed run is indistinguishable from one that was never interrupted. - finished: list[Trace] = [] + finished: list[AgentGraph] = [] + owed: dict[int, int] | None = None if config.resume is not None: - # Resume incomplete group-reward tasks whole so every rollout is present. - group = bool(tasks) and bool(discover_decorated(tasks[0], "group_reward")) finished, owed = resume.load( - out, [t.data.idx for t in tasks], config.num_rollouts, group + out, + [task.data.idx for task in tasks], + config.num_rollouts, + complete=runner.topology.complete, ) - if not owed: # already complete - report it and exit successfully + if not owed: print(resume.nothing_to_resume_msg(out, len(tasks), config.num_rollouts)) raise SystemExit(0) tasks = [task for task in tasks if owed.get(task.data.idx)] logger.info( - "resuming %s: %d task(s), %d rollout(s) owed", + "resuming %s: %d task(s), %d invocation(s) owed", out, len(tasks), sum(owed.values()), @@ -63,96 +56,82 @@ async def run_eval(env: Environment, config: EvalConfig) -> list[Trace]: else: save_config(config, out) logger.info( - "running %dx%d rollouts on %s", + "running %dx%d topology invocation(s) on %s", len(tasks), config.num_rollouts, config.model, ) - start = time.time() logger.info("results: %s", out) - + client = resolve_client(config.client) + ctx = ModelContext(client=client, model=config.model, sampling=config.sampling) + semaphore = ( + asyncio.Semaphore(config.max_concurrent) if config.max_concurrent else None + ) + start = time.time() write_lock = asyncio.Lock() - - async def on_complete(trace: Trace) -> None: - await append_trace(out, trace, write_lock) - - # Shared tool servers (if any) come up once here and their URLs flow into every rollout - # (non-shared ones start per rollout inside the episodes); the interception pool comes up - # here too, so concurrent rollouts share its servers + tunnels rather than one each. Build - # episodes inside `serving` so each rollout is wired to those resources at construction. - async with env.serving(): - episodes = [ - env.episode( - task, ctx, n=owed[task.data.idx] if owed else config.num_rollouts - ) - for task in tasks - ] - rollouts = [resume.Finished(trace) for trace in finished] + [ - rollout for episode in episodes for rollout in episode.rollouts - ] - push_state = None - if config.push and config.rich: - from verifiers.v1.push import PushState - - push_state = PushState() - display = ( - dashboard(rollouts, config, start, push=push_state) - if config.rich - else contextlib.nullcontext() + rollouts = [resume.Finished(trace) for trace in graph_traces(finished)] + + async def run_instance(task) -> AgentGraph: + graph = await runner.run_instance( + task, + ctx, + semaphore, + on_rollout=rollouts.append if config.rich else None, ) - async with display: - results = await asyncio.gather( - *(episode.run(semaphore, on_complete) for episode in episodes) - ) - traces = finished + [ - trace for episode_traces in results for trace in episode_traces + await append_graph(out, graph, write_lock) + return graph + + push_state = None + if config.push and config.rich: + from verifiers.v1.push import PushState + + push_state = PushState() + display = ( + dashboard(rollouts, config, start, push=push_state) + if config.rich + else contextlib.nullcontext() + ) + try: + async with runner.serving(), display: + instances = [ + run_instance(task) + for task in tasks + for _ in range(owed[task.data.idx] if owed else config.num_rollouts) ] - if ( - push_state is not None - ): # upload off the event loop so the view keeps refreshing + completed = await asyncio.gather(*instances) + graphs = finished + completed + if push_state is not None: from verifiers.v1.push import push_traces push_state.started = True - await asyncio.to_thread(push_traces, traces, config, push_state) - await client.close() - return traces + await asyncio.to_thread( + push_traces, graph_traces(graphs), config, push_state + ) + return graphs + finally: + await client.close() -async def run_eval_server(config: EvalConfig) -> list[Trace]: - """Run evaluation through the env-server worker pool.""" +async def run_eval_server(config: EvalConfig) -> list[AgentGraph]: + """Run independent graph invocations through the existing env-server worker pool.""" import multiprocessing as mp from functools import partial - from verifiers.v1.utils.logging import setup_logging from verifiers.v1.env import pool_serve_kwargs from verifiers.v1.serve import EnvClient, env_config_data, serve_env + from verifiers.v1.utils.logging import setup_logging - legacy = config.is_legacy - server_kwargs = ( - { - "env_id": config.id, - "env_args": config.args, - "extra_env_kwargs": config.extra_env_kwargs, - } - if legacy - else {"config_data": env_config_data(config)} # picklable across the spawn - ) - # The pool broker + workers are spawned (fresh interpreters, no logging) — hand them - # the same loguru setup the main process uses (stderr + the run's log file) so their - # rollout logs come back and land in the output dir. + server_kwargs = {"config_data": env_config_data(config)} level = "DEBUG" if config.verbose else "INFO" log_file = str(output_path(config) / "eval.log") mpctx = mp.get_context("spawn") address_queue: mp.Queue = mpctx.Queue() - # Death pipe: serve_env (and, transitively, its workers/tunnels/sandboxes) self-terminates - # if this main process dies abruptly. We keep parent_conn; its close — even on our SIGKILL — - # signals death to the child's watch (see _arm_teardown). Mirrors the broker -> worker pipe. parent_conn, child_conn = mpctx.Pipe() proc = mpctx.Process( target=serve_env, kwargs=dict( **pool_serve_kwargs(config.pool), - legacy=legacy, + legacy=False, address="tcp://127.0.0.1:0", address_queue=address_queue, death_pipe=child_conn, @@ -162,90 +141,75 @@ async def run_eval_server(config: EvalConfig) -> list[Trace]: daemon=False, ) proc.start() - child_conn.close() # the child holds its end; we keep parent_conn so our exit closes it + child_conn.close() + client = None try: address = await asyncio.to_thread(address_queue.get, timeout=600) client = EnvClient(address=address) await client.wait_for_server_startup(timeout=600) info = await client.info() - group_scored = info.requires_group_scoring - idxs = list(range(info.num_tasks)) - if config.shuffle: - random.Random(_SHUFFLE_SEED).shuffle(idxs) - if config.num_tasks is not None: - idxs = idxs[: config.num_tasks] + positions = _selected(list(range(info.num_tasks)), config) + task_ids = [info.task_ids[position] for position in positions] out = output_path(config) - finished: list[Trace] = [] + finished: list[AgentGraph] = [] if config.resume is not None: - finished, owed = resume.load(out, idxs, config.num_rollouts, group_scored) - if not owed: # already complete - report it and exit successfully - print(resume.nothing_to_resume_msg(out, len(idxs), config.num_rollouts)) + finished, owed = resume.load(out, task_ids, config.num_rollouts) + if not owed: + print( + resume.nothing_to_resume_msg( + out, len(positions), config.num_rollouts + ) + ) raise SystemExit(0) - idxs = [idx for idx in idxs if owed.get(idx)] + selected = [ + (position, task_id) + for position, task_id in zip(positions, task_ids) + if owed.get(task_id) + ] logger.info( - "resuming %s: %d task(s), %d rollout(s) owed", + "resuming %s: %d task(s), %d invocation(s) owed", out, - len(idxs), + len(selected), sum(owed.values()), ) else: - owed = {idx: config.num_rollouts for idx in idxs} + selected = list(zip(positions, task_ids)) + owed = {task_id: config.num_rollouts for task_id in task_ids} save_config(config, out) logger.info( - "running %dx%d rollouts via the env-server %s pool on %s", - len(idxs), + "running %dx%d topology invocation(s) via the env-server %s pool on %s", + len(selected), config.num_rollouts, config.pool.type, config.model, ) logger.info("results: %s", out) - request_concurrency = config.max_concurrent - if request_concurrency and group_scored: - # max_concurrent is a rollout resource bound, not a request-throughput target. - # A group is indivisible, so one oversized group must still be allowed to run. - request_concurrency = max(1, request_concurrency // config.num_rollouts) semaphore = ( - asyncio.Semaphore(request_concurrency) if request_concurrency else None + asyncio.Semaphore(config.max_concurrent) if config.max_concurrent else None ) write_lock = asyncio.Lock() - async def run_group_unit(idx: int) -> list[Trace]: + async def run_unit(position: int) -> AgentGraph: async with semaphore or contextlib.nullcontext(): - traces = await client.run_group( - task_idx=idx, - n=config.num_rollouts, + graph = await client.run( + task_idx=position, client=config.client, model=config.model, sampling=config.sampling, ) - for trace in traces: - await append_trace(out, trace, write_lock) - return traces + await append_graph(out, graph, write_lock) + return graph - async def run_rollout_unit(idx: int) -> list[Trace]: - async with semaphore or contextlib.nullcontext(): - trace = await client.run_rollout( - task_idx=idx, - client=config.client, - model=config.model, - sampling=config.sampling, - ) - await append_trace(out, trace, write_lock) - return [trace] - - # A group-scored task must run its rollouts together (cross-rollout scoring) → - # one `run_group` request per task (one worker); otherwise rollouts are - # independent → one `run_rollout` request each, which the broker round-robins - # (least-busy) across workers — mirrors the prime-rl dispatcher. - units = ( - [run_group_unit(i) for i in idxs] - if group_scored - else [run_rollout_unit(i) for i in idxs for _ in range(owed[i])] - ) - results = await asyncio.gather(*units) - await client.close() - return finished + [trace for unit_traces in results for trace in unit_traces] + units = [ + run_unit(position) + for position, task_id in selected + for _ in range(owed[task_id]) + ] + completed = await asyncio.gather(*units) + return finished + completed finally: + if client is not None: + await client.close() proc.terminate() with contextlib.suppress(Exception): await asyncio.to_thread(proc.join, 10) diff --git a/verifiers/v1/cli/output.py b/verifiers/v1/cli/output.py index d991d33700..b20242a601 100644 --- a/verifiers/v1/cli/output.py +++ b/verifiers/v1/cli/output.py @@ -1,14 +1,13 @@ -"""On-disk output: traces.jsonl (one full trace per line) + config.toml. +"""On-disk output: traces.jsonl + config.toml. -The trace is the full data dump — written verbatim, consumed by the platform -(visualization) and prime-rl (training). config.toml is the run's resolved EvalConfig, +Native v1 writes one completed `AgentGraph` per line, including graphs produced by the +built-in single-agent topology. Legacy and replay utilities retain flat trace helpers. +`config.toml` is the run's resolved EvalConfig, written in the same format the CLI reads (`@ config.toml`), so a run is re-runnable from its own output. Aggregates (avg reward, etc.) are cheap to recompute from results, so they aren't stored. -The runner writes `config.toml` once up front (`save_config`) and then appends each -trace to `traces.jsonl` as it completes (`append_trace`), so a long run's results are -durable as they land rather than only at the end. +The runner writes `config.toml` once and appends each graph as soon as it completes. """ import asyncio @@ -19,22 +18,28 @@ from pydantic import BaseModel, TypeAdapter from verifiers.v1.configs.eval import EvalConfig +from verifiers.v1.topology import AgentGraph from verifiers.v1.trace import Trace from verifiers.v1.utils.aio import run_shielded TRACES_FILE = "traces.jsonl" -"""Filename each run's full per-rollout traces are written to (one JSON trace per line).""" +"""Native graph records, or flat traces for isolated legacy/replay commands.""" CONFIG_FILE = "config.toml" """Filename a run's resolved config is written to (re-runnable via `@ config.toml`).""" def output_path(config: EvalConfig) -> Path: - """Where this run writes: `outputs/----/` (or the explicit - `--output-dir`). The per-run `uuid` leaf means runs never overwrite each other.""" + """Where this run writes: `outputs/----/` — a topology + run uses `outputs/--/` (its agents bind their own harnesses) — + or the explicit `--output-dir`. The per-run `uuid` leaf means runs never overwrite + each other.""" if config.output_dir is not None: return config.output_dir - name = f"{config.taskset.name}--{config.model.replace('/', '--')}--{config.harness.name}" + if config.topology is not None: + name = f"{config.topology.name}--{config.model.replace('/', '--')}" + else: + name = f"{config.taskset.name}--{config.model.replace('/', '--')}--{config.harness.name}" return Path("outputs") / name / config.uuid @@ -43,7 +48,14 @@ def write_config(config: BaseModel, results_dir: Path) -> Path: path. mode="json" makes values TOML-friendly (Path -> str, etc.); exclude_none drops the nulls TOML can't represent.""" results_dir.mkdir(parents=True, exist_ok=True) - toml = tomli_w.dumps(config.model_dump(mode="json", exclude_none=True)) + data = config.model_dump(mode="json", exclude_none=True) + if getattr(config, "topology", None) is not None: + # A topology run never consults the taskset × harness pair; persisting their + # (default) sections would make the saved config claim knobs the run ignored — + # and trip the harness-with-topology guard on an `eval @ config.toml` re-run. + data.pop("taskset", None) + data.pop("harness", None) + toml = tomli_w.dumps(data) config_path = results_dir / CONFIG_FILE config_path.write_text(toml) return config_path @@ -64,19 +76,39 @@ def write_trace(results_dir: Path, trace: Trace) -> None: f.write(data + b"\n") +def write_graph(results_dir: Path, graph: AgentGraph) -> None: + """Serialize and append one invocation record in the worker thread.""" + with (results_dir / TRACES_FILE).open("ab") as f: + f.write(json.dumps(graph.to_record()).encode() + b"\n") + + def read_traces(results_dir: Path, trace_type: type) -> list[Trace]: - """Load a run's saved traces from `traces.jsonl`, typed as `trace_type` — the inverse of - `write_trace`. Used by `replay` to re-score finished rollouts (pass the task's typed - `Trace[...]`, or `Trace[WireTaskData, ...]` to read any taskset's traces without importing it).""" + """Load typed traces from native graph records or isolated flat-trace outputs.""" adapter = TypeAdapter(trace_type) traces: list[Trace] = [] with (results_dir / TRACES_FILE).open(encoding="utf-8") as f: for line in f: if line.strip(): - traces.append(adapter.validate_python(json.loads(line))) + record = json.loads(line) + rows = record.get("traces", [record]) + traces.extend(adapter.validate_python(row) for row in rows) return traces +async def append_graph( + results_dir: Path, graph: AgentGraph, lock: asyncio.Lock +) -> None: + """Append one fully scored invocation without blocking the event loop.""" + + async def persist() -> None: + async with lock: + await asyncio.to_thread(write_graph, results_dir, graph) + + # Run lock acquisition and the worker to completion even under cancellation, so + # finalized graphs are never lost mid-write (`run_shielded` re-raises the cancellation). + await run_shielded(persist()) + + async def append_trace(results_dir: Path, trace: Trace, lock: asyncio.Lock) -> None: """Append one finished trace without blocking the event loop. The run's shared lock preserves whole-line ordering, and awaiting the worker preserves per-trace durability.""" diff --git a/verifiers/v1/cli/replay.py b/verifiers/v1/cli/replay.py index 01ab66aa56..c9d62adaa2 100644 --- a/verifiers/v1/cli/replay.py +++ b/verifiers/v1/cli/replay.py @@ -2,8 +2,8 @@ Replay clears each trace's scores and recomputes everything computable from the saved transcript — trace-only handlers plus the layered config's judges. Runtime-requiring -signals (and group rewards, which need the whole group) don't run offline, so a replay -carries offline scores only; the source run keeps the runtime-recorded values. Its saved +signals don't run offline, so a replay carries offline scores only; the source run keeps +the runtime-recorded values. Its saved config is the base for replay-specific overrides. """ diff --git a/verifiers/v1/cli/resolve.py b/verifiers/v1/cli/resolve.py index 60662433f0..5a9e2011a0 100644 --- a/verifiers/v1/cli/resolve.py +++ b/verifiers/v1/cli/resolve.py @@ -44,23 +44,28 @@ def extract_id(argv: list[str], field: str, default: str = "") -> str: def narrow_config(base: type, argv: list[str]) -> type: - """`base` (an `EnvConfig` subclass) with `taskset`/`harness` narrowed to the config types - of the ids given on the CLI — so the single `cli()` parse stays typed and `-h` renders - them. A field whose id isn't on the CLI is left as the base type for `EnvConfig` to resolve - from a `@ file.toml` (never pre-narrowed to a type the config could then contradict). - Absent a config file, the harness falls back to the taskset's bundled harness (if it ships - one) else `default`, so bare `-h` renders the harness that will actually run.""" + """`base` (an `EnvConfig` subclass) with `taskset`/`harness`/`topology` narrowed to the + config types of the ids given on the CLI — so the single `cli()` parse stays typed and + `-h` renders them. A field whose id isn't on the CLI is left as the base type for the + config to resolve from a `@ file.toml` (never pre-narrowed to a type the config could + then contradict). Absent a config file, the harness falls back to the taskset's bundled + harness (if it ships one) else `default`, so bare `-h` renders the harness that will + actually run — unless a topology is chosen, whose agents bind their own harnesses.""" taskset_id = extract_id(argv, "taskset") harness_id = extract_id(argv, "harness") - if not harness_id and not references_config_file(argv): + topology_id = extract_id(argv, "topology") + if not harness_id and not topology_id and not references_config_file(argv): harness_id = vf.default_harness_id(taskset_id) annotations: dict[str, type] = {} fields: dict[str, object] = {} for field, resolve, ident in ( ("taskset", vf.taskset_config_type, taskset_id), ("harness", vf.harness_config_type, harness_id), + ("topology", vf.topology_config_type, topology_id), ): - if ident: + # Only narrow fields the base config declares; narrowing must never invent a field + # the config would then silently swallow. + if ident and field in base.model_fields: ftype = resolve(ident) annotations[field] = ftype fields[field] = ftype(id=ident) diff --git a/verifiers/v1/cli/serve.py b/verifiers/v1/cli/serve.py index 94038bc7b4..58478df9a8 100644 --- a/verifiers/v1/cli/serve.py +++ b/verifiers/v1/cli/serve.py @@ -5,7 +5,6 @@ from pydantic_config import cli -from verifiers.v1.utils.logging import setup_logging from verifiers.v1.cli.resolve import ( extract_id, narrow_config, @@ -15,8 +14,9 @@ from verifiers.v1.configs.serve import ServeConfig from verifiers.v1.env import pool_serve_kwargs from verifiers.v1.serve import serve_env +from verifiers.v1.utils.logging import setup_logging -USAGE = "usage: uv run serve [] [--harness.id ] [--id (legacy)] [options] [@ file.toml]" +USAGE = "usage: uv run serve [] [--harness.id ] | --topology.id | --id (legacy) [options] [@ file.toml]" def main(argv: list[str] | None = None) -> None: @@ -30,6 +30,7 @@ def main(argv: list[str] | None = None) -> None: legacy_id = any(a == "--id" or a.startswith("--id=") for a in argv) # v0 env id if ( not extract_id(argv, "taskset") + and not extract_id(argv, "topology") and not legacy_id and not references_config_file(argv) ): diff --git a/verifiers/v1/configs/eval.py b/verifiers/v1/configs/eval.py index 5bdd8a4859..a1c4f82dfa 100644 --- a/verifiers/v1/configs/eval.py +++ b/verifiers/v1/configs/eval.py @@ -32,13 +32,13 @@ class EvalConfig(EnvServerConfig): "group_size", "rollouts_per_example", "num_rollouts", "r" ), ) - """Rollouts per task. A task with `@group_reward`s requires at least two.""" + """Independent topology invocations per seed task.""" shuffle: bool = Field(False, validation_alias=AliasChoices("shuffle", "s")) """Shuffle tasks before taking the first `num_tasks`.""" max_concurrent: int | None = Field( 128, validation_alias=AliasChoices("max_concurrent", "c") ) - """Max rollouts in flight at once.""" + """Max agent runs in flight in-process, or graph requests through the server.""" verbose: bool = Field(False, validation_alias=AliasChoices("verbose", "v")) """Log at debug level instead of the default info.""" dry_run: bool = False @@ -58,10 +58,9 @@ class EvalConfig(EnvServerConfig): """Where to write the run (config.toml + traces.jsonl). None = a fresh per-run dir under `outputs/----/` (so runs never overwrite each other).""" resume: Path | None = Field(None, exclude=True) - """Set by `--resume `: re-run missing or errored rollouts, or an incomplete - group-scored task as a whole group, appending to that run's own results. The run's saved - config is loaded verbatim, so `--resume` takes no other arguments. Excluded from the - saved config.""" + """Set by `--resume `: re-run missing or errored graph invocations, appending + to that run's own results. The saved config is loaded verbatim, so `--resume` takes + no other arguments. Excluded from the saved config.""" @model_validator(mode="after") def reject_rich_with_server(self): diff --git a/verifiers/v1/decorators.py b/verifiers/v1/decorators.py index 75797d4ecf..44dc3d106f 100644 --- a/verifiers/v1/decorators.py +++ b/verifiers/v1/decorators.py @@ -69,38 +69,37 @@ def stop(func: F | None = None, priority: int = 0) -> F | Callable[[F], F]: @overload -def metric(func: F, priority: int = 0) -> F: ... +def metric(func: F, priority: int = 0, agent: str | None = None) -> F: ... @overload -def metric(func: None = None, priority: int = 0) -> Callable[[F], F]: ... -def metric(func: F | None = None, priority: int = 0) -> F | Callable[[F], F]: - """Mark a metric `(self, trace) -> float` (recorded, not summed).""" - decorator = mark("metric", metric_priority=priority) - return decorator if func is None else decorator(func) - - -@overload -def reward(func: F, weight: float = 1.0, priority: int = 0) -> F: ... -@overload -def reward( - func: None = None, weight: float = 1.0, priority: int = 0 +def metric( + func: None = None, priority: int = 0, agent: str | None = None ) -> Callable[[F], F]: ... -def reward( - func: F | None = None, weight: float = 1.0, priority: int = 0 +def metric( + func: F | None = None, priority: int = 0, agent: str | None = None ) -> F | Callable[[F], F]: - """Mark a weighted per-rollout reward returning a float or keyed scores.""" - decorator = mark("reward", reward_priority=priority, _vf_weight=weight) + """Mark a metric `(self, trace) -> float` (recorded, not summed). On a `Topology`, + `agent=` scopes it to the named agent's traces (see `Topology.score`).""" + decorator = mark("metric", metric_priority=priority, _vf_agent=agent) return decorator if func is None else decorator(func) @overload -def group_reward(func: F, weight: float = 1.0, priority: int = 0) -> F: ... +def reward( + func: F, weight: float = 1.0, priority: int = 0, agent: str | None = None +) -> F: ... @overload -def group_reward( - func: None = None, weight: float = 1.0, priority: int = 0 +def reward( + func: None = None, weight: float = 1.0, priority: int = 0, agent: str | None = None ) -> Callable[[F], F]: ... -def group_reward( - func: F | None = None, weight: float = 1.0, priority: int = 0 +def reward( + func: F | None = None, + weight: float = 1.0, + priority: int = 0, + agent: str | None = None, ) -> F | Callable[[F], F]: - """Mark a weighted group reward returning one score per trace.""" - decorator = mark("group_reward", group_reward_priority=priority, _vf_weight=weight) + """Mark a weighted per-rollout reward returning a float or keyed scores. On a + `Topology`, `agent=` scopes it to the named agent's traces (see `Topology.score`).""" + decorator = mark( + "reward", reward_priority=priority, _vf_weight=weight, _vf_agent=agent + ) return decorator if func is None else decorator(func) diff --git a/verifiers/v1/dialects/chat.py b/verifiers/v1/dialects/chat.py index c0771538c3..a02339255e 100644 --- a/verifiers/v1/dialects/chat.py +++ b/verifiers/v1/dialects/chat.py @@ -317,6 +317,12 @@ def extend( # A None completion seeds the opening turn (no model message yet) — only the user turn(s). messages = [*body.get("messages", [])] if completion is not None: - messages.append(completion["choices"][0]["message"]) + message = dict(completion["choices"][0]["message"]) + if message.get("content") is None and not message.get("tool_calls"): + # A reasoning-only turn (null content, no tool calls) is valid *output* but + # rejected as re-sent *input* ("content is required unless tool_calls") — + # coerce to the empty string so a simulated conversation can continue. + message["content"] = "" + messages.append(message) messages.extend(message_to_wire(m) for m in user_messages) return {**body, "messages": messages} diff --git a/verifiers/v1/env.py b/verifiers/v1/env.py index 184e1f75fa..8d1c5295c4 100644 --- a/verifiers/v1/env.py +++ b/verifiers/v1/env.py @@ -1,20 +1,13 @@ -"""Compose a taskset and harness into runnable episodes.""" +"""Native environment configuration and execution-policy helpers.""" -import contextlib import logging -from typing import Annotated, Literal +from typing import TYPE_CHECKING, Annotated, Literal from pydantic import Field, SerializeAsAny, model_validator from pydantic_config import BaseConfig from verifiers.v1.harness import Harness, HarnessConfig -from verifiers.v1.clients import ModelContext -from verifiers.v1.decorators import discover_decorated -from verifiers.v1.episode import Episode -from verifiers.v1.types import ID -from verifiers.v1.interception import InterceptionPool, RolloutLimits from verifiers.v1.retries import RetryConfig -from verifiers.v1.rollout import Rollout from verifiers.v1.runtimes import ( RuntimeConfig, SubprocessConfig, @@ -22,8 +15,11 @@ ) from verifiers.v1.task import Task from verifiers.v1.taskset import Taskset, TasksetConfig +from verifiers.v1.types import ID from verifiers.v1.utils.generic import generic_type -from verifiers.v1.mcp import SharedToolServer, serve_shared + +if TYPE_CHECKING: + from verifiers.v1.topology import TopologyConfig class TimeoutConfig(BaseConfig): @@ -56,7 +52,7 @@ class ElasticPoolConfig(BaseConfig): max_workers: int | None = None """Upper bound on workers (None = unbounded).""" multiplex: int = Field(128, ge=1) - """Rollouts per worker for the scale-up trigger: add a worker once in-flight rollouts + """Graph requests per worker for the scale-up trigger: add a worker once in-flight requests reach 90% of `workers * multiplex`.""" @@ -78,13 +74,20 @@ def pool_serve_kwargs(pool: StaticPoolConfig | ElasticPoolConfig) -> dict: class EnvConfig(BaseConfig): - """The taskset that loads tasks and the harness that runs them.""" + """What to run: a taskset × harness pair (lowered into the built-in single-agent + topology), or an explicit multi-agent `topology`.""" # SerializeAsAny: these hold resolved subclasses (e.g. MathConfig, DefaultHarnessConfig); # without it model_dump() narrows to the base type and drops the subclass fields, so the # env-server subconfig the orchestrator writes would lose taskset/harness-specific knobs. taskset: SerializeAsAny[TasksetConfig] = TasksetConfig() harness: SerializeAsAny[HarnessConfig] = HarnessConfig(id="default") + topology: "SerializeAsAny[TopologyConfig] | None" = None + """A multi-agent topology to run *instead of* the single `taskset` × `harness` pair + (see `verifiers.v1.topology`), selected by `--topology.id`. Seeds come from its + `taskset` slot (`--topology.taskset.id `); each agent binds its own + harness/routing (`--topology..harness.id`, ...). Forward-referenced: env.py is + below topology.py, which finalizes this model (`model_rebuild`) when it loads.""" timeout: TimeoutConfig = TimeoutConfig() retries: RetryConfig = RetryConfig() max_turns: int | None = None @@ -131,15 +134,27 @@ def env_id(self) -> str: @model_validator(mode="before") @classmethod def _resolve_plugins(cls, data): - """Resolve the generic `taskset` / `harness` to its specific config type by `id`, so - env-specific fields validate against the real plugin config (no untyped args dict).""" + """Resolve the generic `topology` / `taskset` / `harness` to its specific config + type by `id`, so env-specific fields validate against the real plugin config (no + untyped args dict). Topology first, before the default harness is manufactured + below — so a *user-supplied* `harness` alongside a topology is still + distinguishable, and refused (it would be silently ignored; agents bind their + own harnesses).""" from verifiers.v1.loaders import ( default_harness_id, harness_config_type, narrow_plugin_field, taskset_config_type, + topology_config_type, ) + if isinstance(data, dict) and data.get("topology"): + if data.get("harness"): + raise ValueError( + "`--harness.*` is ignored under a topology — each agent binds its " + "own harness (`--topology..harness.*`); drop the flag." + ) + narrow_plugin_field(data, "topology", topology_config_type) narrow_plugin_field(data, "taskset", taskset_config_type) taskset = data.get("taskset") taskset_id = ( @@ -154,6 +169,17 @@ def _resolve_plugins(cls, data): ) return data + @model_validator(mode="after") + def _check_topology(self): + """A topology replaces the config's own taskset × harness pair — reject + combinations that would silently ignore half the config.""" + if self.topology is not None and (self.taskset.id or self.id): + raise ValueError( + "`--topology.id` runs the topology's own agents; drop `--taskset.id` / " + "`--id` (choose the seed tasks via --topology.taskset.id)." + ) + return self + class EnvServerConfig(EnvConfig): """An env plus how it's *served*: the env definition (`EnvConfig`) and the worker-pool @@ -175,7 +201,7 @@ def resolve_runtime_config( an image must run in a container — refuse subprocess), and apply its `workdir` and requested `resources` to the fields the runtime supports. Precedence is cli/toml > task > default; a resource the runtime doesn't support warns once (deduped via `warned`). Shared - by `Environment.runtime_for` (rollouts) and the `validate` entrypoint.""" + by topology agents and the `validate` entrypoint.""" config = base updates: dict = {} if task.data.image is not None: @@ -211,15 +237,53 @@ def resolve_runtime_config( return config.model_copy(update=updates) if updates else config -def validate_pairing(harness: Harness, taskset: Taskset) -> None: - """Reject an impossible harness/taskset pairing at construction — before any dataset - load, shared-server launch, or first episode. Every check reads class-level facts - (`Task.tools` / `Task.user` / `NEEDS_CONTAINER`, `Taskset.tools` — one task type per - taskset, read off the `Taskset[TaskT, ...]` generic), so a failure here holds for - every row the taskset can produce; on the env server it fails worker startup instead - of every request.""" - task_cls = generic_type(type(taskset), Task, origin=Taskset) or Task - if not harness.SUPPORTS_MCP and (task_cls.tools or type(taskset).tools): +def resolve_stage_timeouts( + timeout: TimeoutConfig, task: Task, runtime_config: RuntimeConfig +) -> TimeoutConfig: + """Resolve a task's per-stage wall-clock budgets into a concrete `TimeoutConfig`: + cli/toml > task > default (None = no limit), with remote sandbox lifetimes capping + the harness stage at 24 hours.""" + harness = ( + timeout.rollout if timeout.rollout is not None else task.data.timeout.harness + ) + if ( + harness is not None + and harness > 24 * 60 * 60 + and not runtime_is_local(runtime_config) + ): + logger.warning( + "task %r resolves to a %.1f-hour harness timeout, but %s sandboxes have a " + "maximum lifetime of 24 hours; capping it at 24 hours", + task.data.idx, + harness / (60 * 60), + runtime_config.type, + ) + harness = 24 * 60 * 60 + return TimeoutConfig( + setup=timeout.setup if timeout.setup is not None else task.data.timeout.setup, + rollout=harness, + finalize=timeout.finalize + if timeout.finalize is not None + else task.data.timeout.finalize, + scoring=timeout.scoring + if timeout.scoring is not None + else task.data.timeout.scoring, + ) + + +def validate_task_pairing( + harness: Harness, + task_cls: type[Task], + runtime_config: RuntimeConfig, + shared_tools: tuple = (), +) -> None: + """Reject an impossible harness/task-class/runtime combination. Every check reads + class-level facts (`Task.tools` / `Task.user` / `NEEDS_CONTAINER`, plus any + taskset-scoped `shared_tools`), so a failure holds for every instance of the class. + `runtime_config` is where the run actually lands — the harness's policy at + construction (`validate_pairing`), the *resolved* config per run for topology agents + (a borrowed box may differ from the harness's default, in either direction).""" + if not harness.SUPPORTS_MCP and (task_cls.tools or shared_tools): raise ValueError( f"Harness {harness.config.id!r} does not support MCP tools, but " f"{task_cls.__name__} exposes tool servers (MCP). Run it with a harness that " @@ -231,158 +295,19 @@ def validate_pairing(harness: Harness, taskset: Taskset) -> None: f"{task_cls.__name__} defines one (Task.user). Run it with a harness that " f"supports user simulation (e.g. --harness.id default), or use tasks without one." ) - if task_cls.NEEDS_CONTAINER and isinstance( - harness.config.runtime, SubprocessConfig - ): + if task_cls.NEEDS_CONTAINER and isinstance(runtime_config, SubprocessConfig): raise ValueError( - f"{task_cls.__name__} needs a container runtime (NEEDS_CONTAINER), but the " - "harness runs on the subprocess runtime; use --harness.runtime.type docker or prime." + f"{task_cls.__name__} needs a container runtime (NEEDS_CONTAINER), but this " + "run resolves to the subprocess runtime; use --harness.runtime.type docker or prime." ) -class Environment: - def __init__(self, config: EnvConfig) -> None: - from verifiers.v1.loaders import load_harness, load_taskset - - self.config = config - self.taskset = load_taskset(config.taskset) - self.harness = load_harness(config.harness) - validate_pairing(self.harness, self.taskset) - # The warning is about the *agent* running arbitrary code on the host: every harness hands - # it local execution (bash/edit, or a CLI agent) except the tool-less `null` chat loop, - # whose program only relays the model and remote MCP tools — so exempt `null`, warn for the - # rest. (`null` still runs its fixed chat-loop program locally, but nothing agent-authored.) - if self.harness.config.id != "null" and isinstance( - self.harness.config.runtime, SubprocessConfig - ): - logger.warning( - "Harness %r is running in the subprocess runtime on the local system. " - "Local files and settings may affect the evaluation; use subprocess only " - "for debugging. Use --harness.runtime.type docker or prime for an isolated " - "run.", - self.harness.config.id, - ) - self.setup_timeout = config.timeout.setup - self.harness_timeout = config.timeout.rollout - self.finalize_timeout = config.timeout.finalize - self.scoring_timeout = config.timeout.scoring - self.limits = RolloutLimits( - max_turns=config.max_turns, - max_input_tokens=config.max_input_tokens, - max_output_tokens=config.max_output_tokens, - max_total_tokens=config.max_total_tokens, - ) - self._warned_resources: set[tuple[str, str]] = set() - self._shared_tools: dict[str, SharedToolServer] = {} - self._interception: InterceptionPool | None = None - """Eval-level serving resources, live only inside `serving()`: shared tool servers - ({name: SharedToolServer}) and the interception pool. `episode()` injects them into every rollout - so neither runner has to thread them through `Episode.run`/`Rollout.run`.""" - - def runtime_for(self, task: Task) -> RuntimeConfig: - """Resolve the runtime config for a task off the harness's runtime (see - `resolve_runtime_config`).""" - return resolve_runtime_config( - self.harness.config.runtime, task, self._warned_resources - ) - - def episode(self, task: Task, ctx: ModelContext, n: int = 1) -> Episode: - """Resolve `task` into a runnable episode of `n` rollouts: pick its runtime - (image + resources) and its - timeouts (cli/toml > task > default, None = no limit), build one `Rollout` per - sample sharing them, and wrap them in an `Episode` (which runs them and applies - the task's `@group_reward`s across their traces). - - A task with `@group_reward`s compares its rollouts, so it needs >=2 of - them — refuse `n < 2` there (rather than silently scoring a group of one). - Harness capability (tools / user sim / container) is class-level and already - checked at construction (`validate_pairing`).""" - if n < 2 and discover_decorated(task, "group_reward"): - raise ValueError( - f"task {task.data.idx!r} defines @group_reward(s), which compare a task's rollouts " - f"and need >=2; got n={n} (pass -r/--num-rollouts >= 2)" - ) - runtime_config = self.runtime_for(task) - setup_timeout = ( - self.setup_timeout - if self.setup_timeout is not None - else task.data.timeout.setup - ) - harness_timeout = ( - self.harness_timeout - if self.harness_timeout is not None - else task.data.timeout.harness - ) - if ( - harness_timeout is not None - and harness_timeout > 24 * 60 * 60 - and not runtime_is_local(runtime_config) - ): - logger.warning( - "task %r resolves to a %.1f-hour harness timeout, but %s sandboxes have a " - "maximum lifetime of 24 hours; capping it at 24 hours", - task.data.idx, - harness_timeout / (60 * 60), - runtime_config.type, - ) - harness_timeout = 24 * 60 * 60 - finalize_timeout = ( - self.finalize_timeout - if self.finalize_timeout is not None - else task.data.timeout.finalize - ) - scoring_timeout = ( - self.scoring_timeout - if self.scoring_timeout is not None - else task.data.timeout.scoring - ) - retries = self.config.retries - rollouts = [ - Rollout( - task=task, - harness=self.harness, - ctx=ctx, - runtime_config=runtime_config, - setup_timeout=setup_timeout, - harness_timeout=harness_timeout, - finalize_timeout=finalize_timeout, - scoring_timeout=scoring_timeout, - limits=self.limits, - shared_tools=self._shared_tools, - interception=self._interception, - ) - for _ in range(n) - ] - return Episode(rollouts, retry=retries.rollout) - - @contextlib.asynccontextmanager - async def serving(self): - """Hold the env-level serving resources for the duration of an eval: the shared tool - servers (built once, see `shared_tools`) and the interception pool. Stash them so - every `episode()` built inside this context injects them into its rollouts — that's - what keeps both eval runners (in-process and env-server) on one serving path. Build - episodes inside this context; the resources are torn down on exit.""" - async with ( - self.shared_tools() as shared, - self.interception_pool() as interception, - ): - self._shared_tools = shared - self._interception = interception - try: - yield - finally: - self._shared_tools = {} - self._interception = None - - def interception_pool(self) -> InterceptionPool: - return InterceptionPool(self.harness.config.runtime, self.config.multiplex) - - @contextlib.asynccontextmanager - async def shared_tools(self): - servers = self.taskset.tool_servers() - if not servers: - yield {} - return - harness_is_local = runtime_is_local(self.harness.config.runtime) - async with serve_shared(servers, harness_is_local=harness_is_local) as shared: - yield shared +def validate_pairing(harness: Harness, taskset: Taskset) -> None: + """Reject an impossible harness/taskset pairing at construction — before any dataset + load, shared-server launch, or first episode (see `validate_task_pairing`; one task + type per taskset, read off the `Taskset[TaskT, ...]` generic). On the env server this + fails worker startup instead of every request.""" + task_cls = generic_type(type(taskset), Task, origin=Taskset) or Task + validate_task_pairing( + harness, task_cls, harness.config.runtime, shared_tools=type(taskset).tools + ) diff --git a/verifiers/v1/episode.py b/verifiers/v1/episode.py deleted file mode 100644 index 9eb2d41fbd..0000000000 --- a/verifiers/v1/episode.py +++ /dev/null @@ -1,55 +0,0 @@ -"""Run and group-score all rollouts for one task.""" - -from __future__ import annotations - -import asyncio -from collections.abc import Awaitable, Callable -from contextlib import nullcontext -from typing import TYPE_CHECKING - -from verifiers.v1.decorators import discover_decorated -from verifiers.v1.retries import run_with_retry -from verifiers.v1.rollout import Phase, Rollout -from verifiers.v1.trace import Trace -from verifiers.v1.utils.memory import trim_memory_periodically - -if TYPE_CHECKING: - from verifiers.v1.retries import RolloutRetryConfig - - -class Episode: - def __init__(self, rollouts: list[Rollout], retry: RolloutRetryConfig) -> None: - if not rollouts: - raise ValueError("an episode needs at least one rollout (n >= 1)") - self.rollouts = rollouts - self.task = rollouts[0].task - self.retry = retry - - async def run( - self, - semaphore: asyncio.Semaphore | None = None, - on_complete: Callable[[Trace], Awaitable[None]] | None = None, - ) -> list[Trace]: - """Run rollouts; delay completion callbacks only when group scoring needs all of them.""" - group_scored = bool(discover_decorated(self.task, "group_reward")) - - async def run_one(rollout: Rollout) -> Trace: - async with semaphore or nullcontext(): - trace = await run_with_retry(rollout, self.retry) - if not group_scored: # reward already final → don't wait for the group - rollout.phase = Phase.DONE - if on_complete is not None: - await on_complete(trace) - # hand freed per-turn request bodies (base64 images) back to the OS - await trim_memory_periodically() - return trace - - traces = await asyncio.gather(*(run_one(r) for r in self.rollouts)) - if group_scored: - await self.task.score_group(traces) # cross-rollout @group_rewards - for rollout in self.rollouts: - rollout.phase = Phase.DONE - for trace in traces: - if on_complete is not None: - await on_complete(trace) - return traces diff --git a/verifiers/v1/errors.py b/verifiers/v1/errors.py index d84fa4798a..b3331ab7fd 100644 --- a/verifiers/v1/errors.py +++ b/verifiers/v1/errors.py @@ -64,7 +64,12 @@ class SandboxError(RolloutError): class TaskError(RolloutError): - """Task-authored code raised — `setup`, `finalize`, or a `@reward`/`@metric`/`@group_reward`.""" + """Task-authored code raised — `setup`, `finalize`, or a `@reward`/`@metric`.""" + + +class TopologyError(RolloutError): + """Topology-authored code raised — `go` or the topology's declared instance scoring. + Captured on the instance's `AgentGraph` (episode failures stay on their traces).""" class InterceptionError(RolloutError): diff --git a/verifiers/v1/harnesses/direct/__init__.py b/verifiers/v1/harnesses/direct/__init__.py new file mode 100644 index 0000000000..a2c862b5db --- /dev/null +++ b/verifiers/v1/harnesses/direct/__init__.py @@ -0,0 +1,7 @@ +"""direct — v1's built-in in-process harness: a chat loop with no subprocess and no tools. +The cheapest episode (the judge in the `llm-judge` topology). Resolved by id via +`load_harness`.""" + +from verifiers.v1.harnesses.direct.harness import DirectHarness, DirectHarnessConfig + +__all__ = ["DirectHarness", "DirectHarnessConfig"] diff --git a/verifiers/v1/harnesses/direct/harness.py b/verifiers/v1/harnesses/direct/harness.py new file mode 100644 index 0000000000..5012fb275f --- /dev/null +++ b/verifiers/v1/harnesses/direct/harness.py @@ -0,0 +1,90 @@ +"""The built-in direct harness: an in-process chat loop — no subprocess, no uv script, no tools. + +The cheapest possible episode: the chat loop runs inside the eval process itself, POSTing to +its interception endpoint like any program would — so the trace, stops, limits, and per-agent +routing all work unmodified — but with nothing to provision or launch. Cost per episode is +essentially the model call itself, which makes an agent-as-judge (the `llm-judge` +topology's judge) as cheap as a plain judge call while still producing a real, inspectable +trace. + +Tool-less by design (`null` is the subprocess chat loop with MCP tools); a model that calls a +tool anyway gets an error tool-result and one chance to answer directly.""" + +from openai import AsyncOpenAI + +from verifiers.v1.clients import ModelContext +from verifiers.v1.dialects.chat import message_to_wire +from verifiers.v1.harness import Harness, HarnessConfig +from verifiers.v1.runtimes import ProgramResult, Runtime +from verifiers.v1.trace import Trace + + +class DirectHarnessConfig(HarnessConfig): + """The built-in direct harness: an in-process chat loop. Nothing runs in the runtime, so + it works on any runtime with zero setup — but the *episode's* code (tool servers, scoring + scripts) still uses the runtime normally.""" + + +class DirectHarness(Harness[DirectHarnessConfig]): + APPENDS_SYSTEM_PROMPT = True + SUPPORTS_MCP = False + SUPPORTS_USER_SIM = True + SUPPORTS_MESSAGE_PROMPT = True + + async def launch( + self, + ctx: ModelContext, + trace: Trace, + runtime: Runtime, + endpoint: str, + secret: str, + mcp_urls: dict[str, str], + ) -> ProgramResult: + system_prompt, prompt = self.resolve_prompt(trace.task.data) + messages: list[dict] = ( + [{"role": "system", "content": system_prompt}] if system_prompt else [] + ) + if isinstance(prompt, str): + messages.append({"role": "user", "content": prompt}) + elif prompt is not None: + messages.extend(message_to_wire(m) for m in prompt) + client = AsyncOpenAI(base_url=endpoint, api_key=secret) + bounced = False + try: + while True: + try: + completion = await client.chat.completions.create( + model=ctx.model, messages=messages + ) + except Exception: + # A refused turn (a @stop fired, a budget hit) surfaces as an HTTP error; + # unlike a subprocess program we can see the live trace, so treat a stopped + # rollout as the clean exit it is. A real failure re-raises (classified by + # `Harness.run`, with `RolloutSession.error` carrying the true cause). + if trace.stop_condition is not None: + break + raise + message = completion.choices[0].message + messages.append(message.model_dump(exclude_none=True)) + if not message.tool_calls: + break + if ( + bounced + ): # exactly one chance to answer directly — never an open loop + break + bounced = True + # Tool-less by design: bounce the hallucinated call back with an error result. + messages.extend( + { + "role": "tool", + "tool_call_id": call.id, + "content": "error: no tools are available; answer directly", + } + for call in message.tool_calls + ) + finally: + await client.close() + return ProgramResult(exit_code=0, stdout="", stderr="") + + +__all__ = ["DirectHarness", "DirectHarnessConfig"] diff --git a/verifiers/v1/harnesses/null/program.py b/verifiers/v1/harnesses/null/program.py index 39c66755b6..3b9026afc9 100644 --- a/verifiers/v1/harnesses/null/program.py +++ b/verifiers/v1/harnesses/null/program.py @@ -97,8 +97,14 @@ async def main() -> None: client = AsyncOpenAI(base_url=args.base_url, api_key=args.api_key) config = json.loads(args.mcp_config or "{}") async with AsyncExitStack() as stack: + # Bounded: the streamable-HTTP handshake can wedge (an intermittent mcp-SDK + # race), and an unbounded initialize would silently eat the whole rollout + # budget as a 0-turn harness_timeout. Failing loudly makes it a classified, + # retryable program error instead. tools, dispatch = ( - await connect_mcp(stack, config) if config.get("mcpServers") else ([], {}) + await asyncio.wait_for(connect_mcp(stack, config), timeout=60) + if config.get("mcpServers") + else ([], {}) ) messages = ( [{"role": "system", "content": args.system_prompt}] diff --git a/verifiers/v1/interception/server.py b/verifiers/v1/interception/server.py index 45a2c15c0b..ce7f75f04c 100644 --- a/verifiers/v1/interception/server.py +++ b/verifiers/v1/interception/server.py @@ -149,6 +149,10 @@ async def refused(self) -> str | None: call halts the harness (its model call errors out); Harness.run treats it as clean. A task that ends a trajectory from `trace.state` does it with its own `@stop` (run here generically), so the interception server holds no opinion about the state's contents.""" + if self.trace.stop_condition is not None: + # A stopped trace takes no more turns — however it was stopped (a prior limit + # or @stop, or an out-of-band end like `Session.end`). + return self.trace.stop_condition if (limit := self.limits.reached(self.trace)) is not None: self.trace.stop(limit) logger.debug("limit %r reached: id=%s", limit, self.trace.id) diff --git a/verifiers/v1/legacy.py b/verifiers/v1/legacy.py index cd991ef81c..c7b992af27 100644 --- a/verifiers/v1/legacy.py +++ b/verifiers/v1/legacy.py @@ -21,16 +21,18 @@ import zmq import zmq.asyncio +from verifiers.v1 import graph from verifiers.v1.clients.config import ClientConfig, TrainClientConfig from verifiers.v1.serve.server import EnvServer from verifiers.v1.serve.types import ( + BaseResponse, + LegacyInfoResponse, RunGroupRequest, RunGroupResponse, RunRolloutRequest, RunRolloutResponse, ) from verifiers.v1.task import WireTaskData -from verifiers.v1 import graph from verifiers.v1.trace import Error, TimeSpan, Timing, Trace, TraceTask from verifiers.v1.types import ( AssistantMessage, @@ -401,6 +403,19 @@ async def _run_group(self, req: RunGroupRequest) -> RunGroupResponse: ] return RunGroupResponse(traces=traces) + async def _dispatch(self, route: str, raw: dict) -> BaseResponse: + if route == "info": + return LegacyInfoResponse( + num_tasks=len(self.tasks), + task_ids=list(range(len(self.tasks))), + requires_group_scoring=self.requires_group_scoring, + ) + if route == "run_rollout": + return await self._run_rollout(RunRolloutRequest.model_validate(raw)) + if route == "run_group": + return await self._run_group(RunGroupRequest.model_validate(raw)) + return await super()._dispatch(route, raw) + # --- in-process v0 eval (the `eval` CLI's `--id` path) ------------------------- @@ -439,7 +454,6 @@ async def run_legacy_eval(config) -> list[Trace]: import random from verifiers import load_environment - from verifiers.v1.cli.output import append_trace, save_config from verifiers.v1.utils.install import ensure_installed diff --git a/verifiers/v1/loaders.py b/verifiers/v1/loaders.py index 3e1e04bf08..081bd3f4e8 100644 --- a/verifiers/v1/loaders.py +++ b/verifiers/v1/loaders.py @@ -3,7 +3,7 @@ import importlib import importlib.util from types import ModuleType -from typing import Callable +from typing import TYPE_CHECKING, Callable from pydantic_config import BaseConfig @@ -14,6 +14,9 @@ from verifiers.v1.task import Task from verifiers.v1.taskset import Taskset, TasksetConfig +if TYPE_CHECKING: + from verifiers.v1.topology import Topology, TopologyConfig + def narrow_plugin_field( data: dict, @@ -140,6 +143,32 @@ def judge_config_type(judge_id: str) -> type[JudgeConfig]: return judge_config_cls(judge_class(judge_id)) +def import_topology(topology_id: str) -> ModuleType: + return _import_plugin(topology_id, "topology", "verifiers.v1.topologies") + + +def topology_class(topology_id: str) -> "type[Topology]": + """The topology's `Topology` subclass, exported via its module's `__all__`.""" + from verifiers.v1.topology import Topology + + return _plugin_class(import_topology(topology_id), Topology, "topology") + + +def load_topology(config: "TopologyConfig") -> "Topology": + """Build the topology for a config by dispatching on its `id` (the topology id).""" + return topology_class(config.id)(config) + + +def topology_config_type(topology_id: str) -> "type[TopologyConfig]": + """Resolve the topology's config specialization through its MRO.""" + from verifiers.v1.topology import Topology, TopologyConfig + + return ( + generic_type(topology_class(topology_id), TopologyConfig, origin=Topology) + or TopologyConfig + ) + + def task_type(taskset_id: str) -> type[Task]: """The taskset's `Task` subclass from its `Taskset[TaskT, ConfigT]` generic — no data is loaded, so replay can cheaply recover the task data type. Falls back to diff --git a/verifiers/v1/mcp/launch.py b/verifiers/v1/mcp/launch.py index 15b32418ed..45a9810f4a 100644 --- a/verifiers/v1/mcp/launch.py +++ b/verifiers/v1/mcp/launch.py @@ -285,7 +285,7 @@ async def serve_shared(toolsets: list[Toolset], harness_is_local: bool = True): """Start the taskset-scoped (shared) tool servers ONCE for a whole eval, each in its OWN `runtime`, and yield `{name: SharedToolServer}` reachable by every rollout's harness. Reachability mirrors a per-rollout tool, but there's no single harness runtime to read - locality off — the caller (`Environment.shared_tools`) passes the harness runtime's + locality off — the caller (`TopologyRunner.serving`) passes the harness runtime's `harness_is_local`, so a host tool gets one host bridge (tunnel) when the harness runs remotely, and a remote tool runtime publishes its own URL. Torn down when the eval ends. A shared server is task-agnostic — the taskset carries no per-row data — so its `setup` @@ -298,8 +298,7 @@ async def serve_shared(toolsets: list[Toolset], harness_is_local: bool = True): name = toolset.server_name if name in servers: raise ToolsetError( - f"duplicate shared tool server name '{name}' in Taskset.tools — " - f"give one a distinct TOOL_PREFIX" + f"duplicate shared tool server name '{name}' in Taskset.tools — give one a distinct TOOL_PREFIX" ) if type(toolset).setup_task is not ServerBase.setup_task: logger.warning( diff --git a/verifiers/v1/push.py b/verifiers/v1/push.py index 886c4ac67f..25233255c6 100644 --- a/verifiers/v1/push.py +++ b/verifiers/v1/push.py @@ -146,7 +146,11 @@ def compute_metrics() -> dict[str, Any]: "avg_error": sum(t.has_error for t in traces) / n if n else 0.0, } - env_name = config.taskset.id or config.id + env_name = ( + config.topology.id + if config.topology is not None + else config.taskset.id or config.id + ) metrics = compute_metrics() counts: dict[int, int] = {} samples = [] diff --git a/verifiers/v1/retries.py b/verifiers/v1/retries.py index dd717c734d..5da34a7a2d 100644 --- a/verifiers/v1/retries.py +++ b/verifiers/v1/retries.py @@ -12,7 +12,7 @@ from __future__ import annotations import logging -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Protocol from pydantic import Field from pydantic_config import BaseConfig @@ -27,12 +27,18 @@ ) if TYPE_CHECKING: - from verifiers.v1.rollout import Rollout from verifiers.v1.trace import Trace logger = logging.getLogger(__name__) +class Runnable(Protocol): + """Anything whole-run retryable: one `run()` producing a complete `Trace` per attempt — + a `Rollout`, or an agent attempt wrapping one (`verifiers.v1.agent`).""" + + async def run(self) -> Trace: ... + + def retrying( *, on: type[BaseException] | tuple[type[BaseException], ...] = Exception, @@ -104,13 +110,14 @@ def should_retry(trace: Trace, retry: RolloutRetryConfig) -> bool: async def run_with_retry( - rollout: Rollout, + rollout: Runnable, retry: RolloutRetryConfig, ) -> Trace: """Run the whole rollout, retrying it while its trace ends with a retryable error. - Each retry-causing attempt's error is collected onto the returned trace's `errors`, - so the final trace shows the full history; the last attempt's trace is returned - as-is once attempts run out (or the rollout succeeds / hits a non-retryable error).""" + When the final attempt also fails, the earlier attempts' errors are prepended to the + returned trace's `errors` (the full failure history); when it succeeds, they are + parked in `trace.info["retry_errors"]` instead — the trace must not read as errored + (`has_error`) downstream just because an earlier attempt was retried away.""" if retry.max_retries < 1: return await rollout.run() @@ -138,4 +145,8 @@ def record(state: RetryCallState) -> None: trace = await retrying(rollout.run) if trace.errors: # final attempt errored too → prepend the earlier attempts' trace.errors = history + trace.errors + elif ( + history + ): # success after retries → keep the history without tripping `has_error` + trace.info["retry_errors"] = [error.model_dump() for error in history] return trace diff --git a/verifiers/v1/rollout.py b/verifiers/v1/rollout.py index 94fe82f3de..549c24b056 100644 --- a/verifiers/v1/rollout.py +++ b/verifiers/v1/rollout.py @@ -3,8 +3,8 @@ import time from contextlib import asynccontextmanager from enum import StrEnum +from typing import TYPE_CHECKING -from verifiers.v1.harness import Harness from verifiers.v1.clients import ModelContext from verifiers.v1.decorators import discover_decorated, invoke from verifiers.v1.errors import ( @@ -14,12 +14,14 @@ ToolsetError, boundary, ) +from verifiers.v1.harness import Harness from verifiers.v1.interception import ( InterceptionPool, InterceptionServer, RolloutLimits, RolloutSession, ) +from verifiers.v1.mcp import SharedToolServer, serve_tools, serve_user from verifiers.v1.runtimes import ( HOST, Runtime, @@ -27,10 +29,12 @@ make_runtime, reachable_url, ) -from verifiers.v1.mcp import SharedToolServer, serve_tools, serve_user + +if TYPE_CHECKING: + from verifiers.v1.mcp import Respond from verifiers.v1.state import state_cls from verifiers.v1.task import Task -from verifiers.v1.trace import TraceTask, Trace +from verifiers.v1.trace import Trace, TraceTask logger = logging.getLogger(__name__) @@ -58,6 +62,8 @@ def __init__( limits: RolloutLimits | None = None, shared_tools: dict[str, SharedToolServer] | None = None, interception: InterceptionPool | None = None, + runtime: Runtime | None = None, + user: "Respond | None" = None, ) -> None: self.task = task self.harness = harness @@ -70,6 +76,15 @@ def __init__( self.limits = limits or RolloutLimits() self.shared_tools = shared_tools or {} self.interception = interception + self._borrowed_runtime = runtime + """A live runtime to run in instead of provisioning one (see `Agent.provision`). + The borrower gets its own trace/session/secrets, but the runtime owner keeps + start/stop ownership.""" + self._user = user + """A programmatic user seat (see `verifiers.v1.agent.Session`): an in-process + `Respond` wired straight into the interception session instead of a user-sim + server built from `Task.user`. Exactly one party may hold the user seat — the + caller (`Agent.interact`) refuses tasks that declare their own.""" self.phase = Phase.PENDING self.runtime: Runtime | None = None self.trace: Trace | None = None @@ -110,7 +125,19 @@ async def run(self) -> Trace: self.trace = trace # expose for the --rich dashboard self.phase = Phase.SETUP # leaving the queue: provisioning starts now trace.timing.setup.start = time.time() - self.runtime = make_runtime(self.runtime_config, name=trace.id) + if self._borrowed_runtime is not None and self._borrowed_runtime.stopped: + # A lifetime bug in the borrowing program, not a property of this rollout's + # world: raise to the caller instead of capturing onto the trace. + raise ValueError( + f"borrowed runtime {self._borrowed_runtime.name!r} was already torn " + "down by its owner; keep the provisioning context open for every run " + "placed into the box" + ) + self.runtime = ( + self._borrowed_runtime + if self._borrowed_runtime is not None + else make_runtime(self.runtime_config, name=trace.id) + ) runtime = self.runtime trace.runtime = runtime.info ctx = self.ctx @@ -124,7 +151,8 @@ async def run(self) -> Trace: ) try: session = RolloutSession(ctx, trace, stops, self.limits) - await runtime.start() + if self._borrowed_runtime is None: + await runtime.start() # Task setup and harness provisioning share one setup-stage deadline. setup_deadline = ( None @@ -161,13 +189,16 @@ async def run(self) -> Trace: state_base=state_base, ) as urls, serve_user( - self.task.user_server(), + None if self._user is not None else self.task.user_server(), harness_runtime=runtime, state_port=state_port, state_secret=secret, state_base=state_base, - ) as session.user, + ) as launched_user, ): + session.user = ( + self._user if self._user is not None else launched_user + ) if self.task.data.prompt is None and session.user is None: raise TaskError( "task has no prompt and no user simulator to open the " @@ -210,7 +241,6 @@ async def run(self) -> Trace: self.phase = Phase.SCORING trace.timing.scoring.start = now async with boundary(TaskError, "scoring"): - # Group rewards run later, after the runtime is gone. await asyncio.wait_for( asyncio.gather( self.task.score(trace, runtime), @@ -219,10 +249,19 @@ async def run(self) -> Trace: self.scoring_timeout, ) trace.timing.scoring.end = time.time() - except RolloutError as e: - trace.capture_error(e) except Exception as e: - logger.exception("unexpected error in rollout %s", trace.id) + if self._borrowed_runtime is not None and runtime.stopped: + # The owner tore the borrowed box down mid-run — the same lifetime bug + # as borrowing a stopped runtime, surfaced through the same channel: + # raise to the caller (raw failure chained) instead of capturing a + # misattributed world error onto the trace. + raise ValueError( + f"borrowed runtime {runtime.name!r} was torn down by its owner " + "mid-run; keep the provisioning context open until every run " + "placed into the box has completed" + ) from e + if not isinstance(e, RolloutError): + logger.exception("unexpected error in rollout %s", trace.id) trace.capture_error(e) finally: trace.is_completed = True @@ -234,12 +273,14 @@ async def run(self) -> Trace: ): if span.start and not span.end: span.end = now - try: - await runtime.stop() - except Exception: - logger.warning( - "runtime teardown failed (rollout %s)", trace.id, exc_info=True - ) + if self._borrowed_runtime is None: # a borrowed box is its owner's to stop + try: + await runtime.stop() + except Exception: + logger.warning( + "runtime teardown failed (rollout %s)", trace.id, exc_info=True + ) + self.phase = Phase.DONE logger.info( "rollout done: id=%s task=%s reward=%.3f turns=%d stop=%s", trace.id, diff --git a/verifiers/v1/runtimes/base.py b/verifiers/v1/runtimes/base.py index d27f5f6dfa..73065a83e4 100644 --- a/verifiers/v1/runtimes/base.py +++ b/verifiers/v1/runtimes/base.py @@ -120,6 +120,9 @@ def __init__(self, name: str | None = None) -> None: self.name = name or f"vf-{uuid.uuid4().hex[:12]}" self._uv_interpreters: dict[str, str] = {} self._uv_script_locks: dict[str, asyncio.Lock] = {} + self.stopped = False + """Whether teardown has begun. Borrowed-runtime rollouts refuse a stopped runtime + because its owner has already ended the lifetime they were borrowing.""" @property def type(self) -> str: @@ -139,6 +142,7 @@ async def stop(self) -> None: and an interrupted teardown leaks the container / paid sandbox. Runs `teardown` to completion, then re-raises the cancellation. Framework method — override `teardown`, not this.""" + self.stopped = True await run_shielded(self.teardown()) async def teardown(self) -> None: @@ -200,8 +204,7 @@ async def prepare_uv_script( result = await self.run(["sh", "-c", command], env or {}) if result.exit_code != 0: raise RuntimeError( - "failed to prepare uv script: " - f"{result.stderr.strip()[-2000:]}" + f"failed to prepare uv script: {result.stderr.strip()[-2000:]}" ) self._uv_interpreters[digest] = result.stdout.strip().splitlines()[ -1 diff --git a/verifiers/v1/serve/__init__.py b/verifiers/v1/serve/__init__.py index 01481ed088..8c0e82fa80 100644 --- a/verifiers/v1/serve/__init__.py +++ b/verifiers/v1/serve/__init__.py @@ -8,8 +8,11 @@ HealthResponse, InfoRequest, InfoResponse, + LegacyInfoResponse, RunGroupRequest, RunGroupResponse, + RunRequest, + RunResponse, RunRolloutRequest, RunRolloutResponse, ) @@ -24,6 +27,9 @@ "HealthResponse", "InfoRequest", "InfoResponse", + "LegacyInfoResponse", + "RunRequest", + "RunResponse", "RunRolloutRequest", "RunRolloutResponse", "RunGroupRequest", diff --git a/verifiers/v1/serve/client.py b/verifiers/v1/serve/client.py index db85c38cf0..24da467ac8 100644 --- a/verifiers/v1/serve/client.py +++ b/verifiers/v1/serve/client.py @@ -2,10 +2,9 @@ A DEALER socket + msgpack, with a single receive loop matching responses to per-request futures by `request_id`. Speaks the typed pydantic request/response -models (`serve/types.py`) end-to-end: a request is `model_dump`ed onto the wire -and the reply is `model_validate`d back — `Trace`s come back typed as -`Trace[WireTaskData]` (non-strict task, so env fields survive without importing the -env). Health is just another request (no dedicated probe thread). +models (`serve/types.py`) end-to-end. Native replies load as `AgentGraph`s with wire-typed +tasks and traces, preserving environment-specific fields without importing their packages. +Health is just another request. """ import asyncio @@ -29,10 +28,13 @@ InfoResponse, RunGroupRequest, RunGroupResponse, + RunRequest, + RunResponse, RunRolloutRequest, RunRolloutResponse, ) from verifiers.v1.task import WireTaskData +from verifiers.v1.topology import AgentGraph from verifiers.v1.trace import Trace from verifiers.v1.types import SamplingConfig @@ -75,7 +77,7 @@ async def _request( timeout: float | None = None, ) -> ResponseT: """Send a typed request and validate the reply into `response_type`. A - `timeout` is only used for health polling — rollouts run untimed.""" + `timeout` is only used for health polling — graph requests run untimed.""" self._ensure_receiver() request_id = uuid.uuid4().hex future: asyncio.Future[bytes] = asyncio.get_running_loop().create_future() @@ -136,13 +138,26 @@ async def wait_for_server_startup( ) async def info(self) -> InfoResponse: - """Return the taskset `num_tasks` + whether its tasks group-score.""" + """Return the server's ordered seed-task ids.""" return await self._request(InfoRequest(), InfoResponse) + async def run( + self, task_idx: int, client: ClientConfig, model: str, sampling: SamplingConfig + ) -> AgentGraph: + """Run one independent native v1 invocation and return its agent graph.""" + response = await self._request( + RunRequest( + task_idx=task_idx, client=client, model=model, sampling=sampling + ), + RunResponse, + ) + assert response.graph is not None + return response.graph + async def run_rollout( self, task_idx: int, client: ClientConfig, model: str, sampling: SamplingConfig ) -> Trace[WireTaskData]: - """Run one rollout for `task_idx`; return a typed `Trace[WireTaskData]`.""" + """Run one legacy v0 rollout.""" response = await self._request( RunRolloutRequest( task_idx=task_idx, client=client, model=model, sampling=sampling @@ -159,7 +174,7 @@ async def run_group( model: str, sampling: SamplingConfig, ) -> list[Trace[WireTaskData]]: - """Run `n` rollouts for `task_idx` as a scored group; return typed `Trace[WireTaskData]`s.""" + """Run one legacy v0 rollout group.""" response = await self._request( RunGroupRequest( task_idx=task_idx, n=n, client=client, model=model, sampling=sampling diff --git a/verifiers/v1/serve/pool.py b/verifiers/v1/serve/pool.py index 82fec3693c..7283015ae8 100644 --- a/verifiers/v1/serve/pool.py +++ b/verifiers/v1/serve/pool.py @@ -1,6 +1,6 @@ """Env-server worker pool: a ROUTER broker over N worker processes. -A lone `EnvServer` runs every rollout as an `asyncio.Task` on one event loop, so +A lone `EnvServer` runs every graph request as an `asyncio.Task` on one event loop, so CPU-bound work (renderer tokenization, scoring) competes for that loop. v0 relieved this with a router + worker pool; this reinstates it for v1. @@ -37,7 +37,7 @@ from verifiers.v1.env import EnvConfig from verifiers.v1.serve.server import EnvServer -from verifiers.v1.serve.types import HealthResponse, RunGroupRequest +from verifiers.v1.serve.types import HealthResponse logger = logging.getLogger(__name__) @@ -166,7 +166,7 @@ def _spawn_worker(self) -> None: self._poller.register(dealer, zmq.POLLIN) def _maybe_scale_up(self, in_flight: int) -> None: - """Spawn one more worker when in-flight rollout slots reach 90% of capacity. + """Spawn one more worker when in-flight requests reach 90% of capacity. A new worker starts at `active=0`, so least-busy dispatch funnels the backlog to it as it comes online (a few seconds to load the env) — fine, since we only scale @@ -193,7 +193,7 @@ async def run(self) -> None: # (`max_workers` is a concrete count when elastic is off). for _ in range(1 if self.elastic else (self.max_workers or 1)): self._spawn_worker() - # request_id -> {client_id, worker, rollout_slots} + # request_id -> {client_id, worker} pending: dict[bytes, dict] = {} logger.info( "EnvServerPool up: address=%s workers=%d/%s multiplex=%d elastic=%s", @@ -219,22 +219,13 @@ async def run(self) -> None: [client_id, request_id, _HEALTH] ) else: - # Pool capacity is measured in rollouts; one group request carries n. - rollout_slots = 1 - if method == b"run_group": - with contextlib.suppress(Exception): - request = RunGroupRequest.model_validate( - msgpack.unpackb(payload, raw=False) - ) - rollout_slots = max(1, request.n) worker = min(self.workers, key=lambda w: w["active"]) - worker["active"] += rollout_slots + worker["active"] += 1 pending[request_id] = { "client_id": client_id, "worker": worker, - "rollout_slots": rollout_slots, } - in_flight += rollout_slots + in_flight += 1 # forward without client_id — the DEALER identity is the worker's # `client_id`; we hold the real one in `pending`. await worker["dealer"].send_multipart( @@ -249,8 +240,8 @@ async def run(self) -> None: entry = pending.pop(request_id.bytes, None) if entry is None: continue - entry["worker"]["active"] -= entry["rollout_slots"] - in_flight -= entry["rollout_slots"] + entry["worker"]["active"] -= 1 + in_flight -= 1 with contextlib.suppress(zmq.ZMQError): await self.frontend.send_multipart( [entry["client_id"], request_id, data], copy=False @@ -284,8 +275,15 @@ def _shutdown(self) -> None: def env_config_data(config) -> dict: """The picklable `EnvConfig` fields of a (possibly dynamically-narrowed, unpicklable) config object — ship this across a process boundary, then rebuild via - `EnvConfig.model_validate` (its validator re-resolves the concrete taskset/harness).""" + `EnvConfig.model_validate` (its validator re-resolves the concrete + topology/taskset/harness).""" data = config.model_dump(mode="json") + if data.get("topology") is not None: + # A topology run never consults the taskset × harness pair (agents bind their + # own harnesses); shipping the manufactured defaults would trip the + # harness-with-topology guard on worker-side revalidation. Mirrors write_config. + data.pop("taskset", None) + data.pop("harness", None) return {k: v for k, v in data.items() if k in EnvConfig.model_fields} diff --git a/verifiers/v1/serve/server.py b/verifiers/v1/serve/server.py index 62abcf444d..2cf3da74da 100644 --- a/verifiers/v1/serve/server.py +++ b/verifiers/v1/serve/server.py @@ -11,17 +11,15 @@ from verifiers.v1.clients import ModelContext, resolve_client from verifiers.v1.clients.client import Client from verifiers.v1.clients.config import ClientConfig -from verifiers.v1.decorators import discover_decorated -from verifiers.v1.env import EnvConfig, Environment +from verifiers.v1.env import EnvConfig from verifiers.v1.serve.types import ( BaseResponse, HealthResponse, InfoResponse, - RunGroupRequest, - RunGroupResponse, - RunRolloutRequest, - RunRolloutResponse, + RunRequest, + RunResponse, ) +from verifiers.v1.topology import resolve_topology_runner from verifiers.v1.types import SamplingConfig logger = logging.getLogger(__name__) @@ -32,14 +30,8 @@ def __init__( self, config: EnvConfig, address: str = "tcp://127.0.0.1:5000" ) -> None: self.address = address - self.taskset_id = config.taskset.id - self.env = Environment(config) - self.tasks = self.env.taskset.load() - # One task type per taskset (the authoring contract; its `load()` constructs it), - # so group scoring is a run-wide property. - self.requires_group_scoring = bool(self.tasks) and bool( - discover_decorated(self.tasks[0], "group_reward") - ) + self.runner = resolve_topology_runner(config) + self.tasks = self.runner.tasks self._clients: dict[ tuple[str, str], Client ] = {} # (client_config, model) -> Client @@ -88,25 +80,25 @@ def _context( ) def serving(self): - """Context for the server's eval-level serving resources (shared tool servers + - interception pool), entered for the server's lifetime so they're reused across - requests; episodes built inside it inherit them (see `Environment.serving`). The - legacy v0 bridge overrides this (it runs its own rollouts, with no v1 serving).""" - return self.env.serving() + """Hold topology services for this worker's full request-serving lifetime.""" + return self.runner.serving() - async def _run_rollout(self, req: RunRolloutRequest) -> RunRolloutResponse: + async def _run(self, req: RunRequest) -> RunResponse: ctx = self._context(req.client, req.model, req.sampling) - episode = self.env.episode(self.tasks[req.task_idx], ctx, n=1) - traces = await episode.run() - # Trust the concrete trace; serialize it once before client-side re-typing. - return RunRolloutResponse.model_construct(trace=traces[0]) + graph = await self.runner.run_instance(self.tasks[req.task_idx], ctx) + return RunResponse.model_construct(graph=graph) - async def _run_group(self, req: RunGroupRequest) -> RunGroupResponse: - ctx = self._context(req.client, req.model, req.sampling) - episode = self.env.episode(self.tasks[req.task_idx], ctx, n=req.n) - traces = await episode.run() - # Avoid a dump-and-validate copy for every trusted trace in the group. - return RunGroupResponse.model_construct(traces=traces) + async def _dispatch(self, route: str, raw: dict) -> BaseResponse: + if route == "health": + return HealthResponse() + if route == "info": + return InfoResponse( + num_tasks=len(self.tasks), + task_ids=[task.data.idx for task in self.tasks], + ) + if route == "run": + return await self._run(RunRequest.model_validate(raw)) + return BaseResponse(success=False, error=f"unknown method {route!r}") async def _handle( self, client_id: bytes, request_id: bytes, method: bytes, payload: bytes @@ -114,23 +106,7 @@ async def _handle( try: route = method.decode() raw = msgpack.unpackb(payload, raw=False) - if route == "health": - response: BaseResponse = HealthResponse() - elif route == "info": - response = InfoResponse( - num_tasks=len(self.tasks), - requires_group_scoring=self.requires_group_scoring, - ) - elif route == "run_rollout": - response = await self._run_rollout( - RunRolloutRequest.model_validate(raw) - ) - elif route == "run_group": - response = await self._run_group(RunGroupRequest.model_validate(raw)) - else: - response = BaseResponse( - success=False, error=f"unknown method {route!r}" - ) + response = await self._dispatch(route, raw) except ( Exception ) as e: # a failed request is data, not a crash — report and keep serving @@ -151,11 +127,9 @@ async def _handle( async def run(self) -> None: logger.info( - "EnvServer up: taskset=%s address=%s tasks=%d group_scoring=%s", - self.taskset_id, + "EnvServer up: address=%s tasks=%d", self.address, len(self.tasks), - self.requires_group_scoring, ) poller = zmq.asyncio.Poller() poller.register(self.frontend, zmq.POLLIN) @@ -189,4 +163,4 @@ async def run(self) -> None: await client.close() self.frontend.close() self.ctx.term() - logger.info("EnvServer down: taskset=%s", self.taskset_id) + logger.info("EnvServer down") diff --git a/verifiers/v1/serve/types.py b/verifiers/v1/serve/types.py index 438f0f3c19..a6fca8d900 100644 --- a/verifiers/v1/serve/types.py +++ b/verifiers/v1/serve/types.py @@ -1,16 +1,17 @@ +"""Typed native v1 env-server protocol: one request produces one agent graph.""" + from typing import ClassVar -from pydantic import BaseModel, field_serializer +from pydantic import BaseModel, Field, field_serializer, field_validator from verifiers.v1.clients.config import ClientConfig from verifiers.v1.task import WireTaskData +from verifiers.v1.topology import AgentGraph from verifiers.v1.trace import Trace from verifiers.v1.types import SamplingConfig class BaseRequest(BaseModel): - """`method` is sent as its own route frame, not as payload data.""" - method: ClassVar[str] @@ -33,41 +34,47 @@ class InfoRequest(BaseRequest): class InfoResponse(BaseResponse): num_tasks: int = 0 + task_ids: list[int] = Field(default_factory=list) + + +class LegacyInfoResponse(InfoResponse): requires_group_scoring: bool = False - """Whether tasks must be run and resumed as whole groups.""" -class RunRolloutRequest(BaseRequest): - method: ClassVar[str] = "run_rollout" +class RunRequest(BaseRequest): + method: ClassVar[str] = "run" task_idx: int client: ClientConfig model: str sampling: SamplingConfig +class RunResponse(BaseResponse): + graph: AgentGraph | None = None + + @field_validator("graph", mode="before") + @classmethod + def _load_graph(cls, graph): + return AgentGraph.load(graph) if isinstance(graph, dict) else graph + + @field_serializer("graph") + def _serialize_graph(self, graph: AgentGraph | None) -> dict | None: + return graph.model_dump(mode="python") if graph is not None else None + + +# Legacy v0 bridge protocol. Native v1 never dispatches these routes. +class RunRolloutRequest(RunRequest): + method: ClassVar[str] = "run_rollout" + + class RunRolloutResponse(BaseResponse): trace: Trace[WireTaskData] | None = None - """A trace whose task-specific data is preserved in `model_extra`.""" - @field_serializer("trace") - def _ser_trace(self, trace: "Trace[WireTaskData] | None") -> dict | None: - return trace.model_dump() if trace is not None else None - -class RunGroupRequest(BaseRequest): +class RunGroupRequest(RunRequest): method: ClassVar[str] = "run_group" - task_idx: int n: int - client: ClientConfig - model: str - sampling: SamplingConfig class RunGroupResponse(BaseResponse): traces: list[Trace[WireTaskData]] | None = None - - @field_serializer("traces") - def _ser_traces( - self, traces: "list[Trace[WireTaskData]] | None" - ) -> list[dict] | None: - return [t.model_dump() for t in traces] if traces is not None else None diff --git a/verifiers/v1/services.py b/verifiers/v1/services.py new file mode 100644 index 0000000000..7e2b2c6f0e --- /dev/null +++ b/verifiers/v1/services.py @@ -0,0 +1,62 @@ +"""Run-scoped serving resources shared by rollouts. + +`RunServices` owns the interception pools — too expensive to create per rollout, but +not process-global: an in-process eval enters one for the command; a topology enters one +while its instances run. Shared MCP servers are NOT here: they are taskset-scoped +(`Taskset.tools`, served once by `TopologyRunner.serving` via `serve_shared`) and flow +into rollouts as a plain +`{name: SharedToolServer}` dict. +""" + +import asyncio +import contextlib + +from verifiers.v1.interception import InterceptionPool +from verifiers.v1.runtimes import RuntimeConfig, runtime_is_local + + +class RunServices: + """The active serving scope: interception pools, keyed by the runtime placement a + rollout actually uses. Interception only cares about reachability (local vs remote) + and the runtime family for display, so a task-specific image/resource override does + not fragment the pool.""" + + def __init__(self, multiplex: int = 32) -> None: + self.multiplex = multiplex + self._entered = False + self._stack = contextlib.AsyncExitStack() + self._pools: dict[tuple[str, bool], InterceptionPool] = {} + self._pool_locks: dict[tuple[str, bool], asyncio.Lock] = {} + + async def __aenter__(self) -> "RunServices": + await self._stack.__aenter__() + self._entered = True + return self + + async def __aexit__(self, *exc) -> None: + self._entered = False + self._pools = {} + self._pool_locks = {} + await self._stack.__aexit__(*exc) + + async def pool_for(self, runtime_config: RuntimeConfig) -> InterceptionPool: + """Return a pool compatible with where this rollout's harness actually runs.""" + if not self._entered: + raise RuntimeError("RunServices must be entered before use") + key = (runtime_config.type, runtime_is_local(runtime_config)) + pool = self._pools.get(key) + if pool is not None: + return pool + lock = self._pool_locks.get(key) + if lock is None: + lock = asyncio.Lock() + self._pool_locks[key] = lock + async with lock: + pool = self._pools.get(key) + if pool is not None: + return pool + pool = await self._stack.enter_async_context( + InterceptionPool(runtime_config, self.multiplex) + ) + self._pools[key] = pool + return pool diff --git a/verifiers/v1/task.py b/verifiers/v1/task.py index db955b7bad..0b2204ee01 100644 --- a/verifiers/v1/task.py +++ b/verifiers/v1/task.py @@ -8,7 +8,7 @@ `Task` is the behavior half: a plain class owning runtime prep (`setup`/`finalize`), server declarations (`tools`/`user`), well-formedness (`validate`), and judgement -(`@reward`/`@metric`/`@group_reward` methods, run by `score`/`score_group`). It wraps +(`@reward`/`@metric` methods, run by `score`). It wraps a `TaskData` plus a `TaskConfig`, both plain constructor arguments: task = MyTask(data) # config defaults to the declared type's @@ -21,17 +21,10 @@ on a type field; a taskset yields one task type (its `load` constructs it), and instances differ per row through their data. -The task is the single judgement authority, scored at two granularities (execution -lives in the Rollout — per-rollout — and the Episode — group — which call these): - - `score` runs `@metric`/`@reward` — plus the plugged judges resolved from - `config.judges` (see `verifiers.v1.judge`) — over one trace (in its live runtime). - - `score_group` runs `@group_reward` over all the rollouts of this task at once — - pairwise/preference rewards that compare samples. - -A Task instance is shared across its rollouts (a group's `n` samples hold the same -instance), so hooks and scoring methods must not stash per-rollout state on `self` — -that lives on the trace (`trace.state`, typed via the `Task[..., MyState, ...]` -param). +`score` runs `@metric`/`@reward` — plus the plugged judges resolved from +`config.judges` (see `verifiers.v1.judge`) — over one trace in its live runtime. +Cross-trace judgement within one completed invocation belongs to `Topology`; comparisons +across independent invocations belong to the training algorithm. On the wire only the data (plus the producing class's name, `trace.task.type`) travels: a saved `trace.task.data` reads back as `WireTaskData` @@ -221,7 +214,7 @@ class Task(Generic[DataT, StateT, ConfigT]): Parameterize as `Task[MyData, MyState, MyConfig]`. Construction accepts the row and an optional config; omitting config builds the declared config type. One task - instance is shared across a rollout group, so per-rollout state belongs on the trace. + instance may be reused across invocations, so per-rollout state belongs on the trace. """ NEEDS_CONTAINER: ClassVar[bool] = False @@ -312,22 +305,5 @@ async def score( for judge, result in zip(judges, judge_results): _record_result(trace, judge.reward_name, result, judge.config.weight) - async def score_group(self, traces: list[Trace]) -> None: - rewards = discover_decorated(self, "group_reward") - if not rewards: - return - available = {"task": self.data, "traces": traces} - async with boundary(TaskError, f"task {type(self).__name__} group scoring"): - reward_results = await invoke_all(rewards, available) - for fn, scores in zip(rewards, reward_results): - if len(scores) != len(traces): - raise ValueError( - f"@group_reward {fn.__name__} returned {len(scores)} score(s) " - f"for {len(traces)} rollout(s); it must return one per trace" - ) - weight = getattr(fn, "_vf_weight", 1.0) - for trace, score in zip(traces, scores): - trace.record_reward(fn.__name__, score, weight) - TaskT = TypeVar("TaskT", bound=Task) diff --git a/verifiers/v1/topologies/__init__.py b/verifiers/v1/topologies/__init__.py new file mode 100644 index 0000000000..b84c03ecd2 --- /dev/null +++ b/verifiers/v1/topologies/__init__.py @@ -0,0 +1,27 @@ +"""Built-in topologies, each a plugin module resolved by id (see `verifiers.v1.loaders`): + +- `llm-judge` — any taskset, LLM-judged: a solver agent runs the task, a second + (non-trainable) judge agent — fixed to the in-process `direct` chat loop, one API call — + grades the solver's final answer against the task and its ground truth, and the verdict + lands on the solver's trace as a deferred reward. +- `agentic-judge` — any taskset, judged by a real agent: the solver's entire serialized + trace is uploaded into the judge's own runtime, and the judge (bash+edit `default` + harness by default, configurable) investigates it with tools before committing to a + score. Same verdict contract as `llm-judge`. + +A topology module exports its `Topology` subclass via `__all__`; custom topologies live +under `environments/` or on the Environments Hub, exactly like tasksets and harnesses. +""" + +from verifiers.v1.topologies.agentic_judge import ( + AgenticJudgeConfig, + AgenticJudgeTopology, +) +from verifiers.v1.topologies.llm_judge import LLMJudgeConfig, LLMJudgeTopology + +__all__ = [ + "AgenticJudgeConfig", + "AgenticJudgeTopology", + "LLMJudgeConfig", + "LLMJudgeTopology", +] diff --git a/verifiers/v1/topologies/agentic_judge.py b/verifiers/v1/topologies/agentic_judge.py new file mode 100644 index 0000000000..2e012d6b42 --- /dev/null +++ b/verifiers/v1/topologies/agentic_judge.py @@ -0,0 +1,114 @@ +"""agentic-judge: any taskset, judged by a real agent over the uploaded trace (built-in topology). + +The heavyweight sibling of `llm-judge`: same two agents, same verdict contract (the judge's +score lands on the *solver's* trace as a deferred reward), but the judge is a true agent in +its own runtime. Instead of rendering the attempt into a prompt, `go` serializes the solver's +*entire* trace — the task it was given (prompt plus any ground-truth fields), the complete +conversation, the rewards its own task recorded — and the judge task's `setup` hook uploads +it into the judge's freshly provisioned runtime before the harness runs. The judge then +investigates the file with its tools (the bash+edit `default` harness unless swapped) and +commits to a `SCORE: ` line. + +Everything `llm-judge` locks down is configurable here: the judge's harness +(`--topology.judge.harness.id ...`, and where it runs via `--topology.judge.harness.runtime...`) +and its assignment (`--topology.judge.prompt`, well, `prompt` on the topology config). + + uv run eval --topology.id agentic-judge --topology.taskset.id gsm8k-v1 -n 4 +""" + +import json + +from verifiers.v1.runtimes import Runtime +from verifiers.v1.task import Task, TaskData +from verifiers.v1.topology import AgentConfig, Topology, TopologyRun +from verifiers.v1.topologies.llm_judge import ( + JudgeTask, + LLMJudgeConfig, + LLMJudgeTopology, +) +from verifiers.v1.trace import Trace + +TRACE_PATH = "/tmp/solver_trace.json" +"""Where the solver's serialized trace lands in the judge's runtime.""" + +AGENTIC_JUDGE_PROMPT = """You are grading another agent's rollout on a task. Its full \ +trace has been uploaded to `{path}` as JSON: the task it was given (`task` — the prompt, \ +plus any ground-truth fields such as a reference answer), the complete conversation \ +(`nodes`, one message per node), and any rewards its own task recorded (`rewards`). + +Go look at `{path}` and assess whether the agent solved its task correctly and \ +completely. Investigate however you like — the file can be large, so consider extracting \ +the task and the agent's final messages first. When you have decided, end your reply with \ +a final line of exactly `SCORE: `, where is an integer from 0 (entirely wrong) to \ +10 (flawless).""" + + +class AgenticJudgeData(TaskData): + """The grading assignment's wire half: the solver's entire trace, JSON-serialized + (`Trace.to_record`) — persisted with the judge's own trace, so the record shows + exactly what was judged.""" + + trace_json: str + + +class AgenticJudgeTask(JudgeTask[AgenticJudgeData]): + """A grading assignment over an uploaded trace: the data carries the solver's + serialized trace, and `setup` writes it into the judge's runtime before the + harness runs — so the file is there the moment the judge starts investigating. The + verdict parser and the `committed` stop are inherited from `JudgeTask`.""" + + @classmethod + def from_trace( + cls, trace: Trace, prompt: str = AGENTIC_JUDGE_PROMPT + ) -> "AgenticJudgeTask": + """Wrap a finished episode for agentic judging (the `Task.from_trace` + convention): the assignment (`prompt`, with `{path}` resolved to where the + trace will land) plus the trace itself.""" + return cls( + AgenticJudgeData( + idx=trace.task.data.idx, + prompt=prompt.format(path=TRACE_PATH), + trace_json=json.dumps(trace.to_record()), + ) + ) + + async def setup(self, trace: Trace, runtime: Runtime) -> None: + """Upload the judged trace into the judge's runtime.""" + await runtime.write(TRACE_PATH, self.data.trace_json.encode()) + + +class AgenticJudgeAgentConfig(AgentConfig): + """The agentic judge: a real agent, excluded from training. Runs the bash+edit + `default` harness unless swapped (`--topology.judge.harness.id ...`) — unlike + `llm-judge`, the harness here is the point of the topology, so it's configurable.""" + + trainable: bool = False + + +class AgenticJudgeConfig(LLMJudgeConfig): + judge: AgenticJudgeAgentConfig = AgenticJudgeAgentConfig() + prompt: str = AGENTIC_JUDGE_PROMPT + """The judge's assignment (`{path}` = where the trace was uploaded). A config field, + so the rubric is tunable from the CLI/toml without a custom topology.""" + + +class AgenticJudgeTopology(LLMJudgeTopology, Topology[AgenticJudgeConfig]): + """Inherits the whole verdict contract from `LLMJudgeTopology` (`judge_committed` + metric, weighted `judge` reward on the solver's trace); only the forward arrow — + what the judge is handed — differs.""" + + async def go(self, task: Task, run: TopologyRun) -> None: + solver = await run.agent("solver").run(task) + await run.agent("judge").run( + AgenticJudgeTask.from_trace(solver, self.config.prompt), + parents=[solver], + ) + + +__all__ = [ + "AgenticJudgeAgentConfig", + "AgenticJudgeConfig", + "AgenticJudgeData", + "AgenticJudgeTask", + "AgenticJudgeTopology", +] diff --git a/verifiers/v1/topologies/llm_judge.py b/verifiers/v1/topologies/llm_judge.py new file mode 100644 index 0000000000..b1372474a5 --- /dev/null +++ b/verifiers/v1/topologies/llm_judge.py @@ -0,0 +1,202 @@ +"""llm-judge: any taskset, LLM-judged (built-in topology). + +Two agents: a `solver` — any harness, running the seed tasks (`--topology.taskset.id`) — and +a `judge`, fixed to the in-process `direct` chat loop (an episode ≈ one API call), excluded +from training. `go` peels the judge's inputs off the finished solver episode — the seed +task's framing, its ground truth (an `answer` field, when the taskset carries one), and the +solver's *final message* (the last message of its trace's final branch: the committed +answer, not the reasoning path) — renders them into a `JudgeTask`, and the judge's verdict +lands on the *solver's* trace as a deferred reward. Grading composes with any taskset +without touching its code; the solver task's own `@reward`s still run. + +The judge's model/client/sampling stay configurable (`--topology.judge.model `); only +its harness is pinned down — a judge that runs a real harness and investigates the trace +with tools is the `agentic-judge` topology. (For a verdict baked into a task's own grading, +call `vf.Judge` from the task's `@reward`.) + + uv run eval --topology.id llm-judge --topology.taskset.id gsm8k-v1 -n 4 +""" + +import logging + +from pydantic import SerializeAsAny, model_validator + +from verifiers.v1.decorators import metric, reward, stop +from verifiers.v1.harness import HarnessConfig +from verifiers.v1.harnesses.direct import DirectHarnessConfig +from verifiers.v1.task import DataT, Task, TaskData +from verifiers.v1.topology import ( + AgentConfig, + AgentGraph, + Topology, + TopologyConfig, + TopologyRun, +) +from verifiers.v1.trace import Trace +from verifiers.v1.types import content_text + +logger = logging.getLogger(__name__) + +JUDGE_PROMPT = """You are grading another agent's attempt at a task. + + +{task} + +{reference} + +{attempt} + + +Assess whether the attempt solves the task correctly and completely{against}. Think it \ +through, then end your reply with a final line of exactly `SCORE: `, where is an \ +integer from 0 (entirely wrong) to 10 (flawless).""" + +REFERENCE_SECTION = """ + +{answer} + +""" + + +def parse_score(text: str) -> float | None: + """A `SCORE: ` verdict in `text`, normalized to 0..1 — the last such line + (markdown `**`/backtick emphasis and a `/10` suffix tolerated), clamped to the 0-10 + scale. None when no verdict was committed (the caller decides what that means). + The one verdict grammar shared by everything that asks for a 0-10 score line + (`JudgeTask`, `agentic-judge`, `writer-editors-v1`'s improvement judge).""" + for line in reversed(text.splitlines()): + stripped = line.strip().strip("*`").strip() + if not stripped.upper().startswith("SCORE:"): + continue + try: + score = float(stripped.split(":", 1)[1].split("/")[0].strip()) + except ValueError: + return None + return min(max(score / 10.0, 0.0), 1.0) + return None + + +class JudgeTask(Task[DataT]): + """A grading assignment: `prompt` carries the rendered upstream task + ground truth + + attempt, the class carries the stop and the verdict parser — question and rubric in one + typed object, constructed in `go` from the solver's trace (the forward arrow).""" + + @classmethod + def from_trace(cls, trace: Trace) -> "JudgeTask": + """Render a finished episode into a grading task (the `Task.from_trace` + convention): the upstream task's framing, its ground truth (an `answer` data + field, when the row has one), and the attempt's final message — all read off + the trace, which is self-describing (`trace.task.data`). Pure trace→task code.""" + data = trace.task.data + answer = getattr(data, "answer", None) + return cls( + TaskData( + idx=data.idx, + prompt=JUDGE_PROMPT.format( + task=data.prompt_text, + reference=REFERENCE_SECTION.format(answer=answer) + if answer is not None + else "", + against=" (against the reference answer)" + if answer is not None + else "", + attempt=cls.attempt_text(trace), + ), + ) + ) + + @staticmethod + def attempt_text(trace: Trace) -> str: + """What gets judged: the final message of the trace's final branch (its most recent + full context), as plain text — the solver's committed answer, not its transcript.""" + branches = trace.branches + messages = branches[-1].messages if branches else [] + return content_text(messages[-1].content) if messages else "" + + @staticmethod + def parse_score(trace: Trace) -> float | None: + """The judge's verdict — `parse_score` over its final reply.""" + return parse_score(trace.last_reply) + + @stop + async def committed(self, trace: Trace) -> bool: + """Stop once a verdict line has been emitted, so a judge doesn't keep + investigating past its commitment.""" + return self.parse_score(trace) is not None + + +class JudgeAgentConfig(AgentConfig): + """The judge: fixed to the in-process `direct` chat loop (an episode ≈ one API call), + excluded from training. Its routing stays per-agent config (`--topology.judge.model + `, `--topology.judge.client...`); the harness is the one thing this topology locks — + swapping it is what the `agentic-judge` topology is for.""" + + harness: SerializeAsAny[HarnessConfig] = DirectHarnessConfig(id="direct") + trainable: bool = False + + @model_validator(mode="before") + @classmethod + def _fixed_harness(cls, data): + """Refuse a harness swap with a pointer to the right topology (runs before the + base `_resolve_plugins`, so the id never narrows to a foreign harness config).""" + if isinstance(data, dict): + raw = data.get("harness") + if isinstance(raw, HarnessConfig): + raw = raw.model_dump() + if isinstance(raw, dict) and raw.get("id") not in (None, "", "direct"): + raise ValueError( + "the llm-judge judge harness is fixed (the in-process `direct` chat " + "loop); for a judge that runs a real harness over the solver's trace " + "use `--topology.id agentic-judge`, whose judge harness is configurable" + ) + return data + + +class LLMJudgeConfig(TopologyConfig): + solver: AgentConfig = AgentConfig() + """The agent under evaluation — it runs the seed tasks, so `--topology.taskset.id` is + the one required knob.""" + judge: JudgeAgentConfig = JudgeAgentConfig() + weight: float = 1.0 + """Weight of the judge's verdict in the solver's reward (alongside whatever the + solver task's own `@reward`s already recorded).""" + + +class LLMJudgeTopology(Topology[LLMJudgeConfig]): + async def go(self, task: Task, run: TopologyRun) -> None: + solver = await run.agent("solver").run(task) + await run.agent("judge").run(JudgeTask.from_trace(solver), parents=[solver]) + + @metric(agent="solver") + async def judge_committed(self, trace: Trace, graph: AgentGraph) -> float: + """Whether the judge actually committed to a verdict for this solver trace.""" + return float(self.verdict(trace, graph) is not None) + + @reward(agent="solver") + async def judge(self, trace: Trace, graph: AgentGraph) -> float: + """The judge's verdict, recorded on the *solver's* trace — a missing verdict + (errored judge, or no SCORE line) scores 0, loudly. Weighted by `config.weight` + (config-driven, so applied here rather than via the decorator's static weight).""" + score = self.verdict(trace, graph) + if score is None: + judge = next(iter(graph.children(trace, agent="judge")), None) + logger.warning( + "judge returned no verdict for solver trace %s (%s)", + trace.id, + judge.error.type if judge and judge.error else "no SCORE line", + ) + return (score or 0.0) * self.config.weight + + def verdict(self, trace: Trace, graph: AgentGraph) -> float | None: + """This solver trace's parsed verdict: the score its judge child committed to.""" + judges = graph.children(trace, agent="judge") + return JudgeTask.parse_score(judges[0]) if judges else None + + +__all__ = [ + "JudgeAgentConfig", + "JudgeTask", + "LLMJudgeConfig", + "LLMJudgeTopology", + "parse_score", +] diff --git a/verifiers/v1/topologies/single_agent.py b/verifiers/v1/topologies/single_agent.py new file mode 100644 index 0000000000..b716d8db10 --- /dev/null +++ b/verifiers/v1/topologies/single_agent.py @@ -0,0 +1,5 @@ +"""The built-in one-agent topology.""" + +from verifiers.v1.topology import SingleAgentTopology + +__all__ = ["SingleAgentTopology"] diff --git a/verifiers/v1/topologies/swe_style_judge.py b/verifiers/v1/topologies/swe_style_judge.py new file mode 100644 index 0000000000..c6bde75375 --- /dev/null +++ b/verifiers/v1/topologies/swe_style_judge.py @@ -0,0 +1,132 @@ +"""Shared-sandbox agentic style judging for software-engineering tasks. + +The solver owns one runtime. After its task has been finalized and scored, a +non-trainable judge runs a second harness in that same live runtime. The judge sees the +working tree exactly as the solver left it and grades implementation quality without +receiving the solver's conversation. + + uv run --extra envs eval --topology.id swe-style-judge \ + --topology.taskset.id swebench-verified-v1 \ + --topology.taskset.use-prime-registry true \ + --topology.solver.harness.runtime.type prime \ + -m openai/gpt-5.5 -n 1 -r 1 +""" + +import logging + +from pydantic import SerializeAsAny + +from verifiers.v1.decorators import metric, reward +from verifiers.v1.harness import HarnessConfig +from verifiers.v1.harnesses.default import DefaultHarnessConfig +from verifiers.v1.task import Task, TaskData +from verifiers.v1.topologies.llm_judge import JudgeTask +from verifiers.v1.topology import ( + AgentConfig, + AgentGraph, + Topology, + TopologyConfig, + TopologyRun, +) +from verifiers.v1.trace import Trace + +logger = logging.getLogger(__name__) + +STYLE_JUDGE_PROMPT = """You are reviewing the implementation quality of another coding agent's work. + +You are running in the exact same sandbox and working directory that the coding agent +used. The current filesystem is the artifact to review. Inspect the repository and its +working-tree changes directly with read-only commands such as `git status`, `git diff`, +and targeted file reads. Do not modify any files. Do not inspect hidden verifier data in +`/tests` or `/logs/verifier`. + +The original task was: + + +{task} + + +Grade style and engineering quality, not functional correctness. Use this rubric: + +The task verifier may temporarily replace or reset test files while scoring, so the +current working tree may not retain tests the solver added. Do not penalize missing test +changes or infer that the solver failed to write tests from their absence here. + +- 0-2: scope discipline — focused changes with no unrelated churn. +- 0-3: clarity — readable, idiomatic code with sensible names and structure. +- 0-2: maintainability — robust handling without brittle hacks or needless complexity. +- 0-2: validation hygiene — appropriate tests or checks, without weakening existing tests. +- 0-1: repository hygiene — no generated junk, debug artifacts, or accidental edits. + +Use at most three read-only tool calls. Even if the available evidence is incomplete, +you must finish the review and provide a score. Explain the most important evidence +briefly. End with a final line of exactly +`SCORE: `, where is an integer from 0 to 10. +""" + + +class StyleJudgeTask(JudgeTask): + """A style-review assignment over the solver's still-live working tree.""" + + @classmethod + def from_trace( + cls, trace: Trace, prompt: str = STYLE_JUDGE_PROMPT + ) -> "StyleJudgeTask": + return cls( + TaskData( + idx=trace.task.data.idx, + name=trace.task.data.name, + prompt=prompt.format(task=trace.task.data.prompt_text), + ) + ) + + +class StyleJudgeAgentConfig(AgentConfig): + """A tool-using, read-only-by-instruction judge excluded from training.""" + + harness: SerializeAsAny[HarnessConfig] = DefaultHarnessConfig( + id="default", edit=False + ) + trainable: bool = False + + +class SWEStyleJudgeConfig(TopologyConfig): + solver: AgentConfig = AgentConfig() + judge: StyleJudgeAgentConfig = StyleJudgeAgentConfig() + prompt: str = STYLE_JUDGE_PROMPT + weight: float = 1.0 + + +class SWEStyleJudgeTopology(Topology[SWEStyleJudgeConfig]): + async def go(self, task: Task, run: TopologyRun) -> None: + solver = run.agent("solver") + async with solver.provision(task) as runtime: + solution = await solver.run(task, runtime=runtime) + await run.agent("judge").run( + StyleJudgeTask.from_trace(solution, self.config.prompt), + parents=[solution], + runtime=runtime, + ) + + @metric(agent="solver") + async def style_committed(self, trace: Trace, graph: AgentGraph) -> float: + return float(self.verdict(trace, graph) is not None) + + @reward(agent="solver") + async def style(self, trace: Trace, graph: AgentGraph) -> float: + score = self.verdict(trace, graph) + if score is None: + judge = next(iter(graph.children(trace, agent="judge")), None) + logger.warning( + "style judge returned no verdict for solver trace %s (%s)", + trace.id, + judge.error.type if judge and judge.error else "no SCORE line", + ) + return (score or 0.0) * self.config.weight + + def verdict(self, trace: Trace, graph: AgentGraph) -> float | None: + judges = graph.children(trace, agent="judge") + return StyleJudgeTask.parse_score(judges[0]) if judges else None + + +__all__ = ["SWEStyleJudgeTopology"] diff --git a/verifiers/v1/topology.py b/verifiers/v1/topology.py new file mode 100644 index 0000000000..d328a61de9 --- /dev/null +++ b/verifiers/v1/topology.py @@ -0,0 +1,778 @@ +"""The topology: a surface over which agents interact. + +A `Topology` composes agent runs — one agent consuming one task and producing one trace — +into a multi-agent interaction: which agents exist, how one agent's trace becomes a +downstream agent's task, and how rewards flow backwards once downstream agents have run. +Each agent run is an ordinary `Rollout` (same lifecycle, same error model, same trace), and +tasks are the whole task-side contract — their classes carry the behavior — so a topology +declares named agent bindings, then executes topology-bound agents against tasks. + +The interaction pattern is plain imperative Python in `go` — not a DSL: a loop is rounds, +`asyncio.gather` over `run.agent(...).run(...)` calls is fan-out, and awaiting several traces +before building the next task is fan-in. `go` owns *control flow only*, including the +forward arrow (trace → next task, pure host-side construction); judgement — including the +backward arrow, a reward derived from downstream traces — is declared as +`@vf.reward(agent=...)`/`@vf.metric(agent=...)` methods, run over the finished instance +(see `Topology.score`). + +Running one instance produces an `AgentGraph` — the serialized instance artifact: the +global, causally ordered view over its traces, each linked to its parents (`trace.agent` / +`trace.parents`). A `Trace` stays the per-agent view of one run; the graph is the +cross-agent view of the whole interaction. Interleaving two agents' *execution* inside one +run is deliberately out of scope — an agent run completes before its trace +feeds anything downstream. +""" + +import asyncio +import contextlib +import logging +import uuid +from collections.abc import Callable, Mapping, Sequence +from functools import cached_property +from typing import Generic, TypeVar + +from pydantic import Field, SerializeAsAny, model_validator +from pydantic_config import BaseConfig + +from verifiers.v1.agent import Agent, Parent, Session +from verifiers.v1.clients import Client, ClientConfig, ModelContext, resolve_client +from verifiers.v1.decorators import discover_decorated, invoke +from verifiers.v1.env import EnvConfig, EnvServerConfig, validate_pairing +from verifiers.v1.errors import TopologyError, boundary +from verifiers.v1.harness import Harness, HarnessConfig +from verifiers.v1.interception import RolloutLimits +from verifiers.v1.rollout import Rollout +from verifiers.v1.runtimes import Runtime, SubprocessConfig +from verifiers.v1.services import RunServices +from verifiers.v1.task import Task +from verifiers.v1.taskset import Taskset, TasksetConfig +from verifiers.v1.trace import Error, Trace, TraceTask, WireTrace +from verifiers.v1.types import ID, SamplingConfig, StrictBaseModel +from verifiers.v1.utils.install import env_name +from verifiers.v1.utils.memory import trim_memory_periodically + +logger = logging.getLogger(__name__) + + +def _deep_merge(base: dict, override: dict) -> dict: + """`override` layered onto `base`, recursing into nested dicts — how a partial + `--topology..harness.*` override tunes a pinned harness instead of replacing it.""" + merged = dict(base) + for key, value in override.items(): + if isinstance(value, dict) and isinstance(merged.get(key), dict): + merged[key] = _deep_merge(merged[key], value) + else: + merged[key] = value + return merged + + +def _merge_sampling( + base: SamplingConfig, override: SamplingConfig | None +) -> SamplingConfig: + """Agent sampling overrides layer onto the run sampling, field by field.""" + if override is None: + return base + return base.model_copy(update=override.model_dump(exclude_none=True)) + + +class AgentConfig(BaseConfig): + """One agent in a topology: which harness drives its runs (and where — + `harness.runtime`), and how its model calls are routed. Declared as typed fields of a + `TopologyConfig` subclass — the field *name* is the agent's name in `go` — so every + agent is CLI/toml-addressable (`--topology..harness.id`, + `--topology..model`, ...). An agent carries nothing task-side: the tasks it + consumes (each carrying its own behavior) arrive per run, from the topology's seeds + or constructed in `go`. + + To pin a per-agent default harness, **subclass and set the field default** + (`harness: SerializeAsAny[HarnessConfig] = HarnessConfig(id="direct")`) — pins must + live on an `AgentConfig` subclass, never on the outer topology-config field's default + instance, which a partial override (`--topology..model ...`) would silently + replace (pydantic re-validates the whole field; it never merges into instance + defaults). A pinned harness survives everything except an explicit + `--topology..harness.id`.""" + + harness: SerializeAsAny[HarnessConfig] = HarnessConfig(id="default") + """The program driving this agent's runs, and where it runs (`harness.runtime`) — + per agent, so a judge can run the in-process `direct` chat loop while the solver's + coding agent runs in a container.""" + model: str | None = None + """Model override for this agent (None = the eval's model) — e.g. a stronger judge.""" + client: ClientConfig | None = None + """Client override for this agent (None = the eval's client) — how its model calls are + routed: e.g. a non-trainable judge relayed to a plain API endpoint while the solver + runs against the train client. Per-agent routing lives here, not in extra interception + machinery — sessions are already per-rollout.""" + sampling: SamplingConfig | None = None + """Sampling overrides for this agent, layered over the eval's sampling.""" + trainable: bool = True + """Whether this agent's traces are training samples — stamped onto `Trace.trainable` + so a trainer can filter without consulting the topology config.""" + + @model_validator(mode="before") + @classmethod + def _resolve_plugins(cls, data): + """Resolve the `harness` field. Three cases, in precedence order: + an explicit `id` swaps the harness (narrowed to that harness's own config type); + otherwise a subclass **pin** — detected by value, so any changed default counts, + including a base-typed `HarnessConfig(id="null")` — absorbs partial overrides by + deep-merging them over the pinned default (`--...harness.runtime.type docker` + tunes the pin, never silently replaces it); otherwise the base default narrows to + the `default` harness's config.""" + if not isinstance(data, dict): + return data + from verifiers.v1.loaders import harness_config_type, narrow_plugin_field + + raw = data.get("harness") + if isinstance(raw, BaseConfig): + raw = raw.model_dump() + raw = dict(raw or {}) + default = cls.model_fields["harness"].default + pinned = ( + isinstance(default, HarnessConfig) + and default != AgentConfig.model_fields["harness"].default + ) + if raw.get("id"): # explicit swap always wins + narrow_plugin_field(data, "harness", harness_config_type) + elif pinned: + data["harness"] = harness_config_type(default.id).model_validate( + _deep_merge(default.model_dump(), raw) + ) + else: + narrow_plugin_field(data, "harness", harness_config_type, "default") + return data + + +class DirectAgentConfig(AgentConfig): + """An agent pinned to the in-process `direct` chat loop — tool-less, a run ≈ one + API call. The most common pin, shared so topologies don't each redeclare it; per the + pin contract, partial overrides (`--topology..model ...`) tune it and an + explicit `--topology..harness.id` still swaps it.""" + + harness: SerializeAsAny[HarnessConfig] = HarnessConfig(id="direct") + + +class NullAgentConfig(AgentConfig): + """An agent pinned to the `null` chat loop — a subprocess program WITH the task's MCP + tools but none of its own. The pin for an agent that must call task tools without + being a full coding agent.""" + + harness: SerializeAsAny[HarnessConfig] = HarnessConfig(id="null") + + +class TopologyConfig(BaseConfig): + """Base topology config. Subclass to declare the agents (typed `AgentConfig` fields — + the field name is the agent's name) plus any interaction knobs (fan-out width, number + of rounds, reward weights). Mirrors `TasksetConfig`: the concrete subclass is resolved + by `id`, so its fields surface typed on the CLI/toml.""" + + id: ID = "" + """The topology id, which selects this topology: a built-in (`llm-judge`), a local + package, or an `org/name[@version]` package installed on demand from the Environments + Hub (see `ID`). Set via `--topology.id`.""" + taskset: SerializeAsAny[TasksetConfig] = TasksetConfig() + """The seed source: a taskset (a pure task factory, resolved by id — see + `verifiers.v1.taskset`) whose tasks seed the instances, one instance per seed. Set via + `--topology.taskset.id ` — the same word, slot, and grammar as the single-agent + route's `--taskset.id`. Exclusive-or with overriding `load_tasks`: when this slot can + be set, it IS the seed source, verbatim — a topology that constructs its own seeds + overrides `load_tasks` and is refused this flag (a custom `load_tasks` wanting a + config-driven source declares its own factory field instead).""" + + @property + def name(self) -> str: + """The topology's package name (the id with any org / version stripped).""" + return env_name(self.id) + + @model_validator(mode="before") + @classmethod + def _resolve_taskset(cls, data): + """Narrow the seed `taskset` factory to the config type its `id` resolves to, so its + knobs (`--topology.taskset.split ...`) validate typed.""" + if isinstance(data, dict) and data.get("taskset"): + from verifiers.v1.loaders import narrow_plugin_field, taskset_config_type + + narrow_plugin_field(data, "taskset", taskset_config_type) + return data + + +class SingleAgentTopologyConfig(TopologyConfig): + """Internal lowering target for the user-facing taskset + harness syntax.""" + + id: ID = "single-agent" + agent: AgentConfig = AgentConfig() + + +ConfigT = TypeVar("ConfigT", bound=TopologyConfig) + + +class AgentBinding: + """A topology-registered agent slot: name + config + loaded harness. + + The value exposed inside `Topology.go` is a topology-bound `TopologyAgent`; this + binding is the config-side declaration a topology loads and validates before serving. + """ + + def __init__( + self, name: str, config: AgentConfig, harness: Harness | None = None + ) -> None: + from verifiers.v1.loaders import load_harness + + self.name = name + self.config = config + self.harness = harness if harness is not None else load_harness(config.harness) + + +class AgentGraph(StrictBaseModel): + """One topology instance's artifact: the global, causally ordered view over its traces, + each linked to the traces it was derived from (`trace.agent` names the producing agent, + `trace.parents` its upstream trace ids). Traces append in completion order — a parent + always finishes before a task is derived from it, so the list is topologically sorted. + + The trace's sibling, one level up: what a trace is to an agent run, the graph is to an + instance. A topology run persists one graph per line (`traces.jsonl`), traces nested — + and since each trace carries its own links, the graph is also *recoverable* from a flat + trace dump (one instance = one connected component).""" + + id: str = Field(default_factory=lambda: uuid.uuid4().hex) + """Unique id for this instance, auto-generated per graph.""" + topology: str = "" + """The topology id that produced this instance.""" + task: SerializeAsAny[TraceTask] + """The seed task for this invocation, including its task behavior type.""" + error: Error | None = None + """A failure in the topology's own code (`go` or instance scoring, a `TopologyError`), + recorded instead of raised — agent failures live on their traces, this is for the + composition itself. Traces completed before the failure remain data.""" + traces: list[SerializeAsAny[Trace]] = Field(default_factory=list) + """Every agent trace, in completion (= topological) order.""" + + def add(self, trace: Trace) -> None: + self.traces.append(trace) + + def roots(self) -> list[Trace]: + """The traces with no parents — the entry runs (usually one, on the seed task).""" + return [trace for trace in self.traces if not trace.parents] + + def children(self, trace: Trace, agent: str | None = None) -> list[Trace]: + """The traces derived from `trace` (its direct downstream runs), optionally + only those a named agent produced — the navigation cross-agent scoring lives on + (`graph.children(proposer, agent="solver")`).""" + return [ + t + for t in self.traces + if trace.id in t.parents and (agent is None or t.agent == agent) + ] + + def by_agent(self, agent: str) -> list[Trace]: + """The traces a named agent produced, in completion order.""" + return [trace for trace in self.traces if trace.agent == agent] + + def to_record(self) -> dict: + """A JSON-serializable record of this instance for `traces.jsonl` — each nested + trace dumped like `Trace.to_record` (per-node training tensors stripped).""" + from verifiers.v1.trace import _NODE_DUMP_EXCLUDE + + return self.model_dump( + mode="json", exclude={"traces": {"__all__": _NODE_DUMP_EXCLUDE}} + ) + + @classmethod + def load(cls, data: dict) -> "AgentGraph": + """Load a dumped instance record without the originating packages: each trace is + typed as a `WireTrace` (task-specific fields ride in `task.data.model_extra`).""" + from verifiers.v1.task import WireTaskData + + task = TraceTask[WireTaskData].model_validate(data["task"]) + graph = cls.model_validate({**data, "task": task, "traces": []}) + graph.traces = [WireTrace.model_validate(t) for t in data.get("traces", [])] + return graph + + +def graph_complete(graph: AgentGraph) -> bool: + """The conservative instance-validity default (`Topology.complete`'s base rule): + no instance-level error and no errored trace. Module-level so consumers without a + live topology (the server-mode eval client) apply the same rule.""" + return graph.error is None and not any(t.has_error for t in graph.traces) + + +class Topology(Generic[ConfigT]): + """Generic over its config type, so `self.config` is fully typed in subclasses. + Subclass: declare `AgentConfig` fields on your config, implement `go` (control flow) + and `@vf.reward(agent=...)`/`@vf.metric(agent=...)` methods (judgement). Seeds come + from the config's `taskset` slot, or override `load_tasks` to construct them.""" + + def __init__(self, config: ConfigT) -> None: + self.config = config + + @cached_property + def taskset(self) -> Taskset | None: + """The configured seed taskset, loaded once, or None for self-seeding topologies.""" + if not self.config.taskset.id: + return None + from verifiers.v1.loaders import load_taskset + + return load_taskset(self.config.taskset) + + def load_agents(self) -> dict[str, AgentBinding]: + """The topology's agents, one per `AgentConfig` field on the config (in declaration + order). Override only to compose agents programmatically — each is still just + `AgentBinding(name, config)`.""" + return { + name: AgentBinding(name, value) + for name, value in self.config + if isinstance(value, AgentConfig) + } + + @cached_property + def agents(self) -> dict[str, AgentBinding]: + """The loaded agents, built once via `load_agents`. Loading also validates the + topology's declared judgement (`@reward`/`@metric` methods) against them, so a + typo'd or missing agent scope fails at load time, not mid-eval.""" + agents = self.load_agents() + if not agents: + raise ValueError( + f"topology {self.config.id!r} declares no agents: give its config " + "`AgentConfig` fields (or override `load_agents`)" + ) + # Scan the class, not `discover_decorated` (whose getmembers would re-enter this + # very cached_property via the properties it evaluates). Unwrap descriptors so a + # classmethod/staticmethod-wrapped decorator can't slip past validation only to be + # found (and silently mis-scoped) by `score`'s getmembers discovery later. + for klass in type(self).__mro__: + for fn in vars(klass).values(): + fn = getattr(fn, "__func__", fn) + kind = next( + (k for k in ("reward", "metric", "stop") if getattr(fn, k, False)), + None, + ) + if kind is None: + continue + if kind == "stop": + raise ValueError( + f"@stop is not supported on a Topology (found {fn.__name__!r}): stops belong to Task classes" + ) + scope = getattr(fn, "_vf_agent", None) + if scope is None: + raise ValueError( + f"topology @{kind} {fn.__name__!r} declares no agent scope; " + f"topology judgement is per-agent — use @vf.{kind}(agent=...)" + ) + if scope not in agents: + raise ValueError( + f"topology @{kind} {fn.__name__!r} scopes to unknown agent " + f"{scope!r}; this topology defines {sorted(agents)}" + ) + return agents + + def load_tasks(self) -> list[Task]: + """The seed tasks — one topology instance (`go`) runs per seed task. Defaults to + the config's `taskset` slot (`--topology.taskset.id `); override for a + topology that constructs its own seeds.""" + if not self.config.taskset.id: + raise ValueError( + f"topology {self.config.id!r} has no seed tasks: set --topology.taskset.id " + ", or override `load_tasks` to construct them" + ) + assert self.taskset is not None + return self.taskset.load() + + def complete(self, graph: AgentGraph) -> bool: + """Whether a persisted instance counts as a valid result of this topology — the + verdict consumers read when deciding what to redo or drop (today: `--resume` + re-runs instances that fail it). The default is conservative — no instance-level + error and no errored trace — which is exact for the single-agent lowering, where + the one trace IS the invocation. A topology whose `go` tolerates child failures + overrides this to match (typically `graph.error is None`), else resume redoes + instances it already accepted and scored. A read-only verdict over a finished + graph: what a failed child *means* stays in `go` and the declared rewards.""" + return graph_complete(graph) + + async def score(self, graph: AgentGraph) -> None: + """Run the topology's declared judgement over one completed instance: every + `@metric(agent=...)`, then every `@reward(agent=...)`, each invoked once per + trace the named agent produced — declaring any of `task`/`trace`/`graph` by + parameter name. Runs after `go` returns and before the instance persists, so + every trace (across all rounds and fan-outs) is scored, automatically. + + Ordering contract, chosen for predictability over cleverness: methods run + *sequentially*, metrics before rewards, each phase in (priority, name) order. + A method may read task-recorded rewards (final since the agent run ended) and, + in the rewards phase, any metric — but topology rewards must not read each + other; derive shared inputs from the traces (or a metric) instead. Mirrors + `Task.score` one level up: trace judgement there, instance judgement here.""" + for kind in ("metric", "reward"): + for fn in discover_decorated(self, kind): + weight = getattr(fn, "_vf_weight", 1.0) + for trace in graph.by_agent(getattr(fn, "_vf_agent", None)): + available = { + "task": trace.task.data, # the wire half, as in `Task.score` + "trace": trace, + "graph": graph, + } + result = await invoke(fn, available) + if kind == "metric": + if isinstance(result, Mapping): + trace.record_metrics(result) + else: + trace.record_metric(fn.__name__, result) + elif isinstance(result, Mapping): + for name, value in result.items(): + trace.record_reward(name, value, weight) + else: + trace.record_reward(fn.__name__, result, weight) + + async def go(self, task: Task, run: "TopologyRun") -> None: + """Run one topology instance from seed `task`: the *control flow only* — which + agents run, in what order, with what tasks: + `await run.agent(name).run(task, parents=...)` per agent invocation, + `asyncio.gather` for fan-out, loops for rounds, and + `trace.info[...] = ...` to annotate provenance. The forward arrow lives here — + construct the next agent's typed `Task` from an upstream trace (its typed task, + `last_reply`, `transcript`, or what its `finalize` peeled into `trace.info`). + Judgement belongs in the topology's `@reward(agent=...)`/`@metric(agent=...)` + methods (see `score`); recording a reward imperatively here is the escape hatch, + not the norm. Agent failures come back as data on their traces (`trace.error`), + never as exceptions — `go` decides what a failed child means (drop it, count it + against a pass rate, retry the round, ...).""" + raise NotImplementedError + + +class SingleAgentTopology(Topology[SingleAgentTopologyConfig]): + """The canonical one-agent topology produced from taskset + harness syntax.""" + + async def go(self, task: Task, run: "TopologyRun") -> None: + await run.agent("agent").run(task) + + +def resolve_topology_runner(config: EnvConfig) -> "TopologyRunner": + """Resolve explicit topology or taskset + harness syntax to the canonical runner.""" + if config.topology is not None: + from verifiers.v1.loaders import load_topology + + topology = load_topology(config.topology) + else: + topology = SingleAgentTopology( + SingleAgentTopologyConfig( + taskset=config.taskset, + agent=AgentConfig(harness=config.harness), + ) + ) + return TopologyRunner(topology, config) + + +class TopologyAgent: + """A topology-bound view of a registered executable `Agent`. + + It exposes the public agent's `run` / `provision` surface, but routes completed + traces through the owning `TopologyRun` so the graph, parents, trainability, retries, + and concurrency limit remain topology-owned. + """ + + def __init__(self, run: "TopologyRun", name: str) -> None: + self._run = run + self.name = name + + def _executable(self) -> Agent: + return self._run.executable_agent(self.name) + + @property + def config(self) -> AgentConfig: + return self._run.runner.agent(self.name).config + + @property + def harness(self) -> Harness: + return self._executable().harness + + def provision( + self, task: Task | None = None + ) -> contextlib.AbstractAsyncContextManager[Runtime]: + """Provision a runtime from this agent's policy (resolved for `task` when given) + and tear it down on exit — the box for `run(..., runtime=box)` calls to share, + by this agent or any other (see `Agent.provision`).""" + return self._executable().provision(task) + + async def run( + self, + task: Task, + *, + parents: Sequence[Parent] = (), + runtime: Runtime | None = None, + ) -> Trace: + return await self._run.run_agent( + self.name, task, parents=parents, runtime=runtime + ) + + def interact( + self, + task: Task, + *, + parents: Sequence[Parent] = (), + runtime: Runtime | None = None, + ) -> contextlib.AbstractAsyncContextManager[Session]: + """Hold a live agent run open and yield its `Session` — the + back-and-forth primitive (see `Agent.interact`): `go` converses with the + suspended run via `session.turn(...)`, N sessions compose into games, + debates, negotiations. The completed trace is graph-recorded on close.""" + return self._run.interact_agent( + self.name, task, parents=parents, runtime=runtime + ) + + +class TopologyRun: + """One live topology instance: the execution surface `go` programs against. Owns the + instance's `AgentGraph` and links every agent trace into it; concurrency, retries, and + limits come from the eval config, identically to a single-agent eval.""" + + def __init__( + self, + runner: "TopologyRunner", + task: Task, + agents: dict[str, Agent], + semaphore: asyncio.Semaphore | None = None, + ) -> None: + self.runner = runner + self.graph = AgentGraph( + topology=runner.topology.config.id, + task=TraceTask(type=type(task).__name__, data=task.data), + ) + self._agents = agents + self._semaphore = semaphore + + def agent(self, name: str) -> TopologyAgent: + """The registered agent named `name`, bound to this graph.""" + self.runner.agent(name) # validate spelling with the existing actionable error + return TopologyAgent(self, name) + + def executable_agent(self, name: str) -> Agent: + try: + return self._agents[name] + except KeyError: + self.runner.agent(name) + raise RuntimeError( + f"agent {name!r} is not bound to this topology run" + ) from None + + async def run_agent( + self, + agent: str, + task: Task, + *, + parents: Sequence[Parent] = (), + runtime: Runtime | None = None, + ) -> Trace: + """Run a topology-bound explicit agent and record its trace in this graph.""" + executable = self.executable_agent(agent) + async with self._semaphore or contextlib.nullcontext(): + trace = await executable.run( + task, + parents=parents, + runtime=runtime, + retry=self.runner.config.retries.rollout, + ) + self.graph.add(trace) + await trim_memory_periodically() + return trace + + @contextlib.asynccontextmanager + async def interact_agent( + self, + agent: str, + task: Task, + *, + parents: Sequence[Parent] = (), + runtime: Runtime | None = None, + ): + """Hold a live run of a topology-bound agent open (see `Agent.interact`) and + record its completed trace in this graph on close — even when `go` crashed + mid-interaction (completed traces remain data; `run_instance` captures the + crash as the graph's `TopologyError`). + + Sessions deliberately bypass the rollout-concurrency semaphore: a seat spends + most of an interaction suspended (no model call in flight), and a multi-seat + game holding N slots for its whole duration could deadlock the cap. Instance + concurrency (the eval's `-c`) is what bounds session pressure.""" + executable = self.executable_agent(agent) + session: Session | None = None + try: + async with executable.interact( + task, parents=parents, runtime=runtime + ) as session: + yield session + finally: + if session is not None: + try: + trace = session.trace + except RuntimeError: # the run never started — nothing to record + trace = None + if trace is not None: + self.graph.add(trace) + await trim_memory_periodically() + + +class TopologyRunner: + """Long-lived executor for one topology and its worker-scoped services.""" + + def __init__(self, topology: Topology | TopologyConfig, config: EnvConfig) -> None: + if isinstance(topology, TopologyConfig): + from verifiers.v1.loaders import load_topology + + topology = load_topology(topology) + self.config = config + self.topology = topology + # The seed contract is exclusive-or: the `taskset` slot IS the seed source, verbatim, + # or `load_tasks` is overridden — never both. A self-seeding topology accepting a + # `--topology.taskset.id` it then ignores would silently run a different experiment + # than the config claims, so refuse it up front. + if topology.config.taskset.id and ( + type(self.topology).load_tasks is not Topology.load_tasks + ): + raise ValueError( + f"topology {topology.config.id!r} constructs its own seeds (it overrides " + "`load_tasks`), so `--topology.taskset.id` would be silently ignored; drop " + "it. A topology wanting a config-driven source *inside* a custom " + "`load_tasks` declares its own factory field, not the built-in `taskset` slot." + ) + bindings = self.topology.agents + if isinstance(topology, SingleAgentTopology): + assert topology.taskset is not None + validate_pairing(bindings["agent"].harness, topology.taskset) + for binding in bindings.values(): + if binding.harness.config.id != "null" and isinstance( + binding.harness.config.runtime, SubprocessConfig + ): + logger.warning( + "Harness %r is running in the subprocess runtime on the local system. " + "Local files and settings may affect the evaluation; use subprocess only " + "for debugging. Use --harness.runtime.type docker or prime for an isolated run.", + binding.harness.config.id, + ) + self.tasks = self.topology.load_tasks() + self._services: RunServices | None = None + self._shared_tools: dict = {} + self._override_clients: dict[str, Client] = {} + + def agent(self, name: str) -> AgentBinding: + """The named agent, with an actionable error for a typo in `go`.""" + try: + return self.topology.agents[name] + except KeyError: + raise ValueError( + f"unknown agent {name!r}: topology {self.topology.config.id!r} defines {sorted(self.topology.agents)}" + ) from None + + @contextlib.asynccontextmanager + async def serving(self): + """Hold worker-scoped services while request contexts remain invocation-local.""" + if self._services is not None: + raise RuntimeError("TopologyRunner.serving() is already active") + from verifiers.v1.mcp import serve_shared + from verifiers.v1.runtimes import runtime_is_local + + async with contextlib.AsyncExitStack() as stack: + services = await stack.enter_async_context( + RunServices(self.config.multiplex) + ) + bindings = self.topology.agents + shared_tools: dict = {} + if self.topology.taskset is not None: + servers = self.topology.taskset.tool_servers() + if servers: + # Tunnel unless every agent's harness runs locally — any remote + # seat must still reach the one shared instance. + local = all( + runtime_is_local(b.harness.config.runtime) + for b in bindings.values() + ) + shared_tools = await stack.enter_async_context( + serve_shared(servers, harness_is_local=local) + ) + override_clients: dict[str, Client] = {} + for name, binding in bindings.items(): + if binding.config.client is not None: + client = resolve_client(binding.config.client) + stack.push_async_callback(client.close) + override_clients[name] = client + self._services = services + self._shared_tools = shared_tools + self._override_clients = override_clients + try: + yield + finally: + self._services = None + self._shared_tools = {} + self._override_clients = {} + + def _agents_for( + self, + ctx: ModelContext, + on_rollout: Callable[[Rollout], None] | None = None, + ) -> dict[str, Agent]: + services = self._services + if services is None: + raise RuntimeError( + "TopologyRunner.run_instance() must be called inside TopologyRunner.serving()" + ) + limits = RolloutLimits( + max_turns=self.config.max_turns, + max_input_tokens=self.config.max_input_tokens, + max_output_tokens=self.config.max_output_tokens, + max_total_tokens=self.config.max_total_tokens, + ) + agents: dict[str, Agent] = {} + for name, binding in self.topology.agents.items(): + agent_ctx = ModelContext( + model=binding.config.model or ctx.model, + client=self._override_clients.get(name, ctx.client), + sampling=_merge_sampling(ctx.sampling, binding.config.sampling), + ) + agents[name] = Agent( + binding.harness, + agent_ctx, + binding.harness.config.runtime, + name=name, + trainable=binding.config.trainable, + limits=limits, + timeout=self.config.timeout, + services=services, + shared_tools=self._shared_tools, + on_rollout=on_rollout, + ) + return agents + + async def run_instance( + self, + task: Task, + ctx: ModelContext, + semaphore: asyncio.Semaphore | None = None, + on_rollout: Callable[[Rollout], None] | None = None, + ) -> AgentGraph: + """Run one topology instance — a single `go` over one seed task, then the declared + instance judgement (`Topology.score`) — and return its agent graph. A failure in + topology-authored code is classified (`TopologyError`) and captured on the graph, + never raised: completed traces remain data, and sibling instances keep + running (mirrors `Rollout.run`'s a-bad-rollout-is-data stance one level up).""" + run = TopologyRun(self, task, self._agents_for(ctx, on_rollout), semaphore) + try: + async with boundary( + TopologyError, f"topology {self.topology.config.id!r} go" + ): + await self.topology.go(task, run) + # Instance judgement: the declared @reward/@metric methods over the finished + # graph. Skipped when `go` itself failed — a broken instance isn't scored. + async with boundary( + TopologyError, f"topology {self.topology.config.id!r} scoring" + ): + await self.topology.score(run.graph) + except TopologyError as e: + logger.exception("topology instance failed (seed task %s)", task.data.idx) + run.graph.error = Error(type=type(e).__name__, message=str(e)) + return run.graph + + +# `EnvConfig.topology` is annotated as a forward reference — env.py sits *below* this +# module and can't import it. Finalize the models here, where `TopologyConfig` exists; +# anything importing `verifiers.v1` (or any submodule — the package init runs first and +# imports this module) sees the completed models. +EnvConfig.model_rebuild() +EnvServerConfig.model_rebuild() diff --git a/verifiers/v1/trace.py b/verifiers/v1/trace.py index 0bf973ef30..6008f33898 100644 --- a/verifiers/v1/trace.py +++ b/verifiers/v1/trace.py @@ -23,6 +23,7 @@ from verifiers.v1.types import ( AssistantMessage, Messages, + SamplingConfig, StrictBaseModel, ToolMessage, Usage, @@ -196,13 +197,28 @@ class Trace(StrictBaseModel, Generic[DataT, StateT]): """Unique id for this rollout, auto-generated per trace.""" task: TraceTask[DataT] """The task being solved: its class name (`task.type`) + its row (`task.data`).""" + agent: str | None = None + """The topology agent that produced this trace (the `Topology` config field name, + e.g. `"solver"`; the built-in single-agent topology uses `"agent"`).""" + parents: list[str] = Field(default_factory=list) + """Trace ids of the upstream agent runs this one was derived from — the agent-graph + links. A topology's traces plus these links ARE the agent graph (see + `verifiers.v1.topology`); empty for a root run.""" + trainable: bool = True + """Whether this trace is a training sample. Set from the producing agent's + `AgentConfig.trainable`, so a trainer can drop e.g. a judge agent's traces without + consulting the topology config. Always True outside a topology.""" + sampling: SamplingConfig | None = None + """The resolved sampling configuration used by this agent invocation. This includes + topology-agent overrides, so training consumers can recover the actual sampling + temperature and other policy metadata from the trace itself.""" runtime: RuntimeInfo | None = None """The runtime's full config plus its provisioned resource ID.""" nodes: list[MessageNode] = Field(default_factory=list) """The message graph; branches are derived views and storage stays linear in turns.""" rewards: dict[str, float] = Field(default_factory=dict) - """Weighted contributions from task rewards, group rewards, and judges.""" + """Weighted contributions from task rewards, topology rewards, and judges.""" metrics: dict[str, float] = Field(default_factory=dict) """Unweighted metrics from tasks, harnesses, and judges.""" info: dict[str, Any] = Field(default_factory=dict)