Skip to content
Draft
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
fe05534
feat(v1): multi-agent topologies — imperative go(), agent graphs, def…
eligotts Jul 7, 2026
0c5abd3
chore: minimize the diff — revert non-direct-path changes
eligotts Jul 7, 2026
ef93168
Merge remote-tracking branch 'origin/main' into verifiers-topology
codex Jul 7, 2026
f29722d
feat(v1): make agents explicit topology primitives
codex Jul 8, 2026
441786b
Merge remote-tracking branch 'origin/main' into verifiers-topology
codex Jul 8, 2026
48ca1ee
style(v1): format topology agent changes
codex Jul 8, 2026
6cd1602
style(v1): match CI ruff formatting
codex Jul 8, 2026
4267506
refactor(v1): require explicit topology agents
codex Jul 8, 2026
4f67a63
refactor(v1): tighten topology agent boundary
codex Jul 8, 2026
0e3017a
add shared runtime topology example
codex Jul 8, 2026
1e6c827
refactor(v1): one owner for serving resources — lean Agent, shared pi…
eligotts Jul 8, 2026
fda9a4c
feat(v1): sessions — agents interacting within each other's episodes
eligotts Jul 10, 2026
51ed225
polish(v1): session review pass — cancellation desync guard, budget h…
eligotts Jul 10, 2026
e2852fa
Merge origin/main: adopt the landed task refactor (#1948) under the t…
eligotts Jul 10, 2026
0055d60
refactor(v1): judge topologies adopt the Task.from_trace convention
eligotts Jul 10, 2026
2a74263
polish(v1): finish the best-practices sweep across examples
eligotts Jul 10, 2026
8e676a4
refactor(v1)!: make topology execution canonical
codex Jul 11, 2026
1d6d649
style: reformat the consolidation at this repo's line length
eligotts Jul 11, 2026
036d526
refactor(v1): topology selection is an EnvConfig fact, not a bolt-on
eligotts Jul 11, 2026
b1d4d24
feat(v1): Topology.complete — the topology owns its instance-validity…
eligotts Jul 11, 2026
471b69b
fix(v1): borrowed-box lifetime bugs raise to the caller; pair against…
eligotts Jul 11, 2026
5e2c72b
feat(v1): preserve graph training metadata
codex Jul 11, 2026
d1e5f5f
style: pin ruff to this repo and reformat the training-metadata commit
eligotts Jul 11, 2026
d4ac86c
docs: extend the deferred-work ledger from this session's reviews
eligotts Jul 11, 2026
44ca2e3
feat(v1): add trainable topology examples
codex Jul 12, 2026
3455a29
fix(v1): drop writer-editors judge; use deterministic shared reward
codex Jul 12, 2026
266a8a7
chore(v1): restore gsm8k/reverse-text; remove code-golf example
codex Jul 12, 2026
aec0880
chore(v1): restore echo fixtures to match main
codex Jul 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# env
.venv
.venv/
venv/
env/
Expand Down
129 changes: 2 additions & 127 deletions assets/lab/environments/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This file mirrors the "Environments" documentation page.

---

This guide walks through building environments in Verifiers, from simple single-turn tasks to complex multi-turn agents with tools. See [Overview](overview.md) for how to initialize a new environment template. For reusable taskset/harness environments, see [BYO Harness](byo-harness.md).
This guide walks through building environments in Verifiers, from simple single-turn tasks to complex multi-turn agents with tools. See [Overview](overview.md) for how to initialize a new environment template.

## Table of Contents

Expand Down Expand Up @@ -35,7 +35,6 @@ This guide walks through building environments in Verifiers, from simple single-
- [Cleanup and Teardown](#cleanup-and-teardown)
- [Signaling Early Termination](#signaling-early-termination)
- [Developing Environments](#developing-environments)
- [v1 Env Shape](#v1-env-shape)
- [pyproject.toml](#pyprojecttoml)
- [Managing Dependencies](#managing-dependencies)
- [Installation](#installation)
Expand Down Expand Up @@ -689,7 +688,6 @@ The `prime env init` command initializes a new environment project:

```bash
prime env init my-env # v0 stub
prime env init my-env --v1 # v1 Taskset/Harness template
```

This creates the following structure:
Expand All @@ -701,128 +699,6 @@ environments/my_env/
└── README.md # documentation template
```

### v1 Env Shape

The v1 template teaches the standard object layout: one taskset class, one
typed `load_taskset(config: MyTasksetConfig)` child factory, and a tiny
`load_environment(config: vf.EnvConfig)` root loader that delegates through
`vf.load_taskset(config=config.taskset)` and
`vf.load_harness(config=config.harness)`. The child factory annotation defines
the taskset config type for TOML, CLI, eval, GEPA, RL, and Hosted Training.

After `prime env init my-env --v1`, edit the generated taskset class:

1. Add task settings to `TasksetConfig`.
2. Return task records from `load_tasks(split=...)`.
3. Return task-owned tools from `load_toolsets` when needed.
4. Add lifecycle, metric, reward, and advantage methods with `@vf.*`.

Add a harness config, harness class, and `load_harness(config:
MyHarnessConfig)` when the environment owns reusable rollout behavior.
Otherwise the generated root loader uses the base harness.

`EnvConfig` is the lightweight envelope for the two child configs. Put
environment knobs on `TasksetConfig` or `HarnessConfig`.

The taskset-only shape is:

```python
import verifiers as vf


class MyTasksetConfig(vf.TasksetConfig):
system_prompt: vf.SystemPrompt = "Answer exactly."


class MyTaskset(vf.Taskset[MyTasksetConfig]):
def load_tasks(self, split: vf.TaskSplit = "train") -> vf.Tasks:
"""Return serializable task records as a list, generator, or Dataset."""
if split == "eval":
return []
return [
{
"prompt": [{"role": "user", "content": "Reverse abc."}],
"answer": "cba",
"max_turns": 1,
}
]

@vf.reward(weight=1.0)
async def correct_answer(self, task: vf.Task, state: vf.State) -> float:
messages = vf.get_messages(state.get("completion") or [], role="assistant")
if not messages:
return 0.0
response = str(messages[-1].content or "").strip()
return float(response == task["answer"])


def load_taskset(config: MyTasksetConfig) -> MyTaskset:
return MyTaskset(config=config)


def load_environment(config: vf.EnvConfig) -> vf.Env:
"""Loader pattern for all Taskset/Harness environments."""
return vf.Env(
taskset=vf.load_taskset(config=config.taskset),
harness=vf.load_harness(config=config.harness),
)
```

With a reusable harness, keep the same explicit object boundary:

```python
class MyHarnessConfig(vf.HarnessConfig):
max_turns: int = 20


class MyHarness(vf.Harness[MyHarnessConfig]):
"""Reusable execution behavior for this environment."""


def load_taskset(config: MyTasksetConfig) -> MyTaskset:
return MyTaskset(config=config)


def load_harness(config: MyHarnessConfig) -> MyHarness:
return MyHarness(config=config)


def load_environment(config: vf.EnvConfig) -> vf.Env:
"""Loader pattern for all Taskset/Harness environments."""
return vf.Env(
taskset=vf.load_taskset(config=config.taskset),
harness=vf.load_harness(config=config.harness),
)
```

Keep v1 dependencies behind the owning taskset or harness. Do not pass
already-instantiated resource objects through environment loaders. Bindings are
allowed wherever the owning taskset, toolset, user, program, or harness wires
callables. `objects` entries should be loader specs: prefer serializable import
paths in config, and use factory callables directly only for Python-only
construction when the dependency cannot be serialized. Required Taskset and
Toolset factory parameters must be supplied through bindings.

Judge-style rewards should read endpoint details from the rollout state:

```python
@vf.reward(weight=1.0)
async def judge_reward(task, state) -> float:
endpoint = state.get_endpoint_config(api="chat")
client = state.get_client(api="chat")
model = str(task.get("judge_model") or endpoint.model)
...
```

Expose at most `judge_model: str | None = None` on the taskset config. Do not
add judge endpoint URL/API-key fields or read `os.environ` inside reward/update
handlers.

For reusable tasksets and harnesses, [BYO Harness](byo-harness.md) is the
canonical v1 implementation guide. It covers ownership, configs, task controls,
system prompts, users, toolsets, programs, sandboxes, artifacts, nested
harnesses, package adapters, and TOML/CLI overrides.

### pyproject.toml

The `pyproject.toml` defines package metadata, dependencies, and evaluation defaults:
Expand Down Expand Up @@ -1017,9 +893,8 @@ Supported third-party environment integrations include:
- **`ReasoningGymEnv`** — wraps [reasoning-gym](https://github.com/open-thought/reasoning-gym) procedural datasets
- **`BrowserEnv`** — unified browser automation via [Browserbase](https://browserbase.com) with DOM and CUA modes
- **`OpenEnvEnv`** — wraps OpenEnv gym and MCP contracts using Prime Sandboxes with prebuilt images referenced from `.build.json`
- **`NeMoGymTaskset` / `NeMoGymHarness`** — packaged v1 taskset/harness adapters for NeMo Gym JSONL rows and rollout collection

These require additional dependencies installed via extras (e.g., `uv add 'verifiers[ta]'` for TextArena, `uv add 'verifiers[browser]'` for BrowserEnv, `uv add 'verifiers[openenv]'` for OpenEnvEnv, `uv add 'verifiers[nemogym]'` for NeMo Gym). The bundled OpenEnv project under `proj/` owns its server dependencies and must be built with `uv run vf-build <env-id>` before evaluation or training.
These require additional dependencies installed via extras (e.g., `uv add 'verifiers[ta]'` for TextArena, `uv add 'verifiers[browser]'` for BrowserEnv, `uv add 'verifiers[openenv]'` for OpenEnvEnv). The bundled OpenEnv project under `proj/` owns its server dependencies and must be built with `uv run vf-build <env-id>` before evaluation or training.

Newer and more experimental environment classes include:

Expand Down
129 changes: 2 additions & 127 deletions environments/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This file mirrors the "Environments" documentation page.

---

This guide walks through building environments in Verifiers, from simple single-turn tasks to complex multi-turn agents with tools. See [Overview](overview.md) for how to initialize a new environment template. For reusable taskset/harness environments, see [BYO Harness](byo-harness.md).
This guide walks through building environments in Verifiers, from simple single-turn tasks to complex multi-turn agents with tools. See [Overview](overview.md) for how to initialize a new environment template.

## Table of Contents

Expand Down Expand Up @@ -34,7 +34,6 @@ This guide walks through building environments in Verifiers, from simple single-
- [Cleanup and Teardown](#cleanup-and-teardown)
- [Signaling Early Termination](#signaling-early-termination)
- [Developing Environments](#developing-environments)
- [v1 Env Shape](#v1-env-shape)
- [pyproject.toml](#pyprojecttoml)
- [Managing Dependencies](#managing-dependencies)
- [Installation](#installation)
Expand Down Expand Up @@ -688,7 +687,6 @@ The `prime env init` command initializes a new environment project:

```bash
prime env init my-env # v0 stub
prime env init my-env --v1 # v1 Taskset/Harness template
```

This creates the following structure:
Expand All @@ -700,128 +698,6 @@ environments/my_env/
└── README.md # documentation template
```

### v1 Env Shape

The v1 template teaches the standard object layout: one taskset class, one
typed `load_taskset(config: MyTasksetConfig)` child factory, and a tiny
`load_environment(config: vf.EnvConfig)` root loader that delegates through
`vf.load_taskset(config=config.taskset)` and
`vf.load_harness(config=config.harness)`. The child factory annotation defines
the taskset config type for TOML, CLI, eval, GEPA, RL, and Hosted Training.

After `prime env init my-env --v1`, edit the generated taskset class:

1. Add task settings to `TasksetConfig`.
2. Return task records from `load_tasks(split=...)`.
3. Return task-owned tools from `load_toolsets` when needed.
4. Add lifecycle, metric, reward, and advantage methods with `@vf.*`.

Add a harness config, harness class, and `load_harness(config:
MyHarnessConfig)` when the environment owns reusable rollout behavior.
Otherwise the generated root loader uses the base harness.

`EnvConfig` is the lightweight envelope for the two child configs. Put
environment knobs on `TasksetConfig` or `HarnessConfig`.

The taskset-only shape is:

```python
import verifiers as vf


class MyTasksetConfig(vf.TasksetConfig):
system_prompt: vf.SystemPrompt = "Answer exactly."


class MyTaskset(vf.Taskset[MyTasksetConfig]):
def load_tasks(self, split: vf.TaskSplit = "train") -> vf.Tasks:
"""Return serializable task records as a list, generator, or Dataset."""
if split == "eval":
return []
return [
{
"prompt": [{"role": "user", "content": "Reverse abc."}],
"answer": "cba",
"max_turns": 1,
}
]

@vf.reward(weight=1.0)
async def correct_answer(self, task: vf.Task, state: vf.State) -> float:
messages = vf.get_messages(state.get("completion") or [], role="assistant")
if not messages:
return 0.0
response = str(messages[-1].content or "").strip()
return float(response == task["answer"])


def load_taskset(config: MyTasksetConfig) -> MyTaskset:
return MyTaskset(config=config)


def load_environment(config: vf.EnvConfig) -> vf.Env:
"""Loader pattern for all Taskset/Harness environments."""
return vf.Env(
taskset=vf.load_taskset(config=config.taskset),
harness=vf.load_harness(config=config.harness),
)
```

With a reusable harness, keep the same explicit object boundary:

```python
class MyHarnessConfig(vf.HarnessConfig):
max_turns: int = 20


class MyHarness(vf.Harness[MyHarnessConfig]):
"""Reusable execution behavior for this environment."""


def load_taskset(config: MyTasksetConfig) -> MyTaskset:
return MyTaskset(config=config)


def load_harness(config: MyHarnessConfig) -> MyHarness:
return MyHarness(config=config)


def load_environment(config: vf.EnvConfig) -> vf.Env:
"""Loader pattern for all Taskset/Harness environments."""
return vf.Env(
taskset=vf.load_taskset(config=config.taskset),
harness=vf.load_harness(config=config.harness),
)
```

Keep v1 dependencies behind the owning taskset or harness. Do not pass
already-instantiated resource objects through environment loaders. Bindings are
allowed wherever the owning taskset, toolset, user, program, or harness wires
callables. `objects` entries should be loader specs: prefer serializable import
paths in config, and use factory callables directly only for Python-only
construction when the dependency cannot be serialized. Required Taskset and
Toolset factory parameters must be supplied through bindings.

Judge-style rewards should read endpoint details from the rollout state:

```python
@vf.reward(weight=1.0)
async def judge_reward(task, state) -> float:
endpoint = state.get_endpoint_config(api="chat")
client = state.get_client(api="chat")
model = str(task.get("judge_model") or endpoint.model)
...
```

Expose at most `judge_model: str | None = None` on the taskset config. Do not
add judge endpoint URL/API-key fields or read `os.environ` inside reward/update
handlers.

For reusable tasksets and harnesses, [BYO Harness](byo-harness.md) is the
canonical v1 implementation guide. It covers ownership, configs, task controls,
system prompts, users, toolsets, programs, sandboxes, artifacts, nested
harnesses, package adapters, and TOML/CLI overrides.

### pyproject.toml

The `pyproject.toml` defines package metadata, dependencies, and evaluation defaults:
Expand Down Expand Up @@ -1016,9 +892,8 @@ Supported third-party environment integrations include:
- **`ReasoningGymEnv`** — wraps [reasoning-gym](https://github.com/open-thought/reasoning-gym) procedural datasets
- **`BrowserEnv`** — unified browser automation via [Browserbase](https://browserbase.com) with DOM and CUA modes
- **`OpenEnvEnv`** — wraps OpenEnv gym and MCP contracts using Prime Sandboxes with prebuilt images referenced from `.build.json`
- **`NeMoGymTaskset` / `NeMoGymHarness`** — packaged v1 taskset/harness adapters for NeMo Gym JSONL rows and rollout collection

These require additional dependencies installed via extras (e.g., `uv add 'verifiers[ta]'` for TextArena, `uv add 'verifiers[browser]'` for BrowserEnv, `uv add 'verifiers[openenv]'` for OpenEnvEnv, `uv add 'verifiers[nemogym]'` for NeMo Gym). The bundled OpenEnv project under `proj/` owns its server dependencies and must be built with `uv run vf-build <env-id>` before evaluation or training.
These require additional dependencies installed via extras (e.g., `uv add 'verifiers[ta]'` for TextArena, `uv add 'verifiers[browser]'` for BrowserEnv, `uv add 'verifiers[openenv]'` for OpenEnvEnv). The bundled OpenEnv project under `proj/` owns its server dependencies and must be built with `uv run vf-build <env-id>` before evaluation or training.

Newer and more experimental environment classes include:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ class AlphabetSortState(vf.State):
class AlphabetSortUser(vf.User[vf.UserConfig, AlphabetSortState]):
"""Drives the whole conversation by replaying the episode's pre-generated user turns: each
`respond` delivers the next queued turn as a user message, until the queue is exhausted (then it
flags `user_finished`, which the taskset's `@vf.stop` ends the trajectory on). The task carries no
flags `user_finished`, which the task's `@vf.stop` ends the trajectory on). The task carries no
prompt, so the first turn (the opening `respond("")`) delivers the initial sort prompt; the rest
are the follow-ups."""

Expand Down
Loading
Loading