Add reusable OpenEnv taskset support

environments/README.md

@@ -24,6 +24,7 @@ This folder contains installable example environments that showcase common usage
 - **wordle**: Game-style interaction via `TextArenaEnv`; multiple rewards (correctness, partial credit, few-turn bonus) and XML formatting.
 - **wordle_v1**: Wordle on the reusable v1 `TextArenaTaskset`, with Wordle-specific prompt, feedback, and rewards kept in the environment package.
 - **openenv_echo**: OpenEnv MCP integration example using upstream `echo_env`.
+- **openenv_echo_v1**: Thin v1 config over the reusable OpenEnv image taskset, pinned to the official Echo image.
 - **openenv_textarena**: OpenEnv gym integration example using upstream `textarena_env` (default `Wordle-v0`).
 
 ### Tool use
@@ -91,7 +92,7 @@ This folder contains installable example environments that showcase common usage
 - **ToolEnv with real tools**: `wiki_search`, `math_python`
 - **Custom MultiTurnEnv**: `alphabet_sort`, `doublecheck`, `sentence_repeater`, `wordle`
 - **GymEnv integration**: `gem_wordle`
-- **OpenEnv integration (gym + MCP)**: `openenv_textarena`, `openenv_echo`
+- **OpenEnv integration (gym + MCP)**: `openenv_textarena`, `openenv_echo`, `openenv_echo_v1`
 - **CLI agent sandboxes**: `opencode_harbor`, `terminus_harbor`, `hello_mcp_harbor`
 - **MCP integration**: `mcp_search_env`, `hello_mcp_harbor`
 - **Taskset/Harness v1**: use this pattern for new environments that need reusable tasksets, reusable harnesses, framework programs, endpoint interception, or sandboxed Python/command programs. Examples include `dspy_rlm`, `openai_agents_env`, `langchain_deep_agents_wikispeedia`, `reverse_text`, `alphabet_sort`, `wiki_search`, `math_python`, `mcp_search_env`, `opencode_harbor`, `bfcl_v3`, `hello_subagent_v1`, `nested_harness_v1`, `hello_self_judge_v1`, `hello_parallel_sandbox_v1`, `hello_group_reward_v1`, `hello_rlm_v1`, `rlm_swe_v1`, `dspy_flights`, and `wordle-v1`.

environments/openenv_echo_v1/README.md

@@ -0,0 +1,28 @@
+# openenv-echo-v1
+
+OpenEnv's official Echo image as a minimal v1 example. The reusable image lifecycle and
+JSON-RPC-to-MCP bridge live in `verifiers`; this package only pins the image, prompt, and
+resources.
+
+## Develop
+
+1. From the Verifiers checkout root, add both the current framework and this example to the
+   ephemeral `uv run` environment.
+2. Run it with any MCP-capable Verifiers harness and a container runtime:
+
+```bash
+uv run --with-editable . --with-editable environments/openenv_echo_v1 \
+  eval openenv-echo-v1 -n 1 --harness.runtime.type docker
+uv run --with-editable . --with-editable environments/openenv_echo_v1 \
+  eval openenv-echo-v1 -n 1 --harness.runtime.type docker --harness.id rlm
+uv run --with-editable . --with-editable environments/openenv_echo_v1 \
+  eval openenv-echo-v1 -n 1 --harness.runtime.type docker --harness.id codex \
+  -m nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B
+```
+
+## Layout
+
+- `openenv_echo_v1/taskset.py` — a thin config over `vf`'s reusable `OpenEnvTaskset`.
+
+Echo's production MCP contract is unscored, so the reward is neutral while the tool call and
+result remain in the Verifiers trace.

environments/openenv_echo_v1/openenv_echo_v1/__init__.py

@@ -0,0 +1,3 @@
+from openenv_echo_v1.taskset import OpenEnvEchoTaskset
+
+__all__ = ["OpenEnvEchoTaskset"]

environments/openenv_echo_v1/openenv_echo_v1/taskset.py

@@ -0,0 +1,22 @@
+"""OpenEnv's official Echo image as a zero-config example taskset."""
+
+import verifiers.v1 as vf
+from verifiers.v1.tasksets.openenv import OpenEnvConfig, OpenEnvTaskset
+
+ECHO_IMAGE = (
+    "ghcr.io/meta-pytorch/openenv-echo-env@"
+    "sha256:56c55669c00b23a6af6adbcd8dd1fb5da3a276aec186b5c46cb4abeb708afa9c"
+)
+
+
+class OpenEnvEchoConfig(OpenEnvConfig):
+    image: str = ECHO_IMAGE
+    prompt: str = (
+        'Call the echo_message tool with the message "Hello, World!", then return '
+        "the echoed text."
+    )
+    resources: vf.TaskResources = vf.TaskResources(cpu=2, memory=4, disk=10)
+
+
+class OpenEnvEchoTaskset(OpenEnvTaskset, vf.Taskset[vf.Task, OpenEnvEchoConfig]):
+    pass

environments/openenv_echo_v1/pyproject.toml

@@ -0,0 +1,13 @@
+[project]
+name = "openenv-echo-v1"
+version = "0.1.0"
+description = "openenv-echo-v1 — <one-line description>."
+requires-python = ">=3.11"
+dependencies = ["verifiers"]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["openenv_echo_v1"]

tests/v1/test_envs.py

@@ -20,9 +20,10 @@
 ENVIRONMENTS = Path(__file__).parent.parent.parent / "environments"
 
 # v1 tasksets that can't run a plain-CI smoke eval — e.g. they need a docker/prime runtime or
-# clone a corpus CI can't read. Empty: the SWE/container and corpus tasksets live in
-# research-environments now.
-SKIP_EVAL: set[str] = set()
+# clone a corpus CI can't read.
+SKIP_EVAL = {
+    "openenv_echo_v1",  # requires its pinned OpenEnv container image
+}
 
 
 def v1_tasksets() -> list[str]:

tests/v1/test_openenv.py

@@ -0,0 +1,205 @@
+from pathlib import Path
+from unittest.mock import AsyncMock, Mock
+
+import httpx
+import pytest
+from mcp.server.fastmcp import FastMCP
+
+import verifiers.v1 as vf
+from verifiers.v1.env import EnvConfig, Environment
+from verifiers.v1.runtimes import ProgramResult
+from verifiers.v1.runtimes.subprocess import SubprocessConfig, SubprocessRuntime
+from verifiers.v1.tasksets.openenv import OpenEnvConfig, OpenEnvTaskset
+from verifiers.v1.tasksets.openenv.taskset import (
+    OPENENV_SOCKET,
+    OPENENV_URL,
+)
+
+ECHO_EXAMPLE = Path(__file__).parents[2] / "environments" / "openenv_echo_v1"
+
+
+@pytest.mark.asyncio
+async def test_uv_script_ignores_image_system_python() -> None:
+    runtime = SubprocessRuntime(SubprocessConfig())
+    runtime.write = AsyncMock()
+    runtime.run = AsyncMock(
+        side_effect=[
+            ProgramResult(exit_code=0, stdout="", stderr=""),
+            ProgramResult(
+                exit_code=0, stdout="/tmp/openenv-test/bin/python\n", stderr=""
+            ),
+        ]
+    )
+
+    await runtime.prepare_uv_script("# openenv image system-python regression\n")
+
+    prepare_command = runtime.run.await_args_list[1].args[0][2]
+    assert "; unset UV_SYSTEM_PYTHON; uv sync" in prepare_command
+
+
+@pytest.mark.parametrize("harness_id", [None, "rlm", "codex"])
+def test_openenv_uses_an_external_mcp_harness(harness_id: str | None) -> None:
+    harness: dict[str, object] = {"runtime": {"type": "docker"}}
+    if harness_id:
+        harness["id"] = harness_id
+    config = EnvConfig.model_validate(
+        {
+            "taskset": {
+                "id": "openenv",
+                "image": "openenv:test",
+                "prompt": "Use the available tool.",
+                "resources": {"cpu": 2, "memory": 4, "disk": 10},
+            },
+            "harness": harness,
+        }
+    )
+    env = Environment(config)
+    task = env.taskset.load_tasks()[0]
+    toolset = env.taskset.tools(task)[0]
+
+    assert env.harness.config.id == (harness_id or "default")
+    assert OpenEnvConfig.model_fields["image"].is_required()
+    assert OpenEnvConfig.model_fields["prompt"].is_required()
+    assert task.name == "openenv"
+    assert task.image == "openenv:test"
+    assert task.workdir == "/app/env"
+    assert task.resources.cpu == 2
+    assert task.resources.memory == 4
+    assert task.resources.disk == 10
+    assert isinstance(toolset, vf.JSONRPCToolset)
+    assert toolset.config.colocated is True
+    assert toolset.config.endpoint == f"{OPENENV_URL}/mcp"
+    assert toolset.config.uds == OPENENV_SOCKET
+
+
+def test_echo_image_is_owned_by_the_example(monkeypatch) -> None:
+    monkeypatch.syspath_prepend(str(ECHO_EXAMPLE))
+    from openenv_echo_v1.taskset import (
+        ECHO_IMAGE,
+        OpenEnvEchoConfig,
+        OpenEnvEchoTaskset,
+    )
+
+    taskset = OpenEnvEchoTaskset(OpenEnvEchoConfig(id="openenv-echo-v1"))
+    task = taskset.load_tasks()[0]
+
+    assert task.name == "openenv-echo-v1"
+    assert task.image == ECHO_IMAGE
+    assert task.prompt == (
+        'Call the echo_message tool with the message "Hello, World!", then return '
+        "the echoed text."
+    )
+    assert task.resources.cpu == 2
+    assert task.resources.memory == 4
+    assert task.resources.disk == 10
+
+
+@pytest.mark.asyncio
+async def test_openenv_setup_only_starts_its_server() -> None:
+    taskset = OpenEnvTaskset(
+        OpenEnvConfig(id="openenv", image="openenv:test", prompt="Use the tool.")
+    )
+    runtime = AsyncMock()
+
+    await taskset.setup(taskset.load_tasks()[0], runtime)
+
+    runtime.run_background.assert_awaited_once_with(
+        [
+            "/app/.venv/bin/uvicorn",
+            "server.app:app",
+            "--uds",
+            OPENENV_SOCKET,
+        ],
+        {"ENABLE_WEB_INTERFACE": "false"},
+        "/tmp/openenv.log",
+    )
+
+
+@pytest.mark.asyncio
+async def test_codex_harness_receives_openenv_mcp_server() -> None:
+    from verifiers.v1.harnesses.codex import CodexHarness, CodexHarnessConfig
+
+    runtime = Mock()
+    runtime.run_program = AsyncMock(
+        return_value=ProgramResult(exit_code=0, stdout="", stderr="")
+    )
+    harness = CodexHarness(CodexHarnessConfig(id="codex"))
+
+    await harness.launch(
+        vf.RolloutContext(
+            model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B",
+            client=Mock(),
+            sampling=vf.SamplingConfig(),
+        ),
+        vf.Trace(task=vf.Task(idx=0, prompt="Use the Echo tool.")),
+        runtime,
+        endpoint="http://127.0.0.1:9000/v1",
+        secret="secret",
+        mcp_urls={"jsonrpc": "http://127.0.0.1:12345/mcp"},
+    )
+
+    argv, _ = runtime.run_program.await_args.args
+    assert CodexHarness.SUPPORTS_MCP is True
+    assert harness.config.version == "0.116.0"
+    assert 'mcp_servers.jsonrpc.url="http://127.0.0.1:12345/mcp"' in argv
+
+
+@pytest.mark.asyncio
+async def test_jsonrpc_toolset_bridges_tools_to_mcp(monkeypatch) -> None:
+    responses = iter(
+        [
+            {
+                "jsonrpc": "2.0",
+                "id": 1,
+                "result": {
+                    "tools": [
+                        {
+                            "name": "echo",
+                            "description": "Echo text.",
+                            "inputSchema": {
+                                "type": "object",
+                                "properties": {"message": {"type": "string"}},
+                                "required": ["message"],
+                            },
+                        }
+                    ]
+                },
+            },
+            {"jsonrpc": "2.0", "id": 1, "result": {"data": "hello"}},
+        ]
+    )
+    requests = []
+
+    class Client:
+        def __init__(self, **kwargs):
+            pass
+
+        async def __aenter__(self):
+            return self
+
+        async def __aexit__(self, *args):
+            pass
+
+        async def post(self, url, json):
+            requests.append((url, json))
+            return httpx.Response(
+                200,
+                json=next(responses),
+                request=httpx.Request("POST", url),
+            )
+
+    monkeypatch.setattr("verifiers.v1.mcp.toolset.httpx.AsyncClient", Client)
+    toolset = vf.JSONRPCToolset(vf.JSONRPCToolsetConfig(endpoint="http://openenv/mcp"))
+    await toolset.setup()
+
+    mcp = FastMCP("test")
+    toolset._register(mcp)
+    registered = await mcp.list_tools()
+    result = await mcp.call_tool("echo", {"message": "hello"})
+
+    assert registered[0].inputSchema["required"] == ["message"]
+    assert "hello" in result[0].text
+    assert requests[-1][1]["params"] == {
+        "name": "echo",
+        "arguments": {"message": "hello"},
+    }

verifiers/v1/GUIDE.md

@@ -470,6 +470,10 @@ Two setup hooks, plus the shared state:
 A taskset exposes a task's tools via `tools(task) -> list[vf.Toolset]`, constructing each from a
 config field so placement is CLI-tunable (below).
 
+For an existing JSON-RPC endpoint that exposes `tools/list` and `tools/call` but is not a
+streamable-HTTP MCP server, return a `vf.JSONRPCToolset`. It discovers the remote schemas and
+re-exposes the calls as ordinary MCP tools; `uds` optionally connects through a Unix socket.
+
 ## User simulators
 
 A user simulator is a `vf.User[ConfigT]` (or `[ConfigT, StateT]`) with one hook:

verifiers/v1/README.md

@@ -82,6 +82,7 @@ Taskset examples (the `*_v1` packages under `environments/`):
 | `wiki-search-v1` | a **shared** tool server (built once for the eval) + an LLM judge |
 | `deepwiki-v1` | an **existing remote** tool server, by URL |
 | `wordle-v1` | configuring the vendored `textarena` integration |
+| `openenv-echo-v1` | a thin config over the reusable OpenEnv image taskset |
 
 Harness examples (under `environments/`):
 
@@ -112,6 +113,28 @@ uv run eval harbor -n 1 --taskset.ignore-dockerfile --harness.runtime.type docke
 uv run eval harbor -n 1 --taskset.ignore-dockerfile --harness.runtime.type docker --harness.id codex           # the codex CLI agent
 ```
 
+### OpenEnv
+
+The built-in `OpenEnvTaskset` maps a configured image, prompt, workdir, and resources onto an
+ordinary task. It starts the image's OpenEnv server and exposes its JSON-RPC tools through
+`vf.JSONRPCToolset`; the selected Verifiers harness owns the agent loop, OpenEnv owns tool
+execution, and the ordinary model/tool messages form the trace. It deliberately has no default
+image or OpenEnv-specific harness.
+
+`openenv-echo-v1` is the example environment that pins OpenEnv's official Echo image and its
+hello-world prompt:
+
+```bash
+uv run --with-editable . --with-editable environments/openenv_echo_v1 \
+  eval openenv-echo-v1 -n 1 --harness.runtime.type docker
+uv run --with-editable . --with-editable environments/openenv_echo_v1 \
+  eval openenv-echo-v1 -n 1 --harness.runtime.type docker --harness.id rlm
+```
+
+Use any harness with `SUPPORTS_MCP`. Echo's production MCP contract is intentionally unscored,
+so this smoke test returns a neutral `0.0` reward while retaining the hello-world tool call and
+result in the Verifiers trace.
+
 ### Swappable runtime
 
 Where code runs, behind one `Runtime` contract — the same contract backs the harness
@@ -133,9 +156,8 @@ guaranteed cleanup of its resources, even on exit/interrupt.
 A taskset may expose task-specific tools beyond the tools shipping natively with
 the harness as MCP servers. Its placement (separate runtime or colocated with
 harness) is configurable on `taskset.tools` and reachability is handled resolved
-automatically. Tools only run under a harness with `SUPPORTS_MCP` (the `default`
-harness has it; `rlm` doesn't) — an incompatible pairing is refused at load. The tool examples
-each show one placement:
+automatically. Tools only run under a harness with `SUPPORTS_MCP` (including `default`, `codex`,
+and `rlm`) — an incompatible pairing is refused at load. The tool examples each show one placement:
 
 ```bash
 uv run eval glossary-v1 -n 1     # colocated — in the harness's own runtime, localhost (default)

verifiers/v1/__init__.py

@@ -63,6 +63,8 @@
 from verifiers.v1.task import Task, TaskResources, TaskTimeout, WireTask
 from verifiers.v1.taskset import Taskset, TasksetConfig
 from verifiers.v1.mcp import (
+    JSONRPCToolset,
+    JSONRPCToolsetConfig,
     Toolset,
     ToolsetConfig,
     User,
@@ -196,6 +198,8 @@
     "JudgeSamplingConfig",
     "JudgeResponse",
     # mcp
+    "JSONRPCToolset",
+    "JSONRPCToolsetConfig",
     "Toolset",
     "ToolsetConfig",
     # user simulator