Add reusable OpenEnv taskset support

environments/README.md

@@ -24,6 +24,7 @@ This folder contains installable example environments that showcase common usage
 - **wordle**: Game-style interaction via `TextArenaEnv`; multiple rewards (correctness, partial credit, few-turn bonus) and XML formatting.
 - **wordle_v1**: Wordle on the reusable v1 `TextArenaTaskset`, with Wordle-specific prompt, feedback, and rewards kept in the environment package.
 - **openenv_echo**: OpenEnv MCP integration example using upstream `echo_env`.
+- **openenv_echo_v1**: Thin v1 config over the reusable OpenEnv image taskset, pinned to the official Echo image.
 - **openenv_textarena**: OpenEnv gym integration example using upstream `textarena_env` (default `Wordle-v0`).
 
 ### Tool use
@@ -91,7 +92,7 @@ This folder contains installable example environments that showcase common usage
 - **ToolEnv with real tools**: `wiki_search`, `math_python`
 - **Custom MultiTurnEnv**: `alphabet_sort`, `doublecheck`, `sentence_repeater`, `wordle`
 - **GymEnv integration**: `gem_wordle`
-- **OpenEnv integration (gym + MCP)**: `openenv_textarena`, `openenv_echo`
+- **OpenEnv integration (gym + MCP)**: `openenv_textarena`, `openenv_echo`, `openenv_echo_v1`
 - **CLI agent sandboxes**: `opencode_harbor`, `terminus_harbor`, `hello_mcp_harbor`
 - **MCP integration**: `mcp_search_env`, `hello_mcp_harbor`
 - **Taskset/Harness v1**: use this pattern for new environments that need reusable tasksets, reusable harnesses, framework programs, endpoint interception, or sandboxed Python/command programs. Examples include `dspy_rlm`, `openai_agents_env`, `langchain_deep_agents_wikispeedia`, `reverse_text`, `alphabet_sort`, `wiki_search`, `math_python`, `mcp_search_env`, `opencode_harbor`, `bfcl_v3`, `hello_subagent_v1`, `nested_harness_v1`, `hello_self_judge_v1`, `hello_parallel_sandbox_v1`, `hello_group_reward_v1`, `hello_rlm_v1`, `rlm_swe_v1`, `dspy_flights`, and `wordle-v1`.

environments/openenv_echo_v1/README.md

@@ -0,0 +1,28 @@
+# openenv-echo-v1
+
+OpenEnv's official Echo image as a minimal v1 example. The reusable image lifecycle and
+JSON-RPC-to-MCP bridge live in `verifiers`; this package only pins the image, prompt, and
+resources.
+
+## Develop
+
+1. From the Verifiers checkout root, add both the current framework and this example to the
+   ephemeral `uv run` environment.
+2. Run it with any MCP-capable Verifiers harness and a container runtime:
+
+```bash
+uv run --with-editable . --with-editable environments/openenv_echo_v1 \
+  eval openenv-echo-v1 -n 1 --harness.runtime.type docker
+uv run --with-editable . --with-editable environments/openenv_echo_v1 \
+  eval openenv-echo-v1 -n 1 --harness.runtime.type docker --harness.id rlm
+uv run --with-editable . --with-editable environments/openenv_echo_v1 \
+  eval openenv-echo-v1 -n 1 --harness.runtime.type docker --harness.id codex \
+  -m nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B
+```
+
+## Layout
+
+- `openenv_echo_v1/taskset.py` — a thin config over `vf`'s reusable `OpenEnvTaskset`.
+
+Echo's production MCP contract is unscored, so the reward is neutral while the tool call and
+result remain in the Verifiers trace.

environments/openenv_echo_v1/openenv_echo_v1/__init__.py

@@ -0,0 +1,3 @@
+from openenv_echo_v1.taskset import OpenEnvEchoTaskset
+
+__all__ = ["OpenEnvEchoTaskset"]

environments/openenv_echo_v1/openenv_echo_v1/taskset.py

@@ -0,0 +1,22 @@
+"""OpenEnv's official Echo image as a zero-config example taskset."""
+
+import verifiers.v1 as vf
+from verifiers.v1.tasksets.openenv import OpenEnvConfig, OpenEnvTaskset
+
+ECHO_IMAGE = (
+    "ghcr.io/meta-pytorch/openenv-echo-env@"
+    "sha256:56c55669c00b23a6af6adbcd8dd1fb5da3a276aec186b5c46cb4abeb708afa9c"
+)
+
+
+class OpenEnvEchoConfig(OpenEnvConfig):
+    image: str = ECHO_IMAGE
+    prompt: str = (
+        'Call the echo_message tool with the message "Hello, World!", then return '
+        "the echoed text."
+    )
+    resources: vf.TaskResources = vf.TaskResources(cpu=2, memory=4, disk=10)
+
+
+class OpenEnvEchoTaskset(OpenEnvTaskset, vf.Taskset[vf.Task, OpenEnvEchoConfig]):
+    pass

environments/openenv_echo_v1/pyproject.toml

@@ -0,0 +1,13 @@
+[project]
+name = "openenv-echo-v1"
+version = "0.1.0"
+description = "openenv-echo-v1 — <one-line description>."
+requires-python = ">=3.11"
+dependencies = ["verifiers"]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["openenv_echo_v1"]

tests/v1/test_envs.py

@@ -20,9 +20,10 @@
 ENVIRONMENTS = Path(__file__).parent.parent.parent / "environments"
 
 # v1 tasksets that can't run a plain-CI smoke eval — e.g. they need a docker/prime runtime or
-# clone a corpus CI can't read. Empty: the SWE/container and corpus tasksets live in
-# research-environments now.
-SKIP_EVAL: set[str] = set()
+# clone a corpus CI can't read.
+SKIP_EVAL = {
+    "openenv_echo_v1",  # requires its pinned OpenEnv container image
+}
 
 
 def v1_tasksets() -> list[str]:

verifiers/v1/GUIDE.md

@@ -470,6 +470,10 @@ Two setup hooks, plus the shared state:
 A taskset exposes a task's tools via `tools(task) -> list[vf.Toolset]`, constructing each from a
 config field so placement is CLI-tunable (below).
 
+For an existing JSON-RPC endpoint that exposes `tools/list` and `tools/call` but is not a
+streamable-HTTP MCP server, return a `vf.JSONRPCToolset`. It discovers the remote schemas and
+re-exposes the calls as ordinary MCP tools; `uds` optionally connects through a Unix socket.
+
 ## User simulators
 
 A user simulator is a `vf.User[ConfigT]` (or `[ConfigT, StateT]`) with one hook:

verifiers/v1/README.md

@@ -82,6 +82,7 @@ Taskset examples (the `*_v1` packages under `environments/`):
 | `wiki-search-v1` | a **shared** tool server (built once for the eval) + an LLM judge |
 | `deepwiki-v1` | an **existing remote** tool server, by URL |
 | `wordle-v1` | configuring the vendored `textarena` integration |
+| `openenv-echo-v1` | a thin config over the reusable OpenEnv image taskset |
 
 Harness examples (under `environments/`):
 
@@ -112,6 +113,30 @@ uv run eval harbor -n 1 --taskset.ignore-dockerfile --harness.runtime.type docke
 uv run eval harbor -n 1 --taskset.ignore-dockerfile --harness.runtime.type docker --harness.id codex           # the codex CLI agent
 ```
 
+### OpenEnv
+
+The built-in `OpenEnvTaskset` maps a configured image, OpenEnv contract, workdir, and resources
+onto an ordinary task. It deliberately has no default image or OpenEnv-specific harness.
+
+- `contract = "mcp"` starts the image's OpenEnv server and exposes its JSON-RPC tools through
+  `vf.JSONRPCToolset`; use any harness with `SUPPORTS_MCP`.
+- `contract = "gym"` starts the same server, resets it over OpenEnv's persistent WebSocket API,
+  renders observations as user turns, and sends assistant JSON actions into `step`; use a harness
+  with `SUPPORTS_USER_SIM`.
+
+`openenv-echo-v1` is the example environment that pins OpenEnv's official Echo image and its
+hello-world prompt:
+
+```bash
+uv run --with-editable . --with-editable environments/openenv_echo_v1 \
+  eval openenv-echo-v1 -n 1 --harness.runtime.type docker
+uv run --with-editable . --with-editable environments/openenv_echo_v1 \
+  eval openenv-echo-v1 -n 1 --harness.runtime.type docker --harness.id rlm
+```
+
+Echo's production MCP contract is intentionally unscored, so this smoke test returns a neutral
+`0.0` reward while retaining the hello-world tool call and result in the Verifiers trace.
+
 ### Swappable runtime
 
 Where code runs, behind one `Runtime` contract — the same contract backs the harness
@@ -133,9 +158,8 @@ guaranteed cleanup of its resources, even on exit/interrupt.
 A taskset may expose task-specific tools beyond the tools shipping natively with
 the harness as MCP servers. Its placement (separate runtime or colocated with
 harness) is configurable on `taskset.tools` and reachability is handled resolved
-automatically. Tools only run under a harness with `SUPPORTS_MCP` (the `default`
-harness has it; `rlm` doesn't) — an incompatible pairing is refused at load. The tool examples
-each show one placement:
+automatically. Tools only run under a harness with `SUPPORTS_MCP` (including `default`, `codex`,
+and `rlm`) — an incompatible pairing is refused at load. The tool examples each show one placement:
 
 ```bash
 uv run eval glossary-v1 -n 1     # colocated — in the harness's own runtime, localhost (default)

verifiers/v1/__init__.py

@@ -63,6 +63,8 @@
 from verifiers.v1.task import Task, TaskResources, TaskTimeout, WireTask
 from verifiers.v1.taskset import Taskset, TasksetConfig
 from verifiers.v1.mcp import (
+    JSONRPCToolset,
+    JSONRPCToolsetConfig,
     Toolset,
     ToolsetConfig,
     User,
@@ -196,6 +198,8 @@
     "JudgeSamplingConfig",
     "JudgeResponse",
     # mcp
+    "JSONRPCToolset",
+    "JSONRPCToolsetConfig",
     "Toolset",
     "ToolsetConfig",
     # user simulator

verifiers/v1/env.py

@@ -231,24 +231,17 @@ def resolve_runtime_config(
 class Environment:
     def __init__(self, config: EnvConfig) -> None:
         from verifiers.v1.loaders import load_harness, load_taskset
-        from verifiers.v1.taskset import Taskset
 
         self.config = config
         self.taskset = load_taskset(config.taskset)
         self.harness = load_harness(config.harness)
-        if (
-            not self.harness.SUPPORTS_MCP
-            and type(self.taskset).tools is not Taskset.tools
-        ):
+        if not self.harness.SUPPORTS_MCP and self.taskset.has_tools():
             raise ValueError(
                 f"Harness {self.harness.config.id!r} does not support MCP tools, but taskset "
                 f"{self.taskset.config.id!r} exposes tool servers (MCP). Run it with a harness "
                 f"that supports MCP (e.g. --harness.id default), or use a taskset without tools."
             )
-        if (
-            not self.harness.SUPPORTS_USER_SIM
-            and type(self.taskset).user is not Taskset.user
-        ):
+        if not self.harness.SUPPORTS_USER_SIM and self.taskset.has_user():
             raise ValueError(
                 f"Harness {self.harness.config.id!r} does not drive a user simulator, but taskset "
                 f"{self.taskset.config.id!r} defines one (Taskset.user). Run it with a harness that "

verifiers/v1/harnesses/codex/harness.py

@@ -2,11 +2,12 @@
 
 Codex only speaks the streaming OpenAI Responses API, so it reaches the interception server as a
 custom model provider (`wire_api = responses`) pointed at the rollout endpoint — served by the
-Responses dialect + SSE relay (see `verifiers.v1.dialects.responses`). The binary is the static
-musl release, so it drops into any linux container with no runtime deps; its bearer token (the
-session secret) is read from an env var.
+Responses dialect + SSE relay (see `verifiers.v1.dialects.responses`). Task MCP servers are passed
+as Codex MCP config. The binary is the static musl release, so it drops into any linux container
+with no runtime deps; its bearer token (the session secret) is read from an env var.
 """
 
+import json
 import logging
 import shlex
 
@@ -39,13 +40,14 @@
 class CodexHarnessConfig(HarnessConfig):
     """The Codex CLI harness — which codex release to install in the runtime."""
 
-    version: str = "0.137.0"
-    """Codex release to install (the `rust-v<version>` GitHub release); pinned for reproducibility."""
+    version: str = "0.116.0"
+    """Codex release to install. The last release before custom-provider MCP namespace calls
+    regressed in 0.117."""
 
 
 class CodexHarness(Harness[CodexHarnessConfig]):
     APPENDS_SYSTEM_PROMPT = False  # TODO
-    SUPPORTS_MCP = False  # TODO
+    SUPPORTS_MCP = True
 
     async def setup(self, runtime: Runtime) -> None:
         logger.info("codex: ensuring codex %s is installed", self.config.version)
@@ -83,6 +85,11 @@ async def launch(
             for tool in self.config.disabled_tools or []
             for arg in ("--disable", tool)
         ]
+        mcp_config = [
+            arg
+            for name, url in mcp_urls.items()
+            for arg in ("-c", f"mcp_servers.{name}.url={json.dumps(url)}")
+        ]
         # `-c` values parse as TOML, falling back to a raw string (so the url / `responses`
         # come through literally); `requires_openai_auth=false` parses as a bool.
         argv = [
@@ -104,6 +111,7 @@ async def launch(
             f"model_providers.{PROVIDER}.wire_api=responses",
             "-c",
             f"model_providers.{PROVIDER}.requires_openai_auth=false",
+            *mcp_config,
             *tool_config,
             prompt,
         ]

verifiers/v1/mcp/__init__.py

@@ -16,11 +16,18 @@
     serve_user,
 )
 from verifiers.v1.mcp.server import ServerBase
-from verifiers.v1.mcp.toolset import Toolset, ToolsetConfig
+from verifiers.v1.mcp.toolset import (
+    JSONRPCToolset,
+    JSONRPCToolsetConfig,
+    Toolset,
+    ToolsetConfig,
+)
 from verifiers.v1.mcp.user import User, UserConfig
 
 __all__ = [
     "ServerBase",
+    "JSONRPCToolset",
+    "JSONRPCToolsetConfig",
     "Toolset",
     "ToolsetConfig",
     "User",