feat(v1): GEPA prompt optimization for native v1 environments

[Ziems]

What

Adds GEPA (Genetic-Pareto) system-prompt optimization for native v1 environments as a new gepa CLI command. The existing v0 path (vf-gepa / verifiers/gepa/) is untouched.

uv run gepa --taskset.id reverse-text-v1 --model google/gemini-3-flash-preview --num-train 20 --num-val 10

Why

GEPA only worked against the legacy v0 Environment.generate() API. v1 composes Taskset + Harness + Environment with no generate() and no dataset split, so there was no way to optimize a native v1 taskset. This wires GEPA into the v1 execution path (Environment.episode() / Episode.run() / Trace).

How

• verifiers/v1/gepa/ — a self-contained package (adapter, config, dataset, reflection, runner); the CLI is a thin cli/gepa.py entrypoint. Imports nothing from v0.
• run_gepa is async like run_eval: it holds env.serving() open with an async with and runs the blocking gepa.optimize() via asyncio.to_thread, so the loop stays free. GEPAv1Adapter (GEPA's evaluate / make_reflective_dataset protocol) marshals each rollout batch back onto the loop with run_coroutine_threadsafe — the one sync↔async hop the synchronous third-party gepa API forces. Candidates are injected via Task.model_copy(update={"system_prompt": ...}).
• GEPAConfig inherits EnvConfig (like EvalConfig); parsing is pure pydantic_config.cli(GEPAConfig) — it resolves the taskset/harness subconfigs and renders -h itself, no argv rewriting.
• Output matches eval: every rollout's Trace streams to results.jsonl via the same on_complete → append_trace hook run_eval uses (each trace records its candidate in task.system_prompt); per-iteration progress logs through v1 logging; the best prompt prints to stdout.
• Task selection uses a shared verifiers/v1/utils/sampling.py::sample_tasks (also adopted by run_eval / run_eval_server / validate / debug / the v0 bridge); split_tasks takes two disjoint train/val slices from it.

Scope / non-goals

• v1-native only — a legacy v0 --id env is rejected (use vf-gepa).
• No multi-taskset optimization (v0's EnvGroup has no v1 equivalent); single taskset per run.
• No env-server / --server worker pool — GEPA runs in-process.
• Harnesses with no system-message input (APPENDS_SYSTEM_PROMPT=False, e.g. coding agents) warn (the candidate folds into the user prompt) rather than block. Tasksets that bake instructions into prompt instead of Task.system_prompt (e.g. gsm8k-v1) need --initial-prompt.

Testing

• Verified live end-to-end against echo-v1 and reverse-text-v1: rollouts stream to results.jsonl, all traces scored with no errors, and GEPA proposes/optimizes the system prompt.
• ruff + pre-commit clean.
• Unit tests were removed for now (see review thread) — to be re-added against the reworked v1-native shape.

Files

• verifiers/v1/gepa/ — adapter, config, dataset, reflection, runner
• verifiers/v1/cli/gepa.py — the gepa entrypoint
• verifiers/v1/utils/sampling.py — shared sample_tasks helper
• pyproject.toml — register the gepa console script

Open question: docs & skill (feedback wanted)

Bugbot flagged that this doesn't update the user-facing docs (docs/) or the GEPA skill (skills/optimize-with-environments/SKILL.md), both of which currently describe the v0 prime gepa run / vf-gepa flow. I deliberately left those out of this PR and want a maintainer's call on where they should go, because I'm not sure of the intended structure:

• docs/training.md has a "Prompt Optimization with prime gepa run" section. Should the v1 gepa command be (a) a new subsection there, (b) folded into the existing section as a v0/v1 split, or (c) documented wherever the other v1 CLIs (eval, serve, replay) are described? I couldn't find an existing v1-CLI doc home to match.
• skills/optimize-with-environments/SKILL.md is v0-centric (prime gepa run, v0 env config). Should it gain a parallel v1 path, be rewritten around gepa + GEPAv1Config, or stay v0-only for now?

Happy to do the doc/skill update in this PR or a follow-up once we agree on placement — just let me know the preferred structure.

🤖 Generated with Claude Code

---

Note

Medium Risk
New feature with many LLM rollout/reflection API calls and async/sync bridging in the adapter; no changes to auth or v0 GEPA path, but misconfiguration can burn large eval budgets.

Overview
Adds a gepa console entrypoint (alongside existing vf-gepa for v0) that runs Genetic-Pareto optimization of Task.system_prompt on native v1 tasksets via Environment.episode() / Trace rewards.

GEPAv1Adapter implements GEPA’s sync adapter protocol: candidates are injected with Task.model_copy, rollouts run under a single persistent event loop that keeps env.serving() open for the whole optimize() call, and make_reflective_dataset builds teacher-LM records from traces. GEPAv1Config extends EnvConfig (like eval) with model, reflection model, train/val sizes, metric budget, and output options; the CLI mirrors eval resolution and rejects legacy v0 --id envs.

Train/val splits are carved from taskset.load_tasks() with seedable shuffle; seed prompts come from --initial-prompt or the first task system_prompt, with warnings when the harness does not append a real system message. The runner wires gepa.api.optimize, sync reflection LM calls from v1 client config, GEPADisplay, and v0 save_gepa_results under outputs/gepa/.... tests/v1/test_gepa.py covers splits, adapter behavior, config/CLI guards, and an optional e2e run.

^{Reviewed by Cursor Bugbot for commit 8629d8f. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add GEPA prompt optimization CLI for native v1 environments

• Introduces a gepa CLI command (via verifiers/v1/cli/gepa/) that runs GEPA prompt optimization over verifiers v1 environments, replacing the legacy vf-gepa path for v0 environments.
• GEPAv1Adapter wraps the v1 environment to evaluate candidate system prompts with controlled concurrency, collecting scores and traces for each candidate.
• GEPAv1Config provides a Pydantic-validated config model with fields for model, client, train/val split sizes, budget, reflection model, seed prompt, and output paths.
• run_gepa orchestrates the full pipeline: task loading, seed prompt resolution, train/val splitting, optimization via gepa.api.optimize, result saving, and display lifecycle.
• The CLI supports --dry-run (writes config only), positional taskset selection, and rejects legacy v0 environments with a redirect to vf-gepa.

Changes since #1952 opened

• Refactored verifiers.v1.gepa to run asynchronously with run_gepa coroutine that manages shared environment serving, streams traces to results.jsonl via an on_complete callback during optimization, offloads blocking gepa.api.optimize to a worker thread via asyncio.to_thread, and transformed GEPAv1Adapter from a context-managed class to a dataclass that accepts an event loop, optional semaphore, and optional on_complete callback instead of managing its own serving lifecycle [4b7e9be]
• Restructured verifiers.v1.cli.gepa from a package with multiple modules to a single module with a new main function that parses GEPAConfig via prime-pydantic-config, rejects legacy environments, supports --dry-run mode with config writing, installs SIGTERM handler, constructs Environment, and runs run_gepa via asyncio.run [4b7e9be]
• Standardized task sampling across run_debug, run_eval, run_eval_server, run_validate, and run_legacy_eval coroutines by replacing inline manual shuffling with random.Random(0) and slicing logic with calls to new verifiers.v1.utils.sampling.sample_tasks utility that uses fixed seed _SHUFFLE_SEED=0 [4b7e9be]
• Renamed GEPAv1Config to GEPAConfig, replaced run_dir field with output_dir (aliased to o), and updated type signatures in build_reflection_lm function to reference the renamed configuration class [4b7e9be]
• Reformatted code for readability [81bd1e0]

📊 Macroscope summarized 8629d8f. 8 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 8629d8f. Configure here.}

[cursor]

Missing user-facing documentation

Medium Severity

This change adds a new core gepa console entrypoint for native v1 tasksets, but the user-facing docs under docs/ (for example evaluation workflow and reference material) are not updated to describe uv run gepa, how it differs from vf-gepa, or its config flags.

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 8629d8f. Configure here.}

[Ziems]

Intentionally deferred — see the "Open question: docs & skill" section in the PR description. I wasn't sure where the v1 gepa docs belong (new subsection in docs/training.md, a v0/v1 split of the existing section, or a v1-CLI doc home), so I've asked a maintainer to pick the structure and will add it here or in a follow-up.

[cursor]

GEPA skill not updated

Medium Severity

The new v1 gepa command and its config contract are not reflected in skills/optimize-with-environments/SKILL.md, which still centers on prime gepa run and v0-style env config rather than uv run gepa with GEPAv1Config.

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 8629d8f. Configure here.}

[Ziems]

Intentionally deferred — see the "Open question: docs & skill" section in the PR description. Whether SKILL.md should gain a parallel v1 path or be rewritten around gepa + GEPAv1Config is a maintainer call; I'll update it here or in a follow-up once placement is agreed.

[macroscopeapp]

Approvability

Verdict: Needs human review

2 blocking correctness issues found. This PR adds a complete new GEPA prompt optimization CLI for v1 environments (~500+ lines across 7 new files). New features of this scope warrant human review. Additionally, unresolved comments identify potential resource leaks and logic issues that should be addressed.

^{You can customize Macroscope's approvability policy. Learn more.}

[mikasenghaas]

same here, let's make the output format v1 native (trace)

[Ziems]

Done. Leaving this one open for discussion though. Right now it saves all rollouts from all prompt candidates, so theres lots of data but its not super useful. For GEPA to be properly useful you really care about:

1. How it performs on the valset with the default prompt
2. How it performs on the valset with the optimized prompt

So I think saving these two things and their associated rollouts is probably more useful than saving all of the rollouts done during optimization.

[xeophon]

inherit these funcs from v1 vs. reimplementing them?

[Ziems]

I made a helper since the shuffle and split pattern is used pretty often in GEPA and elsewhere. Not sure if this is what you meant. Leaving this one open for discussion.

[macroscopeapp]

🟠 High cli/gepa.py:25

A bare command like uv run gepa reverse-text-v1 --model ... silently uses the default taskset.id instead of selecting the requested taskset, so the run targets the wrong (or empty) taskset. Unlike the other v1 CLIs, main() passes argv straight to cli(GEPAConfig, ...) without first calling with_positional_taskset(), so the leading positional token is never rewritten into --taskset.id. Apply with_positional_taskset() to argv before parsing so positional taskset ids resolve correctly.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/cli/gepa.py around line 25:

A bare command like `uv run gepa reverse-text-v1 --model ...` silently uses the default `taskset.id` instead of selecting the requested taskset, so the run targets the wrong (or empty) taskset. Unlike the other v1 CLIs, `main()` passes `argv` straight to `cli(GEPAConfig, ...)` without first calling `with_positional_taskset()`, so the leading positional token is never rewritten into `--taskset.id`. Apply `with_positional_taskset()` to `argv` before parsing so positional taskset ids resolve correctly.

[macroscopeapp]

🟡 Medium gepa/runner.py:33

logger.info("gepa config:\n%s", config.model_dump_json(...)) logs the full resolved GEPAConfig, which includes client.headers and reflection_client.headers. When those headers contain authentication tokens for custom model endpoints, the secrets are written to the CLI logs/stdout, exposing credentials to anyone who can read the terminal output or collected logs. Consider redacting header values before logging the config.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/gepa/runner.py around line 33:

`logger.info("gepa config:\n%s", config.model_dump_json(...))` logs the full resolved `GEPAConfig`, which includes `client.headers` and `reflection_client.headers`. When those headers contain authentication tokens for custom model endpoints, the secrets are written to the CLI logs/stdout, exposing credentials to anyone who can read the terminal output or collected logs. Consider redacting header values before logging the config.

[macroscopeapp]

🟡 Medium gepa/reflection.py:16

build_reflection_lm() falls back to config.client when reflection_client is unset, but that config can be a TrainClientConfig targeting a vLLM /inference/v1/generate endpoint. The function unconditionally constructs an OpenAI client and calls chat.completions.create, so a gepa run with --client.type train (and no reflection_client override) sends chat-completions requests to an incompatible engine and the reflection step fails. Consider requiring an explicit reflection_client when the rollout client is a TrainClientConfig, or routing through TrainClient instead of a raw OpenAI client in the fallback path.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/gepa/reflection.py around line 16:

`build_reflection_lm()` falls back to `config.client` when `reflection_client` is unset, but that config can be a `TrainClientConfig` targeting a vLLM `/inference/v1/generate` endpoint. The function unconditionally constructs an `OpenAI` client and calls `chat.completions.create`, so a `gepa` run with `--client.type train` (and no `reflection_client` override) sends chat-completions requests to an incompatible engine and the reflection step fails. Consider requiring an explicit `reflection_client` when the rollout client is a `TrainClientConfig`, or routing through `TrainClient` instead of a raw `OpenAI` client in the fallback path.

[macroscopeapp]

🟡 Medium gepa/adapter.py:75

_run_batch always calls self.env.episode(task, self.ctx, n=1), so evaluating any taskset that uses @group_reward raises ValueError because group-reward scoring requires at least two rollouts per task. This crashes the entire gepa optimization run on valid v1 tasksets instead of optimizing them. Consider passing an n that satisfies the taskset's minimum rollout count, or validating the taskset up front and rejecting group-reward tasksets with a clear error.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/gepa/adapter.py around line 75:

`_run_batch` always calls `self.env.episode(task, self.ctx, n=1)`, so evaluating any taskset that uses `@group_reward` raises `ValueError` because group-reward scoring requires at least two rollouts per task. This crashes the entire `gepa` optimization run on valid v1 tasksets instead of optimizing them. Consider passing an `n` that satisfies the taskset's minimum rollout count, or validating the taskset up front and rejecting group-reward tasksets with a clear error.