fix: retry transient corruption in default (bash+edit) harness program

[samsja]

Problem

The default harness (verifiers/v1/harnesses/default/program.py) — the fallback harness with bash and edit tools — makes model API calls with max_retries=0 and no retry logic. Intermittent corruption of model responses on a flaky interception tunnel crashes the whole rollout on a single bad response.

A corrupted response surfaces as:

• openai.APIResponseValidationError — SDK response-schema validation failure
• pydantic.ValidationError — raw schema failures (e.g. finish_reason='error')
• ConnectionResetError — raw resets / truncated streams

Fix

Apply the same fix from rlm-harness PR #111 to the default harness program:

• Add _RETRYABLE tuple with the same error types: APIConnectionError, APITimeoutError, InternalServerError, NotFoundError, RateLimitError, APIResponseValidationError, ValidationError, ConnectionResetError
• Add _RETRY_DELAYS backoff schedule (15, 30, 60, 90, 120) — same as rlm
• Wrap the chat() model call with retry-on-_RETRYABLE, mirroring rlm's call_with_retries
• Update the max_retries=0 comment to explain the distinction: SDK retries stay off (avoid re-sampling committed turns), but transient corruption that fails before the turn is committed is safely retried

Why this is safe

The interception server resets its pending-turn state before each model turn, so a retried call doesn't double-record a turn. This is the same reasoning as rlm-harness PR #111: corruption errors happen before the turn is committed (the response is malformed, so it can't be committed), so retrying is safe. Legitimate 4xx (auth, bad request) are NOT in _RETRYABLE and fail fast.

Verification

import ast
ast.parse(open("verifiers/v1/harnesses/default/program.py").read())  # parses cleanly

---

Note

Medium Risk
Changes rollout behavior under network or schema glitches—worst case adds several minutes of backoff per model turn—but scope is limited to the default harness program and mirrors an existing rlm-harness pattern.

Overview
Adds application-level retries around model calls in the default harness chat() so a single corrupted or dropped response on a flaky interception tunnel no longer aborts the whole rollout.

chat() now catches a fixed set of transient failures (APIConnectionError, timeouts, rate limits, 5xx, APIResponseValidationError, Pydantic ValidationError, ConnectionResetError), sleeps through backoff 15 → 30 → 60 → 90 → 120 seconds between attempts, then makes one final chat.completions.create if all delayed retries fail. The OpenAI client still uses max_retries=0 so the SDK does not re-sample turns that may already be committed; only errors that fail before a valid turn is returned are retried.

The previous inline comment that ruled out client-side retries was removed from main(); behavior is unchanged there aside from relying on the new chat() loop.

^{Reviewed by Cursor Bugbot for commit df52693. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Retry transient errors in default harness chat() with exponential backoff

Wraps the client.chat.completions.create call in program.py with a retry loop over delays of 15, 30, 60, 90, and 120 seconds before a final attempt. Retries on OpenAI transient errors (APIConnectionError, APITimeoutError, InternalServerError, RateLimitError, APIResponseValidationError), pydantic.ValidationError, and ConnectionResetError.

^{Macroscope summarized df52693.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR reverses a deliberate design decision by adding retry logic where it was explicitly avoided to prevent 'dead-end branch' issues in trace handling. The runtime behavior change and its potential impact on the tracing system warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}