EvalPort mapping for the governed-evolution eval (EvalCase/EvalRunner → TestCase/Grader/ResultSet)

[adhabnr-ux]

Hi — I read through harness-eval/src/main/kotlin/dev/androidagent/harness/eval/ (Eval.kt, EvalRunner.kt, MarkdownWorkspace.kt) to understand the governed-evolution gate: EvalCase (id, userInput, expectedOutputContains), EvalRunner.compare() running the same fixed cases against a baseline MarkdownWorkspace and an overlaid candidate, and EvalComparison producing a REJECT/PROMOTE/UNCHANGED verdict from regressions/improvements between the two List<EvalCaseResult>. It's a genuinely well-built, deterministic mechanism (FixedAgentClock + SequentialAgentIdGenerator per case, single-regression-vetoes-promotion rule) — not something I'd normally open an issue about on a repo this young, but the data shape maps cleanly onto an open spec I maintain, so I wanted to ask before assuming it's wanted.

Who I am / disclosure: I maintain EvalPort (Apache 2.0), an open interchange format for portable LLM evaluation test cases, graders, suites, and results — JSON Schema–defined, with Python/TS SDKs and 37 real to_openeval()/from_openeval() adapter packages under adapters/. I'm not affiliated with this project; this is an unsolicited suggestion, feel free to close it if it's not useful.

Why the mapping is clean

Your type	EvalPort type (schema/)
EvalCase.id, .userInput	TestCase.id, TestCase.input
EvalCase.expectedOutputContains: List<String>	one TestCase.graders[] entry per string, each a type: "contains" grader (params.substring)
EvalCaseResult.output	ResultSet.results[].actual_output
EvalCaseResult.passed / .missingExpectations	ResultSet.results[].passed + per-string grader_results[] (grader_id, type: "contains", score, passed)
baseline vs. candidate List<EvalCaseResult>	two ResultSets sharing suite_id, distinguished by run_id (e.g. "baseline" / "candidate")

That last row is a real gap in the current spec, worth flagging honestly: EvalPort's ResultSet (schema/resultset.json) has no native "two-run diff" concept — there's no field for your REJECT/PROMOTE/UNCHANGED verdict or regressions()/improvements(). The cleanest fit today is carrying that as ResultSet.metadata on the candidate run (free-form, additionalProperties: true), not a first-class field. If you're interested, this is close enough to the open "how should ResultSet represent repeated/paired runs" question already being discussed at evalport#22 that your use case would be a genuinely useful data point there.

Sketch, using your real field names, respecting your "no third-party runtime dependencies" style rule (CONTRIBUTING.md) — plain string building, no serialization library added:

// EvalCase -> EvalPort TestCase (schema/testcase.json)
fun EvalCase.toEvalPortTestCaseJson(): String {
    val graders = expectedOutputContains.mapIndexed { i, substring ->
        """{"id":"$id-gr$i","type":"contains","params":{"substring":${substring.jsonQuoted()}}}"""
    }
    return """{"id":"$id","input":${userInput.jsonQuoted()},"graders":[${graders.joinToString(",")}]}"""
}

// EvalCaseResult -> one entry of ResultSet.results[] (schema/resultset.json)
fun EvalCaseResult.toEvalPortResultJson(case: EvalCase): String {
    val graderResults = case.expectedOutputContains.mapIndexed { i, substring ->
        val ok = substring !in missingExpectations
        """{"grader_id":"$caseId-gr$i","type":"contains","score":${if (ok) 1 else 0},"passed":$ok}"""
    }
    return """{"test_case_id":"$caseId","actual_output":${output.jsonQuoted()},
        "grader_results":[${graderResults.joinToString(",")}],"passed":$passed}"""
}

This would sit as a small optional exporter — outside harness-eval's core comparison logic, maybe a separate module or example under harness-eval — not a change to the gate itself. The upside: a governed-evolution run could emit a spec-valid ResultSet pair (validatable against EvalPort's own validateResultSet()) alongside your existing renderReport() text output, which is portable to other eval tooling and diffable outside this repo.

Comparable real adapters, if useful reference for shape: braintrust-openeval-adapter converts an already-scored Eval() result (comparable to your baseline/candidate List<EvalCaseResult>, since your cases are already run and scored, not just defined) to/from EvalPort; autogen-openeval-adapter is the reference shape EvalPort's own README points contributors to for a first adapter (to_openeval(), from_openeval(), tests against the real validator, a README).

Happy to draft this as a small PR (JSON export functions + a couple of unit tests, no new dependency) if it's something you'd want — or happy to just close this if it's not a direction you're interested in. Either way, thanks for a genuinely careful piece of eval infrastructure to read through.

— Sahi, independent contributor (not affiliated with this project)