feat(test-harness): one machine-written integration PR summary

[nebasuke]

PR adding the status of an integration test run as a comment. My goal is that it should be clear if an integration test was successful (with the goals in mind). In particular, the comment states the following:

• Whether it is output preserving -> good in case of a performance improvement that should not have impact on output
• Whether we have introduced new test failures -> generally should be a fail.
• Numbers on compile time, gas reports along with sensible defaults to allow for jitter.

The original reports are preserved, and are linked in the PR comment.

Claude Summary

Why

The integration workflow posted three link-only PR comments; all analysis of the XLSX reports happened offline (the local solx-perf-review skill's parse_reports.py). Nothing on the PR told you whether output changed, what failed, or how compile-time moved.

What

A single machine-written comment (integration-test-summary), rendered in-tree in Rust from the converter's native Benchmark model — no XLSX round-trip, no Python. Designed so a reviewer can call pass/fail from the comment alone, using the same decision procedure as the offline review.

• New render_summary (solx-benchmark-converter/src/output/markdown.rs) + solx-benchmark-summary binary.
• solx-dev now writes {foundry,hardhat}-benchmark.json next to each XLSX (one shared writer); the workflow emits solx-tester-benchmark.json. The summary reads those.
• The three link comments collapse into one comment (reusing mshick/add-pr-comment via message-path), also mirrored to $GITHUB_STEP_SUMMARY.
• XLSX artifacts are unchanged — still uploaded, still linked (by their real file names) from the summary table.

The comment

Three independent verdict lines, each with its numbers:

• Output invariance — bytecode size (deterministic, all suites) + solx-tester gas (deterministic REVM). A size change informs, never hides: ⚠️ headline + the largest movers listed inline (a size change means semantics possibly changed — but intentional codegen PRs exist, so it is not a red ❌). Foundry/Hardhat gas is fuzz/CREATE-noisy → shown as jitter diff-count + median (never the max, which is always a huge meaningless CREATE-deploy outlier), never gated.
• New failures — PR vs main per paired run; pre-existing baselines are stated, only regressions alarm (❌), and the regressed projects are listed inline. A PR run with no main counterpart is surfaced as unbaselined (⚪/⚠️), not counted as regressions against an imaginary zero.
• Harness health — a suite that errored, produced a corrupt report, or whose data matches no recognized toolchain naming is an explicit ❌ line + row; one bad suite never suppresses the healthy suites' results, and the comment never shows a green checkmark over empty data.

Every percentage states its direction (positive = PR larger / PR slower); no internal toolchain names leak (00.solc → "vs solc", mode keys humanized, report links named per suite).

Example (output-preserving, full matrix):

✅ Output-preserving — bytecode size identical (54,480 comparisons), solx-tester gas identical (63,720).
✅ No new failures — Foundry's 685 / Hardhat's 40 failures already present on main.

Suite	New failures	Size Δ	Gas Δ	Report
solx-tester	✅ 0	✅ 0 of 49,508	✅ 0 of 63,720	solx-tester-report.xlsx ↓
Foundry · 25 proj	✅ 0 (685 pre-existing)	✅ 0 of 4,972	⚪ jitter 90 of 2,486, median 0.4% (not gated)	foundry-report.xlsx ↓
Hardhat · 3 proj	✅ 0 (40 pre-existing)	⚪ not collected	⚪ not collected	hardhat-report.xlsx ↓

Plus: a compile-time tripwire table (aggregate / median per pipeline, columns from the pipelines present in the data, ⚠️ at ≥5%, project outliers at ≥15%, "within noise" line when quiet — authoritative deltas stay with ci:compile-benchmark), and an always-expanded full-matrix table "Bytecode size — PR vs baselines" with vs solc / vs released solx columns, summed over contracts built by both toolchains only so a baseline that failed some builds is excluded rather than skewing the percentage.

Failure behaviour

• A suite that runs but regresses vs main writes its report first, so it's included (with the regressed projects listed) and turns the verdict red.
• A suite that hard-errors (no *-benchmark.json, or a corrupt one) gets an explicit "❌ Suite errored" verdict line and table row; the healthy suites still post.
• A suite skipped by an earlier hard failure is omitted (its flag isn't passed), not mislabeled as errored; a missing main-side tester baseline renders the PR data as unbaselined instead of "suite errored".
• Losing the summary itself fails the job visibly (no continue-on-error on the generation step).
• Benchmark data matching no recognized toolchain naming is a loud "❌ Harness error" — the guard against naming drift silently disabling the gate (the fix(benchmark): solx-tester skips E codegen; Hardhat ignores the configured solx #497 class).

Verification

• 16 converter unit tests (three-verdict split, new-vs-main failure pairing incl. pre-existing-only and unbaselined cases, unrecognized-naming guard, common-contracts baseline sums, gas-jitter median, compile outlier / within-noise / median, errored-suite row, named links, commas); clippy + fmt clean.
• Binary exercised end-to-end on synthetic full-matrix pass data, a combined output-changed + new-failures + errored-suite scenario, and a corrupt-suite-JSON scenario (healthy suites still render).
• actionlint + shellcheck clean (also fixed a pre-existing useless-cat).
• No Cargo.lock change (deps already present; --frozen safe).

Real-data proof is this PR's own CI run — apply ci:integration (+ ci:integration-benchmark-full for the full matrix). Supersedes the offline parse_reports.py.

[github-actions]

🧪 Integration tests — full matrix · PR vs main

⚠️ Output changed — 1 of 63,720 solx-tester gas comparisons differ. If this PR is meant to be output-preserving, investigate before merging.
✅ No new failures — Foundry's 685 / Hardhat's 40 failures already present on main.

Suite	New failures	Size Δ	Gas Δ	Report
solx-tester	✅ 0	✅ 0 of 49,508	⚠️ 1 of 63,720	solx-tester-report.xlsx ↓
Foundry · 25 proj	✅ 0 (685 pre-existing)	✅ 0 of 4,972	⚪ jitter 80 of 5,214, median <0.1% (not gated)	foundry-report.xlsx ↓
Hardhat · 3 proj	✅ 0 (40 pre-existing)	⚪ not collected	⚪ not collected	hardhat-report.xlsx ↓

solx-tester — largest gas changes:

• solx-solidity/test/libsolidity/semanticTests/structs/delete_struct.sol [Yul MzB3 0.8.34] 85,899 → 85,902 (+0.0%)

Compile time — wall-clock tripwire, positive = PR slower (authoritative Δ in ci:compile-benchmark)

Suite	legacy (agg / median)	viaIR (agg / median)
Foundry · 25 proj	+0.1% / +0.2%	-0.1% / +0.2%
Hardhat · 3 proj	+0.2% / +0.3%	+0.7% / +0.7%

Within noise — no suite ≥ 5%, no project ≥ 15%.

Bytecode size — PR vs baselines (positive = PR larger; contracts built by both only)

Suite	Pipeline	vs solc	vs released solx
Foundry · 25 proj	legacy	-13.0%	-0.0%
Foundry · 25 proj	viaIR	-1.6%	+0.0%

---

Suites run the release solx binary. Foundry/Hardhat gas jitters run-to-run (fuzz/invariant tests, CREATE-context deploys), so it never gates.

[nebasuke]

Currently there is a small gas change. I've deliberately NOT rebased, to show how it looks in the case of a change.

Claude Summary

The gas diff is a stale-base artifact, not a change from this PR

The integration summary flags structs/delete_struct.sol [Yul MzB3 0.8.34] 85,899 → 85,902 (+3 gas). It's a base-drift artifact — nothing in this PR moves compiler output:

1. This PR can't change codegen. The diff only touches reporting/harness code — integration-tests.yaml, solx-benchmark-converter/*, and solx-dev/src/test/*. No compiler crate, no solx-solidity submodule, no LLVM. Both sides compile identical bytecode.

2. The branch is 12 commits behind main, including a revm major bump (fix(deps): update rust crate revm to v41 #541, revm 40 → 41):

   	revm pin	Cargo.lock
   merge-base (fa8cf78d)	40.0	40.0.3
   main tip	41.0	41.0.0

3. solx-tester measures gas by executing bytecode through revm (gas.tx_gas_used()). The harness builds the tester on each side, so the PR side runs the bytecode under revm 40.0.3 and the main side runs the same bytecode under 41.0.0. The +3 is a revm 40→41 gas-accounting change (delete_struct hits SSTORE/refund paths), applied to one side only because the branch predates the bump.

I also checked the solx-solidity submodule move (#530) — that's just a #include <cstdint> GCC 16 build fix, not codegen, so it's ruled out.