Add end-to-end CPU vs torch bench harness + 5-scene report (Issue #21)

[s-sasaki-earthsea-wizard]

Summary

Adds an end-to-end CPU vs torch bench harness for smallbaselineApp.py and the combined 5-scene report it produced, addressing the fork-side Issue #21. The bench measures what a production user sees when they flip the two opt-in GPU solver flags (mintpy.networkInversion.solver = torch from PR #8, mintpy.topographicResidual.solver = torch from PR #20) and runs the full 18-step pipeline end-to-end.

The pre-existing per-step direct-call benches (report_baseline.md, report_torch.md, report_large_scene.md, reports/dem_error/) cover the GPU-dispatched steps in isolation but do not show the combined effect with both flags on simultaneously and they do not validate the integration of two sequential GPU dispatches through the intermediate HDF5 products. This bench fills that gap.

Headline numbers — GPU-able subtotal (sum of invert_network + correct_topography wall)

Scene	Pixels	K ifgrams	D dates	cpu / torch	Speedup	Product gates
FernandinaSenDT128	270 k	288	98	659.4 s / 10.3 s	63.89×	7 / 7 PASS
GalapagosSenDT128	3.40 M	490	98	3043.6 s / 98.7 s	30.84×	7 / 7 PASS
SanFranBaySenD42	326 k	1297	333	1086.9 s / 23.7 s	45.86×	6 / 6 PASS
KujuAlosAT422F650	226 k	167	24	39.6 s / 6.7 s	5.92×	1 / 7 PASS (geo product PASS)
SanFranSenDT42	1.04 M	505	114	92.5 s / 17.6 s	5.24×	0 / 6 PASS (no geo product)

Speedup range 5 — 64× depending on scene size and inversion mode. The 30 — 50× band on the two production-scale scenes (Galapagos, SanFranBay) is the most representative number for a maintainer evaluating the dispatch on their own data. Detailed wall tables (top 8 steps per scene) and the closed-form scaling interpretation are in reports/report_end_to_end_bench.md.

Commits

SHA	Subject
7c422fe	fixtures: enable correct_topography GPU solver in all 5 _torch.txt
e71cbe5	run_bench.sh: allow PYTHON_BIN and SBA_BIN env override
dca097b	Add end-to-end CPU vs torch bench harness (Issue #21)
366182f	e2e bench: fix dataset keys, scope regression to 6 monitored steps
20c7db6	.gitignore: cover top-level logs_* files in addition to directories
105d868	e2e bench: route Fernandina to SSD scene root (warm-SSD alignment)
90de39a	e2e bench: add paired cpu fixtures, fix scene layouts, drop SanFranSF pixelwiseGeometry=no
0fbf71b	Add end-to-end CPU vs torch bench report (Issue #21)

Design highlights

Paired cpu / torch fixtures

The first version of the harness used docs/templates/<scene>.txt for the cpu side and fixtures/<scene>_torch.txt for the torch side. For three scenes the two files diverged on non-solver overrides inherited from the Issue #19 Tier 1 direct-call bench (troposphericDelay.method = no, pixelwiseGeometry = no, save.hdfEos5 = no), so the cpu and torch runs were not configured identically.

Repaired in 90de39a by adding paired fixtures/<scene>_{cpu,torch}.txt for all 5 scenes. By design the two now differ only by the 2 mintpy.*.solver = torch lines — this is the syntactic invariant that makes the wall comparison fair. Verified mechanically:

for s in Fernandina Galapagos Kuju SanFranBay SanFranSF; do
  diff <(grep -v '^#' fixtures/${SCENE}_cpu.txt | filter solver) \
       <(grep -v '^#' fixtures/${SCENE}_torch.txt | filter solver)
done
# all empty

One scene = one shell invocation

bash scripts/run_end_to_end_bench.sh <scene>

drives 6 phases in order: workdir cleanup → cpu run → ERA5 cache symlink (if downloaded) → torch run → step wall diff → 7-product HDF5 diff → headline summary. Idempotent re-run via FRESH=1.

Galapagos inputs/*.h5 pre-seed

The raw ISCE inputs (merged/, master/, baselines/) referenced by the Galapagos template were cleaned up after the Issue #6 large-scene bench, leaving only the already-loaded ifgramStack.h5 and geometryRadar.h5. The harness pre-seeds these into both workdirs via symlink, which triggers smallbaselineApp.py's update-mode skip on load_data so the bench measures the full pipeline against the pre-existing stack.

Two-gate regression detector

tools/compare_step_walls.py uses an AND condition (relative ±5% AND absolute floor 2.0 s) on the 6 monitored CPU-only steps. Suppresses false positives on sub-second steps where ordinary scheduling noise easily crosses ±5% while keeping real regressions visible. The 16 CPU-only steps are split into "monitored" (the 6 listed in the Issue #21 acceptance criteria) and "unmonitored" (the rest); the CPU-only subtotal ratio is computed over all of them as an I/O / cache control headline.

Product numerical gate

3 of 5 scenes pass all 6 — 7 product gates at float32 round-off (max observed rms/scale 9.56e-6 on Fernandina's demErr.h5). Kuju and SanFranSF radar-coord products fail at rms/scale 1 — 7%; the Kuju geocoded velocity (filtered through maskTempCoh.h5) passes at 1.38e-7. The pattern points to scipy.lstsq minimum-norm vs cholesky_ex near-rank-deficient behaviour on pixels excluded downstream by the temporal-coherence mask but not by upstream NaN filters. Making the diff mask-aware is a sibling-repo follow-up; it does not affect any wall measurement. Full diagnosis in report §4.2.

Test plan

• All 5 scenes ran to completion under bash scripts/run_end_to_end_bench.sh <scene> with FRESH=1
• Per-step wall numbers in the report match logs_e2e_<scene>_walls_diff.json totals (regenerable from logs_e2e_<scene>_{cpu,torch}/summary.tsv)
• Product numerical gates (3 PASS, 2 with documented mask-related radar-coord divergence) match logs_e2e_<scene>_products_diff.json
• diff against non-solver lines is empty for every cpu/torch fixture pair
• No upstream docs/templates/ files were modified
• The previous Fernandina smoke test result (commit 105d868 → 63.89×) reproduces on the repaired harness using the new FernandinaSenDT128_cpu.txt fixture