correct_topography: add batched-Cholesky GPU solver (opt-in via mintpy.topographicResidual.solver)#20
Merged
Conversation
Add estimate_dem_error_pixelwise_batch in src/mintpy/gpu/dem_error.py, a CUDA-batched replacement for the per-pixel scipy.linalg.lstsq loop in the pixelwise-geometry branch of correct_dem_error_patch (mintpy.dem_error). The solver materializes a (K, num_date, num_param) design tensor per chunk and solves the normal equations via cuSolver-batched Cholesky, mirroring the style of mintpy.gpu.ifgram_inversion (no einsum; @ + .transpose(-1, -2)). Rank-deficient pixels are detected via cholesky_ex info codes and zeroed so NaN/Inf cannot propagate. phase_velocity=True is not yet implemented and raises explicitly. The caller-side dispatch in mintpy.dem_error is added separately so the CPU path is byte-for-byte unchanged here. tests/test_gpu_dem_error.py covers the scene-mean (D, 1) and per-pixel (D, K) baseline cases against a per-pixel CPU reference loop on small synthetic fixtures (num_date=98, num_pixel=1000). The rms / max-abs differences are printed for inspection; numerical equivalence is not asserted at this stage. Unsupported solver and phase_velocity=True error paths are also covered. Assisted-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a solver kwarg (default 'cpu') to correct_dem_error_patch and route the pixelwise-geometry branch to mintpy.gpu.dem_error.estimate_dem_error_pixelwise_batch when solver != 'cpu'. The CPU per-pixel loop is preserved byte-for-byte when solver='cpu', and the mean-geometry branch (already pixel-batched on CPU via scipy multi-RHS lstsq) is untouched. The mintpy.gpu.dem_error import is local to the dispatch branch so mintpy.dem_error remains importable on torch-less environments. CLI flag and template-key wiring for the new solver kwarg are intentionally deferred to a follow-up commit; for now the dispatch is reachable only via direct kwarg passthrough from a future correct_dem_error caller. Assisted-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both mintpy.gpu.ifgram_inversion and mintpy.gpu.dem_error reduce a per-pixel weighted least-squares problem to batched normal equations plus cuSolver-batched Cholesky. They duplicated the torch/CUDA probe, SUPPORTED_SOLVERS, the VRAM-aware chunk sizer, and the Cholesky core with rank-deficient fallback. Move those into _common.py and import them from each solver. ifgram_inversion._solve_cholesky is reduced to a thin wrapper that assembles the weighted system; dem_error's _solve_cholesky_batched is replaced with a direct call to the shared core. Public APIs (estimate_timeseries_batch, estimate_dem_error_ pixelwise_batch, is_solver_available) are unchanged. Verified by running tests/test_gpu_ifgram_inversion.py (6) and tests/test_gpu_dem_error.py (4): 10/10 pass; the synthetic CPU vs GPU diffs for dem_error are unchanged (delta_z rms/|cpu|max ~ 3e-6, ts_cor ~ 6e-9, ts_res ~ 7e-5). Assisted-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wire mintpy.topographicResidual.solver = cpu|torch (and gpuChunkSize) through the CLI and template-read path so users can opt into the GPU dispatch shipped in the prior three commits via smallbaselineApp.py / dem_error.py without monkeypatching. Mirrors the mintpy.networkInversion.solver pattern from the invert_network step: * cli/dem_error.py adds --solver / --gpu-chunk-size argparse flags and reads both as template keys in read_template2inps. * dem_error.py: correct_dem_error_patch accepts a new chunk_size kwarg and passes it through to mintpy.gpu.dem_error.estimate_dem_error_pixelwise_batch; correct_dem_error reads inps.solver and inps.gpuChunkSize and packs them into data_kwargs alongside the existing G_defo / phase_velocity. * defaults/smallbaselineApp.cfg adds mintpy.topographicResidual.solver and .gpuChunkSize entries with the same doc block style as the networkInversion variant. * defaults/smallbaselineApp_auto.cfg resolves auto to cpu / 0. solver and gpuChunkSize are deliberately NOT added to dem_error.py's config_keys list (used for update-mode skip metadata), matching invert_network's choice: CPU and GPU outputs are numerically equivalent within float32 round-off (verified across 5 scenes at rms / |cpu|.max < 1e-5; see sibling mintpy-benchmark PR #9), so an existing demErr.h5 from solver=cpu does not need rebuilding when the user later switches to solver=torch. Smoke checks: - pytest tests/test_gpu_dem_error.py: 4 passed - dem_error.py --help shows new flags in a solver argument group - E2E on FernandinaSenDT128 SSD: --solver cpu (11.2s) vs --solver torch (32.2s), delta_z rms/|cpu|.max = 1.48e-6, ts_cor rms/|cpu|.max = 6.70e-8 - pre-commit run --all-files: all hooks pass Assisted-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a parallel section to the existing invert_network coverage: * Title generalised from "for the network inversion" to "for GPU acceleration", since two steps now ship a torch solver. * Intro paragraph lists both opted-in steps + which branch each affects (invert_network: all weighted-LSQ; correct_topography: pixelwise-geometry only — the mean-geometry branch is already pixel-batched on CPU and stays there). * New §3 "Enable on correct_topography" mirrors §2's CLI / template / example-data subsections. * §5 Performance gets a subsection for correct_topography pointing at the Issue #19 Tier 1 survey + per-scene reports on the sibling repo. Headline numbers (2-2.5x on tutorial-scale, 6.15x on Galapagos K=3.4M, gate <1e-5 across 5 scenes) and the storage caveat (NAS can erase the speedup at small scenes; Fernandina 2.56x SSD vs 0.97x NAS) are quoted so users have realistic expectations before opting in. * Wiki link to the GPU Speedup Scaling Model page for the closed-form model behind the empirical curve. Assisted-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12 tasks
This was referenced May 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in GPU dispatch on the
correct_topographystep, mirroring theinvert_networkGPU solver pattern shipped earlier in this fork (PR #8 / upstream insarlab/MintPy#1490).Toggle:
mintpy.topographicResidual.solver = torch(template) or--solver torch(CLI). Defaultauto→cpu, so existing setups are unaffected.Closes #17.
What's in this PR
Five commits, structured top-down for review:
0214a3fgpu: add batched DEM-error pixelwise solver— newsrc/mintpy/gpu/dem_error.pywithestimate_dem_error_pixelwise_batch(batched normal-equation + Cholesky on(K, P, P)per chunk, auto VRAM sizing, rank-deficient pixel handling) andis_solver_available373135adem_error: dispatch pixelwise branch to torch solver— incorrect_dem_error_patch,elif solver != 'cpu':routes to the GPU path; CPUrange_dist.size == 1mean-geometry branch and scipy per-pixel loop are byte-for-byte unchanged60c9e3agpu: factor shared helpers into mintpy.gpu._common—auto_chunk_size,is_solver_available, etc. lifted out ofifgram_inversionso both GPU steps share themadc0504dem_error: add CLI flag + cfg key for solver dispatch—cli/dem_error.pygains--solver+--gpu-chunk-size,dem_error.pyreadsinps.solver/inps.gpuChunkSizeand passes them intodata_kwargs, defaults cfg files wiremintpy.topographicResidual.solver+gpuChunkSizekeysc01a399docs/gpu.md: document correct_topography GPU dispatch— title generalised, new §3 + §5 subsection, performance / storage caveat / wiki linkPlus 1 new test file (
tests/test_gpu_dem_error.py, 4 tests) that pin the solver / fallback / phase_velocity contracts.Why now — bench evidence
Across 5 InSAR scenes (sibling PR #9, Issue #19 Tier 1) the GPU dispatch is shown to be:
rms / |cpu|.max < 1e-5on every scene; tightest margin is 4.33e-6 (SanFranBaydelta_z), loosest is 3.12e-10 (SanFranSFts_cor).geometryGeo.h5(geo) andgeometryRadar.h5(radar) both work without code changes.Speedup curve (all on local SSD, RTX 5080):
The closed-form model behind this curve is on the wiki: GPU Speedup Scaling Model. Short version: K is the GPU-parallelisable batch dimension; D and P scale CPU/GPU walls similarly. Expect ≳ 4× speedup once K ≳ 1M.
Design choices
einsumnot used — matches the existingmintpy.gpu.ifgram_inversionconvention (idiom not used elsewhere in upstream codebase).phase_velocity=TrueraisesNotImplementedErrorin the GPU branch. Production default isFalse; covering velocity-domain residuals can come later if there's demand.solver=torchwithout a visible CUDA device raises immediately; the GPU Quick Start wiki page explains setup.pixelwiseGeometry=no) stays on CPU — it's already pixel-batched in numpy and no per-pixel loop exists to accelerate.solver/gpuChunkSizenot added toconfig_keysfor the update-mode skip check, matchinginvert_network's decision; numerical equivalence within float32 round-off means rebuilding an existingdemErr.h5because the user flipped the solver is unnecessary.Test plan
pytest tests/test_gpu_dem_error.py— 4 passeddem_error.py --helpshows new--solver/--gpu-chunk-sizeflags under asolverargument groupdem_error.py --solver cpu(11.2 s) vs--solver torch(32.2 s) on FernandinaSenDT128 SSD copy with explicit-s 20170910 20180613(P=6); CPU/GPU agree atdelta_zrms/|cpu|.max = 1.48e-6 andts_cor6.70e-8pre-commit run --all-files— all 13 hooks pass + 1 skip (json)Out of scope (deliberate follow-ups)
phase_velocity=TrueGPU support — not in this PR.upstream/mainafter this fork PR merges, bundled with themintpy.gpusubpackage refactor; per Wiki Upstream-PR-Checklist.Related
docs/gpu.md— user-facing guideinvert_network, applies equally here)🤖 Generated with Claude Code