Parallelize PR system tests via GitHub Actions matrix.

Add release_test_shard_1/2 covering the same cases as release_test,
and run them as separate matrix jobs for clearer logs and cheaper reruns.

Author: PranjalManhgaye

.github/workflows/system-tests-pr.yml

@@ -145,12 +145,20 @@ jobs:
           } >> "$GITHUB_STEP_SUMMARY"
 
   run-system-tests:
-    name: Trigger system tests
+    name: Trigger system tests (shard ${{ matrix.shard }})
     if: ${{ github.event.label.name == 'trigger-system-tests' }}
     needs: gather-refs
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - shard: 1
+            suites: release_test_shard_1
+          - shard: 2
+            suites: release_test_shard_2
     uses: precice/tutorials/.github/workflows/run_testsuite_workflow.yml@develop
     with:
-      suites: release_test
+      suites: ${{ matrix.suites }}
       build_args: "PLATFORM:ubuntu_2404,\
         PRECICE_REF:${{ needs.gather-refs.outputs.ref-precice }},\
         PYTHON_BINDINGS_REF:${{ needs.gather-refs.outputs.ref-python-bindings }},\

tools/tests/README.md

@@ -54,6 +54,12 @@ To be able to fill in the right case tuple into the `tests.yaml`, you can use th
 
 ## Running the system tests on GitHub Actions
 
+Pull-request system tests run as a GitHub Actions matrix over `release_test_shard_1` and
+`release_test_shard_2` (together they cover the same cases as `release_test`). When a shard
+fails, re-run only the failed matrix job. With multiple `precice-tests-vm` runners, shards
+may execute concurrently; overlapping Docker image builds are possible and acceptable unless
+CI reports failures.
+
 Go to Actions > [Run Testsuite (manual)](https://github.com/precice/tutorials/actions/workflows/run_testsuite_manual.yml) to see this workflow.
 
 After bringing these changes to `master`, the manual triggering option should be visible on the top right. Until that happens, we can only trigger this workflow manually from the [GitHub CLI](https://github.blog/changelog/2021-04-15-github-cli-1-9-enables-you-to-work-with-github-actions-from-your-terminal/):

tools/tests/tests.yaml

@@ -551,6 +551,61 @@ test_suites:
       - *water-hammer_fluid1d-left-nutils_fluid3d-right-openfoam
       - *wolf-sheep-soil-creep_soil-creep-landlab_wolf-sheep-grass-mesa
 
+  # Shards of release_test for parallel CI (union = release_test, no duplicates).
+  release_test_shard_1:
+    tutorials:
+      - *breaking-dam-2d_fluid-openfoam_solid-calculix
+      - *channel-transport_fluid-openfoam_transport-nutils
+      - *channel-transport-reaction_fluid-fenics_chemical-fenics
+      - *elastic-tube-1d_fluid-cpp_solid-cpp
+      - *elastic-tube-1d_fluid-cpp_solid-python
+      - *elastic-tube-1d_fluid-fortran_solid-fortran
+      - *elastic-tube-1d_fluid-fortran-module_solid-fortran-module
+      - *elastic-tube-1d_fluid-python_solid-python
+      - *elastic-tube-3d_fluid-openfoam_solid-calculix
+      - *flow-around-controlled-moving-cylinder_controller-fmi_fluid-openfoam_solid-python
+      - *flow-over-heated-plate_fluid-openfoam_solid-fenics
+      - *flow-over-heated-plate_fluid-openfoam_solid-nutils
+      - *flow-over-heated-plate_fluid-openfoam_solid-openfoam
+      - *flow-over-heated-plate_fluid-su2_solid-openfoam
+      - *flow-over-heated-plate-nearest-projection_fluid-openfoam_solid-openfoam
+      - *flow-over-heated-plate-partitioned-flow_fluid1-openfoam_fluid2-openfoam_solid-openfoam
+      - *flow-over-heated-plate-two-meshes_fluid-openfoam_solid-calculix
+      - *free-flow-over-porous-media_free-flow-dumux_porous-media-dumux
+      - *heat-exchanger_fluid-inner-openfoam_solid-calculix_fluid-outer-openfoam
+      - *heat-exchanger-simplified_fluid-top-openfoam_fluid-bottom-openfoam_solid-calculix
+      - *multiple-perpendicular-flaps_fluid-openfoam_solid-upstream-dealii_solid-downstream-dealii
+      - *oscillator_mass-left-fmi_mass-right-fmi
+      - *oscillator_mass-left-python_mass-right-python
+      - *oscillator-overlap_mass-left-python_mass-right-python
+
+  release_test_shard_2:
+    tutorials:
+      - *partitioned-backwards-facing-step_fluid1-openfoam_fluid2-openfoam
+      - *partitioned-elastic-beam_dirichlet-calculix_neumann-calculix
+      - *partitioned-heat-conduction_dirichlet-fenics_neumann-fenics
+      - *partitioned-heat-conduction_dirichlet-openfoam_neumann-openfoam
+      - *partitioned-heat-conduction-complex_dirichlet-fenics_neumann-fenics
+      - *partitioned-heat-conduction-direct_dirichlet-nutils_neumann-nutils
+      - *partitioned-heat-conduction-overlap_left-fenics_right-fenics
+      - *partitioned-pipe_fluid1-openfoam-pimplefoam_fluid2-openfoam-pimplefoam
+      - *partitioned-pipe_fluid1-openfoam-sonicliquidfoam_fluid2-openfoam-sonicliquidfoam
+      - *partitioned-pipe-multiscale_fluid1d-left-nutils_fluid3d-right-openfoam
+      - *partitioned-pipe-two-phase_fluid1-openfoam_fluid2-openfoam
+      - *perpendicular-flap_fluid-fake_solid-fake
+      - *perpendicular-flap_fluid-openfoam_solid-calculix
+      - *perpendicular-flap_fluid-openfoam_solid-dealii
+      - *perpendicular-flap_fluid-openfoam_solid-fenics
+      - *perpendicular-flap_fluid-openfoam_solid-openfoam
+      - *perpendicular-flap_fluid-su2_solid-fenics
+      - *quickstart_openfoam_cpp
+      - *resonant-circuit_capacitor-python_coil-python
+      - *turek-hron-fsi3_fluid-openfoam_solid-dealii
+      - *volume-coupled-diffusion_source-fenics_drain-fenics
+      - *volume-coupled-flow_fluid-openfoam_source-nutils
+      - *water-hammer_fluid1d-left-nutils_fluid3d-right-openfoam
+      - *wolf-sheep-soil-creep_soil-creep-landlab_wolf-sheep-grass-mesa
+
   # These test suites take longer to run. They are available, but not regularly executed.
   extra:
     tutorials:

tools/tests/validate_release_test_shards.py

@@ -0,0 +1,51 @@
+#!/usr/bin/env python3
+"""Verify release_test_shard_1/2 partition release_test without gaps or duplicates."""
+
+from metadata_parser.metdata import Tutorials
+from systemtests.Systemtest import Systemtest
+from systemtests.SystemtestArguments import SystemtestArguments
+from systemtests.TestSuite import TestSuites
+from paths import PRECICE_TESTS_DIR, PRECICE_TUTORIAL_DIR
+
+
+def _cases_for_suite(suite_name: str) -> list[str]:
+    tutorials = Tutorials.from_path(PRECICE_TUTORIAL_DIR)
+    suites = TestSuites.from_yaml(PRECICE_TESTS_DIR / "tests.yaml", tutorials)
+    suite = suites.get_by_name(suite_name)
+    if suite is None:
+        raise SystemExit(f"Unknown test suite: {suite_name}")
+
+    build_args = SystemtestArguments.from_args(None)
+    cases: list[str] = []
+    for tutorial in suite.cases_of_tutorial:
+        for case, reference_result in zip(
+                suite.cases_of_tutorial[tutorial],
+                suite.reference_results[tutorial]):
+            systemtest = Systemtest(tutorial, build_args, case, reference_result)
+            cases.append(str(systemtest))
+    return sorted(cases)
+
+
+def main() -> None:
+    release = _cases_for_suite("release_test")
+    shard_1 = _cases_for_suite("release_test_shard_1")
+    shard_2 = _cases_for_suite("release_test_shard_2")
+    combined = sorted(shard_1 + shard_2)
+
+    if release != combined:
+        only_release = set(release) - set(combined)
+        only_shards = set(combined) - set(release)
+        duplicates = len(shard_1) + len(shard_2) - len(set(shard_1 + shard_2))
+        raise SystemExit(
+            f"Shard mismatch: release_test={len(release)}, "
+            f"shard_1={len(shard_1)}, shard_2={len(shard_2)}, "
+            f"duplicates={duplicates}, only_in_release={len(only_release)}, "
+            f"only_in_shards={len(only_shards)}")
+
+    print(
+        f"OK: release_test ({len(release)} cases) = "
+        f"shard_1 ({len(shard_1)}) + shard_2 ({len(shard_2)})")
+
+
+if __name__ == "__main__":
+    main()