Mpi evaluationmanager dev by nychiang · Pull Request #761 · llnl/hiop

nychiang · 2026-05-20T21:02:48Z

Redesign EvaluationManager to work with MPI on multiple nodes, and add comprehensive testing

Major fixes:
- Fix MPI hanging by switching to OpenMPI and python -m mpi4py.futures
- Fix missing profiling output by storing timing data as instance variables

New features:
- Comprehensive scaling test infrastructure (submit_bo_scaling.sh)
- BODriverEX_mpi.py: Simple 2D test problem for MPI scaling
- Thread/Process/MPI executor test scripts
- Complete documentation suite

Details in COMMIT_MESSAGE.txt and CHANGELOG.md

Files changed:
- hiopbbpy/utils/evaluation_manager.py (profiling fix)
- hiopbbpy/utils/util.py (debug output)
- test_multinode_mpi.sh (OpenMPI + proper launcher)
- Created: BODriverEX_mpi.py, scaling test scripts, documentation

Tested on LLNL LC (Dane) with 1-64 workers, single and multi-node.

bug fixed Fix multi-node MPI worker detection in EvaluationManager Fixed critical bug in _get_num_workers() that prevented correct worker counting for true multi-node MPI execution (mpiexec -n N where N > 1). Key changes: - evaluation_manager.py: Query MPI.COMM_WORLD.Get_size() - 1 instead of relying solely on MPI4PY_FUTURES_MAX_WORKERS environment variable - util.py: Enhanced MPIEvaluator documentation with multi-node examples - EvaluationManagerCI.py: Added -e flag to test thread/process/mpi executors with proper rank handling for multi-node MPI Before: Multi-node profiling incorrectly reported 1 worker After: Correctly reports N-1 workers for mpiexec -n N Backward compatible - legacy single-node MPI mode still supported. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

EvaluationManagerCI.py now handles all executor types (thread/process/mpi) making the separate MPI-only example redundant. update fix CI

Major fixes: - Fix MPI hanging by switching to OpenMPI and python -m mpi4py.futures - Fix missing profiling output by storing timing data as instance variables New features: - Comprehensive scaling test infrastructure (submit_bo_scaling.sh) - BODriverEX_mpi.py: Simple 2D test problem for MPI scaling - Thread/Process/MPI executor test scripts - Complete documentation suite Details in COMMIT_MESSAGE.txt and CHANGELOG.md Files changed: - hiopbbpy/utils/evaluation_manager.py (profiling fix) - hiopbbpy/utils/util.py (debug output) - test_multinode_mpi.sh (OpenMPI + proper launcher) - Created: BODriverEX_mpi.py, scaling test scripts, documentation Tested on LLNL LC (Dane) with 1-64 workers, single and multi-node.

thartland · 2026-05-20T22:33:37Z

+if __name__ == "__main__":
+  do_profiling = True
+
+  for prob_type in prob_type_l:


If prob_type_l contains one element do we want to keep this loop over elements of prob_type_l?

umm... let's keep it since we can easily switch prob_type_l to a list with more than one elements

thartland · 2026-05-28T15:40:05Z

+
+**BODriverEX_mpi.py**: Simple 2D LpNorm optimization
+- 64 initial samples, 20 BO iterations
+- No xfoil dependency, fast evaluations


Remove "No xfoil dependency"

thartland

See inline comments.

Remove all legacy xfoil references from shell scripts and documentation, and clean up unused variables in response to reviewer feedback. Changes: - Update job names in test scripts from xfoil_* to eval_mgr_* - Remove unused HIOP_XFOIL_JOB_ROOT and TOTAL_TASKS variables - Remove xfoil references from README.md and TESTING_GUIDE.md - Remove references to non-existent files (README_EvaluationManager.md, README_XFOIL.md) - Remove ds4mems package dependency mention Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

Remove all legacy xfoil references from shell scripts and documentation, and clean up unused variables in response to reviewer feedback. Changes: - Update job names in test scripts from xfoil_* to eval_mgr_* - Remove unused HIOP_XFOIL_JOB_ROOT and TOTAL_TASKS variables - Remove xfoil references from README.md and TESTING_GUIDE.md - Remove references to non-existent files (README_EvaluationManager.md, README_XFOIL.md) - Remove ds4mems package dependency mention

Remove remaining xfoil sections that were missed in the previous cleanup: - Production xfoil BO section - xfoil Problems subsection with file references - xfoil_bo/submit_bo_xfoil.sbatch reference

nychiang · 2026-05-28T20:52:18Z

See inline comments.

All the comments have been addressed in the new commits.

The --no-cache flag was preventing Spack from using cached dependency builds, causing 'compiler-wrapper not installed' errors. Dependencies installed in the previous step should be available from cache for the package-only build.

thartland

Looks good!

nychiang added 3 commits April 10, 2026 13:26

create branch

f7e98f7

update

e0eec20

add example

ccd6930

nychiang marked this pull request as ready for review May 20, 2026 21:02

nychiang requested review from cnpetra and thartland May 20, 2026 21:04

nychiang and others added 3 commits May 20, 2026 14:06

Update reference to point to EvaluationManagerCI.py

49743d8

EvaluationManagerCI.py now handles all executor types (thread/process/mpi) making the separate MPI-only example redundant. update fix CI

nychiang force-pushed the mpi-evaluationmanager-dev branch from 9c8e5f9 to 80c54df Compare May 20, 2026 21:07

thartland reviewed May 20, 2026

View reviewed changes

Comment thread src/Drivers/hiopbbpy/test_process.sh Outdated

thartland reviewed May 20, 2026

View reviewed changes

Comment thread src/Drivers/hiopbbpy/test_thread.sh Outdated

thartland reviewed May 20, 2026

View reviewed changes