Open MPI doesn't support non-blocking reduction colls on GPU

[cedricchevalier19]

Okay, so the issue is that Open MPI does not implement non-blocking reduction operations with CUDA buffers.
This is referenced in several places:

• Open MPI's documentation:
  • https://docs.open-mpi.org/en/v5.0.x/tuning-apps/networking/cuda.html#which-mpi-apis-do-not-work-with-cuda-aware
  • https://docs.open-mpi.org/en/v5.0.x/tuning-apps/networking/cuda.html#which-mpi-apis-do-not-work-with-cuda-aware-ucx (I think we are falling in this case?)
• Open MPI issues:
  • CUDA-aware Ireduce and Iallreduce operations for GPU tensors segfault open-mpi/ompi#9845
  • coll collectives segfaults on CUDA buffers open-mpi/ompi#12045

Some work was done (started by @devreal 🙂), but it does not seem to have been picked up by anyone on the Open MPI team?
Initial work: open-mpi/ompi#12318, then split into:

1. Add stream operations to accelerator components open-mpi/ompi#12356
2. Add CUDA/HIP implementations of reduction operators open-mpi/ompi#12569

Do we want to voice our needs and open an issue on the Open MPI repo too?

Originally posted by @dssgabriel in #198 (comment)