OpenMPI on Setonix
Open MPI is an open-source implementation of the Message Passing Interface (MPI), developed and maintained through a collaborative effort involving partners from academia, research institutions, and industry. This community-driven approach enables Open MPI to incorporate a broad range of expertise and innovations from across the high-performance computing ecosystem. Comprehensive user and developer documentation, including usage examples, is available on the Open MPI website at https://www.open-mpi.org.
There is unsupported OpenMPI on Setonix. In order to use it one should unload the vendor provided cray-mpich and cray-libsci and load openmpi/5.0.5. The openmpi/5.0.5 lua file resides in the /software/setonix/unsupported/ area
module unload cray-mpich cray-libsci
module use /software/setonix/unsupported/
module load openmpi/5.0.5
The C++ code which runs MPI ping-pong and Allreduce benchmarks
The example code below runs MPI ping-pong and Allreduce benchmarks using GPU-resident (device) buffers across two GPU nodes (16 GPUs total). The fabric provider is set to CXI by exporting FI_PROVIDER=cxi. For the build step, we force the Open MPI C++ wrapper to use the HIP compiler by setting OMPI_CXX=hipcc, ensuring that mpicxx compiles and links through the hipcc wrapper (and therefore picks up the correct ROCm/HIP flags and libraries).