...
- review updated versions of software in the system-wide stack provided by Pawsey and HPE
- reinstall any software managed by the group
- rebuild containers previously built on Pawsey-provided images for multi-node MPI support
- update Slurm batch scripts to reflect updated versions of software
Further details are provided in the following sections.
What is new in CPE 23.02
The Cray Programming Environment (CPE) is the programming environment provided by the vendor on Setonix. It has been updated to version 23.03 and includes newer MPI libraries, newer Cray (14.0.3 -> 15.0.1) and GCC (12.1.0 -> 12.2.0) compilers. The AOCC compiler and programming environment should NOT be used as it is unstable.
...
where <DATE_TAG> is the date of the related software stack deployment in the YYYY.MM format. When you select which pawseyenv version to use, you select also what user and project installations are visible.
GPU software
The new CPE comes with ROCm 5.2.3, replacing version 5.0.2. In addition, the AMD GPU driver has also been updated and should resolve many of the ROCm issues observed on the system.
GPU-enabled builds of several applications have been added to the stack (or will be in the incoming weeks). These replace the containerised deployments that were previously present on Setonix.
...
Machine learning frameworks such as Tensorflow and PyTorch are provided by means of containers and made visible using the module system. Once the respective module is loaded, the python3 interpreter from within the container has access to the Tensorflow or PyTorch Python modules.
Users own software will need to be recompiled/rebuilt
...
Researchers need to recompile their own
...
software that has been built previously on Setonix. This is necessary because the CPE has newer versions of various libraries, to ensure the best possible performance and avoid i
For own software installed with Spack, do this:
- module load spack/0.19.0
- spack uninstall software@version
- spack clean -a && spack -d install software@version
Containers
As the programming environment has changed, containers that make use of MPI may suffer incompatibilities with the new host MPI libraries. So, if users own containers are reporting libraries incompatibilities or suffering execution issues, they may need to rebuild their containers based on a MPICH image that has been tested against the new CPE.
We have made available a new container MPICH 3.4.3 base image built from Ubuntu 20.04 that does not suffer from incompatibilities with the new host libraries. The recipe (Dockerfile) of this base image can be found here: https://github.com/PawseySC/pawsey-containers/blob/master/mpi/mpich-base/buildmpich.dockerfile and the build image has been uploaded already to our quay.io registry. Then, users with own MPI containers will need to update the base image in their own Dockerfile recipies recipes to start from:
FROM quay.io/pawsey/mpich-base:3.4.3_ubuntu20.04
Users
New ROCm 5.2.3
...
can check some practical examples of this in our recently rebuilt containers of OpenFOAM. For example: