During June 2024 maintenance (4th June,2024), Setonix has been updated to the new Cray Programming Environment (CPE) 23.09 which includes newer MPI libraries and a newer Cray compiler version (16.0.1). In preparation for this, Pawsey has prepared a new software stack, which became available immediately after the maintenance via the default pawseyenv/2024.05 module. The purpose of this page is to consolidate in one place all the important changes, highlights and recommended actions.
Researcher actions
Changes in the CPE, Cray-MPICH, and Pawsey provided software stack immediately applied to all users after June 2024 maintenance. Therefore, every researcher should follow these steps to “upgrade” their own tools and scripts towards the new conditions:
Check for version changes in the system-wide software stack provided by Pawsey and HPE for the tool that are part of your workflows.
Check for the correct execution of any software managed by the research group. In case of failure, then reinstall (see sections below).
Check for the correct execution of containers previously built on Pawsey-provided images for multi-node MPI support. In case of failure, then rebuild from the latest Pawsey-provided image.
Update Slurm batch scripts to reflect the new versions of software
(Please refer to the general documentation if you are unsure on how to proceed on any of the proposed steps.)
Further details about the changes are provided in the following sections. We are also including a section at the end for listing “known issues” that have been discovered after maintenance (and that we are working to solve).
What is new in CPE 23.09
The Cray Programming Environment (CPE) is the programming environment provided by the vendor on Setonix. It has been updated to version 23.09, and includes an updated MPI library, and newer Cray compiler (15.0.1 → 16.0.1). The GCC compiler version remains the same. We still recommend against using the AOCC compiler and programming environment (PrgEnv-aocc) due to on-going issues.
Software stack
Pawsey Software Stack
The new version of the software stack is 2024.05, and it replaced 2023.08 software stack. The new software stack has been installed using a newer Spack major release, v0.21.0. We have compiled several packages using the Cray compiler and updated the versions of most packages in both the GNU and Cray programming environments. Users can check for the new versions of the software they used via the modulecommands like:
module avail <moduleName>
In the following expandable sections there is a complete list of changes for the two recommended programming environments:
2023.08 (Old stack for PrgEnv-gnu)
2024.05 (New stack for PrgEnv-gnu)
2023.08 (Old stack for PrgEnv-gnu)
2024.05 (New stack for PrgEnv-gnu)
adios2/2.9.0-hdf5
adios2/2.9.2-hdf5
-
adios2/2.9.2 (without hdf5 support)
alphafold/2.2.3
alphafold/2.3.0
amber/2022
amber/2024
-
amdblis/3.0.1
apr-util/1.6.1
apr-util/1.6.3
arpack-ng/3.8.0
arpack-ng/3.9.0
awscli/1.27.84
awscli/1.29.41
beast2/2.6.7
beast2/2.7.4
blaspp/2022.07.00 and blaspp/2023.08.25
blaspp/2023.11.05
boost/1.80.0-c++14-python
boost/1.83.0-c++14-python
boost/1.80.0-c++98-python
boost/1.83.0-c++98-python
caliper/2.7.0
caliper/2.10.0
cdo/2.1.0
cdo/2.2.2
cfitsio/4.1.0
cfitsio/4.3.0
cmake/3.24.3
cmake/3.27.7
cp2k/2023.1
cp2k/2023.2
cp2k-amd-gfx90a/8.2
cp2k-amd-gfx90a/2022.2
cppzmq/4.7.1
cppzmq/4.9.0
feh/3.3
feh/3.10
ffmpeg/4.4 and ffmpeg/4.4.1
ffmpeg/4.4.1
-
fits2idia/0.1.15
gnuplot/5.4.2
gnuplot/5.4.3
go/1.19.10
go/1.21.3
gromacs/2022.5
gromacs/2022.5-double
gromacs/2023
gromacs/2022.5-mixed
-
gromacs/2023-double
hdf5/1.12.2-api-v110
hdf5/1.14.3-api-v110
hdf5/1.12.2-api-v112
hdf5/1.14.3-api-v112
hdf5/1.12.2-api-v18
hdf5/1.14.3-api-v18
hdf5/1.12.2-parallel-api-v110
hdf5/1.14.3-parallel-api-v110
hdf5/1.12.2-parallel-api-v112
hdf5/1.14.3-parallel-api-v112
hdf5/1.12.2-parallel-api-v18
hdf5/1.14.3-parallel-api-v18
hpc-python-container/2022.03
Removed (No usage)
hpc-python-container/2022.03-hdf5mpi
Removed (No usage)
hpcviewer/2022.10
hpcviewer/2023.07
hpl/2.3
hpl/2.3 and hpl/2.3+openmp
hpx/1.8.1
hpx/1.9.1
kokkos/3.7.02-hpx
kokkos/4.1.00-hpx
-
kokkos/4.1.00-openmp
-
kokkos-amd-gfx90a/3.7.02-openmp
-
kokkos-amd-gfx90a/4.1.00-openmp
lammps/20230615
lammps/20230802
-
lammps/20230802.3
-
lammps-amd-gfx90a/20230802.3
-
lz4/1.9.4
-
magma-amd-gfx90a/2.7.2
-
metis/5.1.0
-
metis/5.1.0-64bit
namd-amd-gfx90a/2.15a2-20211101
namd-amd-gfx90a/2.15a2
nano/6.3
nano/7.2
netcdf-c/4.9.0
netcdf-c/4.9.2
netcdf-fortran/4.6.0
etcdf-fortran/4.6.1
netlib-lapack/3.10.1
netlib-lapack/3.11.0
nextflow/23.04.3 and nextflow/23.10.0
nextflow/23.10.0
-
ninja/1.11.1
omniperf/1.0.6
omniperf/1.0.10
omnitrace/1.10.2
omnitrace/1.11.2-rocm-5.2.3
openblas/0.3.21
openblas/0.3.24
opencv/4.6.0
opencv/4.8.0
openjdk/17.0.5_8
openjdk/17.0.8.1_1
osu-micro-benchmarks/5.7.1
osu-micro-benchmarks/7.3
-
parmetis/4.0.3
-
parmetis/4.0.3-64bit
pennylane/0.33.0
Will be available as containers
pennylane-lightning/0.33.1
Will be available as containers
pennylane-lightning-amd-gfx90a/0.32.0
Will be available as containers
perl/5.36.0
perl/5.38.0
petsc/3.18.1-complex, petsc/3.19.5-complex and petsc/3.19.5-nocomplex
petsc/3.20.1-complex and petsc/3.20.1-nocomplex
plasma/22.9.29
plasma/23.8.2
py-boto3/1.18.12
py-boto3/1.26.26
py-cython/0.29.30
py-cython/0.29.36
py-cython/0.29.32
py-cython/3.0.4
py-dask/2021.6.2
py-dask/2023.4.1
py-h5py/3.7.0
py-h5py/3.8.0
py-healpy/1.14.0
Removed (No usage)
py-ipython/8.5.0
py-ipython/8.14.0
py-matplotlib/3.6.2
py-matplotlib/3.8.1
py-mpi4py/3.1.4-py3.10.10
py-mpi4py/3.1.5-py3.11.6
py-netcdf4/1.5.8
py-netcdf4/1.6.2
py-numba/0.56.0
py-numba/0.57.0
py-numpy/1.22.4
py-numpy/1.24.4
py-numpy/1.23.4
py-numpy/1.26.1
py-pandas/1.5.1
py-pandas/1.5.3 and py-pandas/2.1.2
py-pip/23.1.2-py3.10.10
py-pip/23.1.2-py3.11.6
py-pip/23.1.2-py3.10.8
py-pip/23.1.2-py3.9.15
py-plotly/5.2.2
py-plotly/5.14.1
py-scikit-learn/1.1.3
py-scikit-learn/1.3.2
py-scipy/1.8.1
py-scipy/1.11.3
py-setuptools/59.4.0-py3.10.10
py-setuptools/59.4.0-py3.11.6
py-setuptools/59.4.0-py3.10.8
py-setuptools/59.4.0-py3.9.15
py-setuptools/68.0.0-py3.10.10
py-setuptools/68.0.0-py3.11.6
python/3.10.10
py-setuptools/68.0.0-py3.9.15
python/3.10.8
python/3.11.6
pytorch/2.1.0-rocm5.6.0
pytorch/2.2.0-rocm5.7.3
pytorch/2.2.0-rocm5.7.3
-
pytorch/2.2.0-rocm5.7.3
-
qiskit/0.13.0
Will be available as containers
qiskit-amd-gfx90a/0.13.0
Will be available as containers
r/4.2.2
r/4.3.0
rclone/1.62.2
rclone/1.63.1
-
reframe/4.0.4
-
rocm/5.4.6
-
rocm/5.5.3
-
rocm/5.6.1
-
rocm/5.7.0
-
rocm/5.7.1
-
rocm/5.7.3
rust/1.60.0
rust/1.78.0
-
scotch/7.0.4
-
scotch/7.0.4-metis
shpc/0.1.23
shpc/0.1.28
singularity/3.11.4-askap
singularity/4.1.0-askap
singularity/3.11.4-askap-gpu
singularity/4.1.0-askap-gpu
singularity/3.11.4-mpi
singularity/4.1.0-mpi
singularity/3.11.4-mpi-gpu
singularity/4.1.0-mpi-gpu
singularity/3.11.4-nohost
singularity/4.1.0-nohost
singularity/3.11.4-nompi
singularity/4.1.0-nompi
singularity/3.11.4-slurm
singularity/4.1.0-slurm
singularity/ashley
-
singularityce/3.11.4
singularityce/4.1.0
slate/2022.07.00 and slate/2023.08.25
slate/2023.11.05
spack/0.19.0
spack/0.21.0
squashfs/4.5.1
squashfs/4.6.1
subversion/1.14.1
subversion/1.14.2
trilinos/13.4.0
trilinos/15.0.0
-
trilinos-amd-gfx90a/15.0.0
-
wcslib/7.3-nocfitsio
xerces-c/3.2.3-transcoder-gnuiconv
xerces-c/3.2.4-transcoder-gnuiconv
-
xerces-c/3.2.4-transcoder-iconv
2023.08 (Old stack for PrgEnv-cray)
2024.05 (New stack For PrgEnv-cray)
2023.08 (Old stack for PrgEnv-cray)
2024.05 (New stack For PrgEnv-cray)
alphafold/2.2.3
alphafold/2.3.0
adios2/2.9.2-hdf5
amdblis/3.0.1
awscli/1.27.84
awscli/1.29.41
blaspp/2023.08.25
blaspp/2023.11.05
boost/1.80.0-c++14-python
boost/1.83.0-c++14-python
boost/1.80.0-c++98-python
boost/1.83.0-c++98-python
caliper/2.10.0
cmake/3.24.3
cmake/3.27.7
cp2k-amd-gfx90a/2022.2
cray-hdf5/1.12.2.7
cray-hdf5-parallel/1.12.2.7
cray-mpixlate/1.0.2
cray-parallel-netcdf/1.12.3.7
feh/3.10
ffmpeg/4.4.1
go/1.19.10
go/1.21.3
hdf5/1.12.2-api-v110
hdf5/1.14.3-api-v110
hdf5/1.12.2-api-v112
hdf5/1.14.3-api-v112
hdf5/1.12.2-api-v18
hdf5/1.14.3-api-v18
hdf5/1.12.2-parallel-api-v110
hdf5/1.14.3-parallel-api-v110
hdf5/1.12.2-parallel-api-v112
hdf5/1.14.3-parallel-api-v112
hdf5/1.12.2-parallel-api-v18
hdf5/1.14.3-parallel-api-v18
hpc-python-container/2022.03
Deprecated
hpc-python-container/2022.03-hdf5mpi
Deprecated
hpcviewer/2023.0
hpx/1.9.1
kokkos/4.1.00-hpx
kokkos/4.1.00-openmp
libzmq/4.3.3
lz4/1.9.4
metis/5.1.0
mpifileutils/0.11.1
namd-amd-gfx90a/2.15a2-20211101
namd-amd-gfx90a/2.15a2
nano/7.2
netcdf-c/4.9.0
netcdf-c/4.9.2
netcdf-fortran/4.6.1
netlib-lapack/3.10.1
netlib-lapack/3.11.0
nextflow/22.10.0
nextflow/23.10.0
ninja/1.11.1
omniperf/1.0.6
omniperf/1.0.10
omnitrace/1.10.2
omnitrace/1.11.2-rocm-5.2.3
openblas/0.3.24
opencv/4.8.0
openjdk/17.0.8.1_1
parallel/20220522
parmetis/4.0.3
perl/5.36.0
perl/5.38.0
petsc/3.20.1-nocomplex
plumed/2.9.0
py-boto3/1.18.12
py-boto3/1.26.26
py-cython/0.29.30
py-cython/0.29.36
py-cython/3.0.4
py-dask/2023.4.1
py-h5netcdf/0.10.0
py-h5py/3.8.0
py-hatchet/1.3.1
py-ipython/8.14.0
py-matplotlib/3.8.1
py-mpi4py/3.1.5-py3.11.6
py-numpy/1.23.4
py-numpy/1.26.1
py-pandas/1.5.3
py-pip/23.1.2-py3.10.10
py-pip/23.1.2-py3.11.6
py-plotly/5.14.1
py-setuptools/59.4.0-py3.10.10
py-setuptools/59.4.0-py3.11.6
py-setuptools/68.0.0-py3.10.10
py-setuptools/68.0.0-py3.11.6
python/3.10.10
python/3.11.6
pytorch/2.1.0-rocm5.6.0
pytorch/2.2.0-rocm5.7.3
pytorch/2.2.0-rocm5.7.
pytorch/2.2.0-rocm5.7.3
rclone/1.62.2
rclone/1.63.1
reframe/3.12.0
reframe/4.0.4
rocm/5.4.3
rocm/5.4.6
rocm/5.5.3
rocm/5.6.1
rocm/5.7.0
rocm/5.7.1
rocm/5.7.3
shpc/0.1.23
shpc/0.1.28
singularity/3.11.4-askap
singularity/4.1.0-askap
singularity/3.11.4-askap-gpu
singularity/4.1.0-askap-gpu
singularity/3.11.4-mpi
singularity/4.1.0-mpi
singularity/3.11.4-mpi-gpu
singularity/4.1.0-mpi-gpu
singularity/3.11.4-nohost
singularity/4.1.0-nohost
singularity/3.11.4-nompi
singularity/4.1.0-nompi
singularity/3.11.4-slurm
singularity/4.1.0-slurm
slate/2023.11.05
spack/0.19.0
spack/0.21.0
tower-agent/0.4.5
tower-cli/0.7.0
trilinos/15.0.0
Using the old 2023.08 software stack
As mentioned above, the new default version of the software stack is 2024.05. However, the old version of the stack (2023.08) will still be available alongside the new version (as a “safety net” for potential corner-case problems). If it becomes necessary to use the old software stack, users can still choose to load it. The main steps of the procedure are: unload the default compiler module, swap the pawseyenv module, and reload the compiler module. When using the PrgEnv-gnu programming environment module, the steps are as follows:
In the Cray programming environment, the old software stack (2023.08) can be accessed similarly. However, the previous version of the CCE compiler, cce/15.0.1, used to build the 2023.08 software stack must be reloaded:
Note that while the old software stack is still available, the old versions of some dependencies are not (like MPI libraries). Therefore, the use of the old stack may result in dynamic linking against dependencies which now have versions newer than what they were compiled against. While our testing has not identified any major issues, we still recommend to reserve the use of the old software stack as a last resource for corner case problems.
If you have no other choice but to use the old software stack, please submit a ticket to helpdesk describing the issues that are forcing you into that, so that we can take the necessary actions to fix all the problems introduced with the new software stack.
Research group/user’s own installed software
Based on our testing, users should be able to use any existing local software installations without recompilation, but there may be exceptions. In such cases, user installations of software using spack/0.19.0 will have to be re-executed using spack/0.21.0. Software built or installed by other means will most likely be affected as well, especially if depending on MPI and the Pawsey software stack. Please feel free to reach out to the Pawsey helpdesk for support.
Spack
Spack package manager is our recommended tool for installing packages. Spack has been upgraded from version 0.19.0 to 0.21.0. The new version brings over 7400 software recipes, bug fixes, more supported applications and libraries, and further support for AMD GPUs.
User-private and project-wide software stack directories
For installations using spack, the user-private and project-wide installations are now versioned by the system-wide software stack deployment date. For instance, user-private software is now located at
$MYSOFTWARE/setonix/<DATE_TAG>
where <DATE_TAG> is the date of the related software stack deployment in the YYYY.MM format. The particular version of pawseyenv used determines which user and project installations are visible.
GPU software
Several newer versions of ROCm are now available (up to version 5.7.3). We recommend the use of rocm/5.7.3, which is the version used to build the GPU applications in the new software stack.
Purge policy on /scratch
To ensure ongoing system viability, the following updates to the /scratch filesystem policy are being made from Monday 10 June, 2024:
Purge Policy Change: Files will be automatically deleted after 21 days of inactivity (previously 30 days) to encourage data management best practices.
Quota Implementation: We'll introduce quotas of 1PB capacity per project and 1 million files per user to promote efficient space utilisation.
Known Issues
Module tensorflow/rocm5.6-tf2.12
Problem: This containerised module has shown some problems with the updated environment causing tensorflow scripts to crash in several different ways. The root cause of the problem has not been diagnosed yet.
Temporary fix: If users face problems running their tensorflow scripts, then they should load the old software stack (as described in the sections above) before loading the tensorflow module in their jobscripts (or interactive sessions).