June 2024 Software Update - Important Information

During June 2024 maintenance (4th June,2024), Setonix has been updated to the new Cray Programming Environment (CPE) 23.09 which includes newer MPI libraries and a newer Cray compiler version (16.0.1). In preparation for this, Pawsey has prepared a new software stack, which became available immediately after the maintenance via the default pawseyenv/2024.05 module. The purpose of this page is to consolidate in one place all the important changes, highlights and recommended actions.

Researcher actions

Changes in the CPE, Cray-MPICH, and Pawsey provided software stack immediately applied to all users after June 2024 maintenance. Therefore, every researcher should follow these steps to “upgrade” their own tools and scripts towards the new conditions:

  • Check for version changes in the system-wide software stack provided by Pawsey and HPE for the tool that are part of your workflows.

  • Check for the correct execution of any software managed by the research group. In case of failure, then reinstall (see sections below).

  • Check for the correct execution of containers previously built on Pawsey-provided images for multi-node MPI support. In case of failure, then rebuild from the latest Pawsey-provided image.

  • Update Slurm batch scripts to reflect the new versions of software

(Please refer to the general documentation if you are unsure on how to proceed on any of the proposed steps.)

Further details about the changes are provided in the following sections. We are also including a section at the end for listing “known issues” that have been discovered after maintenance (and that we are working to solve).

What is new in CPE 23.09

The Cray Programming Environment (CPE) is the programming environment provided by the vendor on Setonix. It has been updated to version 23.09, and includes an updated MPI library, and newer Cray compiler (15.0.1 → 16.0.1). The GCC compiler version remains the same. We still recommend against using the AOCC compiler and programming environment (PrgEnv-aocc) due to on-going issues.

Software stack

Pawsey Software Stack

The new version of the software stack is 2024.05, and it replaced 2023.08 software stack. The new software stack has been installed using a newer Spack major release, v0.21.0. We have compiled several packages using the Cray compiler and updated the versions of most packages in both the GNU and Cray programming environments. Users can check for the new versions of the software they used via the modulecommands like:

module avail <moduleName>

In the following expandable sections there is a complete list of changes for the two recommended programming environments:

2023.08 (Old stack for PrgEnv-gnu)

2024.05 (New stack for PrgEnv-gnu)

2023.08 (Old stack for PrgEnv-gnu)

2024.05 (New stack for PrgEnv-gnu)

adios2/2.9.0-hdf5 

adios2/2.9.2-hdf5 

 

adios2/2.9.2 (without hdf5 support)

alphafold/2.2.3  

alphafold/2.3.0

amber/2022

amber/2024 

-

amdblis/3.0.1

apr-util/1.6.1

apr-util/1.6.3

arpack-ng/3.8.0

arpack-ng/3.9.0

awscli/1.27.84

awscli/1.29.41

beast2/2.6.7

beast2/2.7.4

blaspp/2022.07.00 and blaspp/2023.08.25

blaspp/2023.11.05

boost/1.80.0-c++14-python

boost/1.83.0-c++14-python

boost/1.80.0-c++98-python

boost/1.83.0-c++98-python

caliper/2.7.0

caliper/2.10.0

cdo/2.1.0

cdo/2.2.2 

cfitsio/4.1.0

cfitsio/4.3.0 

cmake/3.24.3

cmake/3.27.7

cp2k/2023.1

cp2k/2023.2

cp2k-amd-gfx90a/8.2

cp2k-amd-gfx90a/2022.2

cppzmq/4.7.1  

cppzmq/4.9.0 

feh/3.3  

feh/3.10

ffmpeg/4.4 and ffmpeg/4.4.1

ffmpeg/4.4.1

-

fits2idia/0.1.15

gnuplot/5.4.2

gnuplot/5.4.3

go/1.19.10

go/1.21.3

gromacs/2022.5

gromacs/2022.5-double

gromacs/2023

gromacs/2022.5-mixed

-

gromacs/2023-double

hdf5/1.12.2-api-v110

hdf5/1.14.3-api-v110

hdf5/1.12.2-api-v112

hdf5/1.14.3-api-v112

hdf5/1.12.2-api-v18

hdf5/1.14.3-api-v18

hdf5/1.12.2-parallel-api-v110

hdf5/1.14.3-parallel-api-v110

hdf5/1.12.2-parallel-api-v112

hdf5/1.14.3-parallel-api-v112

hdf5/1.12.2-parallel-api-v18

hdf5/1.14.3-parallel-api-v18

hpc-python-container/2022.03

Removed (No usage)

hpc-python-container/2022.03-hdf5mpi

Removed (No usage)

hpcviewer/2022.10

hpcviewer/2023.07

hpl/2.3

hpl/2.3 and hpl/2.3+openmp

hpx/1.8.1

hpx/1.9.1 

kokkos/3.7.02-hpx

kokkos/4.1.00-hpx 

-

kokkos/4.1.00-openmp

-

kokkos-amd-gfx90a/3.7.02-openmp

-

kokkos-amd-gfx90a/4.1.00-openmp

lammps/20230615

lammps/20230802 

-

lammps/20230802.3 

-

lammps-amd-gfx90a/20230802.3

-

lz4/1.9.4

-

magma-amd-gfx90a/2.7.2

-

metis/5.1.0

-

metis/5.1.0-64bit

namd-amd-gfx90a/2.15a2-20211101

namd-amd-gfx90a/2.15a2

nano/6.3

nano/7.2 

netcdf-c/4.9.0

netcdf-c/4.9.2 

netcdf-fortran/4.6.0 

etcdf-fortran/4.6.1

netlib-lapack/3.10.1 

netlib-lapack/3.11.0 

nextflow/23.04.3  and nextflow/23.10.0

nextflow/23.10.0

-

ninja/1.11.1

omniperf/1.0.6

omniperf/1.0.10

omnitrace/1.10.2

omnitrace/1.11.2-rocm-5.2.3

openblas/0.3.21

openblas/0.3.24

opencv/4.6.0

opencv/4.8.0

openjdk/17.0.5_8 

openjdk/17.0.8.1_1 

osu-micro-benchmarks/5.7.1

osu-micro-benchmarks/7.3 

-

parmetis/4.0.3

-

parmetis/4.0.3-64bit

pennylane/0.33.0

Will be available as containers

pennylane-lightning/0.33.1

Will be available as containers

pennylane-lightning-amd-gfx90a/0.32.0

Will be available as containers

perl/5.36.0

perl/5.38.0

petsc/3.18.1-complex, petsc/3.19.5-complex and petsc/3.19.5-nocomplex

petsc/3.20.1-complex and petsc/3.20.1-nocomplex

plasma/22.9.29 

plasma/23.8.2 

py-boto3/1.18.12

py-boto3/1.26.26

py-cython/0.29.30

py-cython/0.29.36

py-cython/0.29.32

py-cython/3.0.4

py-dask/2021.6.2

py-dask/2023.4.1

py-h5py/3.7.0 

py-h5py/3.8.0

py-healpy/1.14.0

Removed (No usage)

py-ipython/8.5.0

py-ipython/8.14.0

py-matplotlib/3.6.2

py-matplotlib/3.8.1

py-mpi4py/3.1.4-py3.10.10

py-mpi4py/3.1.5-py3.11.6

py-netcdf4/1.5.8

py-netcdf4/1.6.2

py-numba/0.56.0

py-numba/0.57.0

py-numpy/1.22.4

py-numpy/1.24.4

py-numpy/1.23.4

py-numpy/1.26.1

py-pandas/1.5.1

py-pandas/1.5.3 and py-pandas/2.1.2

py-pip/23.1.2-py3.10.10

py-pip/23.1.2-py3.11.6

py-pip/23.1.2-py3.10.8

py-pip/23.1.2-py3.9.15

py-plotly/5.2.2

py-plotly/5.14.1

py-scikit-learn/1.1.3

py-scikit-learn/1.3.2

py-scipy/1.8.1

py-scipy/1.11.3

py-setuptools/59.4.0-py3.10.10

py-setuptools/59.4.0-py3.11.6

py-setuptools/59.4.0-py3.10.8

py-setuptools/59.4.0-py3.9.15

py-setuptools/68.0.0-py3.10.10

py-setuptools/68.0.0-py3.11.6

python/3.10.10

py-setuptools/68.0.0-py3.9.15

python/3.10.8

python/3.11.6

pytorch/2.1.0-rocm5.6.0

pytorch/2.2.0-rocm5.7.3

pytorch/2.2.0-rocm5.7.3

 

pytorch/2.2.0-rocm5.7.3

 

qiskit/0.13.0

Will be available as containers

qiskit-amd-gfx90a/0.13.0

Will be available as containers

r/4.2.2 

r/4.3.0

rclone/1.62.2 

rclone/1.63.1

 

reframe/4.0.4 

 

rocm/5.4.6

 

rocm/5.5.3

 

rocm/5.6.1

 

rocm/5.7.0

 

rocm/5.7.1

 

rocm/5.7.3

rust/1.60.0

rust/1.78.0

 

scotch/7.0.4

 

scotch/7.0.4-metis

shpc/0.1.23

shpc/0.1.28

singularity/3.11.4-askap

singularity/4.1.0-askap

singularity/3.11.4-askap-gpu

singularity/4.1.0-askap-gpu

singularity/3.11.4-mpi

singularity/4.1.0-mpi

singularity/3.11.4-mpi-gpu

singularity/4.1.0-mpi-gpu

singularity/3.11.4-nohost

singularity/4.1.0-nohost

singularity/3.11.4-nompi

singularity/4.1.0-nompi

singularity/3.11.4-slurm

singularity/4.1.0-slurm

singularity/ashley

-

singularityce/3.11.4

singularityce/4.1.0

slate/2022.07.00 and slate/2023.08.25

slate/2023.11.05 

spack/0.19.0

spack/0.21.0

squashfs/4.5.1 

squashfs/4.6.1 

subversion/1.14.1

subversion/1.14.2

trilinos/13.4.0

trilinos/15.0.0

 

trilinos-amd-gfx90a/15.0.0

 

wcslib/7.3-nocfitsio 

xerces-c/3.2.3-transcoder-gnuiconv

xerces-c/3.2.4-transcoder-gnuiconv 

 

xerces-c/3.2.4-transcoder-iconv

2023.08 (Old stack for PrgEnv-cray)

2024.05 (New stack For PrgEnv-cray)

2023.08 (Old stack for PrgEnv-cray)

2024.05 (New stack For PrgEnv-cray)

alphafold/2.2.3  

alphafold/2.3.0

 

adios2/2.9.2-hdf5 

 

amdblis/3.0.1

awscli/1.27.84 

awscli/1.29.41 

 

blaspp/2023.08.25                                       

 

blaspp/2023.11.05  

boost/1.80.0-c++14-python

boost/1.83.0-c++14-python

boost/1.80.0-c++98-python

boost/1.83.0-c++98-python  

 

caliper/2.10.0 

cmake/3.24.3

cmake/3.27.7

 

cp2k-amd-gfx90a/2022.2  

 

cray-hdf5/1.12.2.7

 

cray-hdf5-parallel/1.12.2.7

 

cray-mpixlate/1.0.2 

 

cray-parallel-netcdf/1.12.3.7 

 

feh/3.10 

 

ffmpeg/4.4.1

go/1.19.10

go/1.21.3  

hdf5/1.12.2-api-v110

hdf5/1.14.3-api-v110

hdf5/1.12.2-api-v112 

hdf5/1.14.3-api-v112 

hdf5/1.12.2-api-v18 

hdf5/1.14.3-api-v18 

hdf5/1.12.2-parallel-api-v110

hdf5/1.14.3-parallel-api-v110

hdf5/1.12.2-parallel-api-v112

hdf5/1.14.3-parallel-api-v112

hdf5/1.12.2-parallel-api-v18

hdf5/1.14.3-parallel-api-v18

hpc-python-container/2022.03 

Deprecated

hpc-python-container/2022.03-hdf5mpi

Deprecated

 

hpcviewer/2023.0

 

hpx/1.9.1 

 

kokkos/4.1.00-hpx

 

kokkos/4.1.00-openmp

 

libzmq/4.3.3

 

lz4/1.9.4

 

metis/5.1.0

 

mpifileutils/0.11.1

namd-amd-gfx90a/2.15a2-20211101

namd-amd-gfx90a/2.15a2 

 

nano/7.2 

netcdf-c/4.9.0 

netcdf-c/4.9.2 

 

netcdf-fortran/4.6.1 

netlib-lapack/3.10.1

netlib-lapack/3.11.0

 

nextflow/22.10.0 

 

nextflow/23.10.0 

 

ninja/1.11.1 

omniperf/1.0.6

omniperf/1.0.10 

omnitrace/1.10.2

omnitrace/1.11.2-rocm-5.2.3

 

openblas/0.3.24

 

opencv/4.8.0 

 

openjdk/17.0.8.1_1 

 

parallel/20220522 

 

parmetis/4.0.3

perl/5.36.0

perl/5.38.0 

 

petsc/3.20.1-nocomplex 

 

plumed/2.9.0 

py-boto3/1.18.12

py-boto3/1.26.26 

py-cython/0.29.30

py-cython/0.29.36 

 

py-cython/3.0.4

 

py-dask/2023.4.1

 

py-h5netcdf/0.10.0

 

py-h5py/3.8.0

 

py-hatchet/1.3.1

 

py-ipython/8.14.0

 

py-matplotlib/3.8.1

 

py-mpi4py/3.1.5-py3.11.6

py-numpy/1.23.4

py-numpy/1.26.1

 

py-pandas/1.5.3

py-pip/23.1.2-py3.10.10 

py-pip/23.1.2-py3.11.6

 

py-plotly/5.14.1

py-setuptools/59.4.0-py3.10.10

py-setuptools/59.4.0-py3.11.6

py-setuptools/68.0.0-py3.10.10 

py-setuptools/68.0.0-py3.11.6

python/3.10.10

python/3.11.6

pytorch/2.1.0-rocm5.6.0

pytorch/2.2.0-rocm5.7.3

pytorch/2.2.0-rocm5.7.

 

pytorch/2.2.0-rocm5.7.3

 

rclone/1.62.2

rclone/1.63.1

 

reframe/3.12.0

 

reframe/4.0.4

rocm/5.4.3

rocm/5.4.6 

 

rocm/5.5.3

 

rocm/5.6.1

 

rocm/5.7.0

 

rocm/5.7.1

 

rocm/5.7.3

shpc/0.1.23 

shpc/0.1.28

singularity/3.11.4-askap

singularity/4.1.0-askap

singularity/3.11.4-askap-gpu

singularity/4.1.0-askap-gpu

singularity/3.11.4-mpi

singularity/4.1.0-mpi

singularity/3.11.4-mpi-gpu

singularity/4.1.0-mpi-gpu

singularity/3.11.4-nohost

singularity/4.1.0-nohost

singularity/3.11.4-nompi

singularity/4.1.0-nompi

singularity/3.11.4-slurm

singularity/4.1.0-slurm

 

slate/2023.11.05

spack/0.19.0

spack/0.21.0

 

tower-agent/0.4.5

 

tower-cli/0.7.0

 

trilinos/15.0.0

Using the old 2023.08 software stack

As mentioned above, the new default version of the software stack is 2024.05. However, the old version of the stack (2023.08) will still be available alongside the new version (as a “safety net” for potential corner-case problems). If it becomes necessary to use the old software stack, users can still choose to load it. The main steps of the procedure are: unload the default compiler module, swap the pawseyenv module, and reload the compiler module. When using the PrgEnv-gnu programming environment module, the steps are as follows:

module unload gcc/12.2.0 module swap pawseyenv/2024.05 pawseyenv/2023.08 module load gcc/12.2.0

In the Cray programming environment, the old software stack (2023.08) can be accessed similarly. However, the previous version of the CCE compiler, cce/15.0.1, used to build the 2023.08 software stack must be reloaded:

module unload cce/16.0.1 module swap pawseyenv/2024.05 pawseyenv/2023.08 module load cce/15.0.1

Note that while the old software stack is still available, the old versions of some dependencies are not (like MPI libraries). Therefore, the use of the old stack may result in dynamic linking against dependencies which now have versions newer than what they were compiled against. While our testing has not identified any major issues, we still recommend to reserve the use of the old software stack as a last resource for corner case problems.

If you have no other choice but to use the old software stack, please submit a ticket to helpdesk describing the issues that are forcing you into that, so that we can take the necessary actions to fix all the problems introduced with the new software stack.

Research group/user’s own installed software

Based on our testing, users should be able to use any existing local software installations without recompilation, but there may be exceptions. In such cases, user installations of software using spack/0.19.0 will have to be re-executed using spack/0.21.0. Software built or installed by other means will most likely be affected as well, especially if depending on MPI and the Pawsey software stack. Please feel free to reach out to the Pawsey helpdesk for support.

Spack

Spack package manager is our recommended tool for installing packages. Spack has been upgraded from version 0.19.0 to 0.21.0. The new version brings over 7400 software recipes, bug fixes, more supported applications and libraries, and further support for AMD GPUs.

User-private and project-wide software stack directories

For installations using spack, the user-private and project-wide installations are now versioned by the system-wide software stack deployment date. For instance, user-private software is now located at

$MYSOFTWARE/setonix/<DATE_TAG> 

where <DATE_TAG>  is the date of the related software stack deployment in the YYYY.MM  format. The particular version of pawseyenv used determines which user and project installations are visible.

GPU software

Several newer versions of ROCm are now available (up to version 5.7.3). We recommend the use of rocm/5.7.3, which is the version used to build the GPU applications in the new software stack.

 

 

Purge policy on /scratch

To ensure ongoing system viability, the following updates to the /scratch filesystem policy are being made from Monday 10 June, 2024:

  • Purge Policy Change: Files will be automatically deleted after 21 days of inactivity (previously 30 days) to encourage data management best practices.

  • Quota Implementation: We'll introduce quotas of 1PB capacity per project and 1 million files per user to promote efficient space utilisation.

Known Issues

Module tensorflow/rocm5.6-tf2.12

  • Problem: This containerised module has shown some problems with the updated environment causing tensorflow scripts to crash in several different ways. The root cause of the problem has not been diagnosed yet.

  • Temporary fix: If users face problems running their tensorflow scripts, then they should load the old software stack (as described in the sections above) before loading the tensorflow module in their jobscripts (or interactive sessions).