During March 2025 maintenance (4th March,2025), Setonix nodes have been reimaged with COS version 23.11.4 which contains important security patches. The ROCm driver has also been updated from the previous version that supported ROCm 5.5 through 6.1 to a newer driver that supports ROCm 5.7 through 6.3. This ensures researchers can continue to make use of the latest GPU features and has allowed an updated software stack built against ROCm 6.2.4 to be provided with new software versions. To minimise disruption, the existing software stack built against ROCm 5.7.3 will remain the default.
The new software stack is based on the new Cray Programming Environment (CPE) 24.07 which includes newer MPI libraries and newer versions of GCC (13.3.0) and Cray compilers (18.0.0). In preparation for this, Pawsey has prepared a new software stack, which became available immediately after the maintenance via the non-default pawseyenv/2025.03 module. The purpose of this page is to consolidate in one place all the important changes, highlights and recommended actions.
Researcher actions
Since the existing software stack (2024.05) will remain the default, no immediate action is required from users. However, we strongly recommend users to “upgrade” their tools and scripts towards a new software stack which includes newer MPI libraries, newer versions of GCC (13.3.0) and Cray compilers (18.0.0) and the GPU software based on newer version of ROCm, 6.2.4.
Using the new 2025.03 software stack
As mentioned above, the default version of the software stack will remain to be 2024.05. In order to switch to a new software stack the following three steps procedure is required:
When using the PrgEnv-gnu programming environment module, the steps are as follows:
module load cpe/24.07
module swap pawseyenv/2024.05 pawseyenv/2025.03
module load gcc-native/13.2 |
|
Please note that even though the module name is gcc-native/13.2, it actually corresponds to GCC version 13.3.0. Since the module is part of cpe/24.07, Pawsey staff is unable to change its name. |
In the Cray programming environment, the new software stack (2025.03) can be accessed similarly. However, the new version of the CCE compiler, cce/18.0.0, used to build the 2025.03 software stack must be reloaded:
module load PrgEnv-cray/8.5.0
module load cce/18.0.0
module load cpe/24.07
module swap pawseyenv/2024.05 pawseyenv/2025.03
module load cce/18.0.0 |
If you encounter the issues with using the new software stack, please submit a ticket to helpdesk describing the problem, so that we can take the necessary actions to fix it. |
What is new in CPE 24.07
The Cray Programming Environment (CPE) is the programming environment provided by the vendor on Setonix. It has been updated to version 24.07, and includes an updated MPI library, newer Cray compiler (16.0.1 → 18.0.0) and newer GCC compiler (12.2.0 → 13.3.0). We still recommend against using the AOCC compiler and programming environment (PrgEnv-aocc) due to on-going issues.
Software stack
Pawsey Software Stack
The new version of the software stack is 2025.03. The new software stack has been installed using a newer Spack major release, v0.23.0. We have compiled several packages using the Cray compiler and updated the versions of most packages in both the GNU and Cray programming environments. Users can check for the new versions of the software they used via the modulecommands like:
module avail <moduleName> |
In the following expandable sections there is a complete list of changes for the PrgEnv-gnu programming environments:
2024.05 (The default stack for PrgEnv-gnu) | 2025.03 (The new stack for PrgEnv-gnu) |
|---|
adios2/2.8.3-hdf5 adios2/2.9.2 adios2/2.9.2-hdf5 amdblis/3.0.1 ansys-fluids/2022R1 ansys-fluids/2023R1 apr/1.5.2 apr-util/1.6.3 arpack-ng/3.9.0 autoconf/2.69 automake/1.16.5 awscli/1.29.41 beast1/1.10.4 beast2/2.7.4 birli/0.15.1 blaspp/2023.11.05 boost/1.83.0-c++14-python boost/1.83.0-c++98-python calceph/3.5.5 caliper/2.10.0 casacore/3.2.1 casacore/3.3.0 casacore/3.3.0-adios2 casacore/3.3.0-adios2-openmp casacore/3.3.0-openmp casacore/3.4.0 casacore/3.4.0-adios2 casacore/3.4.0-adios2-openmp casacore/3.4.0-openmp casacore/3.5.0-openmp cdo/2.2.2 cfitsio/3.49 cfitsio/4.3.0 charmpp/7.0.0 chgcentre/1.6 cmake/3.27.7 cp2k/2023.2 cpmd/4.3 cppunit/1.14.0 cppzmq/4.9.0 eigen/3.4.0 exabayes/1.5.1 examl/3.0.22 feh/3.10 ffmpeg/4.4.1 fftw/2.1.5 fftw/3.3.10 fits2idia/0.1.15 giant-squid/1.0.3 gnuplot/5.4.3 go/1.21.3 gromacs/2022.5-double gromacs/2022.5-mixed gromacs/2023-double gromacs/2023-mixed gromacs-amd-gfx90a/2023 gsl/2.7.1 hdf5/1.10.7-api-v110 hdf5/1.10.7-api-v18 hdf5/1.10.7-parallel-api-v110 hdf5/1.10.7-parallel-api-v18 hdf5/1.14.3-api-v110 hdf5/1.14.3-api-v112 hdf5/1.14.3-api-v18 hdf5/1.14.3-parallel-api-v110 hdf5/1.14.3-parallel-api-v112 hdf5/1.14.3-parallel-api-v18 hipfort/5.7.1 hpcviewer/2023.07 hpl/2.3 hpl/2.3+openmp hpx/1.9.1 hyperbeam/0.9.3 hyperbeam-amd-gfx90a/0.9.3 hyperdrive/0.4.1 hyperdrive-amd-gfx90a/0.4.1 ior/3.3.0 kokkos/3.7.02-openmp kokkos/4.1.00-hpx kokkos/4.1.00-openmp kokkos-amd-gfx90a/3.7.02-openmp kokkos-amd-gfx90a/4.1.00-openmp lammps/20230802 lammps/20230802.3 lammps-amd-gfx90a/20230802 lammps-amd-gfx90a/20230802.3 libtool/2.4.7 libzmq/4.3.3 log4cxx/0.12.1-c++11 log4cxx/0.12.1-c++17 lz4/1.9.4 magma-amd-gfx90a/2.7.2 mcpp/2.7.2 metis/5.1.0 metis/5.1.0-64bit mpifileutils/0.11.1 mwalib/1.5.0 namd/2.14 namd/2.15a2 nano/7.2 ncl/6.6.2 nco/5.1.6 ncview/2.1.9 nekrs-amd-gfx90a/23.0 nektar/5.0.2 netcdf-c/4.9.2 netcdf-cxx/4.2 netcdf-cxx4/4.3.1 netcdf-fortran/4.6.1 netlib-lapack/3.11.0 netlib-scalapack/2.2.0 nextflow/22.10.0 nextflow/23.10.0 nextflow/24.04.3 ninja/1.11.1 nwchem/7.2.0 omniperf/1.0.10 omnitrace/1.11.2-rocm-5.2.3 openblas/0.3.24 opencv/4.8.0 openfoam/2206 openjdk/17.0.8.1_1 osu-micro-benchmarks/7.3 parallel/20220522 parallel-netcdf/1.12.3 parmetis/4.0.3 parmetis/4.0.3-64bit perl/5.38.0 petsc/3.20.1-complex petsc/3.20.1-nocomplex pgplot/5.2.2 plasma/23.8.2 plumed/2.8.2 plumed/2.9.0 py-astropy/4.2.1 py-astropy/5.1 py-boto3/1.26.26 py-cython/0.29.36 py-cython/3.0.4 py-dask/2023.4.1 py-emcee/3.1.1 py-funcsigs/1.0.2 py-h5netcdf/0.10.0 py-h5py/3.8.0 py-hatchet/1.3.1 py-ipython/8.14.0 py-matplotlib/3.8.1 py-mpi4py/3.1.5-py3.11.6 py-netcdf4/1.6.2 py-numba/0.57.0 py-numpy/1.24.4 py-numpy/1.26.1 py-pandas/1.5.3 py-pandas/2.1.2 py-pip/23.1.2-py3.11.6 py-pip/23.1.2-py3.9.15 py-plotly/5.14.1 py-scikit-learn/1.3.2 py-scipy/1.11.3 py-setuptools/59.4.0-py3.11.6 py-setuptools/59.4.0-py3.9.15 py-setuptools/68.0.0-py3.11.6 py-setuptools/68.0.0-py3.9.15 python/3.11.6 python/3.9.15 quantum-espresso/7.2 r/4.3.0 r/4.4.1 rclone/1.63.1 reframe/3.12.0 reframe/4.0.4 rocm/5.4.6 rocm/5.5.3 rocm/5.6.1 rocm/5.7.0 rocm/5.7.1 rocm/5.7.3 ruby/3.1.0 rust/1.78.0 scotch/7.0.4 scotch/7.0.4-metis slate/2023.11.05 squashfs/4.6.1 subversion/1.14.2 subversion/1.9.7 tower-agent/0.4.5 tower-cli/0.7.0 trilinos/15.0.0 trilinos-amd-gfx90a/15.0.0 vasp/5.4.4 vasp6/6.2.1 vasp6/6.3.0 wcslib/7.3 wcslib/7.3-nocfitsio wcstools/3.9.7 wsclean/2.9 wsclean/3.4-idg-everybeam wsclean/3.4-idg-everybeam-mpi xerces-c/3.2.4-transcoder-gnuiconv xerces-c/3.2.4-transcoder-iconv | adios2/2.8.3-hdf5 adios2/2.9.2-hdf5 adios2/2.10.0 adios2/2.10.0-hdf5 adios2/2.10.2-hdf5 MISSING ansys-fluids/2022R1 ansys-fluids/2023R1 apr/1.7.5 apr-util/1.6.3 arpack-ng/3.9.0 autoconf/2.69 automake/1.16.5 awscli/1.29.41 beast1/1.10.4 beast2/2.7.4 birli/0.15.1 blaspp/2024.10.26 blaspp-amd-gfx90a/2024.10.26 boost/1.83.0-c++11-python boost/1.86.0-c++11-python boost/1.86.0-c++14-python calceph/3.5.5 caliper/2.11.0 casacore/3.2.1 casacore/3.3.0 casacore/3.3.0-adios2 casacore/3.3.0-adios2-openmp casacore/3.3.0-openmp casacore/3.4.0 casacore/3.4.0-adios2 casacore/3.4.0-adios2-openmp casacore/3.4.0-openmp casacore/3.5.0-openmp cdo/2.2.2 cdo/2.4.4 cfitsio/3.49 cfitsio/4.4.0 charmpp/7.0.0 charmpp/8.0.0 chgcentre/1.6 cmake/3.30.5 cp2k/2023.2 cp2k/2024.3 cpmd/4.3 cppunit/1.15.1 cppzmq/4.10.0 eigen/3.4.0 everybeam/0.5.8 exabayes/1.5.1 examl/3.0.22 feh/3.10 ffmpeg/7.0.2 fftw/2.1.5 fftw/3.3.10 MISSING giant-squid/1.0.3 gnuplot/6.0.0 go/1.23.2 gromacs/2022.5-double gromacs/2022.5-mixed gromacs/2023-double gromacs/2023-mixed gromacs/2024.3-double gromacs/2024.3-mixed gromacs-amd-gfx90a/2023 gsl/2.8 hdf5/1.10.11-api-v110 hdf5/1.10.7-api-v110 hdf5/1.10.7-api-v18 hdf5/1.10.7-parallel-api-v110 hdf5/1.10.7-parallel-api-v18 hdf5/1.12.2-parallel-api-v112 hdf5/1.14.3-api-v110 hdf5/1.14.3-api-v112 hdf5/1.14.3-api-v18 hdf5/1.14.3-parallel-api-v110 hdf5/1.14.3-parallel-api-v112 hdf5/1.14.3-parallel-api-v18 hdf5/1.14.5-parallel-api-v112 hpc-python-collection/-py MISSING hpcviewer/2024.09 hpl/2.3 hpl/2.3+openmp hpx/1.9.1 hyperbeam/0.9.3 hyperbeam-amd-gfx90a/0.9.3 hyperdrive/0.4.1-peel hyperdrive-amd-gfx90a/0.4.1-peel idg/1.0.0 ior/3.3.0 kokkos/3.7.02-openmp kokkos/4.4.01-hpx kokkos/4.4.01-openmp kokkos-amd-gfx90a/4.1.00-openmp kokkos-amd-gfx90a/4.4.01-openmp lammps/20230802.4 lammps/20240829.1 lammps-amd-gfx90a/20230802.4 DISCONTINUED libtool/2.4.7 libzmq/4.3.5 log4cxx/1.2.0-c++11 log4cxx/1.2.0-c++17 lz4/1.10.0 magma-amd-gfx90a/2.8.0 mcpp/2.7.2-25-g619046f metis/5.1.0 metis/5.1.0-64bit mpifileutils/0.11.1 mwalib/1.5.0 namd2/2.14 namd3/3.0.1 nano/8.2 ncl/6.6.2 nco/5.1.6 nco/5.2.4 ncview/2.1.9 nekrs-amd-gfx90a/23.0 nektar/5.0.2 netcdf-c/4.9.2 netcdf-cxx/4.2 netcdf-cxx4/4.3.1 netcdf-fortran/4.6.1 netlib-lapack/3.11.0 netlib-scalapack/2.2.0 nextflow/22.10.0 nextflow/24.10.0 ninja/1.12.1 nwchem/7.2.0 nwchem/7.2.3 MISSING MISSING openblas/0.3.24 openblas/0.3.28 opencv/4.10.0 openfoam/2206 openfoam/2312 openjdk/17.0.8.1_1 osu-micro-benchmarks/7.5 parallel/20240822 parallel-netcdf/1.12.3 parmetis/4.0.3 parmetis/4.0.3-64bit perl/5.40.0 petsc/3.21.1-complex petsc/3.21.1-nocomplex pgplot/5.2.2 plasma/23.8.2 plumed/2.8.2 plumed/2.9.0 plumed/2.9.2 presto/5.0.1 py-astropy/4.2.1 py-astropy/5.1 py-boto3/1.34.44 py-cython/0.29.36 py-cython/3.0.11 py-cython/3.0.8 py-dask/2023.4.1 py-emcee/3.1.1 py-funcsigs/1.0.2 py-h5netcdf/1.3.0 py-h5py/3.11.0 py-h5py/3.12.1 py-hatchet/1.4.0 py-ipython/8.14.0 py-matplotlib/3.8.4 py-matplotlib/3.9.2 py-mpi4py/3.1.5-py3.11.6 py-mpi4py/4.0.1-py3.11.6 py-mpi4py/4.0.1-py3.9.15 py-netcdf4/1.6.5 py-numba/0.58.1 py-numpy/1.25.2 py-numpy/1.26.4 py-numpy/2.1.2 py-pandas/1.5.3 py-pandas/2.2.2 py-pip/23.1.2-py3.11.6 py-pip/23.1.2-py3.9.15 py-plotly/5.20.0 py-scikit-learn/1.4.2 py-scipy/1.13.0 py-scipy/1.14.1 py-setuptools/59.4.0-py3.11.6 py-setuptools/59.4.0-py3.9.15 py-setuptools/63.4.3-py3.11.6 py-setuptools/69.2.0-py3.11.6 python/3.11.6 python/3.9.15 quantum-espresso/7.2 quantum-espresso/7.4 r/4.4.1 rclone/1.68.1 reframe/3.12.0 MISSING rocm/5.7.0 rocm/5.7.1 rocm/5.7.3 rocm/6.2.0 rocm/6.2.1 rocm/6.2.4 rocm/6.3.0 rocm/6.3.1 rocm/6.3.2 ruby/3.3.5 rust/1.84.1 scotch/7.0.4 scotch/7.0.4-metis singularityce/3.11.4 singularityce/4.1.0 slate-amd-gfx90a/2024.10.29 squashfs/4.6.1 subversion/1.14.2 DEPRICATED DEPRICATED trilinos/15.0.0 trilinos-amd-gfx90a/15.0.0 vasp/5.4.4 vasp6/6.2.1 vasp6/6.3.0 vasp6/6.4.3 wcslib/7.3 wcslib/7.3-nocfitsio wcstools/3.9.7 MISSING wsclean/3.4-idg-everybeam wsclean/3.4-idg-everybeam-mpi xerces-c/3.3.0-transcoder-gnuiconv xerces-c/3.3.0-transcoder-iconv |
|
Spack
Spack package manager is our recommended tool for installing packages. Spack has been upgraded In the new software stack from version 0.21.0 to 0.23.0. The new version brings over 8325 software recipes, bug fixes, more supported applications and libraries, and further support for AMD GPUs.
User-private and project-wide software stack directories
For installations using spack, the user-private and project-wide installations are now versioned by the system-wide software stack deployment date. For instance, user-private software is now located at
$MYSOFTWARE/setonix/<DATE_TAG>
where <DATE_TAG> is the date of the related software stack deployment in the YYYY.MM format. The particular version of pawseyenv used determines which user and project installations are visible.
GPU software
Several newer versions of ROCm are now available (up to version 6.3.2). We recommend the use of rocm/6.2.4, which is the version used to build the GPU applications in the new software stack.
module avail rocm
-------------------------------------- /software/setonix/2025.03/modules/zen3/gcc/13.3.0/programming-languages ---------------------------------------
rocm/5.7.0 rocm/5.7.1 rocm/5.7.3 rocm/6.2.0 rocm/6.2.1 rocm/6.2.4 rocm/6.3.0 rocm/6.3.1 rocm/6.3.2 |
|
Known Issues in the new 2025.03 stack
py-numpy in PrgEnv-cray environment:
Currently, the py-numpy/1.25.2 , py-numpy/1.26.4 and py-numpy/2.1.2 packages in PrgEnv-cray environment exhibit lower performance. If this impacts your work, please submit a helpdesk ticket, and Pawsey staff will assist you.
Issues with running MWA packages:
Currently, some CPU packages such as MWALib, Hyperbeam and Hyperdrive are looking for AMD GPU libraries (libamdhip64.so.5 and libamdhip64.so.6). As a workaround, users can include the following export statement in their job scripts:
export LD_LIBRARY_PATH=/software/setonix/rocm/5.7.3/lib:/software/setonix/rocm/6.2.4/lib:$LD_LIBRARY_PATH |
Missing software
In the new software stack 2025.03, a few software packages, such as Amber, Fits2Idia, Omnitrace, and Omniperf, are missing and require manual installation. Pawsey staff are currently working on installing these packages.