Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
/
March 2025 Software Update - Important Information
March 2025 Software Update - Important Information
During March 2025 maintenance (4th March,2025), Setonix nodes have been reimaged with COS version 23.11.4 which contains important security patches. The ROCm driver has also been updated from the previous version that supported ROCm 5.5 through 6.1 to a newer driver that supports ROCm 5.7 through 6.3. This ensures researchers can continue to make use of the latest GPU features and has allowed an updated software stack built against ROCm 6.2.4 to be provided with new software versions. To minimise disruption, the existing software stack built against ROCm 5.7.3 will remain the default.
The new software stack is based on the new Cray Programming Environment (CPE) 24.07 which includes newer MPI libraries and newer versions of GCC (13.3.0) and Cray compilers (18.0.0). In preparation for this, Pawsey has prepared a new software stack, which became available immediately after the maintenance via the non-default pawseyenv/2025.03 module. The purpose of this page is to consolidate in one place all the important changes, highlights and recommended actions.
Researcher actions
Since the existing software stack (2024.05) will remain the default, no immediate action is required from users. However, we strongly recommend users to “upgrade” their tools and scripts towards a new software stack which includes newer MPI libraries, newer versions of GCC (13.3.0) and Cray compilers (18.0.0) and the GPU software based on newer version of ROCm, 6.2.4.
Using the new 2025.03 software stack
As mentioned above, the default version of the software stack will remain to be 2024.05. In order to switch to a new software stack the following three steps procedure is required:
load the new CPE module
swap the pawseyenv module to the new one
load the compiler module used in the new software stack
When using the PrgEnv-gnu programming environment module, the steps are as follows:
Please note that even though the module name is gcc-native/13.2, it actually corresponds to GCC version 13.3.0. Since the module is part of cpe/24.07, Pawsey staff is unable to change its name.
In the Cray programming environment, the new software stack (2025.03) can be accessed similarly. However, the new version of the CCE compiler, cce/18.0.0, used to build the 2025.03 software stack must be reloaded:
If you encounter the issues with using the new software stack, please submit a ticket to helpdesk describing the problem, so that we can take the necessary actions to fix it.
What is new in CPE 24.07
The Cray Programming Environment (CPE) is the programming environment provided by the vendor on Setonix. It has been updated to version 24.07, and includes an updated MPI library, newer Cray compiler (16.0.1 → 18.0.0) and newer GCC compiler (12.2.0 → 13.3.0). We still recommend against using the AOCC compiler and programming environment (PrgEnv-aocc) due to on-going issues.
Software stack
Pawsey Software Stack
The new version of the software stack is 2025.03. The new software stack has been installed using a newer Spack major release, v0.23.0. We have compiled several packages using the Cray compiler and updated the versions of most packages in both the GNU and Cray programming environments. Users can check for the new versions of the software they used via the modulecommands like:
module avail <moduleName>
In the following expandable sections there is a complete list of changes for the PrgEnv-gnu programming environments:
2024.05 (The default stack for PrgEnv-gnu)
2025.03 (The new stack for PrgEnv-gnu)
2024.05 (The default stack for PrgEnv-gnu)
2025.03 (The new stack for PrgEnv-gnu)
adios2/2.8.3-hdf5
adios2/2.9.2
adios2/2.9.2-hdf5
amdblis/3.0.1
ansys-fluids/2022R1
ansys-fluids/2023R1
apr/1.5.2
apr-util/1.6.3
arpack-ng/3.9.0
autoconf/2.69
automake/1.16.5
awscli/1.29.41
beast1/1.10.4
beast2/2.7.4
birli/0.15.1
blaspp/2023.11.05
boost/1.83.0-c++14-python
boost/1.83.0-c++98-python
calceph/3.5.5
caliper/2.10.0
casacore/3.2.1
casacore/3.3.0
casacore/3.3.0-adios2
casacore/3.3.0-adios2-openmp
casacore/3.3.0-openmp
casacore/3.4.0
casacore/3.4.0-adios2
casacore/3.4.0-adios2-openmp
casacore/3.4.0-openmp
casacore/3.5.0-openmp
cdo/2.2.2
cfitsio/3.49
cfitsio/4.3.0
charmpp/7.0.0
chgcentre/1.6
cmake/3.27.7
cp2k/2023.2
cpmd/4.3
cppunit/1.14.0
cppzmq/4.9.0
eigen/3.4.0
exabayes/1.5.1
examl/3.0.22
feh/3.10
ffmpeg/4.4.1
fftw/2.1.5
fftw/3.3.10
fits2idia/0.1.15
giant-squid/1.0.3
gnuplot/5.4.3
go/1.21.3
gromacs/2022.5-double
gromacs/2022.5-mixed
gromacs/2023-double
gromacs/2023-mixed
gromacs-amd-gfx90a/2023
gsl/2.7.1
hdf5/1.10.7-api-v110
hdf5/1.10.7-api-v18
hdf5/1.10.7-parallel-api-v110
hdf5/1.10.7-parallel-api-v18
hdf5/1.14.3-api-v110
hdf5/1.14.3-api-v112
hdf5/1.14.3-api-v18
hdf5/1.14.3-parallel-api-v110
hdf5/1.14.3-parallel-api-v112
hdf5/1.14.3-parallel-api-v18
hipfort/5.7.1
hpcviewer/2023.07
hpl/2.3
hpl/2.3+openmp
hpx/1.9.1
hyperbeam/0.9.3
hyperbeam-amd-gfx90a/0.9.3
hyperdrive/0.4.1
hyperdrive-amd-gfx90a/0.4.1
ior/3.3.0
kokkos/3.7.02-openmp
kokkos/4.1.00-hpx
kokkos/4.1.00-openmp
kokkos-amd-gfx90a/3.7.02-openmp
kokkos-amd-gfx90a/4.1.00-openmp
lammps/20230802
lammps/20230802.3
lammps-amd-gfx90a/20230802
lammps-amd-gfx90a/20230802.3
libtool/2.4.7
libzmq/4.3.3
log4cxx/0.12.1-c++11
log4cxx/0.12.1-c++17
lz4/1.9.4
magma-amd-gfx90a/2.7.2
mcpp/2.7.2
metis/5.1.0
metis/5.1.0-64bit
mpifileutils/0.11.1
mwalib/1.5.0
namd/2.14
namd/2.15a2
nano/7.2
ncl/6.6.2
nco/5.1.6
ncview/2.1.9
nekrs-amd-gfx90a/23.0
nektar/5.0.2
netcdf-c/4.9.2
netcdf-cxx/4.2
netcdf-cxx4/4.3.1
netcdf-fortran/4.6.1
netlib-lapack/3.11.0
netlib-scalapack/2.2.0
nextflow/22.10.0
nextflow/23.10.0
nextflow/24.04.3
ninja/1.11.1
nwchem/7.2.0
omniperf/1.0.10
omnitrace/1.11.2-rocm-5.2.3
openblas/0.3.24
opencv/4.8.0
openfoam/2206
openjdk/17.0.8.1_1
osu-micro-benchmarks/7.3
parallel/20220522
parallel-netcdf/1.12.3
parmetis/4.0.3
parmetis/4.0.3-64bit
perl/5.38.0
petsc/3.20.1-complex
petsc/3.20.1-nocomplex
pgplot/5.2.2
plasma/23.8.2
plumed/2.8.2
plumed/2.9.0
py-astropy/4.2.1
py-astropy/5.1
py-boto3/1.26.26
py-cython/0.29.36
py-cython/3.0.4
py-dask/2023.4.1
py-emcee/3.1.1
py-funcsigs/1.0.2
py-h5netcdf/0.10.0
py-h5py/3.8.0
py-hatchet/1.3.1
py-ipython/8.14.0
py-matplotlib/3.8.1
py-mpi4py/3.1.5-py3.11.6
py-netcdf4/1.6.2
py-numba/0.57.0
py-numpy/1.24.4
py-numpy/1.26.1
py-pandas/1.5.3
py-pandas/2.1.2
py-pip/23.1.2-py3.11.6
py-pip/23.1.2-py3.9.15
py-plotly/5.14.1
py-scikit-learn/1.3.2
py-scipy/1.11.3
py-setuptools/59.4.0-py3.11.6
py-setuptools/59.4.0-py3.9.15
py-setuptools/68.0.0-py3.11.6
py-setuptools/68.0.0-py3.9.15
python/3.11.6
python/3.9.15
quantum-espresso/7.2
r/4.3.0
r/4.4.1
rclone/1.63.1
reframe/3.12.0
reframe/4.0.4
rocm/5.4.6
rocm/5.5.3
rocm/5.6.1
rocm/5.7.0
rocm/5.7.1
rocm/5.7.3
ruby/3.1.0
rust/1.78.0
scotch/7.0.4
scotch/7.0.4-metis
slate/2023.11.05
squashfs/4.6.1
subversion/1.14.2
subversion/1.9.7
tower-agent/0.4.5
tower-cli/0.7.0
trilinos/15.0.0
trilinos-amd-gfx90a/15.0.0
vasp/5.4.4
vasp6/6.2.1
vasp6/6.3.0
wcslib/7.3
wcslib/7.3-nocfitsio
wcstools/3.9.7
wsclean/2.9
wsclean/3.4-idg-everybeam
wsclean/3.4-idg-everybeam-mpi
xerces-c/3.2.4-transcoder-gnuiconv
xerces-c/3.2.4-transcoder-iconv
adios2/2.8.3-hdf5
adios2/2.9.2-hdf5
adios2/2.10.0
adios2/2.10.0-hdf5
adios2/2.10.2-hdf5
MISSING
ansys-fluids/2022R1
ansys-fluids/2023R1
apr/1.7.5
apr-util/1.6.3
arpack-ng/3.9.0
autoconf/2.69
automake/1.16.5
awscli/1.29.41
beast1/1.10.4
beast2/2.7.4
birli/0.15.1
blaspp/2024.10.26
blaspp-amd-gfx90a/2024.10.26
boost/1.83.0-c++11-python
boost/1.86.0-c++11-python
boost/1.86.0-c++14-python
calceph/3.5.5
caliper/2.11.0
casacore/3.2.1
casacore/3.3.0
casacore/3.3.0-adios2
casacore/3.3.0-adios2-openmp
casacore/3.3.0-openmp
casacore/3.4.0
casacore/3.4.0-adios2
casacore/3.4.0-adios2-openmp
casacore/3.4.0-openmp
casacore/3.5.0-openmp
cdo/2.2.2
cdo/2.4.4
cfitsio/3.49
cfitsio/4.4.0
charmpp/7.0.0
charmpp/8.0.0
chgcentre/1.6
cmake/3.30.5
cp2k/2023.2
cp2k/2024.3
cpmd/4.3
cppunit/1.15.1
cppzmq/4.10.0
eigen/3.4.0
everybeam/0.5.8
exabayes/1.5.1
examl/3.0.22
feh/3.10
ffmpeg/7.0.2
fftw/2.1.5
fftw/3.3.10
MISSING
giant-squid/1.0.3
gnuplot/6.0.0
go/1.23.2
gromacs/2022.5-double
gromacs/2022.5-mixed
gromacs/2023-double
gromacs/2023-mixed
gromacs/2024.3-double
gromacs/2024.3-mixed
gromacs-amd-gfx90a/2023
gsl/2.8
hdf5/1.10.11-api-v110
hdf5/1.10.7-api-v110
hdf5/1.10.7-api-v18
hdf5/1.10.7-parallel-api-v110
hdf5/1.10.7-parallel-api-v18
hdf5/1.12.2-parallel-api-v112
hdf5/1.14.3-api-v110
hdf5/1.14.3-api-v112
hdf5/1.14.3-api-v18
hdf5/1.14.3-parallel-api-v110
hdf5/1.14.3-parallel-api-v112
hdf5/1.14.3-parallel-api-v18
hdf5/1.14.5-parallel-api-v112
hpc-python-collection/-py
MISSING
hpcviewer/2024.09
hpl/2.3
hpl/2.3+openmp
hpx/1.9.1
hyperbeam/0.9.3
hyperbeam-amd-gfx90a/0.9.3
hyperdrive/0.4.1-peel
hyperdrive-amd-gfx90a/0.4.1-peel
idg/1.0.0
ior/3.3.0
kokkos/3.7.02-openmp
kokkos/4.4.01-hpx
kokkos/4.4.01-openmp
kokkos-amd-gfx90a/4.1.00-openmp
kokkos-amd-gfx90a/4.4.01-openmp
lammps/20230802.4
lammps/20240829.1
lammps-amd-gfx90a/20230802.4
DISCONTINUED
libtool/2.4.7
libzmq/4.3.5
log4cxx/1.2.0-c++11
log4cxx/1.2.0-c++17
lz4/1.10.0
magma-amd-gfx90a/2.8.0
mcpp/2.7.2-25-g619046f
metis/5.1.0
metis/5.1.0-64bit
mpifileutils/0.11.1
mwalib/1.5.0
namd2/2.14
namd3/3.0.1
nano/8.2
ncl/6.6.2
nco/5.1.6
nco/5.2.4
ncview/2.1.9
nekrs-amd-gfx90a/23.0
nektar/5.0.2
netcdf-c/4.9.2
netcdf-cxx/4.2
netcdf-cxx4/4.3.1
netcdf-fortran/4.6.1
netlib-lapack/3.11.0
netlib-scalapack/2.2.0
nextflow/22.10.0
nextflow/24.10.0
ninja/1.12.1
nwchem/7.2.0
nwchem/7.2.3
MISSING
MISSING
openblas/0.3.24
openblas/0.3.28
opencv/4.10.0
openfoam/2206
openfoam/2312
openjdk/17.0.8.1_1
osu-micro-benchmarks/7.5
parallel/20240822
parallel-netcdf/1.12.3
parmetis/4.0.3
parmetis/4.0.3-64bit
perl/5.40.0
petsc/3.21.1-complex
petsc/3.21.1-nocomplex
pgplot/5.2.2
plasma/23.8.2
plumed/2.8.2
plumed/2.9.0
plumed/2.9.2
presto/5.0.1
py-astropy/4.2.1
py-astropy/5.1
py-boto3/1.34.44
py-cython/0.29.36
py-cython/3.0.11
py-cython/3.0.8
py-dask/2023.4.1
py-emcee/3.1.1
py-funcsigs/1.0.2
py-h5netcdf/1.3.0
py-h5py/3.11.0
py-h5py/3.12.1
py-hatchet/1.4.0
py-ipython/8.14.0
py-matplotlib/3.8.4
py-matplotlib/3.9.2
py-mpi4py/3.1.5-py3.11.6
py-mpi4py/4.0.1-py3.11.6
py-mpi4py/4.0.1-py3.9.15
py-netcdf4/1.6.5
py-numba/0.58.1
py-numpy/1.25.2
py-numpy/1.26.4
py-numpy/2.1.2
py-pandas/1.5.3
py-pandas/2.2.2
py-pip/23.1.2-py3.11.6
py-pip/23.1.2-py3.9.15
py-plotly/5.20.0
py-scikit-learn/1.4.2
py-scipy/1.13.0
py-scipy/1.14.1
py-setuptools/59.4.0-py3.11.6
py-setuptools/59.4.0-py3.9.15
py-setuptools/63.4.3-py3.11.6
py-setuptools/69.2.0-py3.11.6
python/3.11.6
python/3.9.15
quantum-espresso/7.2
quantum-espresso/7.4
r/4.4.1
rclone/1.68.1
reframe/3.12.0
MISSING
rocm/5.7.0
rocm/5.7.1
rocm/5.7.3
rocm/6.2.0
rocm/6.2.1
rocm/6.2.4
rocm/6.3.0
rocm/6.3.1
rocm/6.3.2
ruby/3.3.5
rust/1.84.1
scotch/7.0.4
scotch/7.0.4-metis
singularityce/3.11.4
singularityce/4.1.0
slate-amd-gfx90a/2024.10.29
squashfs/4.6.1
subversion/1.14.2
DEPRICATED
DEPRICATED
trilinos/15.0.0
trilinos-amd-gfx90a/15.0.0
vasp/5.4.4
vasp6/6.2.1
vasp6/6.3.0
vasp6/6.4.3
wcslib/7.3
wcslib/7.3-nocfitsio
wcstools/3.9.7
MISSING
wsclean/3.4-idg-everybeam
wsclean/3.4-idg-everybeam-mpi
xerces-c/3.3.0-transcoder-gnuiconv
xerces-c/3.3.0-transcoder-iconv
Spack
Spack package manager is our recommended tool for installing packages. Spack has been upgraded In the new software stack from version 0.21.0 to 0.23.0. The new version brings over 8325 software recipes, bug fixes, more supported applications and libraries, and further support for AMD GPUs.
User-private and project-wide software stack directories
For installations using spack, the user-private and project-wide installations are now versioned by the system-wide software stack deployment date. For instance, user-private software is now located at
$MYSOFTWARE/setonix/<DATE_TAG>
where <DATE_TAG> is the date of the related software stack deployment in the YYYY.MM format. The particular version of pawseyenv used determines which user and project installations are visible.
GPU software
Several newer versions of ROCm are now available (up to version 6.3.2). We recommend the use of rocm/6.2.4, which is the version used to build the GPU applications in the new software stack.
Currently, the py-numpy/1.25.2 , py-numpy/1.26.4 and py-numpy/2.1.2 packages in PrgEnv-cray environment exhibit lower performance. If this impacts your work, please submit a helpdesk ticket, and Pawsey staff will assist you.
Issues with running MWA packages:
Currently, some CPU packages such as MWALib, Hyperbeam and Hyperdrive are looking for AMD GPU libraries (libamdhip64.so.5 and libamdhip64.so.6). As a workaround, users can include the following export statement in their job scripts:
In the new software stack 2025.03, a few software packages, such as Amber, Fits2Idia, Omnitrace, and Omniperf, are missing and require manual installation. Pawsey staff are currently working on installing these packages.