Pawsey software stack

The system-wide software stack managed by Pawsey made available by the pawseyenv module is now versioned by deployment date, using the YYYY.MM format. You can choose which deployment to use, bearing in mind that a deployment can be marked as deprecated or unsupported due to breaking changes induced by the vendor updates to the system.

The new version of the software stack will be 2023.08, and it will replace the 2022.11 that was deployed on Setonix up until now. Unfortunately, due to the underlying changes in the Cray Programming Environment, the 2022.11 cannot be kept on Setonix. User installations of software using spack/0.17.0 will have to be re-executed using spack/0.19.0. Software builds with other means will most likely be affected as well, especially if depending on MPI and the Pawsey software stack.

The newly deployed software stack has been upgraded to include the latest version of many applications and libraries such as gromacs, cp2k, lammps.

Spack

Spack has been upgraded from version 0.17.0 to 0.19.0. The new version brings many bug fixes, more supported applications and libraries, and support to build software that runs on AMD GPUs. The  ~/.spack  directory, where user customisations for Spack are saved,  has been moved and renamed to $MYSOFTWARE/setonix/<DATE_TAG>/.spack_user_config .

The build cache of Spack has been moved to $MYSCRATCH/setonix not to consume the inode quota on $MYSOFTWARE.

Spack configuration now recognises the ROCm installation on the system and, at the same time, ignores incomplete pre-installed packages that were causing issues when compiling software (e.g. gettext).

User-private and project-wide software stack directories

The user-private and project-wide installations are now versioned by the system-wide software stack deployment date. For instance, user-private software is now located at

$MYSOFTWARE/setonix/<DATE_TAG> 

where <DATE_TAG>  is the date of the related software stack deployment in the YYYY.MM  format. When you select which pawseyenv version to use, you select also what user and project installations are visible.

GPU software

The new CPE comes with ROCm 5.2.3, replacing version 5.0.2. In addition, the AMD GPU driver has also been updated and should resolve many of the ROCm issues observed on the system.

GPU-enabled builds of several applications have been added to the stack (or will be in the incoming weeks). These replace the containerised deployments that were previously present on Setonix.

• amber 22
• lammps
• cp2k
• gromacs without gpu-direct
• nekRS

Machine learning frameworks Tensorflow and PyTorch are provided by means of containers and made visible using the module system. Once the respective module is loaded, the python3 interpreter from within the container has access to the Tensorflow or PyTorch Python modules.

CPU software

Very old versions of OpenFOAM are not deployed system wide as containerised modules anymore.

The available modules can be listed with module avail openfoam command.

If users require the use of very old versions of OpenFOAM they may need to build and deploy their own containers. In this case, they can use Pawsey containers as a guide to deploy their own. Please read the documentation section: Use of old versions of OpenFOAM not provided by Pawsey.

It's also important to note that deployed OpenFOAM container modules have been updated to be based on Ubuntu 20.04, as explained in: Containers changes for the latest updates on Setonix.