Versions Compared

Version Old Version 8 New Version 9
Changes made by

Pascal Elahi

Alexis Espinosa

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Starting from July 24th, 2023, Setonix will include compute nodes with the new Cray Programming Environment (CPE) 23.02. This update to CPE will result in better performance of the supercomputer , but will also imply a series of modifications and updates in the whole working environment to users. The purpose of this page is to consolidate in one place all the important points that researchers will need to pay attention, together with the recommended actions or changes to take.

...

GPU-enabled builds of several applications have been added to the stack (or will be in the incoming weeks). These replace the containerised deployments that were previously present on Setonix.

  • amber 22
  • lammps
  • cp2k
  • GROMACS gromacs without GPUgpu-DIRECTdirect
  • nekRS

Machine learning frameworks Tensorflow and PyTorch are provided by means of containers and made visible using the module system. Once the respective module is loaded, the python3 interpreter from within the container has access to the Tensorflow or PyTorch Python modules.

...

Researchers need to recompile their own manually - built software that has been built previously on Setonix. This is necessary because the CPE has newer versions of various libraries, new paths to libraries (specifically MPICH) to ensure the best possible performance and avoid issues. This process also might require an update to any modules loaded since versions will change. 


Conda/Mamba

In general, your software installed using Conda/Mamba should not be affected by the updates. The exception would be if you have installed software using the conda install --use-local option. The --use-local  option uses local files to do the package installation, rather than the external channels that Conda typically uses (e.g. bioconda of conda-forge). The paths of the local files may have changed, and would need to up rebuilt using the updated paths. In general, the default option is to use the external channels such and conda-forge, so we do not expect this to impact many users. 

R

The version of R provided as a module has changed. This may cause issues with your installed R libraries and packages and require you to update your installed versions to be compatible with this newer version of R. 

Python virtual environment


Spack installations

Researchers that installed software with spack/0.17.0  will need to 

...

 will need to load the older software stack to load this particular version of spack and query it to get the previously installed packages. The steps involved are

Column


Code Block
languagebash
themeEmacs
titleInstructions to get information on previous spack builds
# swap the pawseyenv module to the previous software stack 
module swap pawseyenv pawseyenv/2022.11
# load the older spack version
module load spack/0.17.0
# look for modules within your $MYSOFTWARE installation 
hashlist=($(ls -lR ${MYSOFTWARE}/setonix/modules/zen3/| grep .lua | sed "s/.lua//g" | sed "s/-/ /g"| awk '{print $NF}'))
# query spack for these user built packages to get the build information and store it so that it can be used to generate an installation with
# the new spack
for h in ${hashlist[@]}
do
  spack find -v /${h}




  • modulmodule load spack/0.19.0
  • spack uninstall software@version
  • spack clean -a && spack -d install software@version

Containers

The Singularity container platform has been upgraded to version 3.11.4. As before, several configurations are available and need to be chosen depending on your applications. So, for example, for biocontainers that do not use MPI, users should use singularity/3.11.4-nompi; while containers with MPI applications should use singularity/3.11.4-mpi.

In regards to the containers themselves, containers that does not make use of MPI should not suffer execution issues with the upgrades to the CPE and Singularity.

But, as the programming environment has changed, containers that make use of MPI may suffer from incompatibilities with the new host MPI libraries. So, if users own containers are reporting libraries incompatibilities or suffering execution issues, they containers that make use of MPI may need to rebuild their containers be rebuilt based on a Pawsey's new MPICH base image that has been tested against the new CPE.We have made available a  Then, users with their own container with MPI applications will need to update its recipe and rebuild starting from:

FROM quay.io/pawsey/mpich-base:3.4.3_ubuntu20.04

The new base image with MPICH 3.4.3 built from Ubuntu 20.04 that does not suffer from incompatibilities with the new host libraries. The If interested, the recipe (Dockerfile) of for this new base image is publicly available in Pawsey's Git repository: new MPICH-base-image-Dockerfile, and the built Docker image has already been uploaded to Pawsey's registry quay.io/pawsey. Then, users with their own container with applications that make use of MPI will need to update its recipe and rebuild it, starting from:

FROM quay.io/pawsey/mpich-base:3.4.3_ubuntu20.04

A Also, a practical example of an updated recipe is in our recently updated for the OpenFOAM tool can be found in Pawsey's Git repository: Openfoam-v2212-Dockerfile, also publicly available in Pawsey's Git repository.

SLURM 22.05