The operating system of Setonix is CrayOS, based on sles15sp2. The supercomputer comes with key software packages optimised by the vendor for the HPE Cray EX architecture. It is accessible through the module system, like any other software installed system-wide; however, the Pawsey staff relies on the vendor for its maintenance.

Job Scheduler

Setonix adopts the Slurm job scheduler to manage resources and to grant users fair access to those. To know more about job scheduling with Slurm, visit the page Job Scheduling.

Software stack

Pawsey installs and maintains a predefined set of applications and libraries optimised for Setonix, collectively forming the Pawsey-provided software stack. The list of supported software is available in List of Supported Software. For further information, visit Software Stack.

Programming environments

On an HPE system, the Cray Programming Environment (CPE) determines which set of compilers and libraries are used when compiling and linking code. There are three available programming environments, PrgEnv-aocc, PrgEnv-cray and PrgEnv-gnu, that respectively give access to AMD, Cray, GNU (loaded by default) compilers, along with a consistent set of libraries. It is up to the user to decide which programming environment is most suitable for the task at hand.

Vendor-provided libraries

For many of the popular HPC libraries and interfaces, such as BLAS and MPI, HPE Cray provides its own optimised implementations, preinstalled on Setonix. For some of those libraries,  such as HDF5 and netCDF, Pawsey maintains its own builds.

Cray MPICH

Cray MPICH is an MPI implementation optimised by Cray to take advantage of the Slingshot-10 interconnect through libfabric and is optimized for the Cray Programming Environment. It is based on the ANL MPICH implementation, version 3.4. Users access Cray MPICH by loading the module cray-mpich.

Cray LibSci

The Cray Scientific and Math Libraries (CSML, also known as LibSci) are a collection of numerical routines optimized for best performance on Cray systems. All programming environment modules load cray-libsci by default, except where noted. When possible, users should use calls to the CSML routines in your code in place of calls to public-domain or user-written versions. The CSML/LibSci collection contains the following Scientific Libraries:

• BLAS (Basic Linear Algebra Subroutines)
• LAPACK (Linear Algebra Routines)
• ScaLAPACK (Scalable LAPACK)
• NetCDF (Network Common Data Format)
• FFTW3 (the Fastest Fourier Transforms in the West, release 3)

In addition, the Cray LibSci collection contains the Iterative Refinement Toolkit (IRT) developed by Cray.

Versions are provided for all programming environments. The cray-libsci module is loaded by default and Cray LibSci it will link automatically with your code, selecting the appropriate serial or multithreaded variant of the library depending on whether OpenMP is enabled and the call is inside a parallel region. The OMP_NUM_THREADS environment variable can be used to control threading. Single-threaded version can be enforced by linking with -lsci_cray, -lsci_intel, -lsci_gnu for Cray, Intel and GNU compilers respectively.

Module system

A module system is a way of providing users access to a variety of applications, and to different versions of the same, in an easy way. Setonix adopts the LMOD module system. To see the list of currently installed software, use the command:

    $ module avail

For more information on how to interact with modules, visit Modules. For a more general discussion on what software is supported by Pawsey, and how, visit Software Stack.

Environment variables

Pawsey defines a set of environment variables that may be useful when writing batch scripts or simply interacting with the supercomputer.

Cray environment variables

The compilation process on the HPE Cray EX architecture, that Setonix is based on, adopts dynamic linking by default. This is in contrast with previous generations of Cray systems, HPE Cray EX compilers do not allow dynamic linking with libraries. To turn static linking on, set the following environment variable:

    $ export CRAYPE_LINK_TYPE=static