Executive summary

Setonix is the new supercomputer at the Pawsey Supercomputing Research Centre, ranking 15th in the world for performance in the Top500 list and ranking 4th Green500 list for energy efficiency. This migration guide focuses on the major differences between Garrawarla and Setonix, and additional features not previously available to Garrawarla users.

Throughout this guide links are provided to relevant documentation pages in the general Supercomputing Documentationand to the Setonix User Guide, which provides documentation specifically for using GPUs on the Setonix system. The Setonix GPU Partition Quick Start also provides specific details for using the GPUs on Setonix.

Description of main differences between both systems are described in 5 separate categories:

1. System architecture including CPU and GPU technology (see Compute section below),

2. Filesystems shared with other researchers utilising Setonix (see Filesystems section below),

3. Software stack, including available modules, spack and containers (see Software section below),

4. Software development including Cray Programming Environment and AMD GPU tools (see Software Development section below),

5. Workflows and job submission (see Workflows and Job submission section below).

Compute

The Operating System (OS) of Setonix is SUSE Linux Enterprise Server 15 SP4 (SLES15), a newer version of the same OS present on Garrawarla. On top of it, HPE Cray provides the HPE Cray Programming Environment (CPE) with optimised compilers and libraries, including the GCC, AMD and HPE Cray compilers.

A summary of the node types and their specifications for both Garrawarla and Setonix is shown in the table below. The major changes are the shift to AMD CPUs and GPUs, the increase in cores per node and number of nodes but a reduction in the memory per node, and the increase in GPUs available. 

For more details refer to the System overview section of the Setonix User Guide.​

Filesystems

For the description of Filesystems and their use please refer to Pawsey Filesystems and their Use information.

Software

Pawsey provides and maintains a number of prebuilt libraries and applications on Setonix. These are accessible through the LMOD module system. The most popular radio astronomy software is already installed on Setonix, as shown below.

GPU-enabled software will have a -amd-gfx90asuffix in their module name, specifying the GPU architecture they were compiled for. Use these modules on the GPU partition. For instance, Hyperdrive has been compiled both for CPU and GPU.

The MWA Wiki contains additional documentation on running MWA processing on Setonix:

• Tutorial (external link) illustrating how to run hyperdrive and WSClean to create images of the radio sky,

• VCS Data Processing documentation,

• Offline correlation documentation.

Software Installation with Spack

Pawsey mainly uses Spack to build and install software. See How to Install Software for information about installing software on Setonix and how users can use Spack to install additional software or different builds of existing software that are not provided by Pawsey. 

Each user can only install software for themselves or for their group in specific locations on /software filesystem, see Pawsey Filesystems and their Use for more information.

Installing Python packages

Pawsey provides some python packages such as NumPy in the form of modules. Python modules are prefixed with py-. NumPy is available as py-numpy as an example. You can install additional python packages either using Spack or, more conveniently, using pip(available as module py-pip). Pip is configured to install Python modules in a predefined location on your private /software directory. For more information, see Using Python and Installing Python Packages .

Containers

Containers allow users to package an application and all of the software dependencies needed to run it into a single, sandboxed piece of software. Containers enable an application to run quickly and reliably from one computing environment to another. For more information on how to use containers on Setonix please refer to Containers. 

More information

Refer to the Software Stack pages for more detail on using software, modules and containers.

For information specific to Setonix refer to the Software Environment section of the Setonix User Guide.​

Software Development

Setonix provides the HPE Cray Programming Environment (CPE) with optimised compilers and libraries, including the GCC, AMD and HPE Cray compilers. For details on how to compile codes on Setonix please refer to Compiling Code on Setonix. 

Setonix GPU partitions are based on AMD GPU architecture. Codes developed with the use of CUDA for NVIDIA GPUs will not work automatically on Setonix. Transition from CUDA to HIP and ROCm can be made following the instructions provided on the Porting Cuda Codes to HIP page. In addition, Pawsey provides HIP training resources on our Pawsey YouTube (playlists). 

Workflows and Job submission

Just as on Garrawarla, job scheduling is handled by Slurm on Setonix. The different partitions available on Setonix, including MWA dedicated partitions are listed on Running Jobs on Setonix page.  

For more information refer to Job Scheduling in the Supercomputing documentation.

For information specific to Setonix refer to the Running Jobs section, and particularly for GPU jobs the Example Slurm Batch Scripts for Setonix on GPU Compute Nodes page of the Setonix User guide.

For more information on job dependencies, refer to Example Workflows in the Supercomputing documentation.

In addition, researchers are required to explicitly provide the computing account to which the job resources will be charged (e.g., mwavcs  or pawseyNNNN) for all jobs (interactive or batch submission).

Related pages

• Supercomputing Documentation

• Setonix User Guide

• Setonix GPU Partition Quick Start