This page is intended to help users of Garrawarla transition to using the Setonix supercomputer.
This page will be regularly updated based on recommendations of MWA Setonix migration leads and feedback provided by MWA researchers.
Executive summary
Setonix is the new supercomputer at the Pawsey Supercomputing Research Centre, ranking 15th in the world for performance in the Top500 list and ranking 4th Green500 list for energy efficiency. This migration guide focuses on the major differences between Garrawarla and Setonix, and additional features not previously available to Garrawarla users.
Throughout this guide links are provided to relevant documentation pages in the general Supercomputing Documentationand to the Setonix User Guide, which provides documentation specifically for using GPUs on the Setonix system. The Setonix GPU Partition Quick Start also provides specific details for using the GPUs on Setonix.
Description of main differences between both systems are described in 5 separate categories:
System architecture including CPU and GPU technology (see Compute section below),
Filesystems shared with other researchers utilising Setonix (see Filesystems section below),
Software stack, including available modules, spack and containers (see Software section below),
Software development including Cray Programming Environment and AMD GPU tools (see Software Development section below),
Workflows and job submission (see Workflows and Job submission section below).
Compute
The Operating System (OS) of Setonix is SUSE Linux Enterprise Server 15 SP4 (SLES15), a newer version of the same OS present on Garrawarla. On top of it, HPE Cray provides the HPE Cray Programming Environment (CPE) with optimised compilers and libraries, including the GCC, AMD and HPE Cray compilers.
A summary of the node types and their specifications for both Garrawarla and Setonix is shown in the table below. The major changes are the shift to AMD CPUs and GPUs, the increase in cores per node and number of nodes but a reduction in the memory per node, and the increase in GPUs available.
Numbers of nodes provided below describes the total capacity of Garrawarla and Setonix systems. In the case of Setonix, MWA has access to limited number of nodes within dedicated partitions.
Garrawarla | |||||
|---|---|---|---|---|---|
Type | N. Nodes | CPU | Cores Per Node | RAM Per Node | GPUs Per Node |
Login | 2 | n/a | |||
CPU computing | 78 | Intel Xeon Gold 6230 (2.1GHz) | 2x 20 | 384 GB | 1x NVIDIA V100 32GB GPU |
Data movement | n/a | ||||
Setonix | |||||
Login | 4 | AMD Milan | 2x 64 | 256 GB | n/a |
CPU computing | 1592 | AMD Milan (2.45GHz, 280W) | 2x 64 | 256 GB | n/a |
CPU high memory | 8 | AMD Milan (2.45GHz, 280W) | 2x 64 | 1 TB | n/a |
GPU computing | 154 | AMD Trento (2.0GHz, 225W) | 1x 64 | 256 GB | 4x AMD MI250X each with 2 GCDs and 128GB HBM2e |
GPU high memory | 38 | AMD Trento (2.0GHz, 225W) | 1x 64 | 512 GB | 4x AMD MI250X each with 2 GCDs and 128GB HBM2e |
Data movement | 8 | AMD 7502P | 1x 32 | 128 GB | n/a |
For more details refer to the System overview section of the Setonix User Guide.
Filesystems
For the description of Filesystems and their use please refer to Pawsey Filesystems and their Use information.
Software
Pawsey provides and maintains a number of prebuilt libraries and applications on Setonix. These are accessible through the LMOD module system. The most popular radio astronomy software is already installed on Setonix, as shown below.
------------------------------ /software/setonix/2024.05/modules/zen3/gcc/12.2.0/astro-applications ------------------------------- apr-util/1.6.3 cfitsio/4.3.0 (D) py-astropy/4.2.1 apr/1.5.2 chgcentre/1.6 py-astropy/5.1 (D) birli/0.15.1 (D) cppunit/1.14.0 py-emcee/3.1.1 calceph/3.5.5 (D) cppzmq/4.9.0 py-funcsigs/1.0.2 casacore/3.2.1 giant-squid/1.0.3 subversion/1.9.7 casacore/3.3.0-adios2-openmp hyperbeam-amd-gfx90a/0.9.3 subversion/1.14.2 (D) casacore/3.3.0-adios2 hyperbeam/0.9.3 (D) wcslib/7.3-nocfitsio casacore/3.3.0-openmp hyperdrive-amd-gfx90a/0.4.1 wcslib/7.3 (D) casacore/3.3.0 hyperdrive/0.4.1 wcstools/3.9.7 casacore/3.4.0-adios2-openmp libzmq/4.3.3 wsclean/2.9 casacore/3.4.0-adios2 log4cxx/0.12.1-c++11 wsclean/3.4-idg-everybeam-mpi casacore/3.4.0-openmp log4cxx/0.12.1-c++17 (D) wsclean/3.4-idg-everybeam (D) casacore/3.4.0 mcpp/2.7.2 xerces-c/3.2.4-transcoder-gnuiconv casacore/3.5.0-openmp (D) mwalib/1.5.0 (D) xerces-c/3.2.4-transcoder-iconv (D) cfitsio/3.49 pgplot/5.2.2 (D)
GPU-enabled software will have a -amd-gfx90asuffix in their module name, specifying the GPU architecture they were compiled for. Use these modules on the GPU partition. For instance, Hyperdrive has been compiled both for CPU and GPU.
The MWA Wiki contains a tutorial (external link) illustrating how to run hyperdrive and WAClean to create images of the radio sky.
Software Installation with Spack
Pawsey mainly uses Spack to build and install software. See How to Install Software for information about installing software on Setonix and how users can use Spack to install additional software or different builds of existing software that are not provided by Pawsey.
Each user can only install software for themselves or for their group in specific locations on /software filesystem, see Pawsey Filesystems and their Use for more information.
Installing Python packages
Pawsey provides some python packages such as NumPy in the form of modules. Python modules are prefixed with py-. NumPy is available as py-numpy as an example. You can install additional python packages either using Spack or, more conveniently, using pip(available as module py-pip). Pip is configured to install Python modules in a predefined location on your private /software directory. For more information, see Using Python and Installing Python Packages .
Containers
Containers allow users to package an application and all of the software dependencies needed to run it into a single, sandboxed piece of software. Containers enable an application to run quickly and reliably from one computing environment to another. For more information on how to use containers on Setonix please refer to Containers.
More information
Refer to the Software Stack pages for more detail on using software, modules and containers.
For information specific to Setonix refer to the Software Environment section of the Setonix User Guide.
Software Development
Setonix provides the HPE Cray Programming Environment (CPE) with optimised compilers and libraries, including the GCC, AMD and HPE Cray compilers. For details on how to compile codes on Setonix please refer to Compiling Code on Setonix.
Setonix GPU partitions are based on AMD GPU architecture. Codes developed with the use of CUDA for NVIDIA GPUs will not work automatically on Setonix. Transition from CUDA to HIP and ROCm can be made following the instructions provided on the Porting Cuda Codes to HIP page. In addition, Pawsey provides HIP training resources on our Pawsey YouTube (playlists).
Workflows and Job submission
Just as on Garrawarla, job scheduling is handled by Slurm on Setonix. The different partitions available on Setonix, including MWA dedicated partitions are listed on Running Jobs on Setonix page.
For more information refer to Job Scheduling in the Supercomputing documentation.
For information specific to Setonix refer to the Running Jobs section, and particularly for GPU jobs the Example Slurm Batch Scripts for Setonix on GPU Compute Nodes page of the Setonix User guide.
For more information on job dependencies, refer to Example Workflows in the Supercomputing documentation.
In addition, researchers are required to explicitly provide the computing account to which the job resources will be charged (e.g., mwavcs or pawseyNNNN) for all jobs (interactive or batch submission).
You will need to use a different project code for the --account/-A option. More specifically, it is your project code followed by the -gpu suffix. For instance, if your project code is project1234, then you will have to use project1234-gpu.