Resource Overview

Owned by Cristian Di Pietrantonio
Last updated: Dec 13, 2023 by Alexis Espinosa
3 min read


The Pawsey Supercomputing Research Centre provides access to a number of different supercomputing systems for the Australian research and industry communities, as well as international collaborators. This page provides details of these resources, including supercomputing systems and high-performance filesystems.

On this page:

Supercomputing systems

The Pawsey Supercomputing Research Centre operates several supercomputing systems, which physically reside at the Pawsey and are closely integrated with its other infrastructure.

Setonix

Setonix is the flagship supercomputer of the Pawsey Supercomputing Research Centre, and is based on the HPE Cray EX architecture. It ranked as the 15th largest supercomputer in the world on the Top500 list, and the 4th most energy efficient supercomputer in the world on the Green500 list in November 2022. A significant portion of Setonix's processing capability is due to its AMD MI250X GPUs.

A picture of Setonix

Key characteristics

  • Phase 1 provides more than 2 PFLOPs of computing power through CPU-only nodes based on the AMD EPYC Zen 3 architecture.

  • Phase 2 will extend Setonix to include additional CPU nodes as well as a large GPU node partition based on the AMD Instinct MI250X architecture, for a total of 43 PFLOPs of theoretical peak performance (8 PFLOPs in the CPU-only nodes and 35 PFLOPs in the GPU-nodes).

Setonix, has been recognised as one of the greenest supercomputers in the world, after ranking in the top5 on the globally recognised Green500 list. Setonix was also named the most powerful public research supercomputer in the Southern Hemisphere, ranking 15 in the global Top500 ranking as well as 10 in the HPL Mixed-Precision Benchmark in November 2022.

More information at Setonix User Guide.

Garrawarla

The Pawsey Supercomputing Centre has installed a new GPU-enabled system called Garrawarla, a Wajarri word meaning Spider, to enable our Murchison Wide Field Array (MWA) researchers to produce scientific outcomes while the Pawsey Supercomputing System is being procured. This MWA compute cluster provides the latest generation of CPUs and GPUs, high memory bandwidth, and increased memory per node to allow MWA researchers to effectively process large datasets.

Key characteristics

  • 78 nodes each with 40 Intel CPU cores and one NVIDIA V100 GPU.
  • 756 TFLOPs of computing power.
  • Dedicated to the MWA organisation.

More information at Garrawarla User Guide

Filesystems

There are a number of filesystems mounted by Pawsey supercomputers:

HPC filesystems
  • /scratch is a large, high-performance filesystem intended for short-term use by jobs actively running on the system.
  • /home is a smaller filesystem that should only be used for configuration files that are expected by software to be located there.
  • /software is a filesystem that provides the system software, and is also available for project groups to install domain-specific software.
Radioastronomy HPC filesystems
  • /astro is a filesystem that supports the operation of the MWA radio telescopes.
  • /askapbuffer is a filesystem that supports the operation of the ASKAP radio telescopes.


Migration: Filesystems

While Pawsey is migrating from old systems to the new one, Setonix, the following points stand true:

  • The /group filesystem has been decommissioned. All projects should store data on Acacia when using Setonix.

  • Setonix will mount a different /home  and /scratch filesystem than Garrawarla, with the same mount point but different corresponding filesystems.

For more information about filesystems and file management, see File Management.

Object storage

Each project and each user have access to a long-term storage solution for their data implemented via the object storage Acacia. Supercomputing users transfer data to and from Acacia using dedicated commands already installed on supercomputers. For more information, visit Acacia - User Guide .

HPC software

Pawsey installs and supports a range of software packages, which can be managed and accessed through the Modules system. Additional software can be installed by users through Spack or manual builds or by means of Containers. For more information, visit the Software Stack page.

Popular scientific applications

Due to their popularity and importance, several scientific applications and libraries are preinstalled on Pawsey supercomputers. Examples are GROMACS, VASP  OpenFOAM. For the complete list of supported packages, consult the page List of Supported Software or use the command

$ module avail 

to list the software currently installed on the supercomputer.

Development tools

The software supported by Pawsey includes a range of open-source and commercial tools for code development.

Compilers

Compilers convert human-readable source code into machine language, resulting in code that typically runs much faster than scripting and interpreted code.

For more information on compiling code on Pawsey HPC systems, refer to Compiling.

Debuggers

Debuggers help software developers identify bugs and issues with a code. For more details refer to Debugging.

Profilers

Profilers help software developers identify sections of code where the most time is spent, to focus efforts to improve performance. There are a number of command-line tools available for profiling, as well as ARM MAP and Cray Tools.

For more details refer to Profiling.