Versions Compared

Old Version 28

changes.mady.by.user Pascal Elahi

Saved on

compared with

New Version Current

changes.mady.by.user Alexis Espinosa

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Column
Note
titleWork in Progress for Phase-2 Documentation

The content of this section is currently being updated to include material relevant for Phase-2 of Setonix and the use of GPUs.
On the other hand, all the existing material related to Phase-1 and the use of CPU compute nodes can be considered safe, valid and up-to-date.

Excerpt

Setonix is Pawsey's flagship supercomputer based on the HPE Cray EX architecture that was commissioned in 2020 and delivered in two phases over the course of 2022 and 2023.

...

Type

N. Nodes

CPU

Cores Per Node

RAM Per Node

GPUs Per Node

Login

49

AMD Milan2x 64256GBn/a

CPU computing

1592

AMD Milan (2.45GHz, 280W)

2x 64

256GBn/a
CPU high memory8AMD Milan (2.45GHz, 280W)2x 641TBn/a
GPU computing154AMD Trento1 x 64256GB8 GCDs (from 4x "AMD MI250X" cards, each card with 2 GCDs)
GPU high memory38AMD Trento1 x 64512GB8 GCDs (from 4x "AMD MI250X" cards, each card with 2 GCDs)
Data movement811

AMD 7502P

1x 32128Gbn/a

More details regarding the hardware architecture and filesystems are made available in the sections below.

...

Section


Column

Figure 4. GPU node architecture. Note here the that the GPU's shown here are equivalent to a GCD (see here).


Note that each MI250X has two Graphics Compute Dies (GCD) that are accessible as two logical GPUs, for a total of eight per node.

...

For more detail regarding the MI250X GPU architecture, refer to https://www.amd.com/en/technologies/cdna2.the AMD CDNA 2 Architecture Whitepaper.

Anchor
Filesystem
Filesystem

...

    Partition Charge Rate ✕ Max(Cores Proportion, Memory Proportion, GPU Proportion) ✕ N. of nodes requested ✕ Job Elapsed Time (Hours).

...

  • Partition Charge Rate is a constant value associated with each Slurm partition,
  • Core proportion is the number of CPU cores per node requested divided by the total number of CPU cores per node,
  • Memory proportion is the amount of memory per node requested divided by the total amount of memory available per node,
  • GPU proportion is the amount of GPUs requested divided by the total amount of GPUs available per node (remember that for slurm, each GPU is equivalent to a GCD, so each GPU-node has 8 available GPUs to be requested).

For Setonix CPU nodes, the charge rate is 128 SU per node hour, as each CPU node has 128 cores.

For Setonix GPU nodes, the charge rate is 512 SU per node hour, based on the difference in energy consumption between the CPU and GPU node architectures. Since there are fewer GPU nodes than CPU nodes, these GPU nodes are to be used solely for GPU-enabled codes. Thus, resource requests on GPU nodes are slightly different to CPU nodes as all requests are in units of GCDs, with 1 GCD = 1 Slurm GPU. Requests cannot be made based on memory but must be based on the number of GPUs to be used. 

Maintenance

Due to the cutting-edge nature of Setonix, regular and frequent updates of the software stack is expected during the first year of Setonix's operation as further optimisations and improvements are made available.

...