Versions Compared

Old Version 29

changes.mady.by.user Pascal Elahi

Saved on

compared with

New Version Current

changes.mady.by.user Alexis Espinosa

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Column
Note
titleWork in Progress for Phase-2 Documentation

The content of this section is currently being updated to include material relevant for Phase-2 of Setonix and the use of GPUs.
On the other hand, all the existing material related to Phase-1 and the use of CPU compute nodes can be considered safe, valid and up-to-date.

Excerpt

Setonix is Pawsey's flagship supercomputer based on the HPE Cray EX architecture that was commissioned in 2020 and delivered in two phases over the course of 2022 and 2023.

...

Type

N. Nodes

CPU

Cores Per Node

RAM Per Node

GPUs Per Node

Login

49

AMD Milan2x 64256GBn/a

CPU computing

1592

AMD Milan (2.45GHz, 280W)

2x 64

256GBn/a
CPU high memory8AMD Milan (2.45GHz, 280W)2x 641TBn/a
GPU computing154AMD Trento1 x 64256GB8 GCDs (from 4x "AMD MI250X" cards, each card with 2 GCDs)
GPU high memory38AMD Trento1 x 64512GB8 GCDs (from 4x "AMD MI250X" cards, each card with 2 GCDs)
Data movement811

AMD 7502P

1x 32128Gbn/a

More details regarding the hardware architecture and filesystems are made available in the sections below.

...

Section


Column

Figure 4. GPU node architecture. Note here the that the GPU's shown here are equivalent to a GCD (see here).


Note that each MI250X has two Graphics Compute Dies (GCD) that are accessible as two logical GPUs, for a total of eight per node.

...

For more detail regarding the MI250X GPU architecture, refer to https://www.amd.com/en/technologies/cdna2the AMD CDNA 2 Architecture Whitepaper.

Anchor
Filesystem
Filesystem

...

    Partition Charge Rate ✕ Max(Cores Proportion, Memory Proportion, GPU Proportion) ✕ N. of nodes requested ✕ Job Elapsed Time (Hours).

...

  • Partition Charge Rate is a constant value associated with each Slurm partition,
  • Core proportion is the number of CPU cores per node requested divided by the total number of CPU cores per node,
  • Memory proportion is the amount of memory per node requested divided by the total amount of memory available per node,
  • GPU proportion is the amount of GPUs requested divided by the total amount of GPUs available per node (remember that for slurm, each GPU is equivalent to a GCD, so each GPU-node has 8 available GPUs to be requested).

For Setonix CPU nodes, the charge rate is 128 SU per node hour, as each CPU node has 128 cores.

...