...
Table 1. Setonix Node Overview
Type | N. NodesNozdes | CPU | Cores Per Node | RAM Per Node | GPUs Per Node |
---|---|---|---|---|---|
Login | 9 | AMD Milan | 2x 64 | 256GB | n/a |
CPU computing | 1592 | AMD Milan (2.45GHz, 280W) | 2x 64 | 256GB | n/a |
CPU high memory | 8 | AMD Milan (2.45GHz, 280W) | 2x 64 | 1TB | n/a |
GPU computing | 154 | AMD Trento | 1 x 64 | 256GB | 8 GCDs (from 4x "AMD MI250X" cards, each card with 2 GCDs) |
GPU high memory | 38 | AMD Trento | 1 x 64 | 512GB | 8 GCDs (from 4x "AMD MI250X" cards, each card with 2 GCDs) |
Data movement | 11 | AMD 7502P | 1x 32 | 128Gb | n/a |
...
For Setonix GPU nodes, the charge rate is 512 SU per node hour, based on the difference in energy consumption between the CPU and GPU node architectures. Since there are fewer GPU nodes than CPU nodes, these GPU nodes are to be used solely for GPU-enabled codes. Thus, resource requests on GPU nodes are slightly different to CPU nodes as all requests are in units of GCDs, with 1 GCD = 1 Slurm GPU. Requests cannot be made based on memory but must be based on the number of GPUs to be used.
Worked Examples of the Accounting Model
Let’s work through some CPU examples.
Example 1: Memory heavy
You request the following:
Resource | Request |
---|---|
Cores | 32 |
Memory | 128GB |
Nodes | 1 |
Hours | 4 |
On a CPU node, there are 128 cores per node. Core Proportion is calculated as requested cores
/ total cores
, which is 32/128=0.25
On a normal CPU node (i.e. not the highmem
nodes), there is 256GB of memory. Memory Proportion is calculated as requested memory
/ total memory
, which is 128/256=0.5
Which of these is bigger? It’s the memory proportion in this case, so that’s what we will use for the equation. You always use whichever is the biggest proportion. Also remember that the Partition Charge Rate
never changes.
128 x 0.5 x 1 x 4 = 256SU
Partition charge rate x memory proportion x nodes x hours = 256SU
Example 2: CPU heavy
You request the following:
Resource | Request |
---|---|
Cores | 128 |
Memory | 64GB |
Nodes | 1 |
Hours | 4 |
On a CPU node, there are 128 cores per node. Core Proportion is calculated as requested cores
/ total cores
, which is 128/128=1
On a normal CPU node (i.e. not the highmem
nodes), there is 256GB of memory. Memory Proportion is calculated as requested memory
/ total memory
, which is 64/256=0.25
Which of these is bigger? It’s the memory proportion in this case, so that’s what we will use for the equation. You always use whichever is the biggest proportion. Also remember that the Partition Charge Rate
never changes.
128 x 1 x 1 x 4 = 512SU
Partition charge rate x CPU proportion x nodes x hours = 512SU
Example 3: GPU
Resource | Request |
---|---|
GCDs | 3 |
Memory | N/A |
Nodes | 1 |
Hours | 4 |
On a GPU node, there are 8 GCDs per node. The GPU proportion is calculated as requested GCDs/total GCDs
which is 3/8=0.375
You don't need to worry about how much memory to request as this is automatically allocated for you.
512 x 0.375 x 1 x 4 = 768SU
Partition charge rate x memory proportion x nodes x hours = 768SU
Maintenance
Due to the cutting-edge nature of Setonix, regular and frequent updates of the software stack is expected during the first year of Setonix's operation as further optimisations and improvements are made available.
...