/
Example Slurm Batch Scripts for Setonix-Q on GH200 Compute Nodes

Example Slurm Batch Scripts for Setonix-Q on GH200 Compute Nodes

Node architecture

The GH200 node architecture differs from both the CPU-only nodes and the standard GPU nodes (see the node diagram referenced above).

Each GH200 compute node contains four NVIDIA GraceHopper GH200 superchips. Each superchip consists of:

  • Grace CPU: Arm Neoverse V2, 72 cores, with 120 GB LPDDR

  • Hopper GPU: H100, with 96 GB memory

Because the superchip uses a unified memory architecture, each node reports a combined memory total of:

  • 4 × 120 GB (CPU memory) + 4 × 96 GB (GPU memory)

The available memory is 857 GB.

The Grace CPU architecture is Arm, and therefore completely different from the x86_64 CPUs used on the other node types.

These nodes connect to the Slingshot network via two NIC cards for inter-node data transfer. The GH200 nodes are available on the quantum partition associated with the Setonix-Q Pilot merit allocation scheme.

Important: GH200 GPU requests (GCD vs superchip)

On systems with AMD MI250X GPU cards, Pawsey often refers to GCDs (see: Setonix GPU Partition Quick Start ).

For the NVIDIA GH200 nodes, however, each superchip has exactly one H100 GPU. As a result:

  • On GH200 nodes, Slurm “GPU” requests map to the number of superchips requested.

Pawsey does not use the standard Slurm interpretation of --gres=gpu:<number> on these nodes. Instead, Pawsey’s site configuration repurposes this parameter to mean the number of allocation-packs requested.

This is done to achieve best performance by making the allocation-pack the basic schedulable unit. Users should therefore request only the number of allocation-packs that meet their needs.

Each allocation-pack provides one superchip’s resources:

  • 1 x Grace CPU (72 CPU cores)

  • 1 x H100 GPU

  • ~120 GB CPU memory + 96 GB GPU memory

    • with unified memory, this is ~216 GB available to the superchip

Each GPU node also has an attached NVMe device with up to 3500 GiB of usable storage.

Further details of the node architecture are also available on the GPU node architecture page.

Slurm use of GH200 nodes

Pawsey’s allocation model

We follow Setonix GPU partition where CPU cores are allocated based on requesting a GH200 superchip. Options used for resource request via salloc or (#SBATCH pragmas) and the options for the use of resources during execution of the code via srun are separated.

Requesting allocation-packs

With the CLI filter used on the quantum partition, GPU resources should be thought of as "allocation-packs". Each allocation-pack provides 1 whole Grace CPU (72 CPU cores), approximately 117000M of host memory, and 1 H100 GPU.

For most jobs, the recommended way to request resources is simply --gpus=N, where N is the total number of allocation-packs required by the job. Slurm will automatically allocate the corresponding CPU cores and memory. Users should not specify --cpus-per-task or --mem in the allocation request.

The --gpus=N form does not specify node geometry and Slurm will pack the requested GPUs onto as few nodes as possible.

If a specific node/GPU geometry is required, use --nodes=N --gres=gpu:n --ntasks-per-node=72*n. In this form, --gres=gpu:n specifies the number of allocation-packs per node, and --ntasks-per-node must exactly match 72 tasks per requested GPU. For example, 2 GPUs per node requires --ntasks-per-node=144.

The option --gpus-per-node=n follows the same task-count requirement, but is not recommended due to previously observed issues.

Resource allocation and resource use within an srun job step should be treated separately. Users should explicitly specify the resources required by each srun step.

In particular, srun may inherit the job's NumTasks value. Jobs requested with --gpus=N normally have NumTasks=1, while jobs using the explicit --nodes + --gres + --ntasks form have a much larger NumTasks value (72 tasks per GPU). Therefore, users should not rely on a bare srun command.

For srun, Slurm options have their standard meaning, and -n, -c and GPU options should normally be specified explicitly.

The following table provides some examples that will serve as a guide for requesting resources in the GPU nodes. Most of the examples in the table provide are for typical jobs where multiple GPUs are allocated to the job as a whole but each of the tasks spawned by srun is binded and has direct access to only 1 GPU. For applications that require multiple GPUs per task, there 3 examples (*4, *5 & *7) where tasks are binded to multiple GPUs:

Required Resources per Job

New "simplified" way of requesting resources

Total Allocated resources

Charge per hour

The use of full explicit srun options is now required
(only the 1st method for optimal binding is listed here)

1 CPU task (single CPU thread) controlling 1 H100 GPU (Slurm GPU)

#SBATCH --gpus=1

1 allocation-pack =
1 GPU, 72 CPU cores (1 Grace CPU), ~216 GB unified RAM

256 SU

*1

export OMP_NUM_THREADS=1
srun -N 1 -n 1 -c 72 --gres=gpu:1 --gpus-per-task=1 --gpu-bind=closest <executable>

1 CPU task (with 144 CPU threads each) all threads controlling the same 1 GCD

#SBATCH --gpus=2

2 allocation-packs=
2 GPUs, 144 CPU cores (2 Grace CPUs), ~432 GB unified RAM

512 SU

*2

export OMP_NUM_THREADS=80
srun -N 1 -n 1 -c 144 --gres=gpu:1 --gpus-per-task=1 --gpu-bind=closest <executable>

3 CPU tasks (single thread each), each controlling 1 GCD with GPU-aware MPI communication

#SBATCH --gpus=3

3 allocation-packs=
3 GPUs, 216 CPU cores (3 Grace CPUs), ~648 GB unified RAM

768 SU

*3

export MPICH_GPU_SUPPORT_ENABLED=1
export OMP_NUM_THREADS=1
srun -N 1 -n 3 -c 72 --gres=gpu:3 --gpus-per-task=1 --gpu-bind=closest <executable>

8 CPU tasks (single thread each), each controlling 1 GCD with GPU-aware MPI communication

#SBATCH --nodes=2 #SBATCH --exclusive #SBATCH --ntasks-per-node=288

8 allocation-packs=
8 GPU, 576 CPU cores (8 Grace CPUs), ~1728 GB unified RAM

2048 SU

*4

export MPICH_GPU_SUPPORT_ENABLED=1
export OMP_NUM_THREADS=1
srun -N 1 -n 8 -c 72 --gres=gpu:8 --gpus-per-task=1 --gpu-bind=closest <executable>

 

By default, non-exclusive allocation shapes provide 128 GiB of local NVMe storage under /tmp and /var/tmp.

A full-node exclusive allocation using --exclusive --ntasks-per-node=288 provides access to the full local NVMe capacity of approximately 3500 GiB.

Notes for the request of resources:

  • For most jobs, use --gpus=N, where N is the total number of allocation-packs required.

  • Each GPU request automatically allocates 72 CPU cores and approximately 117000M of host memory.

  • Do not specify --cpus-per-task or --mem in the allocation request.

  • --gpus=N does not specify node geometry. Slurm packs GPUs onto as few nodes as possible.

  • If an explicit node geometry is required, use --nodes=N --gres=gpu:n --ntasks-per-node=72*n.

  • When --nodes and --gres are used together, the task count must be explicitly provided and must equal 72 tasks per GPU on each node.

  • The application's actual MPI task count should normally be specified in the srun command, not represented by the allocation-level task count.

  • --gpus-per-node=n follows the same task-count requirement but is not recommended due to previously observed issues.

  • For full-node allocation, use --nodes=N --exclusive --ntasks-per-node=288.

  • The same rules apply to interactive allocations made with salloc.

  • IMPORTANT: In addition to the request parameters shown in the table, users should indeed use other Slurm request parameters related to partition, walltime, job naming, output, email, etc. (Check the examples of the full Slurm batch scripts.)

Notes for the use/management of resources with srun:

  • Note that, for the case of srun, options do have the standard Slurm meaning.

  • The following options need to be explicitly provided to srun and not assumed to be inherited with some default value from the allocation request:

    • For srun, options have their standard Slurm meaning.

    • Always specify the resources required by the job step explicitly.

    • The --nodes (-N) option indicates the number of nodes used by the step.

    • The --ntasks (-n) option indicates the actual number of tasks to be launched by the step.

    • The --cpus-per-task (-c) option is used at the srun level to reserve CPU resources for each task. For one task per GH200 superchip, use -c 72.

    • Prefer --gpus=<total> to specify the total number of GPUs used by the step. This also works for multi-node allocations where the GPU distribution is uneven.

    • The --gpus-per-task option indicates the number of GPUs to be binded to each task spawned by the srun step via the -n option. Note that this option neglects sharing of the assigned GPUs to a task with other tasks. (See cases *4, *5 and *7 and their notes for non-intuitive cases.)

  • And for optimal binding, the following should be used:

    • The --gpu-bind=closest indicates that the chosen GPUs to be binded to each task should be the optimal (physically closest) to the chiplet assigned to each task.

    • IMPORTANT: The use of --gpu-bind=closest will assign optimal binding but may still NOT work and launch execution errors for codes relying OpenMP or OpenACC pragma's for moving data from/to host to/from GPU and attempting to use GPU-to-GPU enabled MPI communication. For those cases, the use of the "manual" optimal binding (method 2) is required. Method 2 is explained later in the main document.

Table 1 Footnotes

  • (*1) This is the only case where srun may work fine with default inherited option values. Nevertheless, it is a good practice to always use full explicit options of srun to indicate the resources needed for the executable. In this case, the settings explicitly "reserve" a whole Grace CPU (-c 72) for the srun task and control the real number of threads with the OMP_NUM_THREADS environment variable. Although the use of gres=gpu, gpus-per-task & gpu-bind is redundant in this case, we keep them for encouraging their use, which is strictly needed in the most of cases (except case *5).

  • (*2) The required CPU threads per task is 14 and that is controlled with the OMP_NUM_THREADS environment variable. But still the two full chiplets (-c 72) are indicated for each srun task.

  • (*3) The settings explicitly "reserve" a whole Grace CPU (-c 72) for each srun task. This provides superchip separation among each of the CPU cores to be allocated for the tasks spawned by srun (-n 3).  The real number of threads is controlled with the OMP_NUM_THREADS variable. The requirement of optimal binding of GPU to corresponding chiplet is indicated with the option --gpu-bind=closest. And, in order to allow GPU-aware MPI communication, the environment variable MPICH_GPU_SUPPORT_ENABLED is set to 1.

  • (*4) All GPUs in the node are requested, which mean all the resources available in the node via the --exclusive allocation option (there is no need to indicate the number of GPUs per node when using exclusive allocation). The use of -c 72 provides a whole superchip separation among each of the CPU cores to be allocated for the tasks spawned by srun (-n 8).  The real number of threads is controlled with the OMP_NUM_THREADS variable. The requirement of optimal binding of GPU to corresponding superchip is indicated with the option --gpu-bind=closest. And, in order to allow GPU-aware MPI communication, the environment variable MPICH_GPU_SUPPORT_ENABLED is set to 1.

General notes:

  • The allocation charge is for the total of allocated resources and not for the ones that are explicitly used in the execution, so all idle resources will also be charged

Achieving optimal GPU binding

As mentioned above, the optimal placement of GPUs and CPU cores for each task is to have direct communication within a superchip. So tasks being executed in cores in Chiplet 0 should be using GPU 4 (Bus D1), tasks in Chiplet 1 should be using GPU 5 (Bus D6), etc.

Method 1: srun parameters

This is the most intuitive (and simple) method for achieving optimal placement of CPUs and GPUs in each task spawned by srun. This method consists in providing the  --gpus-per-task and the --gpu-bind=closest parameters. So, for example, in a MPI job that requires the use of 4 CPU MPI processes (single threaded) with 1 GPU per task, the srun command to be used is:

export MPICH_GPU_SUPPORT_ENABLED=1 export OMP_NUM_THREADS=1 srun -N 1 -n 4 -c 72 --gres=gpu:4 --gpus-per-task=1 --gpu-bind=closest myMPIExecutable

The explanation of this method will be completed in the following sections where a very useful code (test_profile_util) will be used to confirm optimal (or sub-optimal, or incorrect) binding of GPUs (Slurm GPUs) and Grace CPUs for srun job steps.

It is important to be aware that this method works fine for most codes, but not for all. Codes suffering MPI communication errors with this methodology, should try the "manual" binding method described next.

Method 2: Manual binding

For codes relying OpenMP or OpenACC pragma's for moving data from/to host to/from GPU and attempting to use GPU-to-GPU enabled MPI communication, the first method may fail, giving errors similar to:

Terminal N. Example error message for some GPU-aware MPI
some error message

For these codes, the alternative is to use a "manual" method. This second method is more elaborated than the first but, as said, may be the only option for some codes.

In this "manual" method, the  --gpus-per-task and the --gpu-bind parameters (key of the first method) should NOT be provided. And, instead of those two parameters, we use an ordered list of CPU cores in the --cpu-bind option of srun to explicitly indicate the CPU cores where each task will be placed. An example of a "hardcoded" ordered list that would bind correctly the 4 GPUs across a node would be:

CPU_BIND="map_cpu:0,72,144,216"

("map_cpu" is a Slurm indicator of the type of binding to be used. Please read the Slurm documentation for further details.)

According to the node diagram at the top of this page, it is clear that this list consists of 1 CPU core per superchip. Then, the set of commands to use for a job that requires the use of 4 CPU tasks (single threaded) with 1 GPU (logical/Slurm GPU) per task would be:

export MPICH_GPU_SUPPORT_ENABLED=1 export OMP_NUM_THREADS=1 CPU_BIND="map_cpu:0,72,144,216" srun -N 1 -n 4 -c 72 --gres=gpu:4 --cpu-bind=${CPU_BIND} myMPIExecutable

This provides the optimal binding in a job that requires the use of 4 CPU tasks (single threaded) with 1 GPU (logical/Slurm GPU) per task.

For jobs that are hybrid, that is, that require multiple CPU threads per task, the list needs to be modified to be a list of masks instead of CPU core IDs. The explanation of the use of this list of masks will be given in the next subsection that also describes the use of an auxiliary script to generate the lists of CPU cores or mask for general cases.

For jobs that request exclusive use of the GPU nodes, the settings described in the example so far are enough for achieving optimal binding with the "manual" method. This works because the identification numbers of all the GPUs and the CPU cores that will be assigned to the job are known before hand (as all the resources of the node are what is requested). But when the job requires a reduced amount of resources, so that the request shares the rest of the node with other jobs, the GPUs and CPU cores that are to be allocated to the job are not known before submitting the script for execution. And, therefore, a "hardcoded" list of CPU cores that will always work to achieve optimal binding cannot be defined beforehand.

MPI & OpenMP settings

Thread control ( OMP_NUM_THREADS )

As mentioned in the previous section, allocation of resources is granted in "allocation-packs" with 72 cores (1 Grace CPU) per GPU. Also briefly mentioned in previous section is the need of "reserving" chunks of whole superchips (multiples of 72 CPU cores) in the srun command via the --cpus-per-task ( -c ) option. But the use of this option in srun is still more a "reservation" parameter for the srun tasks to be binded to the whole superchip, rather than an indication of the "real number of threads" to be used by the executable. The real number of threads to be used by the executable needs to be controlled by the  OpenMP environment variable OMP_NUM_THREADS. In other words, we use --cpus-per-task to make available whole superchips to the srun task, but use OMP_NUM_THREADS to control the real number of threads per srun task.

For pure MPI-GPU jobs it is recommended to set OMP_NUM_THREADS=1 before executing the srun command and avoid unexpected use of OpenMP threads:

export OMP_NUM_THREADS=1 srun ... -c 72 ...

For GPU codes with hybrid management on the CPU side (MPI + OpenMP + GPU), the environment variable needs to be set to the required number of threads per MPI task. For example, if 4 threads per task are required, then settings should be:

export OMP_NUM_THREADS=4 srun ... -c 72 ...

Also mentioned above is the example of a case where the "real number of threads" is 80 (which is greater than 8) and, therefore, requiring more than one chiplet. In that case, srun should reserve the number of chiplets per task that satisfy the demand using multiples of 72 in the --cpus-per-task (-c) option, together with the set the real number of threads with the OMP_NUM_THREADS environment variable:

export OMP_NUM_THREADS=80 srun ... -c 144 ...

GPU-Aware MPI

Use this for GPU-aware MPI codes

To use GPU-aware Cray MPICH, users must set the following modules and environment variables:

module load PrgEnv-nvidia
module load craype-arm-grace
export MPICH_GPU_SUPPORT_ENABLED=1

Testing code with profile_util

Here we will make use of the profile_util code within these same examples. For this reason, before presenting the full example, we use this section to explain important details of the test code. (If researchers want to test the code by themselves please clone the repo or expand the block below.)