Overview

The Setonix Quantum partition consists of 4 GH200 compute nodes and your allocation can be viewed on origin and used by setting the account to <project_id>-q. This allocation is seperate from the CPU and GPU Setonix allocations.

Nodes

Each GH200 compute node has four NVIDIA GraceHopper GH200 superchips. Each GH200 superchip has a Grace Arm Neoverse V2 CPU with 72 cores with 120 GB of LPDDR and a Hopper H100 GPU with 96 GB of memory. Due to the unified memory architecture of the superchip, each node will report 4x120 + 4x96 of RAM. The available memory is 855GB of usable space. The Grace CPU architecture is completely different from all other nodes, which are x86_64. These nodes are connected to the Slingshot by two NIC cards for data transfer between nodes. These nodes are available on the quantum partition associated with the Setonix-Q Pilot merit allocation scheme. Each Setonix GH200 node has an attached NVMe device with 3575GB usable space.

Open

Figure 1. A GH200 node of Setonix (original image from CSCS Alps documentation).

The GH200 node has small internal latencies between each core of 60-70 ns. The latency is larger across superchips and sockets as seen in the Figure 2. The exact numbers will vary but typical socket 1 to socket n latency will be of order 370 ns, whereas all other sockets-to-socket transfers will have latencies of roughly 500 ns.

 

Open

Figure 2. GH200 node cross-socket latency

Software on GH200 Arm nodes

A GH200 node has a Grace Arm CPU, which is a completely different architecture to the rest of the CPUs on Setonix, which are x86 architecture. Software compiled with other nodes will not run on the GH200 nodes. Thus for software to run on GH200 nodes, it must be compiled on a GH200 node. As a result, the software stack and module paths for Setonix-Q GH200 nodes is quite separate from the other software stack.

To load these modules, you should purge the module environment and do the following steps.

This will set the relevant module paths. Note that this also needs to be done in all sbatch scripts.

Supported Applications

Although much of the applications that can run on the GPU partition (AMD MI250x GPUs, HIP) can run on the GH200 nodes (NVIDIA GraceHopper GH200 superchip, CUDA), these applications are not provided on the quantum partition. We instead provide software focused on quantum simulation.

Module names of GH200 ARM applications exist in a separate module path.

Supported Numerical Libraries

Popular numerical routines and functions have been implemented by NVIDIA to run on their GPU hardware. All of the following are available when loading the PrgEnv-nvidia  module.

Table 2. Popular GPU numerical libraries.

A complete list of available libraries can be found on this page (external site).

Submitting Jobs

You can submit GPU jobs to the quantum Slurm partitions using your Setonix-Q allocation.

Note that you will need to use a different project code for the --account/-A option. More specifically, it is your project code followed by the -quantum suffix. For instance, if your project code is project1234, then you will have to use project1234-q.

Unlike the CPU partitions, the /tmp and /var/tmp directories access NVMe storage, not the tmpfs RAM disk. (The tmpfs filesystem is still available under /dev/shm.) Request a specific amount of NVMe storage in your job script by adding tmp:<some-value>G to the --gres option. Unless using all of the GPUs on the node, please limit your NVMe request to no more than 2679 GiB. You should not be able to use more NVMe space than what has been allocated to you. By default, without any explicit NVMe request, a job should get allocated 128 GiB of the NVMe device. The NVMe device (or the portion used by a job) is cleaned up after the job completes.  IMPORTANT: Migrate any valuable results from the NVMe device before the job completes.

An extensive explanation on the use of the Setonix-Q GH200 nodes (including request by "allocation packs" and the "manual" binding) is in Example Slurm Batch Scripts for Setonix-Q on GH200 Compute Nodes

Compiling software

As mentioned, GH200 node has a Grace Arm CPU, which is a completely different architecture to the rest of the CPUs on Setonix, which are x86 architecture. Software must be compiled on a GH200 node. To compile code, first you should salloc to a GH200 node and load the appropriate Cray Programming Environment (CPE). Due to the ARM architecture of the GH200 nodes, you must purge the module environment from the login nodes and reload the relevant environment. This will eventually be automated with the pawseyenv module.

For CUDA code you can use the Cray compiler wrappers cc/CC/ftn or use nvcc, nvc++, or nvfortran explicitly as appropriate. And, for code making use of OpenMP offloading, you can use:

• cc/CC for c/c++

• ftn for fortran.

When using nvc++, and the NVIDIA compilers, note that the location of the MPI headers and libraries are not automatically included (contrary to the automatic inclusion when using the Cray wrapper scripts). Therefore, if your code also requires MPI, the location of the MPI headers and libraries must be provided to nvc++ as well as the GPU Transport Layer libraries:

We also suggest using the gcc/12.3 toolchain when compiling code to use the newer host compiler.

Accounting

Each GH200 superchip has 1 GPU, which corresponds to a Slurm GPU, is charged 256 SU per hour. This means the use of an entire GH200 node is charged 1024 SU per hour. In general, a job is charged the largest proportion of core, memory, or GPU usage rounded up to 1/4ths of a node (corresponding to an individual GraceHopper GH200 superchip). Note that quantum node usage is accounted against GPU allocations with the -quantum suffix, which are separate to other allocations.

Programming NVIDIA GH200 GPUs

You can program NVIDIA GraceHopper GH200 GPUs using CUDA. The complete NVIDIA documentation on how to program with CUDA can be found here (external site).

Example Jobscripts

The following are some brief examples of requesting GPUs via Slurm batch scripts on Setonix-Q GH200 nodes. For more detail, refer to Example Slurm Batch Scripts for Setonix-Q on GH200 Compute Nodes

Example 2 : Single CPU process that use the 4 GPUs of the node

Full guides

• Example Slurm Batch Scripts for Setonix-Q on GH200 Compute Nodes

• Setonix User Guide

Related pages

• Quantum Hub: Portal Quick Start