...

Several scientific applications are already able to offload computations to the MI250X, many others are in the process of being ported to AMD GPUs. Here is a list of the main ones and their current status.

...

AMD ROCm installations

The main default ROCm installation is rocm/5.0.2  provided by HPE Cray. In addition, Pawsey staff have installed the more recent rocm/5.4.3  from source using ROCm-from-source. It is an experimental installation and users might encounter compilation or linking errors. You are encouraged to explore it during development and to report any issues. For production jobs, however, we currently recommend using rocm/5.0.2.

...

You can submit GPU jobs to the gpu, gpu and -dev and gpu-highmem Slurm partitions using your GPU allocation.

Note that you will need to use a different project code for the --account/-A option. More specifically, it is your project code followed by the -gpu suffix. For instance, if your project code is project1234, then you will have to use project1234-gpu.

GPUs must be explicitly requested to Slurm using the --gres=gpu:<num_gpus>,  --gpus-per-task=<num_gpus> or --gpu-per-node=<num_gpus> options. The --gpus-per-task option is recommended.

Compiling software

If you are using ROCm libraries, such as rocFFT, to offload computations to GPUs, you should be able to use any compiler to link those to your code.

...

Each MI250X GCD, which corresponds to a Slurm GPU, is charged 64SU 64 SU per hour. This means the use of an entire GPU node is charged 512SU 512 SU per hour. In general, a job is charged the largest proportion of core, memory, or GPU usage rounded up to 1/8ths of a node (corresponding to an individual MI250X GCD). Note that GPU node usage is accounted against GPU allocations with the -gpu suffix, which are separate to CPU allocations.

Programming AMD GPUs

You can program AMD MI250X GPUs using HIP, which is the programming framework equivalent to the one of NVIDIA, CUDA. The HIP platform is available after having loaded the rocm  module.

...

.

The complete AMD documentation on how to program with HIP can be found here (external site).

Example Jobscripts

The following are some examples of requesting GPUs via Slurm batch scripts on Setonix. For more detail, refer to Example Slurm Batch Scripts for Setonix on GPU Compute Nodes.

#!/bin/bash --login​
 
​#SBATCH --account=project-gpu​
#SBATCH --partition=gpu​
#SBATCH --ntasks=1​
#SBATCH --ntasks-per-node=1​
#SBATCH --cpus-per-task=8​
#SBATCH --gpus-per-task=1​
#SBATCH --time=24:00:00​
​
srun -c 8 --gpu-bind=closest ./program

#!/bin/bash --login​
 
#SBATCH --account=project-gpu​
#SBATCH --partition=gpu​
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-task=8​
#SBATCH --time=24:00:00​
#SBATCH --exclusive
 
srun -c 8 --gpu-bind=closest ./program 

#!/bin/bash --login​
 
#SBATCH --account=project-gpu​
#SBATCH --partition=gpu​
#SBATCH --ntasks=8​
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-task=1​
#SBATCH --time=24:00:00​
#SBATCH --exclusive
 
srun -c 8 --gpu-bind=closest ./program