...
| Excerpt |
|---|
Pawsey's way for requesting resources on GPU nodes (different to standard Slurm)The request of resources for the GPU nodes has changed dramatically. The main reason for this change has to do with Pawsey's efforts to provide a method for optimal binding of the GPUs to the CPU cores in direct physical connection for each task. For this, we decided to completely separate the options used for resource request via salloc or (#SBATCH pragmas) and the options for the use of resources during execution of the code via srun. | Note |
|---|
| title | Request for the amount of "allocation-packs" required for the job |
|---|
| With a new CLI filter that Pawsey staff had put in place for the GPU nodes, the request of resources in GPU nodes should be thought as requesting a number of "allocation-packs". Each "allocation-pack" provides: - 1 whole CPU chiplet (8 CPU cores)
- a bit less of 32 GB memory (29.44 GB of memory, to be exact, allowing some memory for the system to operate the node) = 1/8 of the total available RAM
- 1 GCD directly connected to that chiplet
For that, the request of resources only needs the number of nodes (–-nodes, -N) and the number of allocation-packs per node (--gres=gpu:number). The total of allocation-packs requested results from the multiplication of these two parameters. Note that the standard Slurm meaning of the second parameter IS NOT used at Pawsey. Instead, Pawsey's CLI filter interprets this parameter as: - the number of requested "allocation-packs" per node
Note that the "equivalent" option --gpus-per-node=number (which is also interpreted as the number of "allocation-packs" per node) is not recommended as we have found some bugs with its use. |
Furthermore, in the request of resources, users should not indicate any other Slurm allocation option related to memory or CPU cores. Therefore, users should not use --ntasks, --cpus-per-task, --mem, etc. in the request headers of the script ( #SBATCH directives), or in the request options given to salloc for interactive sessions. If, for some reason, the requirements for a job are indeed determined by the number of CPU cores or the amount of memory, then users should estimate the number of "allocation-packs" that cover their needs. The "allocation-pack" is the minimal unit of resources that can be managed, so that all allocation requests should be indeed multiples of this basic unit. Pawsey also has some site specific recommendations for the use/management of resources with srun command. Users should explicitly provide a list of several parameters for the use of resources by srun. (The list of these parameters is made clear in the examples below.) Users should not assume that srun will inherit any of these parameters from the allocation request. Therefore, the real management of resources at execution time is performed by the command line options provided to srun. Note that, for the case of srun, options do have the standard Slurm meaning.
| Warning |
|---|
| title | --gpu-bind=closest may NOT work for all applications |
|---|
| Within the full explicit srun options for "managing resources", there are some that help to achieve optimal binding of GPUs to their directly connected chiplet on the CPU. There are two methods to achieve this optimal binding of GPUs. So, together with the full explicit srun options, the following two methods can be used: - Include these two Slurm parameters:
--gpus-per-task=<number> together with --gpu-bind=closest - "Manual" optimal binding with the use of "two auxiliary techniques" (explained later in the main document).
The first method is simpler, but may still launch execution errors for some codes. "Manual" binding may be the only useful method for codes relying OpenMP or OpenACC pragma's for moving data from/to host to/from GPU and attempting to use GPU-to-GPU enabled MPI communication. An example of such a code is Slate. |
The following table provides some examples that will serve as a guide for requesting resources in the GPU nodes: | Required Resources per Job | New "simplified" way of requesting resources | Total Allocated resources | Charge per hour | The use of full explicit srun options is now required
(only the 1st method for optimal binding is listed here) |
|---|
| 1 CPU task (single CPU thread) controlling 1 GCD (Slurm GPU) | #SBATCH --nodes=1
#SBATCH --gres=gpu:1 | 1 allocation-pack =
1 GPU, 8 CPU cores (1 chiplet), 29.44 GB CPU RAM | 64 SU | *1
export OMP_NUM_THREADS=1
srun -N 1 -n 1 -c 8 --gres=gpu:1 --gpus-per-task=1 --gpu-bind=closest <executable>
| 1 CPU task (with 14 CPU threads each) all threads controlling the same 1 GCD | #SBATCH --nodes=1
#SBATCH --gres=gpu:2
| 2 allocation-packs=
2 GPUs, 16 CPU cores (2 chiplets), 58.88 GB CPU RAM | 128 SU | *2
export OMP_NUM_THREADS=14
srun -N 1 -n 1 -c 16 --gres=gpu:1 --gpus-per-task=1 --gpu-bind=closest <executable>
| | 3 CPU tasks (single thread each), each controlling 1 GCD with GPU-aware MPI communication | #SBATCH --nodes=1
#SBATCH --gres=gpu:3 | 3 allocation-packs=
3 GPUs, 24 CPU cores (3 chiplets), 88.32 GB CPU RAM | 192 SU | *3
export MPICH_GPU_SUPPORT_ENABLED=1
export OMP_NUM_THREADS=1
srun -N 1 -n 3 -c 8 --gres=gpu:3 --gpus-per-task=1 --gpu-bind=closest <executable>
| | 2 CPU tasks (single thread each), each task controlling 2 GCDs with GPU-aware MPI communication | #SBATCH --nodes=1
#SBATCH --gres=gpu:4
| 4 allocation-packs=
4 GPU, 32 CPU cores (4 chiplets), 117.76 GB CPU RAM | 256 SU | *4 export MPICH_GPU_SUPPORT_ENABLED=1
export OMP_NUM_THREADS=1
srun -N 1 -n 2 -c 16 --gres=gpu:4 --gpus-per-task=2 --gpu-bind=closest <executable>
| 5 CPU tasks (with 2 CPU threads single thread each) all threads/tasks able to see all 5 GPUs | #SBATCH --nodes=1
#SBATCH --gres=gpu:5
| 5 allocation-packs=
5 GPUs, 40 CPU cores (5 chiplets), 147.2 GB CPU RAM | 320 SU | *5
export MPICH_GPU_SUPPORT_ENABLED=1
export OMP_NUM_THREADS=21
srun -N 1 -n 5 -c 8 --gres=gpu:5 --gpus-per-task=55 <executable>
| | 8 CPU tasks (single thread each), each controlling 1 GCD with GPU-aware MPI communication | #SBATCH --nodes=1
#SBATCH --exclusive | 8 allocation-packs=
8 GPU, 64 CPU cores (8 chiplets), 235 GB CPU RAM | 512 SU | *6 export MPICH_GPU_SUPPORT_ENABLED=1
export OMP_NUM_THREADS=1
srun -N 1 -n 8 -c 8 --gres=gpu:8 --gpus-per-task=1 --gpu-bind=closest <executable>
| | 8 CPU tasks (single thread each), each controlling 4 GCD with GPU-aware MPI communication | #SBATCH --nodes=4
#SBATCH --exclusive | 32 allocation-packs=
4 nodes, each with: 8 GPU, 64 CPU cores (8 chiplets), 235 GB CPU RAM | 2048 SU | *7 export MPICH_GPU_SUPPORT_ENABLED=1
export OMP_NUM_THREADS=1
srun -N 4 -n 8 -c 32 --gres=gpu:8 --gpus-per-task=4 --gpu-bind=closest <executable>
| | 1 CPU task (single thread), controlling 1 GCD but avoiding other jobs to run in the same node for ideal performance test. | #SBATCH --nodes=1
#SBATCH --exclusive | 8 allocation-packs=
8 GPU, 64 CPU cores (8 chiplets), 235 GB CPU RAM | 512 SU | *8 export OMP_NUM_THREADS=1
srun -N 1 -n 1 -c 8 --gres=gpu:1 --gpus-per-task=1 --gpu-bind=closest <executable>
| Notes for the request of resources: - Note that this simplified way of resource request is based on requesting a number of "allocation-packs", so that standard use of Slurm parameters for allocation should not be used for GPU resources.
- The
--nodes (-N) option indicates the number of nodes requested to be allocated. - The
--gres=gpu:number option indicates the number of allocation-packs requested to be allocated per node. (The "equivalent" option --gpus-per-node=number is not recommended as we have found some bugs with its use.) - The
--exclusive option requests all the resources from the number of requested nodes. When this option is used, there is no need for the use of --gres=gpu:number during allocation and, indeed, its use is not recommended in this case. - Users should not include any other Slurm allocation option that may indicate some "calculation" of required memory or CPU cores. The management of resources should only be performed after allocation via
srun options. - The same simplified resource request should be used for the request of interactive sessions with
salloc. - IMPORTANT: In addition to the request parameters shown in the table, users should indeed use other Slurm request parameters related to partition, walltime, job naming, output, email, etc. (Check the examples of the full Slurm batch scripts.)
Notes for the use/management of resources with srun: - Note that, for the case of
srun, options do have the standard Slurm meaning. - The following options need to be explicitly provided to
srun and not assumed to be inherited with some default value from the allocation request:- The
--nodes (-N) option indicates the number of nodes to be used by the srun step. - The
--ntasks (-n) option indicates the total number of tasks to be spawned by the srun step. By default, tasks are spawned evenly across the number of allocated nodes. - The
--cpus-per-task (-c) option should be set to multiples of 8 (whole chiplets) to guarantee that srun will distribute the resources in "allocation-packs" and then "reserving" whole chiplets per srun task, even if the real number is 1 thread per task. The real number of threads is controlled with the OMP_NUM_THREADS environment variable. - The
--gres=gpu:number option indicates the number of GPUs per node to be used by the srun step. (The "equivalent" option --gpus-per-node=number is not recommended as we have found some bugs with its use.) - The
--gpus-per-task option indicates the number of GPUs to be binded to each task spawned by the srun step via the -n option. Note that this option neglects sharing of the assigned GPUs to a task with other tasks. (See cases *4, *5 and *7 and their notes for non-intuitive cases.)
- And for optimal binding, the following should be used:
- The
--gpu-bind=closest indicates that the chosen GPUs to be binded to each task should be the optimal (physically closest) to the chiplet assigned to each task. - IMPORTANT: The use of
--gpu-bind=closest will assign optimal binding but may still NOT work and launch execution errors for codes relying OpenMP or OpenACC pragma's for moving data from/to host to/from GPU and attempting to use GPU-to-GPU enabled MPI communication. For those cases, the use of the "manual" optimal binding (method 2) is required. Method 2 is explained later in the main document.
- (*1) This is the only case where
srun may work fine with default inherited option values. Nevertheless, it is a good practice to always use full explicit options of srun to indicate the resources needed for the executable. In this case, the settings explicitly "reserve" a whole chiplet (-c 8) for the srun task and control the real number of threads with the OMP_NUM_THREADS environment variable. Although the use of gres=gpu, gpus-per-task & gpu-bind is reduntant in this case, we keep them for encouraging their use, which is strictly needed in the most of cases (except case *5). - (*2) The required CPU threads per task is 14 and that is controlled with the
OMP_NUM_THREADS environment variable. But still the two full chiplets (-c 16) are indicated for each srun task. - (*3) The settings explicitly "reserve" a whole chiplet (
-c 8) for each srun task. This provides "one-chiplet-long" separation among each of the CPU cores to be allocated for the tasks spawned by srun (-n 3). The real number of threads is controlled with the OMP_NUM_THREADS variable. The requirement of optimal binding of GPU to corresponding chiplet is indicated with the option --gpu-bind=closest. And, in order to allow GPU-aware MPI communication, the environment variable MPICH_GPU_SUPPORT_ENABLED is set to 1. - (*4) Each task needs to be in direct communication with 2 GCDs. For that, each of the CPU task reserve "two-full-chiplets". The use of
-c 16 "reserves" a "two-chiplets-long" separation among the two CPU cores that are to be used (one for each of the srun tasks, -n 2 ). In this way, each task will be in direct communication to the two logical GPUs in the MI250X card that has optimal connection to the chiplets reserved for each task. The real number of threads is controlled with the OMP_NUM_THREADS variable. The requirement of optimal binding of GPU to corresponding chiplet is indicated with the option --gpu-bind=closest. And, in order to allow GPU-aware MPI communication, the environment variable MPICH_GPU_SUPPORT_ENABLED is set to 1. - (*5) Sometimes, the executable performs all the management of GPUs requested. If all the management logic for the GPUs is performed by the executable, then all the available resources should be exposed to it. In this case the option for optimal binding is not provided. Neither the
--gpus-per-task option is provided, as all the available GPUs are to be shared among tasks. The real number of threads is controlled with the the OMP_NUM_THREADS variable. And, in order to allow GPU-aware MPI communication, the environment variable MPICH_GPU_SUPPORT_ENABLED is set to 1. - (*6) All GPUs in the node are requested, which mean all the resources available in the node via the
--exclusive allocation option (there is no need to indicate the number of GPUs per node when using exclusive allocation). The use of -c 8 provides "one-chiplet-long" separation among each of the CPU cores to be allocated for the tasks spawned by srun (-n 8). The real number of threads is controlled with the OMP_NUM_THREADS variable. The requirement of optimal binding of GPU to corresponding chiplet is indicated with the option --gpu-bind=closest. And, in order to allow GPU-aware MPI communication, the environment variable MPICH_GPU_SUPPORT_ENABLED is set to 1. - (*7) All resources in each node are requested via the
--exclusive allocation option (there is no need to indicate the number of GPUs per node when using exclusive allocation). Each task needs to be in direct communication with 4 GCDs. For that, each of the CPU task reserve "four-full-chiplets". The use of -c 32 "reserves" a "four-chiplets-long" separation among the two CPU cores that are to be used per node (8 srun tasks in total, -n 8 ). The real number of threads is controlled with the OMP_NUM_THREADS variable. The requirement of optimal binding of GPU to corresponding chiplet is indicated with the option --gpu-bind=closest. In this way, each task will be in direct communication to the closest four logical GPUs in the node with respect to the chiplets reserved for each task. And, in order to allow GPU-aware MPI communication, the environment variable MPICH_GPU_SUPPORT_ENABLED is set to 1. The --gres=gpu:8 option assigns 8 GPUs per node to the srun step (32 GPUs in total as 4 nodes are being assigned). - (*8) All GPUs in the node are requested using the --
exclusive option, but only 1 CPU chiplet - 1 GPU "unit" (or allocation-pack) is used in the srun step.
General notes: - The allocation charge is for the total of allocated resources and not for the ones that are explicitly used in the execution, so all idle resources will also be charged
|
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal N. Explaining the use the "hello_jobstep" code from an salloc session (list allocated GPUs) |
|---|
| $ rocm-smi --showhw
======================= ROCm System Management Interface =======================
============================ Concise Hardware Info =============================
GPU DID GFX RAS SDMA RAS UMC RAS VBIOS BUS
0 7408 DISABLED ENABLED DISABLED 113-D65201-042 0000:C9:00.0
1 7408 DISABLED ENABLED DISABLED 113-D65201-042 0000:D1:00.0
2 7408 DISABLED ENABLED DISABLED 113-D65201-042 0000:D6:00.0
================================================================================
============================= End of ROCm SMI Log ============================== |
|
...
Using hello_jobstep code for testing a non-recommended practice
In a first test, we observe what happens when no "management" parameters are given to to srun. So, in this "non-recommended" setting, the output is:
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal N. Explaining the use the "hello_jobstep" code from an salloc session ( "not recommended" use without full srun parameters) |
|---|
| $ export OMP_NUM_THREADS=1; srun -N 1 -n 3 ./hello_jobstep | sort -n
MPI 000 - OMP 000 - HWT 000 - Node nid001004 - RunTime_GPU_ID 0,1,2 - ROCR_VISIBLE_GPU_ID 0,1,2 - GPU_Bus_ID c9,d1,d6
MPI 001 - OMP 000 - HWT 008001 - Node nid001004 - RunTime_GPU_ID 0,1,2 - ROCR_VISIBLE_GPU_ID 0,1,2 - GPU_Bus_ID c9,d1,d6
MPI 002 - OMP 000 - HWT 016002 - Node nid001004 - RunTime_GPU_ID 0,1,2 - ROCR_VISIBLE_GPU_ID 0,1,2 - GPU_Bus_ID c9,d1,d6 |
|
As can be seen, each MPI task have been can be assigned to a CPU core in a different chiplet. But the same chiplet by the scheduler, which is not a recommended practice. Also, all three GCDs (logical/Slurm GPUs) that have been allocated are visible to each of the tasks. Although some codes are able to deal with this kind of available resources, this is not the recommended best practice. The recommended best practice is to assign CPU tasks to different chiplets and to provide only 1 GCD per task and, even more, to provide the optimal bandwidth between CPU and GCD.
Using hello_jobstep code for testing optimal
...
binding for a pure MPI job (single threaded) 1 GPU per task
Starting from the same allocation as above (3 "allocation-packs"), now all the parameters needed to define the correct use of resources are provided to srun. In this case, 3 MPI tasks are to be ran (single threaded) each task making use of 1 GCD (logical/Slurm GPU). As described above, there are two methods to achieve optimal binding. The first method only uses Slurm parameters to indicate how resources are to be used by srun. In this case:
...
Again, there is a difference is in the values of the ROCR_VISIBLE_GPU_IDs in the results of both methods. With the first method, these values are always 0 while, in the second method, these values are the ones given by the wrapper that "manually" selects the GCDs (logical/Slurm GPUs). This difference has proven to be important and may be the reason why the "manual" binding is the only option for codes relying OpenMP or OpenACC pragma's for moving data from/to host to/from GPU and attempting to use GPU-to-GPU enabled MPI communication.
Example scripts for: Exclusive access to the GPU nodes
...
Using hello_jobstep code for testing visibility of all allocated GPUs to each of the tasks
Some codes, like tensorflow and other machine learning engines, require visibility of all GPU resources for an internal-to-the-code management of resources. In that case, optimal binding cannot be provided to the code and then the responsability of optimal binding and communication among the resources is given completely to the code. In that case, the recommended settings for the srun command are:
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal N. Explaining the use the "hello_jobstep" code from an salloc session ( "not recommended" use without full srun parameters) |
|---|
| $ export OMP_NUM_THREADS=1; srun -N 1 -n 3 -c 8 --gres=gpu:3 ./hello_jobstep | sort -n
MPI 000 - OMP 000 - HWT 000 - Node nid001004 - RunTime_GPU_ID 0,1,2 - ROCR_VISIBLE_GPU_ID 0,1,2 - GPU_Bus_ID c9,d1,d6
MPI 001 - OMP 000 - HWT 008 - Node nid001004 - RunTime_GPU_ID 0,1,2 - ROCR_VISIBLE_GPU_ID 0,1,2 - GPU_Bus_ID c9,d1,d6
MPI 002 - OMP 000 - HWT 016 - Node nid001004 - RunTime_GPU_ID 0,1,2 - ROCR_VISIBLE_GPU_ID 0,1,2 - GPU_Bus_ID c9,d1,d6 |
|
As can be seen, each MPI task is assigned to a different chiplet. Also, all three GCDs (logical/Slurm GPUs) that have been allocated are visible to each of the tasks which, for these codes, is what they need to run properly.
Example scripts for: Exclusive access to the GPU nodes with optimal binding
In this section, a series of example slurm job scripts are presented in order for the users to be able to use them as a point of departure for preparing their own scripts. The examples presented here make use of most of the important concepts, tools and techniques explained in the previous section, so we encourage users to take a look into that top section of this page first.
...
| Ui tabs |
|---|
| Ui tab |
|---|
| title | B. Method 1: Optimal binding using srun parameters |
|---|
| For optimal binding using srun parameters the options "--gpus-per-task" & "--gpu-bind=closest" need to be used: 900px
bashEmacsListing N. exampleScript_2NodesExclusive_16GPUs_bindMethod1.shtrue
Now, let's take a look to the output after executing the script: 900px
bashDJangoTerminal N. Output for 16 GPUs job (2 nodes) exclusive access
According to the architecture diagram, this binding configuration is optimal. Method 1 may fail for some applications.This first method is simpler, but may not work for all codes. "Manual" binding (method 2) may be the only reliable method for codes relying OpenMP or OpenACC pragma's for moving data from/to host to/from GPU and attempting to use GPU-to-GPU enabled MPI communication. "Click" in the TAB above to read the script and output for the other method of GPU binding. |
| Ui tab |
|---|
| title | B. Method 2: "Manual" optimal binding of GPUs and chiplets |
|---|
| For "manual" binding, two auxiliary techniques need to be performed: 1) use of a wrapper and 2) generate an ordered list to be used in the --cpu-bind option of srun: 900px
bashEmacsListing N. exampleScript_2NodesExclusive_16GPUs_bindMethod2.shtrue
Note that the wrapper for selecting the GPUs is being created with a redirection to the cat command. Also node that its name uses the SLURM_JOBID environment variable to make this wrapper unique to this job, and that the wrapper is deleted when execution is finalised. Now, let's take a look to the output after executing the script: 900px
bashDJangoTerminal N. Output for 16 GPUs job (2 nodes) exclusive access
According to the architecture diagram, this binding configuration is optimal. "Click" in the TAB above to read the script and output for the other method of GPU binding. |
|
Example scripts for: Shared access to the GPU nodes with optimal binding
Shared node 1 GPU job
Jobs that need only 1 GCD (logical/Slurm GPU) for their execution are going to be sharing the GPU node with other jobs. That is, they will run in shared access, which is the default so no request for exclusive access is performed.
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | Emacs |
|---|
| title | Listing N. exampleScript_1NodeShared_1GPU.sh |
|---|
| linenumbers | true |
|---|
| #!/bin/bash --login
#SBATCH --job-name=1GPUSharedNode
#SBATCH --partition=gpu
#SBATCH --nodes=1 #1 nodes in this example
#SBATCH --gres=gpu:1 #1 GPU per node (1 "allocation-pack" in total for the job)
#SBATCH --time=00:05:00
#SBATCH --account=<yourProject>-gpu #IMPORTANT: use your own project and the -gpu suffix
#(Note that there is not request for exclusive access to the node)
#----
#Loading needed modules (adapt this for your own purposes):
module load PrgEnv-cray
module load rocm craype-accel-amd-gfx90a
echo -e "\n\n#------------------------#"
module list
#----
#Printing the status of the given allocation
echo -e "\n\n#------------------------#"
echo "Printing from scontrol:"
scontrol show job ${SLURM_JOBID}
#----
#Definition of the executable (we assume the example code has been compiled and is available in $MYSCRATCH):
exeDir=$MYSCRATCH/hello_jobstep
exeName=hello_jobstep
theExe=$exeDir/$exeName
#----
#MPI & OpenMP settings
#Not needed for 1GPU:export MPICH_GPU_SUPPORT_ENABLED=1 #This allows for GPU-aware MPI communication among GPUs
export OMP_NUM_THREADS=1 #This controls the real CPU-cores per task for the executable
#----
#Execution
#Note: srun needs the explicit indication full parameters for use of resources in the job step.
# These are independent from the allocation parameters (which are not inherited by srun)
# For optimal GPU binding using slurm options,
# "--gpus-per-task=1" and "--gpu-bind=closest" create the optimal binding of GPUs
# (Although in this case this can be avoided as only 1 "allocation-pack" has been requested)
echo -e "\n\n#------------------------#"
echo "Test code execution:"
srun -l -u -N 1 -n 1 -c 8 --gres=gpu:1 --gpus-per-task=1 --gpu-bind=closest ${theExe} | sort -n
#----
#Printing information of finished job steps:
echo -e "\n\n#------------------------#"
echo "Printing information of finished jobs steps using sacct:"
sacct -j ${SLURM_JOBID} -o jobid%20,Start%20,elapsed%20
#----
#Done
echo -e "\n\n#------------------------#"
echo "Done" |
|
...