Setonix uses the Slurm workload manager to schedule user programs for execution. To learn the generalities of using Slurm to schedule programs in supercomputers, visit the Job Scheduling page. In addition, please read the following subsections discuss the peculiarities of running jobs on Setonix, together with the Example Slurm Batch Scripts for Setonix on CPU Compute Nodes and Example Slurm Batch Scripts for Setonix on GPU Compute Nodes.
Important
It is highly recommended that you specify values for the --nodes, --ntasks, --cpus-per-task and --time options that are optimal for the job and for the system on which it will run. Also, use --mem if the job will not use all the resources in the node: shared access; or --exclusive for allocation of all resources in the requested nodes: exclusive access.
Overview
Each compute node of Setonix share its resources by default to run multiple jobs on the node at the same time, submitted by many users from the same or different projects. We call this configuration shared access and, as mentioned, is the default for Setonix nodes. Nevertheless, users can use slurm options to override the default and explicitly request for exclusive access to the requested nodes.
Nodes are grouped in partitions. Each partition is characterised by a particular configuration of its resources and it is intended for a particular workload or stage of the scientific workflow development. Table 1 shows the list of partitions present on Setonix and their available resources per node.
Each job submitted to the scheduler gets assigned a Quality of Service (QoS) level which determines the priority of the job with respect to the others in the queue. Usually, the default normal QoS applies. Users can boost the priority of their jobs up to 10% of their allocations, using the high QoS, in the following way:
$ sbatch --qos=high myscript.sh
Each project has an allocation for a number of service units (SUs) in a year, which is broken into quarters. Jobs submitted under a project will subtract SUs from the project's allocation. A project that has entirely consumed its SUs for a given quarter of the year will run its jobs in low priority mode for that time period. If a project's SU consumption (for a given quarter) hits the 150% usage mark with respect to its granted allocation, no further jobs will be able to run under the project.
Submitting jobs to the GPU partitions
You will need to use a different project code for the --account/-A option. More specifically, it is your project code followed by the -gpu suffix. For instance, if your project code is project1234, then you will have to use project1234-gpu.
Table 1. Slurm partitions on Setonix
| Name | N. Nodes | Cores per node | Available node-RAM for jobs | GPU chiplets per node | Types of jobs supported | Max Number of Nodes per Job | Max Wall time | Max Number of Concurrent Jobs per User | Max Number of Jobs Submitted per User |
|---|---|---|---|---|---|---|---|---|---|
| work | 1376 | 2x 64 | 230 GB | n/a | Supports CPU-based production jobs. | - | 24h | 256 | 1024 |
| long | 8 | 2x 64 | 230 GB | n/a | Long-running CPU-based production jobs. | 1 | 96h | 4 | 96 |
| highmem | 8 | 2x 64 | 980 GB | n/a | Supports CPU-based production jobs that require a large amount of memory. | 1 | 96h | 2 | 96 |
| debug | 8 | 2x 64 | 230 GB | n/a | Exclusive for development and debugging of CPU code and workflows. | 4 | 1h | 1 | 4 |
| gpu | 124 | 1x 64 | 230 GB | 8 | Supports GPU-based production jobs. | - | 24h | - | - |
| gpu-highmem | 38 | 1x 64 | 460 GB | 8 | Supports GPU-based production jobs requiring large amount of host memory. | - | 24h | - | - |
| gpu-dev | 20 | 1x 64 | 230 GB | 8 | Exclusive for development and debugging of GPU code and workflows. | - | 4h | - | - |
| copy | 7 | 1x 32 | 115 GB | n/a | Copy of large data to and from the supercomputer's filesystems. | - | 48h | 4 | 2048 |
| askaprt | 180 | 2x 64 | 230 GB | n/a | Dedicated to the ASKAP project (similar to work partition) | - | 24h | 8192 | 8192 |
| casda | 1 | 1x 32 | 115 GB | n/a | Dedicated to the CASDA project (similar to copy partition) | - | 24h | 30 | 40 |
| mwa | 10 | 2x 64 | 230 GB | n/a | Dedicated to the MWA projects (similar to work partition) | - | 24h | 1000 | 2000 |
| mwa-asvo | 10 | 2x 64 | 230 GB | n/a | Dedicated to the MWA projects (similar to work partition) | - | 24h | 1000 | 2000 |
| mwa-gpu | 10 | 1x 64 | 230 GB | 8 | Dedicated to the MWA projects (similar to gpu partition) | - | 24h | 1000 | 2000 |
| mwa-asvocopy | 2 | 1x 32 | 115 GB | n/a | Dedicated to the MWA projects (similar to copy partition) | - | 48h | 32 | 1000 |
Table 2. Quality of Service levels applicable to a Slurm job running on Setonix
| Name | Priority Level | Description |
|---|---|---|
| lowest | 0 | Reserved for particular cases. |
| low | 3000 | Priority for jobs past the 100% allocation usage. |
| normal | 10000 | The default priority for production jobs. |
| high | 14000 | Priority boost available to all projects for a fraction (10%) of their allocation. |
| highest | 20000 | Assigned to jobs that are of critical interest (e.g. project part of the national response to an emergency). |
| exhausted | 0 | QoS for jobs for projects that have consumed more than 150% of their allocation. |
Debug and Development Partitions Policy
To ensure the debug and development partitions are available for use by Pawsey researchers, they are strictly reserved for the following activities:- Code porting
- Code debugging
- Code development
- Job script/workflow management script porting, debugging and/or development
These partitions must not be used for the following activities:
- Production runs (i.e., jobs that are intended to generate final results or data for publication, reporting, or use in further analysis)
- Preparatory or test runs, including but not limited to:
- Warm-up/generation of initial conditions for simulations
- Testing configurations, searching for optimal/stabilitiy parameters, or setting up simulations, even if the results will not be used directly.
- Running simulations or experiments to determine production parameters for AI/ML model training (e.g., hyperparameter tuning, configuration testing, validation of stability under different settings).
- Testing code or scripts in ways that mimic production workloads, such as large-scale simulations or model training, that are not explicitly part of the development or debugging process.
Note: This restriction applies regardless of the execution time of the jobs. For instance, jobs that involve testing for numerical stability, parameter optimization, or early-stage simulations should not be conducted on the debug/development partitions, even if the run times are under the partition's walltime limit.
- Warm-up/generation of initial conditions for simulations
- Testing configurations, searching for optimal/stabilitiy parameters, or setting up simulations, even if the results will not be used directly.
- Running simulations or experiments to determine production parameters for AI/ML model training (e.g., hyperparameter tuning, configuration testing, validation of stability under different settings).
- Testing code or scripts in ways that mimic production workloads, such as large-scale simulations or model training, that are not explicitly part of the development or debugging process.
Job Queue Limits
Users can check the limits on the maximum number of jobs that users can run at a time (i.e., MaxJobs) and the maximum number of jobs that can be submitted (i.e., MaxSubmitJobs) for each partition on Setonix using the command:
$ sacctmgr show associations user=$USER cluster=setonix
Additional constraints are imposed on projects that have overused their quarterly allocation.
Executing large jobs
When executing large, multinode jobs on Setonix, the use of the --exclusive option in the batch script is recommended. The addition will result in better resource utilisation within each node assigned to the job.