| Excerpt |
|---|
|
Supercomputing project exampleThis page gives an example in how to handle a data workflow using Acacia and Setonix, with three different jobs connected by dependencies. |
This page gives an example in how to handle a data workflow using Acacia and Setonix, with three different jobs connected by dependencies.
Example workflow on Setonix
The data workflow will consist of:
- A first job submitted to the data transfer nodes (
copy partition) that will stage initial data from Acacia and into the working directory on /scratch - A second job submitted to the compute nodes (
work partition) that will execute a supercomputing job making use of the staged data. We assume that this job will produce new result/data in the same working directory on /scratch. - A third job submitted to the data transfer nodes (
copy partition) that will handle the new results/data and store them into Acacia
Script for staging of initial data into /scratch
Staging can be performed with a slurm job script that makes use of the data-transfer nodes (copy partition) to handle initial data originally stored in Acacia and to be staged into the working directory on /scratch. Note that the original data is packed within a .tar file, so the script also performs the "untarring" of the data:
...
| Note |
|---|
|
- Note the use of variables to store the names of directories, files, buckets, prefixes, objects etc.
- Also note the several checks at the different parts of the script and the redirection to
/dev/null in most of the commands used for checking correctness (as we are not interested in their output). - As messages from the mc client are too verbose when transferring files (even with the
--quiet option), we make use of a redirection of the output messages to /dev/null when performing transfers with this client on scripts. For this reason we often find rclone a better choice for the use of clients in scripts.
|
The slurm job script for performing the supercomputing job makes use of the initial data staged in the previous step. This job requests execution in the compute nodes (work partition).
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | Emacs |
|---|
| title | Listing 2. superExecutionSetonix.sh |
|---|
| linenumbers | true |
|---|
| #!/bin/bash -l
#SBATCH --account=[yourProject]
#SBATCH --job-name=superExecution
#SBATCH --partition=work
#SBATCH --ntasks=[numberOfCoresToUse]
#SBATCH --time=[requiredTime]
#SBATCH --export=none
#--------------
#Load required modules here ...
#---------------
#Defining the working dir
workingDir="$MYSCRATCH/<pathAndNameOfWorkingDirectory"
#---------------
#Entering the working dir
cd $workingDir
#---------------
#Check for the correct staging of the initial conditions if needed ...
#---------------
#Supercomputing execution
srun <theTool> <neededArguments>
#---------------
#Successfully finished
echo "Done"
exit 0 |
|
Script for storing of results into Acacia
Storing can be performed with a slurm job script that makes use of the data-transfer nodes (copy partition) to handle new results/data in the working directory on /scratch and store them in Acacia. Note that data is first packed into a .tar file, and transferred to Acacia afterwards.
| Ui tabs |
|---|
| Ui tab |
|---|
|
900px
bashEmacsListing 3.rclone storeIntoAcaciaTar.shtrue
|
| Ui tab |
|---|
|
900px
bashEmacsListing 3.mc storeIntoAcaciaTar.shtrue
|
|
Coordinating the different steps (scripts) with job dependencies
To coordinate the stage, supercomputing, and store steps, the job scripts are submitted to the scheduler with job dependencies. This is easier to manage with a master script that submits the jobs:
...
| Note |
|---|
|
Note that the final command in the script is the use of the squeue command to display the information of your jobs in the queue and their dependencies. |
Example workflow on other systems
In principle the same scripts can be used in other systems, but they will need to be adapted accordingly. The main adaptation would be the impossiblity of coordinating the different steps through dependencies when scrips are to be executed on different clusters; for example, using the data mover nodes on zeus (copyq) and the compute nodes on magnus (workq) for the supercomputing job.
Related pages