This page gives an example in how to handle a data workflow using Acacia and Setonix, with three different jobs connected by dependencies.
Example workflow on Setonix
The data workflow will consist of:
- A first job submitted to the data transfer nodes (
copypartition) that will stage initial data from Acacia and into the working directory on/scratch - A second job submitted to the compute nodes (
workpartition) that will execute a supercomputing job making use of the staged data. We assume that this job will produce new result/data in the same working directory on/scratch. - A third job submitted to the data transfer nodes (
copypartition) that will handle the new results/data and store them into Acacia
Script for staging of initial data into /scratch
Staging can be performed with a slurm job script that makes use of the data-transfer nodes (copy partition) to handle initial data originally stored in Acacia and to be staged into the working directory on /scratch. Note that the original data is packed within a .tar file, so the script also performs the "untarring" of the data:
Client support
- Note the use of variables to store the names of directories, files, buckets, prefixes, objects etc.
- Also note the several checks at the different parts of the script and the redirection to
/dev/nullin most of the commands used for checking correctness (as we are not interested in their output). - As messages from the mc client are too verbose when transferring files (even with the
--quietoption), we make use of a redirection of the output messages to/dev/nullwhen performing transfers with this client on scripts. For this reason we often find rclone a better choice for the use of clients in scripts.
Script for performing a supercomputing job
The slurm job script for performing the supercomputing job makes use of the initial data staged in the previous step. This job requests execution in the compute nodes (work partition).
#!/bin/bash -l #SBATCH --account=[yourProject] #SBATCH --job-name=superExecution #SBATCH --partition=work #SBATCH --ntasks=[numberOfCoresToUse] #SBATCH --time=[requiredTime] #SBATCH --export=none #-------------- #Load required modules here ... #--------------- #Defining the working dir workingDir="$MYSCRATCH/<pathAndNameOfWorkingDirectory" #--------------- #Entering the working dir cd $workingDir #--------------- #Check for the correct staging of the initial conditions if needed ... #--------------- #Supercomputing execution srun <theTool> <neededArguments> #--------------- #Successfully finished echo "Done" exit 0
Script for storing of results into Acacia
Storing can be performed with a slurm job script that makes use of the data-transfer nodes (copy partition) to handle new results/data in the working directory on /scratch and store them in Acacia. Note that data is first packed into a .tar file, and transferred to Acacia afterwards.
Coordinating the different steps (scripts) with job dependencies
To coordinate the stage, supercomputing, and store steps, the job scripts are submitted to the scheduler with job dependencies. This is easier to manage with a master script that submits the jobs:
#!/bin/bash
scriptsDir=<pathToWhereTheScriptsAre>
echo "Using dependencies to connect execution of several jobs on different partitions"
#----------------------
echo "Submitting the staging job and saving the jobID"
stageJobID=$(sbatch --parsable \
$scriptsDir/stageFromAcaciaTar.sh \
| cut -d ";" -f 1)
echo "stageJobID=$stageJobID"
#----------------------
echo "Submitting the supercomputing job and saving the jobID"
superJobID=$(sbatch --parsable \
--dependency=afterok:$stageJobID \
$scriptsDir/superExecutionSetonix.sh \
| cut -d ";" -f 1)
echo "superJobID=$superJobID"
#----------------------
echo "Submitting the storing job and saving the jobID"
storeJobID=$(sbatch --parsable \
--dependency=afterok:$superJobID \
| cut -d ";" -f 1)
echo "storeJobID=$storeJobID"
#----------------------
echo "Displaying the current status of the jobs"
squeue --jobs="$stageJobID,$superJobID,$storeJobID" \
--Format "JobID:8 ,UserName:10 ,Account:11 ,Partition:10 ,Name:14 ,State:8 ,Reason:12 ,Dependency:30"
Client support
- Note that this is not a slurm job script, but a simple bash script. Then you should not submit it to the scheduler but simply execute it in the command line, then the script is the one which submits the scripts for the different steps to the scheduler.
- Note that the scrip makes use of the cut command to grab the fourth field of the output of the sbatch command (which is the jobID) and then assign it to a variable. In the following submission, the variable with the corresponding jobID is used in the
--dependencyoption. In this case, the dependent jobs will only start execution if the "parent" job ends with success.
When executing the script, you can save the output to a log file for your records pipelining the output with the tee command:
$ ./masterWorkflow.sh | tee log.master.out Using dependencies to connect execution of several jobs on different partitions Submitting the staging job and saving the jobID stageJobID=5534757 Submitting the supercomputing job and saving the jobID superJobID=5534758 Submitting the storing job and saving the jobID storeJobID=5534759 Displaying the current status o the jobs JOBID USER ACCOUNT PARTITION NAME STATE REASON DEPENDENCY 5534759 username pawsey copy storeTar PENDING Dependency afterok:5534758(unfulfilled) 5534758 username pawsey work superExecution PENDING Dependency afterok:5534757(unfulfilled) 5534757 username pawsey copy stageTar PENDING Priority (null)
Client support
Note that the final command in the script is the use of the squeue command to display the information of your jobs in the queue and their dependencies.
Example workflow on other systems
In principle the same scripts can be used in other systems, but they will need to be adapted accordingly. The main adaptation would be the impossiblity of coordinating the different steps through dependencies when scrips are to be executed on different clusters; for example, using the data mover nodes on zeus (copyq) and the compute nodes on magnus (workq) for the supercomputing job.