Supercomputing workflow example
This page gives an example in how to handle a data workflow using Acacia and Setonix, with three different jobs connected by dependencies.
Example workflow on Setonix
The data workflow will consist of:
- A first job submitted to the data transfer nodes (
copypartition) that will stage initial data from Acacia and into the working directory on/scratch - A second job submitted to the compute nodes (
workpartition) that will execute a supercomputing job making use of the staged data. We assume that this job will produce new result/data in the same working directory on/scratch. - A third job submitted to the data transfer nodes (
copypartition) that will handle the new results/data and store them into Acacia
Script for staging of initial data into /scratch
Staging can be performed with a slurm job script that makes use of the data-transfer nodes (copy partition) to handle initial data originally stored in Acacia and to be staged into the working directory on /scratch. Note that the original data is packed within a .tar file, so the script also performs the "untarring" of the data:
Listing 1.rclone stageFromAcaciaTar.sh
#!/bin/bash --login
#---------------
#About this script
#stageFromAcaciaTar.sh : copies a tar object from Acacia and extracts it in the destination path
#---------------
#Requested resources:
#SBATCH --account=[yourProjectName]
#SBATCH --job-name=stageTar.rclone
#SBATCH --partition=copy
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --time=[requiredTime]
#SBATCH --export=NONE
#-----------------
#Loading the required modules
module load rclone/<version> #This example is performed with the use of the rclone.
#-----------------
#Defining variables that will handle the names related to your access, buckets and stored objects in Acacia
profileName=<profileNameGivenToYourProfileOfAccessToAcacia>
bucketName=<bucketInAcaciaContainingTheData>
prefixPath=<prefixPathInBucketUsedToOrginiseTheData>
fullPathInAcacia="${profileName}:${bucketName}/${prefixPath}" #Note the colon(:) when using rclone
#-----------------
#Name of the file to be transferred and auxiliary dir to temporarily place it
tarFileName=<nameOfTheTarFileContainingInitialData>
auxiliaryDirForTars="$MYSCRATCH/tars"
echo "Checking that the auxiliary directory exists"
if ! [ -d $auxiliaryDirForTars ]; then
echo "Trying to create the auxiliary directory as it does not exist"
mkdir -p $auxiliaryDirForTars; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "The auxiliary directory $auxiliaryDirForTars does not exist and can't be created"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
fi
#-----------------
#Working directory in scratch for the supercomputing job
workingDir="$MYSCRATCH/<workingDirectoryForSupercomputingJob>"
echo "Checking that the working directory exists"
if ! [ -d $workingDir ]; then
echo "Trying to create the working directory as it does not exist"
mkdir -p $workingDir; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "The working directory $workingDir does not exist and can't be created"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
fi
#-----------------
#Check if Acacia definitions make sense, and if the object to transfer exist
echo "Checking that the profile exists" rclone config show | grep "${profileName}" > /dev/null; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "The given profileName=$profileName seems not to exist in the user configuration of rclone"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
echo "Checking that bucket exists and that you have read access"
rclone lsd "${profileName}:${bucketName}" > /dev/null; exitcode=$? #Note the colon(:) when using rclone
if [ $exitcode -ne 0 ]; then
echo "The bucket name or the profile name may be wrong: ${profileName}:${bucketName}"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
echo "Checking if the file can be listed in Acacia"
listResult=$(rclone lsl "${fullPathInAcacia}/${tarFileName}")
if [ -z "$listResult" ]; then
echo "Problems occurred during the listing of the file ${tarFileName}"
echo "Check that the file exists in the fullPathInAcacia: ${fullPathInAcacia}/"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
#-----------------
#Perform the transfer of the tar file into the auxiliary directory and check for the transfer
echo "Performing the transfer ... "
srun rclone copy "${fullPathInAcacia}/${tarFileName}" "${auxiliaryDirForTars}/"; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "Problems occurred during the transfer of file ${tarFileName}"
echo "Check that the file exists in the fullPathInAcacia: ${fullPathInAcacia}/"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
#-----------------
#Perform untaring with desired options into the working directory
echo "Performing the untarring ... "
#tarOptions=( --strip-components 8 ) #Avoiding creation of some directories in the path
srun tar -xvzf "${auxiliaryDirForTars}/${tarFileName}" -C $workingDir "${tarOptions[@]}"; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "Problems occurred during the untaring of file ${auxiliaryDirForTars}/${tarFileName}"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
else
echo "Removing the tar file as it has been successfully untarred"
rm $auxiliaryDirForTars/$tarFileName #comment this line when debugging workflow
fi
#-----------------
## Final checks ....
#---------------
#Successfully finished
echo "Done"
exit 0
Client support
- Note the use of variables to store the names of directories, files, buckets, prefixes, objects etc.
- Also note the several checks at the different parts of the script and the redirection to
/dev/nullin most of the commands used for checking correctness (as we are not interested in their output). - As messages from the mc client are too verbose when transferring files (even with the
--quietoption), we make use of a redirection of the output messages to/dev/nullwhen performing transfers with this client on scripts. For this reason we often find rclone a better choice for the use of clients in scripts.
Script for performing a supercomputing job
The slurm job script for performing the supercomputing job makes use of the initial data staged in the previous step. This job requests execution in the compute nodes (work partition).
Listing 2. superExecutionSetonix.sh
#!/bin/bash -l #SBATCH --account=[yourProject] #SBATCH --job-name=superExecution #SBATCH --partition=work #SBATCH --ntasks=[numberOfCoresToUse] #SBATCH --time=[requiredTime] #SBATCH --export=none #-------------- #Load required modules here ... #--------------- #Defining the working dir workingDir="$MYSCRATCH/<pathAndNameOfWorkingDirectory" #--------------- #Entering the working dir cd $workingDir #--------------- #Check for the correct staging of the initial conditions if needed ... #--------------- #Supercomputing execution srun <theTool> <neededArguments> #--------------- #Successfully finished echo "Done" exit 0
Script for storing of results into Acacia
Storing can be performed with a slurm job script that makes use of the data-transfer nodes (copy partition) to handle new results/data in the working directory on /scratch and store them in Acacia. Note that data is first packed into a .tar file, and transferred to Acacia afterwards.
Listing 3.rclone storeIntoAcaciaTar.sh
#!/bin/bash --login
#SBATCH --account=[yourProjectName]
#SBATCH --job-name=storeTar
#SBATCH --partition=copy
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --time=[requiredTime]
#SBATCH --export=NONE
#-----------------
#Loading the required modules
module load rclone/<version> #This example is performed with the use of the rclone.
#-----------------
#Defining variables that will hold the names related to your access, buckets and objects to be stored in Acacia
profileName=<profileNameGivenToYourProfileOfAccessToAcacia>
bucketName=<bucketInAcaciaContainingTheData>
prefixPath=<prefixPathInBucketUsedToOrginiseTheData>
fullPathInAcacia="${profileName}:${bucketName}/${prefixPath}" #Note the colon(:) when using rclone
#-----------------
#Check if Acacia definitions make sense, and if you can transfer objects into the desired bucket
echo "Checking that the profile exists"
rclone config show | grep "${profileName}" > /dev/null; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "The given profileName=$profileName seems not to exist in the user configuration of rclone"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
echo "Checking the bucket exists and that you have writing access"
rclone lsd "${profileName}:${bucketName}" > /dev/null; exitcode=$? #Note the colon(:) when using rclone
if [ $exitcode -ne 0 ]; then
echo "The bucket intended to receive the data does not exist: ${profileName}:${bucketName}"
echo "Trying to create it"
rclone mkdir "${profileName}:${bucketName}"; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "Creation of bucket failed"
echo "The bucket name or the profile name may be wrong: ${profileName}:${bucketName}"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
fi
echo "Checking if a test file can be trasferred into the desired full path in Acacia"
testFile=test_file_${SLURM_JOBID}.txt
echo "File for test" > "${testFile}"
rclone copy "${testFile}" "${fullPathInAcacia}/"; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "The test file $testFile cannot be transferred into ${fullPathInAcacia}"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
echo "Checking if the test file can be listed in Acacia"
listResult=$(rclone lsl "${fullPathInAcacia}/${testFile}")
if [ -z "$listResult" ]; then
echo "Problems occurred during the listing of the test file ${testFile} in ${fullPathInAcacia}"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
echo "Removing test file from Acacia"
rclone delete "${fullPathInAcacia}/${testFile}"; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "The test file $testFile cannot be removed from ${fullPathInAcacia}"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
rm $testFile
#----------------
#Defining the working dir and cd into it
workingDir="$MYSCRATCH/<workingDirectoryOfSupercomputingJob>"
echo "Checking that the working directory exists"
if ! [ -d $workingDir ]; then
echo "The working directory $workingDir does not exist"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
else
cd $workingDir
fi
#----------------
#Defining what to tar and the name and place of the tarfile
whatToTar=( file1 dir1/ dir2/file2 dir3/file* ) #These are inside the working dir
tarFileName=case_A1-newData.tar.gz
auxiliaryDirForTars="$MYSCRATCH/tars"
echo "Checking that the auxiliary directory exists"
if ! [ -d $auxiliaryDirForTars ]; then
echo "Trying to create the auxiliary directory as it does not exist"
mkdir -p $auxiliaryDirForTars; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "The auxiliary directory $auxiliaryDirForTars does not exist and can't be created"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
fi
#----------------
#Taring the indicated files and directories
srun tar -cvzf "${auxiliaryDirForTars}/${tarFileName}" "${whatToTar[@]}"; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "Something went wrong when taring:"
echo "tarFileName=${tarFileName}"
echo "whatToTar=( ${whatToTar[@]} )"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
fi
#-----------------
#Perform the transfer of the tar file into the working directory and check for the transfer
echo "Performing the transfer ... "
srun rclone copy "${auxiliaryDirForTars}/${tarFileName}" "${fullPathInAcacia}/"; exitcode=$?
if [ $exitcode -ne 0 ]; then
echo "Problems occurred during the transfer of file ${tarFileName}"
echo "Check that the file exists in ${workingDir}"
echo "And that nothing is wrong with the fullPathInAcacia: ${fullPathInAcacia}/"
echo "Exiting the script with non-zero code in order to inform job dependencies not to continue."
exit 1
else
echo "Removing the tar file from scratch as it was successfully transferred into Acacia"
echo "Final place in Acacia: ${fullPathInAcacia}/${tarFileName}"
rm "${auxiliaryDirForTars}/${tarFileName}" #Comment this line when debugging workflow
fi
#---------------
# Final checks ...
#---------------
#Successfully finished
echo "Done"
exit 0
Coordinating the different steps (scripts) with job dependencies
To coordinate the stage, supercomputing, and store steps, the job scripts are submitted to the scheduler with job dependencies. This is easier to manage with a master script that submits the jobs:
Listing 4. masterWorkflow.sh
#!/bin/bash
scriptsDir=<pathToWhereTheScriptsAre>
echo "Using dependencies to connect execution of several jobs on different partitions"
#----------------------
echo "Submitting the staging job and saving the jobID"
stageJobID=$(sbatch --parsable \
$scriptsDir/stageFromAcaciaTar.sh \
| cut -d ";" -f 1)
echo "stageJobID=$stageJobID"
#----------------------
echo "Submitting the supercomputing job and saving the jobID"
superJobID=$(sbatch --parsable \
--dependency=afterok:$stageJobID \
$scriptsDir/superExecutionSetonix.sh \
| cut -d ";" -f 1)
echo "superJobID=$superJobID"
#----------------------
echo "Submitting the storing job and saving the jobID"
storeJobID=$(sbatch --parsable \
--dependency=afterok:$superJobID \
| cut -d ";" -f 1)
echo "storeJobID=$storeJobID"
#----------------------
echo "Displaying the current status of the jobs"
squeue --jobs="$stageJobID,$superJobID,$storeJobID" \
--Format "JobID:8 ,UserName:10 ,Account:11 ,Partition:10 ,Name:14 ,State:8 ,Reason:12 ,Dependency:30"
Client support
- Note that this is not a slurm job script, but a simple bash script. Then you should not submit it to the scheduler but simply execute it in the command line, then the script is the one which submits the scripts for the different steps to the scheduler.
- Note that the scrip makes use of the cut command to grab the fourth field of the output of the sbatch command (which is the jobID) and then assign it to a variable. In the following submission, the variable with the corresponding jobID is used in the
--dependencyoption. In this case, the dependent jobs will only start execution if the "parent" job ends with success.
When executing the script, you can save the output to a log file for your records pipelining the output with the tee command:
Terminal 1. Submit multiple dependent jobs
$ ./masterWorkflow.sh | tee log.master.out Using dependencies to connect execution of several jobs on different partitions Submitting the staging job and saving the jobID stageJobID=5534757 Submitting the supercomputing job and saving the jobID superJobID=5534758 Submitting the storing job and saving the jobID storeJobID=5534759 Displaying the current status o the jobs JOBID USER ACCOUNT PARTITION NAME STATE REASON DEPENDENCY 5534759 username pawsey copy storeTar PENDING Dependency afterok:5534758(unfulfilled) 5534758 username pawsey work superExecution PENDING Dependency afterok:5534757(unfulfilled) 5534757 username pawsey copy stageTar PENDING Priority (null)
Client support
Note that the final command in the script is the use of the squeue command to display the information of your jobs in the queue and their dependencies.
Example workflow on other systems
In principle the same scripts can be used in other systems, but they will need to be adapted accordingly. The main adaptation would be the impossiblity of coordinating the different steps through dependencies when scrips are to be executed on different clusters; for example, using the data mover nodes on zeus (copyq) and the compute nodes on magnus (workq) for the supercomputing job.