How to Run Workflows on the Workflow Nodes
The Setonix Workflow Nodes serve as specialised login nodes dedicated to launching long-running workflow manager master tasks, such as those generated by Nextflow and Snakemake. These nodes are exclusively designated for executing master tasks that submit jobs to the SLURM queue. To prevent inadvertent execution of computationally intensive tasks, there are restrictions on the memory and number of cores allocated to each user. This resource is tailored to support workflows that surpass the 96-hour wall-time constraint on the long partition. Users can utilise Screen
or Tmux
on these nodes to maintain an interactive session while their master task is in progress.
There are two workflow nodes, setonix-07 and setonix-08. When running a screen/tmux session that you intend to revisit, make sure to remember the specific node you are working on. This way, you can easily resume your tasks later on.
Instructions
Logging in and using the workflow nodes:
SSH into the Workflow Nodes.
If you intend to launch a new task and are open to using either node, execute
ssh username@setonix-workflow.pawsey.org.au
For returning to a previous session, use
ssh username@setonix-XX.pawsey.org.au
, replacing XX with the node number from your prior session.
Initiating a session: Begin a persistent session by employing tools like
screen
ortmux
. This feature enables you to disconnect and reconnect without disrupting your ongoing tasks. Note the use of the-S/-s
flags to name the session “my_workflow_session”. You have the flexibility to assign any name to your session. Screen and tmux are both available as modules.
Usingscreen
:module load screen/<version>