pshell scripting and HPC

Navigation:

Scripting in pshell

Once approach to executing a series of tasks is to create a file containing all the commands to be executed, for example, "script1" which consists of:

remote private
cd sfx-test2
ls

This file can then be executed by pshell as input commands by running:

./pshell -i script1

pawsey:/> remote private
private:/> cd sfx-test2
private:/sfx-test2/> ls
 30.95 KB | fuse.py
  9.69 KB | keystone.py

Multiple commands on a line in pshell

You can concatenate your commands and separate them with double ampersands '&&'

For example, the scripting example could also be achieved as follows:

./pshell "remote private && cd sfx-test2 && ls"

pawsey:/> remote private
private:/> cd sfx-test2
private:/sfx-test2/> ls
 30.95 KB | fuse.py
  9.69 KB | keystone.py

Authentication and Delegation

If you are doing a lot of scripting connecting to a Mediaflux remote, it is undesirable to populate lots of plain text files with your login details. This is where delegation becomes useful. Delegate credentials are binary tokens that reside in the home directory of the user that creates them. These credentials automatically perform a login on your behalf when interacting with Pawsey data storage. Here's an example of creating a delegate credential that lasts for approximately one month:

pawsey:/projects>delegate 30
Delegating until: 01-Jun-2016 14:13:15
 
pawsey:/projects>exit

From now on, until the delegate expires, there is no need to login to data storage on the machine where this was run. Any pshell sessions that are run will automatically use the delegate to login.

Finally, to destroy all delegate credentials, use:

pawsey:/projects>delegate off
 Delegate credentials removed.

Renewing the delegate token before it expires

You cannot create a new token while you are using an old token. So if you want to renew a token before it expires, you'll first need to delete the old token (use the "delegate off" command) and then log into the system using your username and password (use the "login" command). Once logged in like this, you can create a new delegate token as above.

Scripting on Pawsey Supercomputers

Here is an example slurm script for submitting a data transfer job that copies the contents of the /scratch/mystuff folder to a remote 'xyz' at the /mydir/somepath location:

setonix.pawsey.org.au

#!/bin/bash --login
#SBATCH --account=<my account>
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=4
#SBATCH --time=00:10:00
#SBATCH --partition=copy
#SBATCH --export=NONE

/home/sean/bin/pshell "remote xyz && cd /mydir/somepath && put /scratch/mystuff"