pshell scripting and HPC

Getting pshell

If you do not already have access to pshell, you can use the information from this page: PSHELL Quickstart, eg from setonix:

wget "https://bitbucket.org/datapawsey/mfclient/downloads/pshell"
chmod u+x pshell
./pshell

Scripting in pshell

Once approach to executing a series of tasks is to create a file containing all the commands to be executed, for example, "script1" which consists of:

remote private 
cd sfx-test2
ls

This file can then be executed by pshell as input commands by running:

./pshell -i script1

pawsey:/> remote private
private:/> cd sfx-test2
private:/sfx-test2/> ls
 30.95 KB | fuse.py
  9.69 KB | keystone.py

Multiple commands on a line in pshell

You can concatenate your commands and separate them with double ampersands '&&'
For example, the scripting example could also be achieved as follows:

Authentication and Delegation

If you are scripting workflows, it is undesirable to populate plain text files with your login details. This is where delegation becomes useful. Delegate credentials are binary tokens that reside in the home directory of the user that creates them. These credentials automatically perform a login on your behalf when interacting with Pawsey data storage. Here's an example of creating a delegate credential that lasts for approximately one month:

From now on, until the delegate expires, there is no need to login to data storage on the machine where this was run. Any pshell sessions that are run will automatically use the delegate to login.
Finally, to destroy all delegate credentials, use:

Renewing the delegate token before it expires

You cannot create a new token while you are using an old token. So if you want to renew a token before it expires, you'll first need to delete the old token (use the "delegate off" command) and then log into the system using your username and password (use the "login" command). Once logged in like this, you can create a new delegate token as above.

Scripting on Pawsey Supercomputers

Here is an example slurm script for submitting a data transfer job that copies the contents of the /scratch/mystuff folder to a remote ‘portal' (by default this will be Pawsey’s cool-tier banksia storage) at the /mydir/somepath location:

setonix.pawsey.org.au

User Support Documentation