pshell and scripting

Overview

If you have a supercomputing allocation, then your workflow will involve some of these steps.

The first is how do you perform authentication in the context of a (non-interactive) submitted job - use the delegate command.

pshell can be then run from the command line to perform stand alone commands.

It can also be given a script containing any number of commands that will be executed sequentially.

Exercises

Exercise 1 - create a delegate and check that pshell is using it.

You will only be able to do this exercise if you've logged in with a Pawsey user account. Delegate identities are not allowed to create further delegates.

Solution 1

pshell

login

sean

????

delegate

exit

pshell whoami

Exercise - script a task to download a file in your directory.

Solution 2

Create a plain text file script1.txt containing:

cd Demo/sean
get testfile1

Then do:

iblis:~> pshell -i script1.txt
Reading config [/Users/sean/.mf_config]
input> cd Demo/sean

Remote: /projects/Demo/sean
input> get IMG_0009.jpg

Total files=1, transferring ...  
Progress=100%, rate=0.0 MB/s  Completed.

iblis:~> file IMG_0009.jpg 
IMG_0009.jpg: JPEG image data

Exercise - script a task that attempts to perform something that will fail (eg remove a directory that doesn't exist) and correctly report that the script encountered an error.

Solution 3

Run the following script if you have bash ... sorry Windows users (without WSL.)

#!/bin/bash

pshell "rmdir /idontexist"

if [ $? == 0 ]; then
   echo "All good - proceed further!"
else
   echo "Operation failed!"
fi

Which produces:

Error from server: The namespace '/idontexist' does not exist or is not accessible
Operation failed!

This would typically be a job script on (eg) Pawsey HPC - where you would be doing all such work in a Linux environment.

The idea is that if something in your data setup pipeline fails - don't waste any valuable cpu time on it.

Exercise - adapt the previous exercise to perform an upload that will succeed (eg upload a file from your local desktop) and correctly report that the operation was successful.

Solution 4

You will need two scripts here, one to run pshell (which would be submitted as a job) and another to supply multiple commands to pshell.

The second script is needed as, by default, pshell will try to upload into your current working directory (/projects) which you do not have permission to alter.

First script, call it: upload.txt

cd /projects/Demo/sean
put testfile

Then we can run this:

#!/bin/bash

pshell -i upload.txt

if [ $? == 0 ]; then
   echo "All good - proceed further!"
else
   echo "Operation failed!"
fi