Scripting with pshell

Overview

If you have a supercomputing allocation, then your workflow may involve some of these steps.

The delegate command is intended for performing automated authentication in the context of a (non-interactive) submitted job.

Pshell can be then run from the command line to perform stand alone commands.

Commands with spaces need to be encapsulated with double quotes.

Pshell can also be given a script containing any number of commands that will be executed sequentially.

iblis:~/dev/mfclient> ./pshell.py -h
usage: pshell.py [-h] [-c CONFIG] [-i SCRIPT] [-d] [command]

pshell help

positional arguments:
  command     a single command to execute

optional arguments:
  -h, --help  show this help message and exit
  -c CONFIG   the server in $HOME/.mf_config to connect to
  -i SCRIPT   input text file containing commands
  -d          turns debugging on

Exercises

Exercise 1 - create a delegate and check that pshell is using it.

You will only be able to do this exercise if you've logged in with a Pawsey user account. Delegate identities are not allowed to create further delegates.

Perform the following:

run pshell and log in using your Pawsey credentials
examine your identity
create a delegate
exit and restart pshell
check your identity

Solution to exercise 1

iblis:~/dev/mfclient> pshell
Reading config [/Users/sean/.mf_config]
 === pshell: type 'help' for a list of commands ===
pawsey:offline>login
Username: sean
Password: 

pawsey:/projects>whoami
actor = ivec:sean
  role = ivec
  role = user
  role = Data Team:administer


pawsey:/projects>delegate
Delegating until: 30-May-2017 16:31:10

pawsey:/projects>exit
iblis:~/dev/mfclient> pshell
Reading config [/Users/sean/.mf_config]
 === pshell: type 'help' for a list of commands ===

pawsey:/projects>whoami
actor = delegate
  role = ivec:sean
  role = ivec

Exercise 2 - running commands from the shell

Run pshell from the command line to:

check your identity,
list the contents of the Data Team/testfiles remote folder,
download a single file.

Solution to exercise 2

iblis:~> pshell whoami
Reading config [/Users/sean/.mf_config]
actor = delegate
  role = Data Team:readonly
  role = Demo:readwrite
  role = ivec


iblis:~> pshell "ls /projects/Data Team/testfiles"
Reading config [/Users/sean/.mf_config]
31 items, 29 items per page, remote folder: /projects/Data Team/testfiles
 69776098   | online  |     17 KB | i000769.jpg
 69776099   | online  |  17.02 KB | i000765.jpg
 69776100   | online  |  17.69 KB | i000771.jpg
 69776101   | online  |  17.21 KB | i000767.jpg
etc


iblis:~> pshell "get /projects/Data Team/testfiles/i000769.jpg"
Reading config [/Users/sean/.mf_config]
Total files=1, transferring ...  
Progress=100%, rate=0.0 MB/s  Completed.

iblis:~> file Data\ Team/testfiles/i000769.jpg 
Data Team/testfiles/i000769.jpg: JPEG image data

Exercise 3 - script a task to download a file in your directory.

Perform the following:

create a text file to change into the /projects/Data Team/testfiles directory and download a file
use this script as input to pshell

Solution to exercise 3

Create a plain text file script1.txt containing:

cd Data Team/testfiles
get i000769.jpg

Then do:

iblis:~> pshell -i script1.txt
Reading config [/Users/sean/.mf_config]
input> cd Data Team/sean

Remote: /projects/Data Team
input> get i000769.jpg

Total files=1, transferring ...  
Progress=100%, rate=0.0 MB/s  Completed.

iblis:~> file i000769.jpg
i000769.jpg: JPEG image data

Exercise 4 - script a task that attempts to perform something that will fail (eg remove a directory that doesn't exist) and correctly report that the script encountered an error.

Perform the following:

write a shell (eg bash) script to run a single pshell command to remove the remote folder /idontexist
based on the exit code being 0 if successful and non 0 on failure, report that there was a problem

Solution to exercise 4

Run the following script if you have bash ... sorry Windows users (without WSL.)

#!/bin/bash

pshell "rmdir /idontexist"

if [ $? == 0 ]; then
   echo "All good - proceed further!"
else
   echo "Operation failed!"
fi

Which produces:

Error from server: The namespace '/idontexist' does not exist or is not accessible
Operation failed!

This would typically be a job script on (eg) Pawsey HPC - where you would be doing all such work in a Linux environment and probably using the bash shell.

The idea is that if something in your data setup pipeline fails - abort straight away and don't waste any valuable cpu time on it.