pshell and scripting

Overview

If you have a supercomputing allocation, then your workflow will involve some of these steps.

The first is how do you perform authentication in the context of a (non-interactive) submitted job - use the delegate command.

pshell can be then run from the command line to perform stand alone commands. Note: commands with spaces need to be encapsulated with double quotes.

It can also be given a script containing any number of commands that will be executed sequentially.

iblis:~> pshell -h
usage: pshell [-h] [-c CONFIG] [-i SCRIPT] [-d] [command]

pshell help

positional arguments:
  command

optional arguments:
  -h, --help  show this help message and exit
  -c CONFIG   The server in $HOME/.mf_config to connect to
  -i SCRIPT   Input script file containing commands
  -d          Turn debugging on

Exercises

Exercise 1 - create a delegate and check that pshell is using it.

Perform the following:

run pshell and log in using your Pawsey credentials
examine your identity
create a delegate
exit and restart pshell
check your identity

You will only be able to do this exercise if you've logged in with a Pawsey user account. Delegate identities are not allowed to create further delegates.

Solution 1

iblis:~/dev/mfclient> pshell
Reading config [/Users/sean/.mf_config]
 === pshell: type 'help' for a list of commands ===
pawsey:offline>login
Username: sean
Password: 

pawsey:/projects>whoami
actor = ivec:sean
  role = ivec
  role = user
  role = Data Team:administer


pawsey:/projects>delegate
Delegating until: 30-May-2017 16:31:10

pawsey:/projects>exit
iblis:~/dev/mfclient> pshell
Reading config [/Users/sean/.mf_config]
 === pshell: type 'help' for a list of commands ===

pawsey:/projects>whoami
actor = delegate
  role = ivec:sean
  role = ivec

Exercise 2 - running commands from the shell

Run pshell from the command line to:

check your identity,
list the contents of the Data Team/testfiles remote folder,
download a single file.

Solution to exercise 2

iblis:~> pshell whoami
Reading config [/Users/sean/.mf_config]
actor = delegate
  role = Data Team:readonly
  role = Demo:readwrite
  role = ivec


iblis:~> pshell "ls /projects/Data Team/testfiles"
Reading config [/Users/sean/.mf_config]
31 items, 29 items per page, remote folder: /projects/Data Team/testfiles
 69776098   | online  |     17 KB | i000769.jpg
 69776099   | online  |  17.02 KB | i000765.jpg
 69776100   | online  |  17.69 KB | i000771.jpg
 69776101   | online  |  17.21 KB | i000767.jpg
etc


iblis:~> pshell "get /projects/Data Team/testfiles/i000769.jpg"
Reading config [/Users/sean/.mf_config]
Total files=1, transferring ...  
Progress=100%, rate=0.0 MB/s  Completed.

iblis:~> file Data\ Team/testfiles/i000769.jpg 
Data Team/testfiles/i000769.jpg: JPEG image data

Exercise 3 - script a task to download a file in your directory.

Perform the following:

create a text file to change into the Demo/<yourname> directory and download a file
use this script as input to pshell

Solution 3

Create a plain text file script1.txt containing:

cd Demo/sean
get testfile1

Then do:

iblis:~> pshell -i script1.txt
Reading config [/Users/sean/.mf_config]
input> cd Demo/sean

Remote: /projects/Demo/sean
input> get IMG_0009.jpg

Total files=1, transferring ...  
Progress=100%, rate=0.0 MB/s  Completed.

iblis:~> file IMG_0009.jpg 
IMG_0009.jpg: JPEG image data

Exercise 4 - script a task that attempts to perform something that will fail (eg remove a directory that doesn't exist) and correctly report that the script encountered an error.

Perform the following:

write a shell (eg bash) script to run a single pshell command that will fail
based on the exit code being 0 if successfull and non 0 on failure, report a problem

Solution 4

Run the following script if you have bash ... sorry Windows users (without WSL.)

#!/bin/bash

pshell "rmdir /idontexist"

if [ $? == 0 ]; then
   echo "All good - proceed further!"
else
   echo "Operation failed!"
fi

Which produces:

Error from server: The namespace '/idontexist' does not exist or is not accessible
Operation failed!

This would typically be a job script on (eg) Pawsey HPC - where you would be doing all such work in a Linux environment.

The idea is that if something in your data setup pipeline fails - don't waste any valuable cpu time on it.