Scripting with pshell

Overview

If you have a supercomputing allocation, then your workflow may involve some of these steps.

There are two ways of executing pshell commands non-interactively:

a single command to run
a series of sequential commands supplied via an input script

iblis:~/dev/mfclient> python pshell -h
usage: pshell [-h] [-c CONFIG] [-i SCRIPT] [-d] [command]

pshell help

positional arguments:
  command     a single command to execute

optional arguments:
  -h, --help  show this help message and exit
  -c CONFIG   the server in $HOME/.mf_config to connect to
  -i SCRIPT   input text file containing commands
  -d          turns debugging on

The delegate command is intended for performing automated authentication in the context of a (non-interactive) submitted job.

This means that pshell commands can be run without the need to type in username and password, within the lifetime of the delegate (default 7 days.)

Exercises

Exercise 1 - create a delegate and check that pshell is using it.

This exercise can only be done when logged in with a Pawsey username and password. Delegate identities are not allowed to create further delegates.

Perform the following:

run pshell and log in using your Pawsey credentials
examine your identity
create a delegate
exit and restart pshell
check your identity

iblis:~/dev/mfclient> python pshell Reading config [/Users/sean/.mf_config] === pshell: type 'help' for a list of commands === pawsey:offline>login Username: sean Password: pawsey:/projects>whoami actor = ivec:sean role = ivec role = user role = Data Team:administer pawsey:/projects>delegate Delegating until: 30-May-2017 16:31:10 pawsey:/projects>exit iblis:~/dev/mfclient> pshell Reading config [/Users/sean/.mf_config] === pshell: type 'help' for a list of commands === pawsey:/projects>whoami actor = delegate role = ivec:sean role = ivec

Exercise 2 - running commands from the shell

Run pshell from the command line to:

check your identity,
list the contents of the /projects/Data Team/testfiles remote folder,
download a single file.

iblis:~> python pshell whoami Reading config [/Users/sean/.mf_config] actor = delegate role = Data Team:readonly role = Demo:readwrite role = ivec iblis:~> python pshell "ls /projects/Data Team/testfiles" Reading config [/Users/sean/.mf_config] 31 items, 29 items per page, remote folder: /projects/Data Team/testfiles 69776098 | online | 17 KB | i000769.jpg 69776099 | online | 17.02 KB | i000765.jpg 69776100 | online | 17.69 KB | i000771.jpg 69776101 | online | 17.21 KB | i000767.jpg etc iblis:~> python pshell "get /projects/Data Team/testfiles/i000769.jpg" Reading config [/Users/sean/.mf_config] Total files=1, transferring ... Progress=100%, rate=0.0 MB/s Completed.

Exercise 3 - script a task to download a file in your directory.

Perform the following:

create a text file to change into the /projects/Data Team/testfiles directory and download a file
use this script as input to pshell

Create a plain text file script1.txt containing: cd Data Team/testfiles get i000769.jpg Then do: iblis:~> python pshell -i script1.txt Reading config [/Users/sean/.mf_config] input> cd Data Team/sean Remote: /projects/Data Team input> get i000769.jpg Total files=1, transferring ... Progress=100%, rate=0.0 MB/s Completed.

Exercise 4 - script a task that attempts to perform something that will fail (eg remove a directory that doesn't exist) and correctly report that the script encountered an error.

Perform the following:

write a shell (eg bash) script to run a single pshell command to remove the remote folder /idontexist
based on the exit code being 0 if successful and non 0 on failure, report that there was a problem

Run the following script if you have bash ... sorry Windows users (without WSL.) #!/bin/bash python pshell "rmdir /idontexist" if [ $? == 0 ]; then echo "All good - proceed further!" else echo "Operation failed!" fi Which produces: Error from server: The namespace '/idontexist' does not exist or is not accessible Operation failed! This would typically be a job script on (eg) Pawsey HPC - where you would be doing all such work in a Linux environment and probably using the bash shell. The idea is that if something in your data setup pipeline fails - abort straight away and don't waste any valuable cpu time on it.