Overview
If you have a supercomputing allocation, then your workflow will may involve some of these steps.
The first is how do you perform There are two ways of executing pshell commands non-interactively:
- a single command to run
- a series of sequential commands supplied via an input script
| Code Block |
|---|
iblis:~/dev/mfclient> python pshell -h
usage: pshell [-h] [-c CONFIG] [-i SCRIPT] [-d] [command]
pshell help
positional arguments:
command a single command to execute
optional arguments:
-h, --help show this help message and exit
-c CONFIG the server in $HOME/.mf_config to connect to
-i SCRIPT input text file containing commands
-d turns debugging on |
The delegate command is intended for performing automated authentication in the context of a (non-interactive) submitted job - use the delegate command.
This means that pshell commands can be then run from without the command line to perform stand alone commands.It can also be given a script containing any number of commands that will be executed sequentially.need to type in username and password, within the lifetime of the delegate (default 7 days.)
Exercises
Exercise 1 - create a delegate and check that pshell is using it.
| Warning |
|---|
This exercise can only be done when logged in with a Pawsey username and password. Delegate identities are not allowed to create further delegates. |
Perform the following:
- run pshell and log in using your Pawsey credentials
- examine your identity
- create a delegate
- exit and restart pshell
- check your identity
...
| Expand |
|---|
| title | Solution to exercise 1 |
|---|
|
| Code Block |
|---|
stuffiblis:~/dev/mfclient> python pshell
Reading config [/Users/sean/.mf_config]
=== pshell: type 'help' for a list of commands ===
pawsey:offline>login
Username: sean
Password:
pawsey:/projects>whoami
actor = ivec:sean
role = ivec
role = user
role = Data Team:administer
pawsey:/projects>delegate
Delegating until: 30-May-2017 16:31:10
pawsey:/projects>exit
iblis:~/dev/mfclient> pshell
Reading config [/Users/sean/.mf_config]
=== pshell: type 'help' for a list of commands ===
pawsey:/projects>whoami
actor = delegate
role = ivec:sean
role = ivec
|
|
Exercise 2 - running commands from the shell
Perform the followingRun pshell from the command line to:run pshell from your command prompt
- check your identity,
- list the contents of the /projects/Data Team/testfiles remote folder,
- download a single file.
| Expand |
|---|
| title | Solution to exercise 2 |
|---|
|
| Code Block |
|---|
iblis:~> python pshell whoami
Reading config [/Users/sean/.mf_config]
actor = delegate
role = Data Team:readonly
role = Demo:readwrite
role = ivec
iblis:~> python pshell "ls /projects/Data Team/testfiles"
Reading config [/Users/sean/.mf_config]
31 items, 29 items per page, remote folder: /projects/Data Team/testfiles
69776098 | online | 17 KB | i000769.jpg
69776099 | online | 17.02 KB | i000765.jpg
69776100 | online | 17.69 KB | i000771.jpg
69776101 | online | 17.21 KB | i000767.jpg
etc
iblis:~> python pshell "get /projects/Data Team/testfiles/i000769.jpg"
Reading config [/Users/sean/.mf_config]
Total files=1, transferring ...
Progress=100%, rate=0.0 MB/s Completed.
|
|
Exercise 3 - script a task to download a file in your directory.
Perform the following:
- create a text file to change into the Demo/<yourname> /projects/Data Team/testfiles directory and download a file
- use this script as input to pshell
| Expand |
|---|
| title | Solution to exercise 3 |
|---|
|
Create a plain text file script1.txt containing: | Code Block |
|---|
cd Data DemoTeam/seantestfiles
get testfile1i000769.jpg |
Then do: | Code Block |
|---|
iblis:~> python pshell -i script1.txt
Reading config [/Users/sean/.mf_config]
input> cd Data DemoTeam/sean
Remote: /projects/Demo/seanData Team
input> get IMG_0009i000769.jpg
Total files=1, transferring ...
Progress=100%, rate=0.0 MB/s Completed.
iblis:~> file IMG_0009.jpg
IMG_0009.jpg: JPEG image data |
|
Perform the following:
- write a shell (eg bash) script to run a single pshell command that will failto remove the remote folder /idontexist
- based on the exit code being 0 if successfull successful and non 0 on failure, report that there was a problem
| Expand |
|---|
| title | Solution to exercise 4 |
|---|
|
Run the following script if you have bash ... sorry Windows users (without WSL.) | Code Block |
|---|
#!/bin/bash
python pshell "rmdir /idontexist"
if [ $? == 0 ]; then
echo "All good - proceed further!"
else
echo "Operation failed!"
fi |
Which produces: | Code Block |
|---|
Error from server: The namespace '/idontexist' does not exist or is not accessible
Operation failed! |
This would typically be a job script on (eg) Pawsey HPC - where you would be doing all such work in a Linux environment and probably using the bash shell. The idea is that if something in your data setup pipeline fails - abort straight away and don't waste any valuable cpu time on it. |