From Your Local Computer

Simple transfers with scp

For small file transfers from your local computer, use the secure copy protocol (scp):

1. Open a terminal on your local computer

2. Use the following command
   

   >scp -i /path/to/your/file ~/.ssh/YOUR_KEYPAIR.pem login_name@###.##.##.##:/path/of/your/nimbus/directory

• • Replace /path/to/your/file to the path of the file that you wish to transfer.
  • Replace YOUR_KEYPAIR.pem with the name of your keypair.

  • Replace login_name@###.##.##.## with your username@ip-address.

    Image	login_name
    Ubuntu	ubuntu
    Centos	centos
    Fedora	fedora
    Scientific Linux	root
    Debian	debian

  • Replace /path/of/your/nimbus/directory with the directory on your instance that you wish to transfer the file to.

From IRDS (UWA) over the command line

In many cases, you will be able to transfer from IRDS to your Nimbus instance by using a program such as FileZilla or CyberDuck. However, this will use your local internet bandwidth, which may be an issue if you are not at a University/Institute site and you need to transfer large files. Additionally, if you are on a laptop, moving away from an internet connection may disrupt your upload. An alternative is setting up the transfer directly on Nimbus via 'screen' (https://linuxize.com/post/how-to-use-linux-screen/). Then, you can close your laptop and not worry about your download, or bandwidth.

The following data transfer steps can be be used for automated batch transfers of your data, parallelising using up to the number of CPUs on your VM. However, it is important to note that IRDS will not allow transfer of files with unusual extensions, such as .bam, .bai, .md5, and others. One way around this is to zip/tar your files before transfer. It is also possible to change the extension to an acceptable format for transfer (i.e. change .bam to .bam.txt).

IRDS may close the connection after several hours, so it is important to check your log files to see if you transfer completed successfully. If you use the -c flag as specified below, you will be able to resume failed transfer by simply repeating your transfer script.

Step-by-step guide (once logged in to your Nimbus instance)

Part 1. Set up wget

You will need to make a ~/.wgetrc file for your user name and password.

##Change into your home directory
cd YOUR_HOME_DIRECTORY

##make ~/.wgetrc file
touch .wgetrc 

##change the permissions on the file so only you can view and edit
chmod 600 .wgetrc 

##open .wgetrc for editing
nano .wgetrc 

#Add the following two lines with your credentials, then save the file
--http-user=UWA staff number
--http-passwd=Pheme password

Part 2. Set up your transfer scripts

1. In the directory where you would like to download the files at on Nimbus, make a .txt file with the full paths of the files you would like to transfer (eg. https://unidrive.uwa.edu.au/staff//irds/PATH/TO/YOUR/FILES), with one file path per line. Alternatively, compress the files you would like to transfer into one zip/tar.gz file and use that path instead. Note: the double forward slashes between staff and irds (i.e. staff//irds) are deliberate.

2. Create a batch script to perform the transfer. A semi-automated example is provided below
3. Before running your script, make sure to open a screen session by typing 'screen' into your VM. To detach from a screen session, use the keys 'ctrl + a' and then 'd'. To reconnect to a screen session, use 'screen -r'.

Example batch script:

#!/bin/bash

#editable variables. Change these to suit ypur needs and pawsey account
input_file_list="test_input.txt"
number_of_CPUs="8"

###Add slurm parameters to the job script
echo "#!/bin/bash" >> irds_trf.sh.tmp

#split the input sample list into 16, without breakiung up any text lines.
#Gives prefix of "input-" to the output split files
split -n l/${number_of_CPUs} ${input_file_list} input-

#add the input files to the job script
for file in $(ls input-*)
do
echo "wget -r -np -N -c -i $file &" >> irds_trf.sh.tmp
done

#remove the "&" character from the very last line of the job script
sed '$s/&$//' irds_trf.sh.tmp >> irds_trf.sh

#remove temp file
rm irds_trf.sh.tmp

#start job script
bash irds_trf.sh 2>&1

The -c flag allows easy resuming of failed transfers if they fail. The -i flag reads the list of files you made in Part 2, Step 1. Your user name and password should be read automatically from the .wgetrc file you made earlier.

Note that IRDS tends to disconnect users after several hours, so don't bother setting a long wall time as you will almost certainly get disconnected before that. You will need to check your log files when your job completes to know if the transfer failed at any point. If it did fail, simply restart the script and it will continue where it left off. Make sure to delete the sbatch_irds_trf.sh file before restarting your job.

The log files tend to be very long, so you are best to use tail rather than cat or zless.

Between Nimbus Projects

If you are a member of multiple Nimbus projects, you may want to share data between those projects. For one-off transfers of data, the simplest option is to transfer a storage volume from one project to the other. This option will move (not copy) an existing storage volume from one project to another, using a shared Transfer ID and Authorization Key. If you wish to use this option, please note the following:

• The steps given below are done through the Nimbus dashboard; if you prefer to perform the transfer from the command-line using the openstack client, that can be found here: https://docs.openstack.org/cinder/latest/cli/cli-manage-volumes.html#transfer-a-volume
• You do not need to be a member of both the source and destination project in order to transfer a storage volume between them. Once you have initiated the transfer from the source project, you can give the Transfer ID and Authorization Key to a member of the destination project to complete the transfer.

Source Project - Initiate the Transfer

1. Log in to the Nimbus dashboard (of your source project), and go to the "Volumes" section.
2. If you wish to make a copy of the volume in the source project, do so before you begin the transfer.
   1. You will need to create a snapshot of the volume first, then create a separate volume from that snapshot (be sure to delete the snapshot once you have created the volume copy from it).
   2. Make sure that the status of the volume you wish to transfer is "Available" - i.e. it is not attached to another instance or in use by someone.
   3. Also, check the "Snapshots" page of the dashboard to confirm that there are no snapshots associated with the volume.
3. Click on the drop-down menu to the right hand side of the volume you wish to transfer, and select "Create Transfer".
   1. This will pop up a prompt asking for a "Transfer Name". Give it a simple name (preferably without any spaces in the name), and click on "Create Volume Transfer".
   2. Another window will pop up, giving you a "Transfer ID" and an "Authorization Key". Make a copy of both of these values somewhere (or click on the "Download transfer credentials" button below it to save those details to a text file).
4. Once you have done that, the status of the volume will change to "awaiting-transfer" in the dashboard. At this point, if you decide you don't want to proceed with the transfer, you can click on "Cancel Transfer" to the right of the volume to cancel the process.

Destination Project - Complete the Transfer

1. When you are ready to proceed, click on the down-arrow next to the project name in the top-left corner of the dashboard, to give you a list of all of the Nimbus projects you are a member of. Select the project from the drop-down list that you wish to change to.
2. From the "Volumes" page in the destination project, click on "Accept Transfer" on the right hand side of the dashboard. This will pop up an "Accept Volume Transfer" window.
3. Enter the "Transfer ID" and "Authorization Key" that you copied down from previously, then click on "Accept Volume Transfer". The volume transfer should be pretty much instantaneous, as it is essentially a change of ownership of the volume, rather than copying the volume from one project to another.