Pawsey filesystems are specialised for different uses, and therefore it is important that you transfer your files into the correct location.

Command-line clients for transfers in and out Pawsey filesystems

Command line clients are a convenient way of moving data between computers.

The transfer of files in and out Pawsey filesystems is performed over encrypted data streams. The Secure Copy Protocol (scp) and the Secure File Transfer Protocol (sftp), which are based on the Secure Shell protocol (SSH), are installed in our systems and can send data over a normal SSH connection. Most of the tools we recommend in this page make use of these protocols.

For Linux and macOS operating systems, the command-line clients listed here are available in a terminal or xterm.

For windows operating system, the command-line clients listed here are part of most of the common Linux environments that run on top of windows; that is the "Windows Subsystem for Linux (WSL)", "Cygwin" and the terminal tool "MobaXterm" (see Alternative Programs to Log in from Windows).

GUI clients

GUI clients are a very attractive option for file transfers because of their intuitive framework. They have the advantage that users do not need to remember the several different options for the command-line tools, although, they are still based on the command-line clients listed above. In practice, the combined usage of both GUI and command-line clients within your workflows results in better efficiency.

Transfers between Pawsey filesystems and Acacia

Transfers between Pawsey filesystems and Acacia object storage are performed by tools/clients compatible with the Amazon S3 protocol. In Pawsey clusters, we count with modules of the following S3 compatible clients:

• rclone:

module load rclone/<version>

• aws client:

module load awscli/<version>

In-depth description of the different tools/clients for accessing and transferring data in/out Acacia is in the /wiki/spaces/DATA/pages/54459526.

Best practices for your data and data transfers

Only keep at Pawsey the data that is still going to be needed for your project 

Only keep at Acacia and Pawsey filesystems the files and objects that are going to be utilised during the rest of your project. Those files/results/data that are finalised and are not going to be used anymore at Pawsey supercomputing/cloud and visualisation facilities should be transferred into your own institution storage and removed from our systems. The same applies when a project as a whole is finalised. None of the systems at Pawsey are designed as long term storage.

Don't use login nodes to transfer large amounts of data

Transfers initiated by remote sites should connect to the data-mover nodes with the generic hostname data-mover.pawsey.org.au.

When large data transfers are initiated from within the Pawsey systems, they should use the "copyq" queue.

Prefer the use of TAR files to transfer your data

Transferring clients spend a lot of time figuring out the list of files to be transferred and checking the correctness of each individual transfer. When the transfer involves a large amount of files and directories (for example, the result files of multiple OpenFOAM runs), this "listing and checking" time adds up dramatically. Therefore, it is always much more convenient to use the tar command locally to first pack the files that need be transfered into a single TAR file (or into a very reduced number of TAR files). Then transfer the TAR files and "untar" them (tar -x) afterwards, once they are at Pawsey filesystem

Use tools that can resume transfers if something fails

Interactive use of SCP may be the most appropriate and practical way of transferring a small amount of data, but for larger data transfers, interactive use of SCP is discouraged. It is always preferred to perform large data transfers with tools that can resume transfers if the process/connection fails (for example Rsync or FileZilla).

Use scripts to automatize frequent transfers

For transfer processes that may occur routinely, the use of scripts with command-line tools for automating the processes is a great option.

For automating the connections needed within your script, you can use ssh-keys as indicated in "Secure transfers using ssh-keys" section below.

WinSCP also allows scripting.

Do not preserve the time stamp of your files when transferring them into /scratch

If files have not been accessed for more than 30 days in your own system and then you transfer them into /scratch using a "time stamp preserving" option, this will result in the inadvertent and almost immediate deletion of the recently copied files and then data loss. Deletion will happen because of the 30 days purge policy will identify those files as old.

Do not use spaces or special characterers in the names

The administration of filenames containing spaces and special characters may present problems. We strongly recommend to avoid their use in files that are to be transferred into Pawsey.

Secure transfers using ssh-keys

Transferring data using a protocol based on SSH allows us to protect information and ensure its integrity. However, setting up a proper environment configuration can be tricky; if not done right, security risks arise. This is especially true when one wants to automate copy operations, for example through a SLURM job on data-mover nodes. In such a scenario, a public key-based authentication method is recommended because the SSH client, running on a Pawsey's supercomputer node, will only need the private key to connect to a third-party system's SSH server, which in turn has the correspondent public key to be used to perform a secure handshake. The private key, however, must not be protected by a passphrase otherwise human input is required. There are several issues to address in the described situation.

You must generate a key-pair specifically for this purpose, that is, data transfers to and from Pawsey systems (Use of SSH Keys for Authentication). Let's call this key-pair: COPYPAIR. Do not repurpose an existing key-pair used to log in to Pawsey or other systems (which by the way, should use a passphrase). This allows isolation of unauthorized accesses due to a compromised key-pair.

The SSH server on the third party system should be configured to avoid using COPYPAIR's public key to authorise connections not originating from Pawsey's data-mover machines. This is a powerful capability that protects the third party server from unauthorised use of COPYPAIR from outside the Pawsey network. To enable the discussed feature, users need to edit the COPYPAIR.pub public key file and prepend the following string:

from="data-mover*.pawsey.org.au" no-port-forwarding no-pty

followed by a space and followed by the original key.

Here is an example:

from="data-mover*.pawsey.org.au" no-port-forwarding no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDhGk1QdMVDVao1j9eclHPPhniU5x6rHYBhJp88DJZrEiDM3Kt70+gHvo/fCGaHmOMWQX0hjqLs5uin42VGUW7w3y0FrIBB/hZJro+JKXJzhUJFpTE/wR08CK8DI4c3GrxjrCqNRkd3ff4AOUIgS7VFGcmagg9aAj6iSas1ibvAMLMZuXkVyPcNcKhB+J38atc3u5/zuRqU9QgKGQvTQgLL7lx4CrsHGKd8bPzjdEVDaCoeD1KBdRq/S+am2wvaPwN5wqqgs6hVU83VvZggIBkGRLBbGEeMmnzu8dkG1osqE4S3RCmFVQ8MG9tiOiP0MN/jx/DpckP++NnuamJWcD/Z comment

Common problems with transferred files

Your uploaded files into /scratch have been unexpectedly purged

This may have happened if the original time stamp of the transferred files was preserved after copying them into /scratch (like the -t option for Rsync or the default transfer settings for WinSCP). To avoid this problem, do not use a time-preserving option in your transfer tool.

Problem with ownership of files on /scratch

To resolve this problem, the "group" part of the ownership properties needs to be fixed. Secondly, the setup of the transferring tool needs to be fixed for avoiding the problem to happen again.

Consider the following example where the directory badDirectory resulted with the wrong settings after being transferred. The ls -la command shows the ownership properties of the directory as mickey-mickey instead of mickey-pawsey9999, which is the right "owner-group" combination of ownership properties. The rest of the directories listed have the correct properties.

$ ls -la
total 20
drwxrws---+  5 mickey pawsey9999 4096 Jul 30 16:00 .
drwxr-s---+ 17 mickey pawsey9999 4096 Jul 25 09:25 ..
drwxrws---+  2 mickey mickey     4096 Jul 30 16:00 badDirectory
drwxrws---+  2 mickey pawsey9999 4096 Jul 30 15:52 borra
drwxrws---+  4 mickey pawsey9999 4096 Jul 11 15:53 GS-8819-ONETEP

To resolve the problem, execute the chgrp command (change group) with the "-R" option to apply the change recursively through all the content (files and subdirectories) inside the affected initial directory. The general syntax is:

$ chgrp -R <yourProject> <yourAffectedDirectory>

This fixes that group association of the directory named  badDirectory and all the files in it recursively:

$ chgrp -R pawsey0001 badDirectory
$ ls -la
total 20
drwxrws---+  5 mickey pawsey9999 4096 Jul 30 16:00 .
drwxr-s---+ 17 mickey pawsey9999 4096 Jul 25 09:25 ..
drwxrws---+  2 mickey pawsey9999 4096 Jul 30 16:00 badDirectory
drwxrws---+  2 mickey pawsey9999 4096 Jul 30 15:52 borra
drwxrws---+  4 mickey pawsey9999 4096 Jul 11 15:53 GS-8819-ONETEP

If the problem has expanded extensively into most of your /group directory, you can fix it by using fix.group.permission.sh, which is provided by the module pawseytools. For more information about this tool, see under "File Permissions and Quota" on the Pawsey Filesystems and their Use page.

Finally, in order to avoid this problem to happen again, configure your file transfer program to honour the setgid (set-group identification) default so that newly created files and directories belong to your project on the "group" property. This is explained for several documented tools in the following subsections.

Your institution firewall may block connections

SSH is enabled on Pawsey systems for both incoming and outgoing traffic. This, however, may not be true for some firewalls on connections on the client side. Most university, business and home internet connections only permit outgoing connections, and have their incoming SSH disabled within their firewall. This means that, SCP is always invoked on the client, that is, your Laptop/Desktop to copy the data to/from the Pawsey supercomputers.

Related Pages

• Use of SSH Keys for Authentication
• Transferring Files
• /wiki/spaces/DATA/pages/54459526