/
Transferring Files in/out Pawsey Filesystems

Transferring Files in/out Pawsey Filesystems

Pawsey filesystems are specialised for different uses, and therefore it is important that you transfer your files into the correct location.

On this page:

Command-line clients for transfers in and out Pawsey filesystems

Command line clients are a convenient way of moving data between computers.

The transfer of files in and out Pawsey filesystems is performed over encrypted data streams. The Secure Copy Protocol (scp) and the Secure File Transfer Protocol (sftp), which are based on the Secure Shell protocol (SSH), are installed in our systems and can send data over a normal SSH connection. Most of the tools we recommend in this page make use of these protocols.

For Linux and macOS operating systems, the command-line clients listed here are available in a terminal or xterm.

For windows operating system, the command-line clients listed here are part of most of the common Linux environments that run on top of windows; that is the "Windows Subsystem for Linux (WSL)", "Cygwin" and the terminal tool "MobaXterm" (see Alternative Programs to Connect from Windows).

Use ssh-keys with command-line clients

Pawsey strongly recommends the use of ssh-keys instead of the conventional and less secure method of typing your username and password. For a description of how to set up ssh-keys, see: Use of SSH Keys for Authentication.

SCP (Secure Copy Protocol)

The Secure Copy Protocol, or SCP, is a file transfer network protocol used to move files onto servers, and it fully supports encryption and authentication. SCP uses Secure Shell (SSH) mechanisms for data transfer and authentication to ensure the confidentiality of the data in transit.

SCP is useful to copy few small files and not a lot or very large files. It is not recommended for the transfer of large amounts of data, as it can't resume transfers if the operation/connection is interrupted for any reason.

Syntax

$ scp [options] SourceFileOrDir USER@HOST:DestFileOrDir

$ scp [options] USER@HOST:SourceFileOrDir DestFileOrDir

Here, the name of the "SourceFileOrDir" includes both the path and name of the files/directories to transfer. The "DestFileOrDir" (destination) also includes the path (and possibly the "new" name) of the destination file or directory. "USER" is the username to connect, and "HOST" is the hostname or ip address of the remote computer to connect to. Note the use of the colon (:) separating the HOST and the definition of the path/files on it.

Examples

Terminal 1. Example of transferring a file using scp.
$ scp /myDir/initialConditions.tar.gz [username]@data-mover.pawsey.org.au:/scratch/[project]/[username]

To transfer a whole directory tree recursively, then the option "-r" needs to be used. For example, to backup the whole directory where a user may keep their useful scripts "/software/pawsey9999/mickey/myScripts" into user's own computer local directory "./PawseyStuff":

Terminal 2. Examples of transferring a directory recursively.
$ scp -r mickey@data-mover.pawsey.org.au:/software/pawsey9999/mickey/myScripts ./PawseyStuff

External references

Documentation about the general use of SCP can be found elsewhere.

Rsync

Rsync is a utility for efficiently transferring and synchronizing files across computer systems, by checking the timestamp and size of files.

Rsync is more robust than SCP in the sense that it can resume transfers after failure, and it can also be used for synchronising and backing up.
Pawsey recommend its use within scripts that can be reused for transferring data.

Syntax

$ rsync [options] SourceFileOrDir USER@HOST:DestFileOrDir

$ rsync [options] USER@HOST:SourceFileOrDir DestFileOrDir

Here, the name of the "SourceFileOrDir" includes both the path and name of the files/directories to transfer. The "DestFileOrDir" (destination) also includes the path (and possibly the "new" name) of the destination file or directory. "USER" is the username to connect, and "HOST" is the hostname or ip address of the remote computer to connect to. Note the use of the colon (:) separating the HOST and the definition of the path/files on it.

Rsync has a lot of different options that are documented elsewhere. Pawsey recommends to use the following options (to be included in the above syntax):

-vhsrl --chmod=Dg+s -e ssh

The purpose of the recommended options are as follows:

Table 1. Pawsey’s recommended rsync options.

Option

Purpose

Option

Purpose

-e ssh

Run rsync over SSH

--chmod=Dg+s

Force all directories to get marked by the default setgid.

-v

(verbose) Display messages about the progress of the transfer.

-h

Use human-readable numbers for the sizes of the transferred data displayed by the "verbose" option.

-s

This allows for strange names of files (with spaces or special characters) to be interpreted as part of the name.

Pawsey does not recommend the use of spaces or special characters in filenames.

-r

Copy directories recursively, so that if a directory has been chosen to be transferred, all the contents of the directory will be transferred.

-l

When symlinks are encountered, recreate the symlink on the destination.

Do not use -a option for transferring files into our systems

Contrary to the common recommendations available on the internet, Pawsey does not recommend the use the "-a" option, especially for transferring files into our systems. This is because the behaviour of "-a" can override the default setgid settings of the "group" property (see "problems with ownership" below).

Use preservation of times with care

Rsync has an option, -t, that activates the preservation of modification times of the files into the destination system. This can be useful for Rsync choosing not to transfer files that already exist in the destination system with the same modification dates.  However, this should not be used when transferring files to the /scratch filesystem, where a 21-day purge policy is in place. If files have not been accessed for more than 21 days in your own system and then you transfer them into /scratch using the -t  option, this will result in the inadvertent and almost immediate deletion of the recently copied files and then data loss. Deletion will happen because of the 21 days purge policy will identify those files as old.

Examples

For transferring the local directory in your host computer "~/initialConditions" and all its contents into your personal directory on /scratch, where "[username]" and "[project]" is your username and project respectively:

Terminal 3. Example transferring a whole directory
$ rsync -vhsrl --chmod=Dg+s -e ssh ~/initialConditions [username]@data-mover.pawsey.org.au:/scratch/[project]/[username]

External references

Documentation about the general use of Rsync can be found elsewhere.

 

SFTP (SSH File Transfer Protocol)

SFTP is a protocol and an interactive file transfer client that uses SSH to create a secure connection to the server. Its functionality is very similar to FTP, but not all of the FTP options are available. Not be confused with FTPS (FTP over SSL).

Open and close a connection to Pawsey

The following sftp command can be used to establish a connection from your local machine to the Pawsey data-mover nodes:

Terminal 4. SFTP connection to the data-mover nodes
myLocalComputer:> sftp [username]@data-mover.pawsey.org.au sftp>

Note that after establishing a connection, the prompt will change to "sftp>" indicating that the interactive SFTP session has started.

To close the connection, execute the following:

Terminal 5. Change the current directory from the remote server
sftp> bye

Navigation in the local and remote systems

As for any Linux interactive session, the basic navigation tool for navigating directories in the remote filesystem is cd:

Terminal 6. Change the current directory from the remote server
sftp> cd RemoteDesiredDirectory

In addition, you can check the current directory on the remote server with pwd and ls:

Terminal 7. Check the current directory on the remote server
sftp> pwd Remote working directory: /scratch/pawsey9999/mickey sftp> ls Nek5000 OpenFOAM babadu testingTransfers userSupport

Within the SFTP interactive session, you can also navigate in your local computer by using the prefix "l" for "local" in the commands:

Terminal 8. Navigate the local filesystem
sftp> lpwd Local working directory: /Users/[username] sftp> lls Applications Downloads Movies Tools Boostnote Dropbox Music borra Desktop Library Pictures Documents Public mnt sftp> lcd Downloads

It can be tricky to remember to use lcd, lls and lpwd to navigate in the local system. Therefore, it is recommended to navigate in your local system before establishing the SFTP connection. This way, your current directory in the local computer will be the desired local directory for file transfers and you may not need to navigate from there anymore.

Copy files to Pawsey filesystems

Once an interactive SFTP connection to the data-mover system at Pawsey has been established, and the current directory in the local and remote systems are the desired ones, users can put files into the remote system by executing the put command:

put [options] SourceFileOrDirInLocalSystem [DestFileOrDirInRemoteSystem]

For example:

Terminal 9. Transfer files to the remote filesystem
sftp> put ~/PawseyStuff/initialConditions.tar /scratch/pawsey9999/mickey/case1

Or, if you already navigated (as explained above) into the correct path in the local and the remote computer, then a simple put would be enough:

Terminal 10. Transfer myfile.dat
sftp> put initialConditions.tar

As the general syntax suggests, there is still freedom to choose source path/files and destination path/file names. The following example takes another file from other directory and puts it into your personal directory in /software, even if the current remote directory was in /scratch:

Terminal 11. Transferring files from different source and destination paths
sftp> pwd Remote working directory: /scratch/[project]/[username] sftp> put /otherDir/otherFile.F90 /software/[project]/[username]/someOtherFile.F90

In order to put an entire directory, the option "-r" needs to be used. In this case, the directory "myScripts" will be transferred to the your personal directory in the /scratch filesystem:

Terminal 12. Transfer of a directory recursively
sftp> put -r ./myScripts /scratch/[project]/[username]

Copy files from Pawsey filesystems

The SFTP command for copying data into the local filesystem is get. Its general syntax is:

get [options] SourceFileOrDirInRemoteSystem [DestFileOrDirInLocalSystem]

External references

Documentation about the general use of SFTP can be found elsewhere.

GUI clients

GUI clients are a very attractive option for file transfers because of their intuitive framework. They have the advantage that users do not need to remember the several different options for the command-line tools, although, they are still based on the command-line clients listed above. In practice, the combined usage of both GUI and command-line clients within your workflows results in better efficiency.

Use ssh-keys with GUI transfer tools

Pawsey strongly recommends the use of ssh-keys instead of the conventional and less secure username/password method. In this section we describe how to set up some GUI tools using ssh-keys to authenticate access to Pawsey. For a description of how to set up ssh-keys, see: Use of SSH Keys for Authentication.

Also, by avoiding the use of username/password method, you will not need to save this sensitive data within the tool (which is also never recommended). Furthermore, avoiding the use of username/password method will keep you apart of the common problem of blockage of your account when the transfer tool retries to connect to Pawsey with an old or wrong password.

Always pay attention to the source and destination directories

Most GUI clients will start in your /home directory when you first connect to a remote server, while some will start in your previously accessed directory. This is almost never where you need to put the data for the new session. In most cases you will need to browse to your own /scratch, /home or /software directories.

FileZilla

FileZilla is a fast and reliable file transfer client with an intuitive GUI. It works on multiple platforms (Windows, macOS and Linux) and supports SFTP, which is one of the supported protocols for transferring files to and from Pawsey systems. FileZilla supports simultaneous transfer of multiple files, transfer of large files (>4GB) and transfer resume after connection failure.

Avoid spyware

Only download FileZilla from its official website: https://filezilla-project.org/index.php . As for any software, be careful of not falling into "click tricks" that mislead you to download or install undesired software.

Basic setup

  • Select Site Manager from the top File menu. A window for the "Site Manager" settings will be presented

  • Click on "New Site" on the left panel. Name the site as you want (data-mover is the name used here)

  • On the right panel, choose "Protocol" as: "SFTP - Secure File Transfer Protocol"

  • In "Host", enter: "data-mover.pawsey.org.au" (Leave "Port" blank)

  • In "Logon Type" choose "Key file"

  • In "Key file" select the ssh-key you use to connect to Pawsey (~/.ssh/pawsey_ecdsa_key in this example). Pawsey strongly recommends the use of ssh-keys instead of the conventional and less secure use of username/passwords (please read the note above).

    • After selecting the key, Filezilla will convert it into its own format (.ppk).

    • It will also ask for the passphrase of the key and for the name of the new key (we recommend to use the same name as the original key, except for the .ppk extension)

  • Click "Connect". Enter your username and your password for accessing Pawsey systems.

image-20260127-080638.png
Figure 1. FileZilla's Site Manager

Connecting with username/password

It is possible, but is not our recommendation. When using it, we recommend to never save the password within the tool.

Multiple simultaneous transfers

FileZilla supports simultaneous transfer of multiple files, so should be faster than WinSCP or command-line scp if you have many files to transfer. Multiple streams is most useful over long distances, in which case 12 may be useful. Nevertheless, too many streams will create load on the data-mover node, and may reduce performance (and impact others). Over short distances, up to 4 streams may help improve performance while not creating too much load. You can limit the maximum simultaneous connections in the "Transfer Settings" section of the Site Manager settings window shown above.

Automation using scripts

To our knowledge, FileZilla does not allow for automatic transfers through the use of scripts. (See https://superuser.com/questions/239860/how-do-i-send-a-file-with-filezilla-from-the-command-line). But you can always prepare scripts that make use of command-line clients (explained above) or use other tools that allow this, such as WinSCP described below.

WinSCP

WinSCP is an open source free file transfer client for Windows. It has an intuitive GUI and supports SFTP and SCP protocols among others. Its main function is file transfer between a local and a remote computer. Beyond this, WinSCP offers scripting and basic file manager functionality. We recommend to use the SFTP protocol (default) as it allows to resume transfers after a connection interruption.

Only download WinSCP from its official website: https://winscp.net/eng/index.php . As for any software, be careful of not falling into "click tricks" that mislead you to download or install undesired software.

Basic setup

  • Open a "New Session".

  • Click "New Site".

  • In "File protocol" select "SFTP". You could also use "SCP" if preferred. We recommend SFTP as it supports the resume of truncated transfers and SCP does not, although SCP seems to be faster.

  • In "Host name" enter "data-mover.pawsey.org.au" (Leave default for "Port number").

  • Enter "User name". Do not enter your "Password".

image-20260127-080803.png
Figure 2. Setup start window

  • Do NOT hit Login yet.

  • Click "Advanced" (for setting up the ssh-key authentication). Pawsey strongly recommends the use of ssh-keys instead of the conventional and less secure use of username/passwords (please read the note above).

  • On the left tree panel choose "SSH" and then "Authentication".

  • Now, the ssh-key you use for connecting into Pawsey needs to be translated into PuTTY format (.ppk). For that click on "Tools" on the right panel (just below the Private key selection box) and then choose "Generate New Key Pair with PuTTYgen...".

Figure 3. Advanced site settings (for defining the ssh-key to use)

  • In "Load an existing private key" click "Load" and select the ssh-key you use to connect to Pawsey (C:\Users\yourLocalUser\.ssh\pawsey_ecdsa_key in this example). You may need to look for "All Files (*.*)" to see it. Then click "Open".

  • Type the passphrase for the ssh-key and click "OK".

  • Now you need to save the translated key to the new format. So click "Save private key" and choose an appropriate name with ".ppk" extension (C:\Users\yourLocalUser\.ssh\pawsey_ecdsa_key.ppk in this example). Then click "Save".

Figure 4. PuTTY Key Generator for translating your existing key

  • Close the PuTTY Key Generator window on the top right corner "X".

  • In the "Private key file", the translated key should be already displayed (C:\Users\yourLocalUser\.ssh\pawsey_ecdsa_key.ppk in this example). If it is not, then browse to select it. Then click "OK" in the general window of "Advanced Site Settings" which will return you to the original settings window titled "Login".

  • Now click "Save" button under the "User name". This will open a new window where you can choose a "Site name" for "Saving the session as a site" and click "OK".

Figure 5. Save the session as a site.

  • Now, on the left panel of the main "Login" window, you can select the "Session Site" you have just configured and finally click "Login" to test the connection.

  • You may need to accept cache fingerprints among computers and enter the passphrase for the ssh-key.

Pageant ssh-key agent setup

In order to manage the .ppk key by an agent you will need to add it to Pageant. This tool is part of the tools that can be downloaded from the WinSCP site.

  • After downloading, installing and executing Pageant, right click on the Pageant icon (a little computer with a hat).

  • Choose "View Keys" from the menu and a window with the current list of keys managed by the agent will appear.

  • Then click "Add Key" and select the desired .ppk key.

Figure 6. Pageant Key List

  • Type the passphrase, which will be remembered by Pagent, so that you don't need to type the passphrase every time you start a session with WinSCP.

Basic usage

After a connection has been established, the right panel will initially show the content of the /home/<user> directory. Besides all the navigation options available, the easier to use for the first time is the "Open directory/bookmark" button. This will open a window where you can type the desired path to navigate first. We recommend to first click "Add" to save the bookmark, and then "OK" to navigate into the indicated path.

Figure 7. Directory listings of both the local and remote filesystems.

For transferring files, select and drag, or use the "Upload" and "Download" buttons.

Ending a session

From the top menu choose "Session" and then "Close Session".

Transfer settings (avoid preservation of time stamps)

Use preservation of times with care

WinSCP has a default setting to preserve timestamps (i.e. modification times) after uploading to the destination system. This should not be used when transferring files to the /scratch filesystem, where a 21-day purge policy is in place (see [Conversion Test] Filesystem Policies ). Using time preservation transfer to files that have not been accessed for more than 21 days will result in the deletion of those files by the purge policy, and then data loss.

To avoid the preservation of time stamps:

  • From the top menu click "Options" and then "Preferences"

  • On the left panel tree choose "Transfer" and then click on "Edit".

  • From the many options, unselect the "Preserve timestamp" and check that the permissions coincide with our following recommendation:

Figure 8. Transfer settings and permissions

Connecting with username/password

It is possible, but is not our recommendation. When using it, we recommend to never save the password within the tool.

Cyberduck

Cyberduck is described as "libre server" and cloud storage browser for macOS and Windows. It supports SFTP among many other transfer protocols to remote clusters and cloud platforms. Cyberduck has a simple and practical GUI, although it appears less user-friendly than FileZilla or WinSCP.

Only download Cyberduck from its official website: https://cyberduck.io/. As for any software, be careful of not falling into "click tricks" that mislead you to download or install undesired software.

Basic setup

  • Click the "Open Connection" button at the top of the Cyberduck window.

  • Select "SFTP (SSH File Transfer Protocol)" from the drop-down selection box at the top of the window.

  • In "Server" enter: "data-mover.pawsey.org.au" (leave default number for "Port").

  • In "Username" enter your username for Pawsey systems.

  • In "Password" leave it blank.

  • In "SSH Private Key" select the ssh-key you use to connect to Pawsey (~/.ssh/pawsey_ecdsa_key in this example). Pawsey strongly recommends the use of ssh-keys instead of the conventional and less secure use of username/passwords (please read the note above).

  • Activate the option: "Add to Keychain". This will save the passphrase of the ssh-key into your keychain and avoid the need for typing the passphrase for each transfer.

  • Click connect:

figure9bis.png
Figure 9. SFTP connection settings.

  • Provide the passphrase for your ssh-key and select "Save Password" to keep the passphrase registered and avoid the need of typing it for each transfer.

figure10bis.png
Figure 10. Passphrase to your ssh-key

  • Accept the SSH fingerprint by selecting "Allow". Here you can tick to "Always" remember the key.

figure11bis.png
Figure 11. Accept SSH fingerprint

Connecting with username/password

It is possible, but is not our recommendation. When using it, we recommend to never save the password within the tool.

Basic use

The navigation window of Cyberduck shows the directory structure of the server you have connected to. It is handy to open an additional window of your preferred navigation GUI on your own computer to be able to display the files and directory structure on your local filesystem.

Transfers can be performed by selecting files and the dragging the selection from one window to the other.

On the Cyberduck window, you can also select files and right click to choose an operation on them.

Make use of bookmarks

Once you have navigated to the desired path in the server, it is extremely useful to save bookmarks of it for future use.

Transfers between Pawsey filesystems and Acacia

Transfers between Pawsey filesystems and Acacia object storage are performed by tools/clients compatible with the Amazon S3 protocol. In Pawsey clusters, we count with modules of the following S3 compatible clients:

  • rclone:

module load rclone/<version>

  • aws client:

module load awscli/<version>

In-depth description of the different tools/clients for accessing and transferring data in/out Acacia is in the Acacia - User Guide.

Best practices for your data and data transfers

Only keep at Pawsey the data that is still going to be needed for your project 

Only keep at Acacia and Pawsey filesystems the files and objects that are going to be utilised during the rest of your project. Those files/results/data that are finalised and are not going to be used anymore at Pawsey supercomputing/cloud and visualisation facilities should be transferred into your own institution storage and removed from our systems. The same applies when a project as a whole is finalised. None of the systems at Pawsey are designed as long term storage.

Don't use login nodes to transfer large amounts of data

Transfers initiated by remote sites should connect to the data-mover nodes with the generic hostname data-mover.pawsey.org.au.

When large data transfers are initiated from within the Pawsey systems, they should use the "copyq" queue.

Prefer the use of TAR files to transfer your data

Transferring clients spend a lot of time figuring out the list of files to be transferred and checking the correctness of each individual transfer. When the transfer involves a large amount of files and directories (for example, the result files of multiple OpenFOAM runs), this "listing and checking" time adds up dramatically. Therefore, it is always much more convenient to use the tar command locally to first pack the files that need be transfered into a single TAR file (or into a very reduced number of TAR files). Then transfer the TAR files and "untar" them (tar -x) afterwards, once they are at Pawsey filesystem

Use tools that can resume transfers if something fails

Interactive use of SCP may be the most appropriate and practical way of transferring a small amount of data, but for larger data transfers, interactive use of SCP is discouraged. It is always preferred to perform large data transfers with tools that can resume transfers if the process/connection fails (for example Rsync or FileZilla).

Use scripts to automatize frequent transfers

For transfer processes that may occur routinely, the use of scripts with command-line tools for automating the processes is a great option.

For automating the connections needed within your script, you can use ssh-keys as indicated in "Secure transfers using ssh-keys" section below.

WinSCP also allows scripting.

Do not preserve the time stamp of your files when transferring them into /scratch

If files have not been accessed for more than 21 days in your own system and then you transfer them into /scratch using a "time stamp preserving" option, this will result in the inadvertent and almost immediate deletion of the recently copied files and then data loss. Deletion will happen because of the 21 days purge policy will identify those files as old.

Do not use spaces or special characterers in the names

The administration of filenames containing spaces and special characters may present problems. We strongly recommend to avoid their use in files that are to be transferred into Pawsey.

Secure transfers using ssh-keys

Transferring data using a protocol based on SSH allows us to protect information and ensure its integrity. However, setting up a proper environment configuration can be tricky; if not done right, security risks arise. This is especially true when one wants to automate copy operations, for example through a SLURM job on data-mover nodes. In such a scenario, a public key-based authentication method is recommended because the SSH client, running on a Pawsey's supercomputer node, will only need the private key to connect to a third-party system's SSH server, which in turn has the correspondent public key to be used to perform a secure handshake. The private key, however, must not be protected by a passphrase otherwise human input is required. There are several issues to address in the described situation.

You must generate a key-pair specifically for this purpose, that is, data transfers to and from Pawsey systems (Use of SSH Keys for Authentication). Let's call this key-pair: COPYPAIR. Do not repurpose an existing key-pair used to log in to Pawsey or other systems (which by the way, should use a passphrase). This allows isolation of unauthorized accesses due to a compromised key-pair.

The SSH server on the third party system should be configured to avoid using COPYPAIR's public key to authorise connections not originating from Pawsey's data-mover machines. This is a powerful capability that protects the third party server from unauthorised use of COPYPAIR from outside the Pawsey network. To enable the discussed feature, users need to edit the COPYPAIR.pub public key file and prepend the following string:

Listing 2. Edit the public key file
from="data-mover*.pawsey.org.au" no-port-forwarding no-pty

followed by a space and followed by the original key.

Here is an example:

Listing 3. Example public key
from="data-mover*.pawsey.org.au" no-port-forwarding no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDhGk1QdMVDVao1j9eclHPPhniU5x6rHYBhJp88DJZrEiDM3Kt70+gHvo/fCGaHmOMWQX0hjqLs5uin42VGUW7w3y0FrIBB/hZJro+JKXJzhUJFpTE/wR08CK8DI4c3GrxjrCqNRkd3ff4AOUIgS7VFGcmagg9aAj6iSas1ibvAMLMZuXkVyPcNcKhB+J38atc3u5/zuRqU9QgKGQvTQgLL7lx4CrsHGKd8bPzjdEVDaCoeD1KBdRq/S+am2wvaPwN5wqqgs6hVU83VvZggIBkGRLBbGEeMmnzu8dkG1osqE4S3RCmFVQ8MG9tiOiP0MN/jx/DpckP++NnuamJWcD/Z comment

Common problems with transferred files

Your uploaded files into /scratch have been unexpectedly purged

This may have happened if the original time stamp of the transferred files was preserved after copying them into /scratch (like the -t option for Rsync or the default transfer settings for WinSCP). To avoid this problem, do not use a time-preserving option in your transfer tool.

Problem with ownership of files on /scratch

If the "group" property has been overriden

By default, your personal directory in /scratch (easily accessible with the environmental variables $MYSCRATCH) is set to belong to: "<yourUsername>:<yourProject>" in the "owner:group" properties of the directory. This combination of ownership properties has been set through setgid.  Also, by default, all files and subdirectories created under your personal directory in /scratch should inherit the same ownership properties, that is, they should belong to: "<yourUsername>:<yourProject>" too.

Unfortunately, some file transfer programs override our setgid defaults for ownership properties. The typical unfortunate change is to set "<yourUsername>:<yourUsername>" instead of "<yourUsername>:<yourProject> for the "owner:group" properties (see terminal 1). This makes the files abide by the quota restrictions as set on /home and won't be able to make use of extended quotas as on /scratch or /astro. You may receive errors during an upload if this is the case.

To resolve this problem, the "group" part of the ownership properties needs to be fixed. Secondly, the setup of the transferring tool needs to be fixed for avoiding the problem to happen again.

Consider the following example where the directory badDirectory resulted with the wrong settings after being transferred. The ls -la command shows the ownership properties of the directory as mickey-mickey instead of mickey-pawsey9999, which is the right "owner-group" combination of ownership properties. The rest of the directories listed have the correct properties.

Terminal 1. Listing group ownership for a set of files
$ ls -la total 20 drwxrws---+ 5 mickey pawsey9999 4096 Jul 30 16:00 . drwxr-s---+ 17 mickey pawsey9999 4096 Jul 25 09:25 .. drwxrws---+ 2 mickey mickey 4096 Jul 30 16:00 badDirectory drwxrws---+ 2 mickey pawsey9999 4096 Jul 30 15:52 borra drwxrws---+ 4 mickey pawsey9999 4096 Jul 11 15:53 GS-8819-ONETEP

To resolve the problem, execute the chgrp command (change group) with the "-R" option to apply the change recursively through all the content (files and subdirectories) inside the affected initial directory. The general syntax is:

$ chgrp -R <yourProject> <yourAffectedDirectory>

This fixes that group association of the directory named  badDirectory and all the files in it recursively:

Terminal 2. Example of changing the group ownership of a directory
$ chgrp -R pawsey0001 badDirectory $ ls -la total 20 drwxrws---+ 5 mickey pawsey9999 4096 Jul 30 16:00 . drwxr-s---+ 17 mickey pawsey9999 4096 Jul 25 09:25 .. drwxrws---+ 2 mickey pawsey9999 4096 Jul 30 16:00 badDirectory drwxrws---+ 2 mickey pawsey9999 4096 Jul 30 15:52 borra drwxrws---+ 4 mickey pawsey9999 4096 Jul 11 15:53 GS-8819-ONETEP

If the problem has expanded extensively into most of your /group directory, you can fix it by using fix.group.permission.sh, which is provided by the module pawseytools. For more information about this tool, see under "File Permissions and Quota" on the Pawsey Filesystems and their Use page.

Finally, in order to avoid this problem to happen again, configure your file transfer program to honour the setgid (set-group identification) default so that newly created files and directories belong to your project on the "group" property. This is explained for several documented tools in the following subsections.

Your institution firewall may block connections

SSH is enabled on Pawsey systems for both incoming and outgoing traffic. This, however, may not be true for some firewalls on connections on the client side. Most university, business and home internet connections only permit outgoing connections, and have their incoming SSH disabled within their firewall. This means that, SCP is always invoked on the client, that is, your Laptop/Desktop to copy the data to/from the Pawsey supercomputers.

Related Pages