Content Comparison

SCP (Secure Copy Protocol)

The Secure Copy Protocol, or SCP, is a file transfer network protocol used to move files onto servers, and it fully supports encryption and authentication. SCP uses Secure Shell (SSH) mechanisms for data transfer and authentication to ensure the confidentiality of the data in transit.

Syntax

$ scp [options] SourceFileOrDir USER@HOST:DestFileOrDir

$ scp [options] USER@HOST:SourceFileOrDir DestFileOrDir

Here, the name of the "SourceFileOrDir" includes both the path and name of the files/directories to transfer. The "DestFileOrDir" (destination) also includes the path (and possibly the "new" name) of the destination file or directory. "USER" is the username to connect, and "HOST" is the hostname or ip address of the remote computer to connect to. Note the use of the colon (:) separating the HOST and the definition of the path/files on it.

Examples

To transfer the file "/myDir/initialConditions.tar.gz" from a local computer into your personal directory in "/scratch", where "[username]" and "[project]" is your username and project respectively:

Rsync

Rsync is a utility for efficiently transferring and synchronizing files across computer systems, by checking the timestamp and size of files.

Syntax

$ rsync [options] SourceFileOrDir USER@HOST:DestFileOrDir

$ rsync [options] USER@HOST:SourceFileOrDir DestFileOrDir

Here, the name of the "SourceFileOrDir" includes both the path and name of the files/directories to transfer. The "DestFileOrDir" (destination) also includes the path (and possibly the "new" name) of the destination file or directory. "USER" is the username to connect, and "HOST" is the hostname or ip address of the remote computer to connect to. Note the use of the colon (:) separating the HOST and the definition of the path/files on it.

Rsync has a lot of different options that are documented elsewhere. Pawsey recommends to use the following options (to be included in the above syntax):

-vhsrl --chmod=Dg+s -e ssh


The purpose of the recommended options are as follows:

Table 1. Pawsey's recommended options.

Examples

For transferring the local directory in your host computer "~/initialConditions" and all its contents into your personal directory on /scratch, where "[username]" and "[project]" is your username and project respectively:

External references

Documentation about the general use of Rsync can be found elsewhere.

SFTP (Secure File Transfer Protocol)

SFTP is an interactive transfer client that uses SSH to create a secure connection to the server. Its functionality is very similar to FTP, but not all of the FTP options are available. (Not be confused with FTPS (FTP over SSL)).

Open and Close a connection to Pawsey

The following sftp command can be used to establish a connection from your local machine to the Pawsey data-mover nodes:

Note that after establishing a connection, the prompt will change to "sftp>" indicating that the interactive SFTP session has started.

To close the connection, execute the following:

Navigation in the local and remote systems

As for any Linux interactive session, the basic navigation tool for navigating directories in the remote filesystem is cd:

In addition, you can check the current directory on the remote server with pwd and ls:

Within the SFTP interactive session, you can also navigate in your local computer by using the prefix "l" for "local" in the commands:

Copy files to Pawsey filesystems

Once an interactive SFTP connection to the data-mover system at Pawsey has been established, and the current directory in the local and remote systems are the desired ones, users can put files into the remote system by executing the put command:

put [options] SourceFileOrDirInLocalSystem [DestFileOrDirInRemoteSystem]

For example:

Or, if you already navigated (as explained above) into the correct path in the local and the remote computer, then a simple put would be enough:

As the general syntax suggests, there is still freedom to choose source path/files and destination path/file names. The following example takes another file from other directory and puts it into your personal directory in /software, even if the current remote directory was in /scratch:

In order to put an entire directory, the option "-r" needs to be used. In this case, the directory "myScripts" will be transferred to the your personal directory in the /scratch filesystem:

Copy files from Pawsey filesystems

The SFTP command for copying data into the local filesystem is get. Its general syntax is:

get [options] SourceFileOrDirInRemoteSystem [DestFileOrDirInLocalSystem]

External references

Documentation about the general use of SFTP can be found elsewhere.

FileZilla

FileZilla is a fast and reliable file transfer client with an intuitive GUI. It works on multiple platforms (Windows, macOS and Linux) and supports SFTP, which is one of the supported protocols for transferring files to and from Pawsey systems. FileZilla supports simultaneous transfer of multiple files, transfer of large files (>4GB) and transfer resume after connection failure.

Basic setup

• Select Site Manager from the top File menu. A window for the "Site Manager" settings will be presented
• Click on "New Site" on the left panel. Name the site as you want (data-mover is the name used here)
• On the right panel, choose "Protocol" as: "SFTP - Secure File Transfer Protocol"
• In "Host", enter: "data-mover.pawsey.org.au" (Leave "Port" blank)
• In "Logon Type" choose "Key file"
• In "Key file" select the ssh-key you use to connect to Pawsey (~/.ssh/pawsey_ecdsa_key in this example). Pawsey strongly recommends the use of ssh-keys instead of the conventional and less secure use of username/passwords (please read the note above).
  • After selecting the key, Filezilla will convert it into its own format (.ppk).
  • It will also ask for the passphrase of the key and for the name of the new key (we recommend to use the same name as the original key, except for the .ppk extension)
• Click "Connect". Enter your username and your password for accessing Pawsey systems.

Figure 1. FileZilla's Site Manager

Connecting with username/password

It is possible, but is not our recommendation. When using it, we recommend to never save the password within the tool.

Multiple simultaneous transfers

FileZilla supports simultaneous transfer of multiple files, so should be faster than WinSCP or command-line scp if you have many files to transfer. Multiple streams is most useful over long distances, in which case 12 may be useful. Nevertheless, too many streams will create load on the data-mover node, and may reduce performance (and impact others). Over short distances, up to 4 streams may help improve performance while not creating too much load. You can limit the maximum simultaneous connections in the "Transfer Settings" section of the Site Manager settings window shown above.

Automation using scripts

To our knowledge, FileZilla does not allow for automatic transfers through the use of scripts. (See https://superuser.com/questions/239860/how-do-i-send-a-file-with-filezilla-from-the-command-line). But you can always prepare scripts that make use of command-line clients (explained above) or use other tools that allow this (like WinSCP next tab).

WinSCP

WinSCP is an open source free file transfer client for Windows. It has an intuitive GUI and supports SFTP and SCP protocols among others. Its main function is file transfer between a local and a remote computer. Beyond this, WinSCP offers scripting and basic file manager functionality. We recommend to use the SFTP protocol (default) as it allows to resume transfers after a connection interruption.

Basic setup

• Open a "New Session"
• Click "New Site"
• In "File protocol" select "SFTP". You could also use "SCP" if preferred. We recommend SFTP as it supports the resume of truncated transfers and SCP does not, although SCP seems to be faster.
• In "Host name" enter "data-mover.pawsey.org.au" (Leave default for "Port number")
• Enter "User name". Do not enter your "Password"

Figure 2. Setup start window

• Do NOT hit Login yet.
• Click "Advanced" (for setting up the ssh-key authentication). Pawsey strongly recommends the use of ssh-keys instead of the conventional and less secure use of username/passwords (please read the note above).
• On the left tree panel choose "SSH" and then "Authentication"
• Now, the ssh-key you use for connecting into Pawsey needs to be translated into PuTTY format (.ppk). For that click on "Tools" on the right panel (just below the Private key selection box) and then choose "Generate New Key Pair with PuTTYgen...".

Figure 3. Advance site settings (for defining the ssh-key to use)

• In "Load an existing private key" click "Load" and select the ssh-key you use to connect to Pawsey (C:\Users\yourLocalUser\.ssh\pawsey_ecdsa_key in this example). You may need to look for "All Files (*.*)" to see it. Then click "Open".
• Type the passphrase for the ssh-key and click "OK".
• Now you need to save the translated key to the new format. So click "Save private key" and choose an appropriate name with ".ppk" extension (C:\Users\yourLocalUser\.ssh\pawsey_ecdsa_key.ppk in this example). Then click "Save"

Figure 4. PuTTY Key Generator for translating your existing key

• Close the PuTTY Key Generator window on the top right corner "X"
• In the "Private key file", the translated key should be already displayed (C:\Users\yourLocalUser\.ssh\pawsey_ecdsa_key.ppk in this example). If it is not, then browse to select it. Then click "OK" in the general window of "Advanced Site Settings" which will return you to the original settings window titled "Login".
• Now click "Save" button under the "User name". This will open a new window where you can choose a "Site name" for "Saving the session as a site" and click "OK"

 
Figure 5. Save the session as a site.

• Now, on the left panel of the main "Login" window, you can select the "Session Site" you have just configured and finally click "Login" to test the connection.
• You may need to accept cache fingerprints among computers and enter the passphrase for the ssh-key.

Pageant ssh-key agent setup

In order to manage the .ppk key by an agent you will need to add it to Pageant. This tool is part of the tools that can be downloaded from the WinSCP site.

• After downloading, installing and executing Pageant, right click on the Pageant icon (a little computer with a hat).
• Choose "View Keys" from the menu and a window with the current list of keys managed by the agent will appear.
• Then click "Add Key" and select the desired .ppk key

Figure 6. Pageant Key List

• Type the passphrase, which will be remembered by Pagent, so that you don't need to type the passphrase every time you start a session with WinSCP

Basic usage

After a connection has been established, the right panel will initially show the content of the /home/<user> directory. Besides all the navigation options available, the easier to use for the first time is the "Open directory/bookmark" button. This will open a window where you can type the desired path to navigate first. We recommend to first click "Add" to save the bookmark, and then "OK" to navigate into the indicated path.

Figure 7. Directory listings of both the local and remote filesystems.

For transferring files, select and drag, or use the "Upload" and "Download" buttons.

Ending a session

From the top menu choose "Session" and then "Close Session".

Transfer settings (avoid preservation of time stamps)

To avoid the preservation of time stamps:

• From the top menu click "Options" and then "Preferences"
• On the left panel tree choose "Transfer" and then click on "Edit".
• From the many options, unselect the "Preserve timestamp" and check that the permissions coincide with our following recommendation:

Figure 8. Transfer settings and permissions

Connecting with username/password

It is possible, but is not our recommendation. When using it, we recommend to never save the password within the tool.

Cyberduck

Cyberduck is described as "libre server" and cloud storage browser for macOS and Windows. It supports SFTP among many other transfer protocols to remote clusters and cloud platforms. Cyberduck has a simple and practical GUI, although it appears less user-friendly than FileZilla or WinSCP.

Basic setup

• Click the "Open Connection" button at the top of the Cyberduck window.
• Select "SFTP (SSH File Transfer Protocol)" from the drop-down selection box at the top of the window.
• In "Server" enter: "data-mover.pawsey.org.au" (leave default number for "Port")
• In "Username" enter your username for Pawsey systems
• In "Password" leave it blank
• In "SSH Private Key" select the ssh-key you use to connect to Pawsey (~/.ssh/pawsey_ecdsa_key in this example). Pawsey strongly recommends the use of ssh-keys instead of the conventional and less secure use of username/passwords (please read the note above)
• Activate the option: "Add to Keychain". This will save the passphrase of the ssh-key into your keychain and avoid the need for typing the passphrase for each transfer.
• Click connect:
  

Figure 7. SFTP connection settings.

• Provide the passphrase for your ssh-key and select "Save Password" to keep the passphrase registered and avoid the need of typing it for each transfer.

Figure 8. Passphrase to your ssh-key

• Accept the SSH fingerprint by selecting "Allow". Here you can tick to "Always" remember the key.

Figure 9. Accept SSH fingerprint.

Connecting with username/password

It is possible, but is not our recommendation. When using it, we recommend to never save the password within the tool.

Basic use

The navigation window of Cyberduck shows the directory structure of the server you have connected to. It is handy to open an additional window of your preferred navigation GUI on your own computer to be able to display the files and directory structure on your local filesystem.

Transfers can be performed by selecting files and the dragging the selection from one window to the other.

On the Cyberduck window, you can also select files and right click to choose an operation on them.