...
Therefore, it is common to transfer data between 1) Acacia and your own/institutional computer/cluster, and 2) Acacia and Pawsey filesystems. Both kind of transfers are performed by tools/clients which are compatible with Amazon S3 protocol. In-depth description of the different tools/clients for accessing and transferring data in/out Acacia is in the Acacia - User Guide /wiki/spaces/DATA/pages/54459526.
Use of filesystems
Even if Acacia may usually be the first choice of transfer of data in/out Pawsey, users will still need to transfer in/out a variety of files directly between their own/institutional computer/cluster and the Pawsey filesystems. Such needs may be the uploading of code/software source files into /software
for compilation, or the downloading of result files recently created in a working directory in /scratch
towards your own local computer for your own needs.
...
Use of the data-mover nodes
Pawsey supercomputers count with provides dedicated data-mover nodes for the transfer of data in and out Pawsey filesystems. These nodes are accessible through the generic hostname: data-mover.pawsey.org.au
, or through the copy
partition for batch processing through the scheduler. into, between, and out of, its storage systems.
These nodes are distinct from the login nodes, allowing ensuring that the load for incurred in transferring large files or a large amount amounts of data will not impact other users. All standard
Pawsey's POSIX filesystems, such as /scratch
, /software
and /home
, and Acacia its object storage, Acacia, are all accessible from these data-mover nodes.
The individual data-mover nodes are accessible, interactively, via SSH login to the generic hostname: data-mover.pawsey.org.au
, or through the copy
partition for batch processing through the Slurm scheduler.
Section | |||||||||
---|---|---|---|---|---|---|---|---|---|
|