Use of Acacia object storage
Acacia object storage has been designed to store research data for the whole duration of your project. Therefore, when you need to transfer data in/out Pawsey, it is very likely that Acacia be the first choice.
Acacia is not a filesytem, but an object storage. Once stored in Acacia, data is immutable. So, in order to interact with your data, it should be first transferred from Acacia into an adequate Pawsey filesystem and then be used for supercomputing, visualisation or data analysis jobs. The new results and data generated from these jobs can be also transferred from the Pawsey filesystem and stored in Acacia.
Therefore, it is common to transfer data between 1) Acacia and your own/institutional computer/cluster, and 2) Acacia and Pawsey filesystems. Both kind of transfers are performed by tools/clients which are compatible with Amazon S3 protocol. In-depth description of the different tools/clients for accessing and transferring data in/out Acacia is in the Acacia - User Guide.
Use of filesystems
Even if Acacia may usually be the first choice of transfer of data in/out Pawsey, users will still need to transfer in/out a variety of files directly between their own/institutional computer/cluster and the Pawsey filesystems. Such needs may be the uploading of code/software source files into /software for compilation, or the downloading of result files recently created in a working directory in /scratch towards your own local computer for your own needs.
When transferring data into Pawsey filesystems (either from users own/institutional computer/cluster or from Acacia) users should adhere to their correct use:
/home- filesystem to store configuration files. NOT for software installations./software- filesystem to store project-specific software installations./scratch- filesystem to store temporary working data. A 30 day purge policy applies to this filesystem./astro- filesystem to store temporary working data for MWA astronomy project.
Remember that none of the Pawsey filesystems are designed for storing project data for the whole duration of your project.
A description of the different tools/commands to perform transfers is in the Transferring Files in/out Pawsey Filesystems page.
Use of the data-mover nodes
Pawsey supercomputers count with data-mover nodes for the transfer of data in and out Pawsey filesystems. These nodes are accessible through the generic hostname: data-mover.pawsey.org.au, or through the copy partition for batch processing through the scheduler. These nodes are distinct from the login nodes, allowing that the load for transferring large files or a large amount of data will not impact other users. All standard filesystems, such as /scratch, /software and /home, and Acacia object storage are accessible from these data-mover nodes.