Use of Acacia object storage

Acacia object storage has been designed to store research data for the whole duration of your project. Therefore, when you need to transfer data in/out Pawsey, it is very likely that Acacia be the first choice.

Acacia is not a filesytem, but an object storage. Once stored in Acacia, data is immutable. So, in order to interact with your data, it should be first transferred from Acacia into an adequate Pawsey filesystem and then be used for supercomputing, visualisation or data analysis jobs. The new results and data generated from these jobs can be also transferred from the Pawsey filesystem and stored in Acacia.

Therefore, it is common to transfer data between 1) Acacia and your own/institutional computer/cluster, and 2) Acacia and Pawsey filesystems. Both kind of transfers are performed by tools/clients which are compatible with Amazon S3 protocol. In-depth description of the different tools/clients for accessing and transferring data in/out Acacia is in the /wiki/spaces/DATA/pages/54459526.

Use of filesystems

Even if Acacia may usually be the first choice of transfer of data in/out Pawsey, users will still need to transfer in/out a variety of files directly between their own/institutional computer/cluster and the Pawsey filesystems. Such needs may be the uploading of code/software source files into /software for compilation, or the downloading of result files recently created in a working directory in /scratch towards your own local computer for your own needs.

When transferring data into Pawsey filesystems (either from users own/institutional computer/cluster or from Acacia) users should adhere to their correct use:

• /home - filesystem to store configuration files. NOT for software installations.
• /software - filesystem to store project-specific software installations.
• /scratch - filesystem to store temporary working data. A 30 day purge policy applies to this filesystem.
• /astro - filesystem to store temporary working data for MWA astronomy project.

Remember that none of the Pawsey filesystems are designed for storing project data for the whole duration of your project.

A description of the different tools/commands to perform transfers is in the Transferring Files in/out Pawsey Filesystems page.

Use of the data-mover nodes

Pawsey provides dedicated data-mover nodes for the transfer of data into, between, and out of, its storage systems.
These nodes are distinct from login nodes, ensuring that the load incurred in transferring large files or large amounts of data will not impact other users.
Pawsey's POSIX filesystems, such as /scratch, /software and /home, and its object storage, Acacia, are all accessible from these data-mover nodes.
The individual data-mover nodes are accessible, interactively, via SSH login to the generic hostname: data-mover.pawsey.org.au, or through the copy partition for batch processing through the Slurm scheduler.