...
Before you begin, review the recommendations and policies in the following pages:
- HPC Filesystems on the Resource Overview page (mainly the "Filesystems" and "Object Storage" sections)
- Pawsey Filesystems and their UsageUse
- Filesystem Policies
The /scratch
filesystem should be used user programs to write and read files, especially the large ones, during a job. This filesystem is tuned to deliver high bandwidth for data access. Datasets should not be stored on /scratch
after a job completes. Unnecessary data should be removed from /scratch
and important data should be copied to a more appropriate place, for example the Acacia object storage, local institutional storage or long term storage. This process can be automated with the use of the job dependencies feature of the Slurm scheduler.
An example involving the staging of data from Acacia, executing a supercomputing job and storing results back into Acacia is available in the Acacia Workflow Examples /wiki/spaces/DATA/pages/54459952.