These policies ensure that every user can efficiently take advantage of Pawsey filesystems.
Staff access to project directories
Pawsey's supercomputing specialists have access to directories of a project, such as /scratch/<projectcode>
. This streamlines user support, reduces the time required to resolve Helpdesk queries, and eliminates the need for users to email large files to Pawsey staff.
Pawsey supercomputing specialists do not have access to a user's home directory (/home/<username>
) or to a project's storage allocation on Acacia.
When raising a request for help, users can choose not to let helpdesk staff access project directories by clearly stating in the ticket that access to project files is not permitted. Note that this could increase the time taken to resolve technical queries or limit the supercomputing experts' ability to assist.
/scratch purge policy
Files on the /scratch filesystem are deleted automatically after 30 days from their last access. This action is referred to as purging. It is part of a dynamic allocation system Pawsey implements to maximise the availability of space for large-scale, data-intensive computations.
You can check your usage of the /scratch filesystem in terms of number of files and kilobytes. Run the following command:
$ lfs quota /scratch
$ lfs quota /scratch
Disk quotas for user aelwell (uid 20701):
Filesystem kbytes quota limit grace files quota limit grace
/scratch 1460 0 0 - 14 0 0 -
The /scratch filesystem is a shared resource that suffers low performance when the number of files increases too much. Users should actively remove their files if no longer in use, rather than waiting for the system to delete them.
To prevent performance impacts on other users, there are limits of 1 PB capacity per project and 1 million files per user on /scratch.
In exceptional circumstances, if the /scratch filesystem is close to being full, files that are less than 30 days old may be deleted. However, in such circumstances, Pawsey contacts users by email to advise of the situation and to give them time to transfer their data files to other storage facilities.
Data should be moved out of the scratch filesystem as soon as is practically possible, into longer-term storage such as Acacia or third-party storage facilities.
With a purge policy, users are able to use more of the scratch system when they need it. For example, if projects A and B each require 500 TB of space to run one big job, Project A can use that space at one time, and project B can use that same disk space at another time (when the data for Project A has been removed by the user or purged by the system). Under an allocation policy, these two projects together would need to be allocated half of the available resources, leaving little for the rest of the projects on the machine.
Many other supercomputing facilities also utilise a scratch purge policy, including CSCS (Swiss National Supercomputing Centre), NERSC (Lawrence Berkeley Laboratory, USA) and OLCF (Oak Ridge Leadership Computing Facility, USA).
/home usage
Each user has exclusive access to the directory /home/$USER, which intended for user-specific configuration files such as login profiles and shell configurations. The /home filesystem is implemented with a Network File System (NFS). Each user may only store up to 1 GB of data and up to 10,000 inodes (that is, a limit on the number of files and directories). You can use the command quota -s to check whether you are approaching those limits. The /home filesystem has a significantly lower capacity and performance compared to /scratch, hence it is not suitable to execute batch scripts or to store data or software installations.
Do not add software environments commands, such as module load and setting environment variables, in login profiles (for instance, .bashrc). Instead, add those commands in batch scripts that are submitted to the scheduler. This results in more portable and reproducible workflows that can be shared more easily with colleagues and Pawsey staff that may be providing assistance.
/software usage
The /software filesystem provides configuration files, batch script templates and system-wide software installations.
For the Pawsey supported software stack, users should not access these files directly. The module system should be used to add the relevant directories to the shell environment. See the Modules documentation page for more details.
Each project has access to the directory /software/projects/<project-code>, intended for project-specific software installations and batch script templates. There is a limit of 256 GB per project and 100,000 files per user on /software.
Related pages
External links
- CSCS (Swiss National Supercomputing Centre)
- NERSC (Lawrence Berkeley Laboratory, USA)
- OLCF (Oak Ridge Leadership Computing Facility, USA)