Excerpt
There are multiple filesystems mounted to each of Pawsey's supercomputers. Each of these filesystems are designed for particular use cases. This page provides a detailed description of these filesystems.

...

/home - which should be used to store software configuration files that cannot be easily located elsewhere.
/software - Lustre filesystem which should contain both Pawsey and researcher software installations and Slurm batch scripts.
/scratch - Lustre filesystem which should contain working data in use by jobs that are actively queued and running on the supercomputer
/astro - Lustre filesystem which supports operational radio astronomy observatory work

These filesystems can be viewed using the df command from the login nodes:

...

Apart from /home, all are Lustre distributed filesystems. Lustre is an open-source, high performance parallel file system optimised for high throughput.

...

Note

icon	false
title	Migration: Filesystems

While Pawsey is migrating to Setonix, there are existing filesystems still in use by Garrawarla.

The existing /astro filesystem, which will be replaced by the new /scratch filesystem on Setonix.

The filesystems are different in many ways and are designed to facilitate different activities in supercomputing. The intended usage for The filesystems are different in many ways and are designed to facilitate different activities in supercomputing. The intended usage for each of them is explained below. Use outside of these purposes may cause poor performance for a particular activity as well as create detrimental impacts to other users.

...

Column

Note
The `/home` filesystem is intended to be used to store relatively small numbers of important system files such as your Linux profile and shell configuration. It is not suitable for launching jobs.

Current usage of the /home filesystem can be viewed by executing the quota command:

...

Insert excerpt

	Visual Studio Code for Remote Development
	Visual Studio Code for Remote Development
nopanel	true

Further explanation about quotas, permissions and copy (cp) vs move (mv) of files and directories is given in the sections below.

Software filesystem

The /software filesystem is a Lustre file system with much higher throughput than /home. It is intended for software installations and Slurm batch script templates. Each project has an associated directory on the filesystem whose path is /software/projects/<project>. Within a project directory, each project member has his or her own directory whose full path, /software/projects/<project>/<username>, is contained in the MYSOFTWARE environment variable.

...

Column

Tip

title	Use Acacia to store your data

The scratch file system is not intended for long-term storage, is not backed up and is purged on a regular basis. If you wish to retain files, move them to the Acacia object storage.

Warning

title	Warning:

Files which have not been accessed for the purge period of 30 21 days will be deleted automatically and WILL BE LOST. See Filesystem Policies.
(Note that this was changed since Monday 10th of June 2024, removing the previous rule of 1-month)

Warning

title	Do not Never use the `touch` command to avoid purge

The use of the touch command to avoid purging generates overloading of the metadata servers of the /scratch filesystem. Remember that /scratch is shared among all users. Therefore, users should respect the best practices to avoid overloading of metadata servers at all times. Overloading of the metadata servers dramatically degrade performance affecting all users at the same time. Pawsey reserves the right to revoke access to users that do not respect this best practice. See Filesystem Policies.

As mentioned above, users should incorporate regular file movement from /scratch into Acacia for long-term storage.

...

Leaving files to be removed by the 3021-day purge policy places an unnecessary load on the filesystem as the system is scanned for these files, and causes less capacity to be available for other users.

...

These reference datasets will be exempt from the /scratch purge policy.

The specific Some of the bioinformatics reference datasets available are:

...

Files created in a user's /home are accessible only to that user.
Files created in a user's /software , /scratch or /astro and /scratch directories are accessible only to that user and to members of the same project.

...

Column

Table 1. Pawsey filesystems: capacity, file limit and duration

Filesystem	Capacity Limit	File Limit	Duration
`/home`	1 GB per user	10k files per user	Active project allocation
`/software`	256 GB per project	100k files per user	Active project allocation
`/scratch`	1 PB per project	1M files per user	30 21 days from last modification (see Filesystem Policies)

The default group membership for files and directories that are created in /home is the user's primary group, which is the same as their user ID. Files and directories that are created in any of the Lustre filesystems are associated by default with the user's project ID.

...

A user is always a member of their own primary group (which is the same as their own username) and can also be a member of more than one project. This is important to know because files created with a group associated to a username rather than a project are limited to a default quota of 1GB and there can be at most 100 of them.

Column

width	900px

Warning
If you encounter a write error, compiler error, or file transfer error on the `/scratch` or `/software` filesystems, this is likely because the files are counting against your personal group quota rather than your project's group quota.

Tip

You should proactively and regularly monitor both file count and quota usage across the filesystems. This practice will reduce your likelihood of hitting the quota limits; whenever this happens, no files can be written until usage is brought back below quota.

As regards the /home filesystem, regularly check for and clean unneeded files, which may be generated by software as temporary or cache files. These are often stored in hidden directories (their name starts with a dot).

File permissions and ownerships are also important to consider. Here are the The default permissions of a file named myscript.sh that was created in a user's home directoryfiles created by a user on any of the Setonix filesystems is the same, but the default ownerships are different. An example of default properties of a file in /home filesystem is as below:

Column

width	900px

Code Block

language	bash
theme	DJango
title	Terminal 4. List the default file permissions and ownerships for a file created in on Setonix /home

$ ls -ld myscript.sh
-rwxr-xr-xrw-r--r-- 1 username username 2 Nov 30 16:33 myscript.sh

Recall that Linux file permissions are broken down into three groups of three:

...

rwx

...

The first set of permissions determines what actions can be performed by the owner of the file. In this case username is the owner, and is allowed to read (r), write (w), and execute the file (x).

...

r-x

...

Note that there are 10 characters describing the permissions. The first character is not really a permission, but an indication of the type "file". So a "-" in the first character indicates that myscript.sh is indeed a file. (A "d" would indicate it is not a file but a directory, an "l" would indicate it is a link, etc.). The rest nine characters indicate the permissions of the file, and these permissions are broken down into three groups of three:

`-`	The `-` in the first character indicates this is a file.
rwx	The first set of permissions determines what actions can be performed by other users who belong to the same group as the owner of the file. The group here is the primary or default group of the file's owner, which is `username`. Group members are allowed to read and execute.	r-x	The final set of permissions apply to all other users. While the permissions are set to read and execute, the top-level user directory ( `/home/username` ) is locked to just the user, so no others are In this case `username` is the owner, and is allowed to read (r), write (w), and execute the file (x).
r--	The second set of permissions determines what actions can be performed by other users who belong to the same group as the file. The group here is the primary or default group of the file's owner, which is `username`. Group members are allowed to read only.
r--	The final set of permissions apply to all other users. While the permissions are set to read, the top-level user directory is locked to just the user or project, so others are not able to read, write, or execute files in another usergroup's home directorydirectories.

Now look at the difference for An example of default properties of a file created in /softwarefilesystem is as below:

Column

width	900px

Code Block

language	bash
theme	DJango
title	Terminal 5X. List the default file permissions for a file created in on Setonix /software & /scratch

$ ls -ld swscriptotherscript.sh
-rwxr-xr-xrw-r--r-- 1 username projectgroup 2 Nov 30 16:5133 swscriptotherscript.sh

The file So, as said, default permissions are the same as before, but with different group ownership: projectgroup. Other members of projectgroup will be able to read and execute this script. Similar to myscript.sh , the permissions for "all other users" are set to read and execute (r-x), but the top level group directory ( /software/projects/swgroup ) is locked to just the group so that others who are not in the group cannot access any files within it:, but not the default ownerships (the next two words after the permissions).

The next two words after the permissions are, respectively, the owner-name and the group-name of the file. In the case of files created in /home, both the owner and the group are assigned to the username by default. And in the case of files created in /software and /scratch, the default for the group-name is the projectgroup for your project.

Column

width	900px

Code Block

language	bash
theme	DJango
title	Terminal 6. Group ownership of a file

$ ls -ld /software/projects/projectgroup
drwxrws--- 46 root projectgroup 4096 Nov 29 09:06 /software/projects/projectgroup

Note there is a new flag in the group permissions, the SETGID flag (s). With the SETGID flag set on the directory, whenever a user creates a new file under /software/projectgroup, the group ownership is set to the same as the group owner of the directory, as opposed to setting the group ownership to the user who created it. So, in the example above, any file created under /software/projectgroup will have a group ownership of projectgroup instead of username.

The SETGID flag on your project's group directory is set when Pawsey staff first set up the new project so there's no need for users to modify this. However, there are situations where a user might accidentally modify permissions or ownership when moving files. For example, if a user moves a file from /home to /software (instead of copying it) the group ownership is not changed:

Column

width	900px

Tip

It is important to note because the quota limits explained in the section above are really counted on the group-name of the files and not the owner-name. Then, the quota limit of all the files that have groupname = username (added among all our filesytems) is 10k files, which is very limited. This limitation has frequently hit the quota limit for users that do not follow our recommendations for installing software, because some installation tools/commands may override the default group-name of /scratch and software and assign it as username instead of projectgroup. (Check the related pages at the bottom of this page.)

Warning

For the same reason, when transferring files between filesystems, we do not recommend the use of the mv (move) command, because this command preserves the original group-name. Instead, we recommend the use of the cp command (or other tools that allow the assignment of the default ownerships for the destination filesystem). When using cp, do not use the -a or -p flags. If you want to preserve timestamps, use cp --preserve=timestamps. For example, considering the files in $HOME as above:

$ echo "foo" > foo.txt

Code Block

language	bash
theme	DJango
title	Terminal 7. Group ownership after moving a file from /home to /software

X. Do not use mv when transferring files to another filesystem

$ ls -ld foo$HOME/myscript.txtsh
-rw-r--r-- 1 username username 42 Nov 2930 17:02 foo.txt16:33 myscript.sh

$ cp $HOME/myscript.sh $MYSOFTWARE/myscript_cp.sh        #good practice
$ mv foo$HOME/myscript.txtsh $MYSOFTWARE/myscript_mv.sh        #bad  practice

$ ls -ld $MYSOFTWARE/foo.txtmyscript*
-rw-r--r-- 1 username usernameprojectgroup 46 NovDec 2910 1710:03 /software/projects/projectgroupproject/username/foo.txt

In terminal 5, foo.txt was created in a directory on /home. As a result, the group ownership is set to user's group ( username ). The file was then moved it to the /software filesystem, and you can see that the original permissions and group remained. The file foo.txt will count against the user's quota instead of the project, even though it is located in /software.

The solution is to use the copy command (cp) instead of move (mv) when transferring files from /home to /software. This is because cp actually creates a new file, which inherits the SETGID flag from the top-level group directory:

...

width	900px

Code Block

language	bash
theme	DJango
title	Terminal 8. Group ownership after copying a file from /home to /software

$ echo "bar" > bar.txt
$ ls -ld bar.txt
-rw-r--r-- 1 username username 4 Nov 29 17:05 bar.txt
$ cp bar.txt $MYSOFTWARE
$ ls -ld $MYSOFTWARE/bar.txt
-rw-r--r-- 1 username projectname 4 Nov 29 17:06 /software/projects/projectname/username/bar.txt

When transferring files between filesystems, you will see the above behaviour and require this workaround. When using cp, do not use the -a or -p flags. If you want to preserve timestamps, use cp --preserve=timestamps.

File transfer programs like WinSCP can also cause issues with permissions and groups. You should consult the documentation of your preferred transfer program. rsync users should avoid using the -a and -p flags; these flags will preserve permissions of the source files, which may conflict with the default behaviour on Pawsey systems. Some additional information about file transfer programs is at: Transferring Files in/out Pawsey Filesystems.

Pawsey provides a tool that lets you fix file and directory permissions on /software. The fix.group.permission.sh script is available in the pawseytools module, which is loaded by default. To use it, enter the script name followed by your group name. For example, if your project ID is projectgroup you would enter this:

$ fix.group.permission.sh projectgroup

...

Note

title	Notes:

This script might take some time to complete.
It will only fix files and directories owned by the user executing the command ($USER).
You can only run one instance of the script at a time.

There is a manual way of doing this in your own area using the find command. Replace projectgroup with your project ID and username with your user name.

...

width	900px

Code Block

language	bash
theme	DJango
title	Terminal 9. Fix file and directory permissions on /software

$ find /software/projects/projectgroup/username ! -group projectgroup -exec chgrp projectgroup \{} \;
$ find /software/projects/projectgroup/username -type d ! -perm /g=s -exec chmod g+s \{} \;

Code Block

language	bash
theme	DJango
title	Terminal 10. Fix file and directory permissions on /scratch

$ find /scratch/projectgroup/username ! -group projectgroup -exec chgrp projectgroup \{} \;
$ find /scratch/projectgroup/username -type d ! -perm /g=s -exec chmod g+s \{} \;

The extra tests for the find commands in terminal 7 and terminal 8 speed up the process for many files, by only changing files and directories that need to be changed.

Astronomy filesystem

The Astonomy Filesystem /astro is a Lustre filesystem provided for the scratch space needs of the MWA group who perform computations on the Garrawarla cluster.

It is an SGI/HPE provided cluster of nodes backed by DDN storage. The system currently contains 2 Metadata servers (MDS) with 2 Metadata targets (MDT). It has 4 Object Store servers (OSS) for storing data and they have 48 Object Store targets (OST). This gives approximately 2.7 PB of usable storage. It has a possible read and write speed of over 10GB/s and Pawsey staff have been easily getting 7-8GB/s when using only the four copyq nodes to transfer data around using dcp. The single threaded IO speed is greater than on /scratch due to the updated version of Lustre.

The expandability of Lustre means that the filesystem can be expanded, without downtime, by adding more OSS's and OST Disk behind them in groups of 2 (for high availability).

Location

The Astronomy filesystem is mounted on all Garrawarla nodes and Setonix data mover nodes as /astro. The top level directory has directories for all the areas that /astro has:

...

width	900px

Code Block

language	bash
theme	DJango
title	Terminal 11. /astro directories

$ ls -l /astro/
total 20
drwxrws--- 36 root mwaeor     4096 Oct 28 09:52 mwaeor
drwxrws--- 12 root mwaops     4096 Sep 30 16:31 mwaops
drwxrws--- 47 root mwasci     4096 Dec 11 16:56 mwasci
drwxrwsr-x 57 root mwavcs     4096 Dec 16 16:46 mwavcs
drwxrws--- 27 root pawsey0001 4096 Dec 17 12:22 pawsey0001

The pawsey0001 directory is for Pawsey testing of the system and can be set up in different ways as needed. It will not often be used.

Quotas

At the time of writing MWA have requested that mwaeor, mwavcs, mwaops and mwasci are assigned 370 TB, 580 TB, 20 TB and 600 TB respectively.

To check the current quota use the following command:

$ lfs quota -g projectcode /astro

Usage

To check usage you can use the df command to check the entire filesystem. This command gives a breakdown by OST and a summary at the bottom.

...

width	900px

Code Block

language	bash
theme	DJango
title	Terminal 12. Using the df command to check usage

$ lfs df -h /astro/
UUID                       bytes        Used   Available Use% Mounted on
astrofs-MDT0000_UUID      542.1G       26.0G      479.4G   6% /astro[MDT:0]
astrofs-MDT0001_UUID      542.1G       32.7G      472.7G   7% /astro[MDT:1]
astrofs-OST0000_UUID       57.7T       26.6T       28.1T  49% /astro[OST:0]
astrofs-OST0001_UUID       57.7T       26.4T       28.4T  49% /astro[OST:1]
astrofs-OST0002_UUID       57.7T       25.9T       28.8T  48% /astro[OST:2]
astrofs-OST0003_UUID       57.7T       28.6T       26.2T  53% /astro[OST:3]
astrofs-OST0004_UUID       57.7T       26.9T       27.9T  50% /astro[OST:4]
astrofs-OST0005_UUID       57.7T       26.6T       28.2T  49% /astro[OST:5]
astrofs-OST0006_UUID       57.7T       26.3T       28.4T  49% /astro[OST:6]
astrofs-OST0007_UUID       57.7T       26.8T       28.0T  49% /astro[OST:7]
astrofs-OST0008_UUID       57.7T       26.8T       27.9T  50% /astro[OST:8]
astrofs-OST0009_UUID       57.7T       26.3T       28.5T  48% /astro[OST:9]
astrofs-OST000a_UUID       57.7T       26.6T       28.2T  49% /astro[OST:10]
astrofs-OST000b_UUID       57.7T       26.8T       28.0T  49% /astro[OST:11]
astrofs-OST000c_UUID       57.7T       25.6T       29.2T  47% /astro[OST:12]
astrofs-OST000d_UUID       57.7T       27.1T       27.6T  50% /astro[OST:13]
astrofs-OST000e_UUID       57.7T       27.0T       27.8T  50% /astro[OST:14]
astrofs-OST000f_UUID       57.7T       26.5T       28.3T  49% /astro[OST:15]
astrofs-OST0010_UUID       57.7T       37.3T       17.5T  69% /astro[OST:16]
astrofs-OST0011_UUID       57.7T       38.1T       16.7T  70% /astro[OST:17]
astrofs-OST0012_UUID       57.7T       37.5T       17.3T  69% /astro[OST:18]
astrofs-OST0013_UUID       57.7T       38.0T       16.7T  70% /astro[OST:19]
astrofs-OST0014_UUID       57.7T       38.0T       16.7T  70% /astro[OST:20]
astrofs-OST0015_UUID       57.7T       37.1T       17.6T  68% /astro[OST:21]
astrofs-OST0016_UUID       57.7T       37.1T       17.6T  68% /astro[OST:22]
astrofs-OST0017_UUID       57.7T       38.0T       16.7T  70% /astro[OST:23]
astrofs-OST0018_UUID       57.7T       37.4T       17.4T  69% /astro[OST:24]
astrofs-OST0019_UUID       57.7T       37.7T       17.1T  69% /astro[OST:25]
astrofs-OST001a_UUID       57.7T       38.3T       16.5T  70% /astro[OST:26]
astrofs-OST001b_UUID       57.7T       37.4T       17.4T  69% /astro[OST:27]
astrofs-OST001c_UUID       57.7T       37.5T       17.2T  69% /astro[OST:28]
astrofs-OST001d_UUID       57.7T       37.1T       17.7T  68% /astro[OST:29]
astrofs-OST001e_UUID       57.7T       37.8T       17.0T  70% /astro[OST:30]
astrofs-OST001f_UUID       57.7T       37.8T       16.9T  70% /astro[OST:31]
astrofs-OST0020_UUID       57.6T       34.6T       20.2T  64% /astro[OST:32]
astrofs-OST0021_UUID       57.6T       34.3T       20.5T  63% /astro[OST:33]
astrofs-OST0022_UUID       57.6T       34.9T       19.9T  64% /astro[OST:34]
astrofs-OST0023_UUID       57.6T       33.6T       21.1T  62% /astro[OST:35]
astrofs-OST0024_UUID       57.6T       33.5T       21.2T  62% /astro[OST:36]
astrofs-OST0025_UUID       57.6T       35.0T       19.7T  64% /astro[OST:37]
astrofs-OST0026_UUID       57.6T       33.7T       21.0T  62% /astro[OST:38]
astrofs-OST0027_UUID       57.6T       34.1T       20.6T  63% /astro[OST:39]
astrofs-OST0028_UUID       57.6T       33.5T       21.2T  62% /astro[OST:40]
astrofs-OST0029_UUID       57.6T       33.6T       21.1T  62% /astro[OST:41]
astrofs-OST002a_UUID       57.6T       34.2T       20.5T  63% /astro[OST:42]
astrofs-OST002b_UUID       57.6T       33.8T       21.0T  62% /astro[OST:43]
astrofs-OST002c_UUID       57.6T       34.9T       19.8T  64% /astro[OST:44]
astrofs-OST002d_UUID       57.6T       34.1T       20.7T  63% /astro[OST:45]
astrofs-OST002e_UUID       57.6T       33.8T       20.9T  62% /astro[OST:46]
astrofs-OST002f_UUID       57.6T       34.5T       20.2T  64% /astro[OST:47]
 
filesystem_summary:         2.7P        1.5P        1.0P  60% /astro

Related pages

...

myscript_cp.sh
-rw-r--r-- 1 username username     2 Nov 30 16:33 /software/projects/project/username/myscript_mv.sh

File transfer programs like WinSCP can also cause issues with permissions and groups. You should consult the documentation of your preferred transfer program. rsync users should avoid using the -a and -p flags; these flags will preserve permissions of the source files, which may conflict with the default behaviour on Pawsey systems. Some additional information about file transfer programs is at: Transferring Files in/out Pawsey Filesystems.

Pawsey provides a tool that lets you fix file and directory ownerships on /software. The fix.group.permission.sh script is available in the pawseytools module, which is loaded by default. To use it, enter the script name followed by your group name. For example, if your project ID is projectgroup you would enter this:

$ fix.group.permission.sh projectgroup

Column

Note

title	Notes:

This script might take some time to complete.
It will only fix files and directories owned by the user executing the command ($USER).
You can only run one instance of the script at a time.

There is a manual way of doing this in your own area using the find command. Replace projectgroup with your project ID and username with your user name.

Column

width	900px

Code Block

language	bash
theme	DJango
title	Terminal 5. Fix file and directory permissions on /software

$ find /software/projects/projectgroup/username ! -group projectgroup -exec chgrp projectgroup \{} \;
$ find /software/projects/projectgroup/username -type d ! -perm /g=s -exec chmod g+s \{} \;

Code Block

language	bash
theme	DJango
title	Terminal 6. Fix file and directory permissions on /scratch

$ find /scratch/projectgroup/username ! -group projectgroup -exec chgrp projectgroup \{} \;
$ find /scratch/projectgroup/username -type d ! -perm /g=s -exec chmod g+s \{} \;

Related pages

External links

Lustre home page

...

Versions Compared

Old Version 30

New Version Current

Key

Software filesystem

Astronomy filesystem

Location

Quotas

Usage

Related pages

Related pages

External links

Page Comparison

Versions Compared

Old Version 30

New Version Current

Key

Software filesystem

Astronomy filesystem

Location

Quotas

Usage

Related pages

Related pages

External links