Versions Compared

Old Version 26

changes.mady.by.user Alexis Espinosa

Saved on

compared with

New Version 27

changes.mady.by.user Alexis Espinosa

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Excerpt

There are multiple filesystems mounted to each of Pawsey's supercomputers. Each of these filesystems are designed for particular use cases. This page provides a detailed description of these filesystems.

...

Column


Panel
titleOn this page:

Table of Contents


Overview

The following filesystems are available from one or more Pawsey supercomputing systems:

...

Apart from /home, all are Lustre distributed filesystems. Lustre is an open-source, high performance parallel file system optimised for high throughput. 

Column


Note
iconfalse
titleMigration: Filesystems

While Pawsey is migrating to Setonix, there are existing filesystems still in use by Garrawarla.

  • The existing /astro filesystem, which will be replaced by the new /scratch filesystem on Setonix.


...

Column


Warning

Pawsey filesystems are not backed up. Ensure you have a backup of your important files.


Home filesystem

The home filesystem should be used to store software configuration files. It is a Networked FileSystem (NFS). Each user has a default login directory in the /home filesystem with a quota of 1 GB and 10,000 individual files.

...

Due to its small quota limit and low performance, the /home filesystem is not suitable for launching or storing production work. Files such as software installations and Slurm batch scripts should be stored on the /software filesystem. Working data, such as job input and output, should use the /scratch file system.

What to do if you exceeded your quota

First thing to do is to identify those directories that contain a large number of files or those files that are too large and are consuming your quota. Then delete them.

Identifying subdirectories with a large number of files

You can use the following command that finds the subdirectories recursively and list them in descending order of containing files. Execute this command from your $HOME directory:

...

Then you can check the file $MYSCRATCH/homeSubdirectoriesRanked.out and decide what subdirectories to remove. Note that the output is written in $MYSCRATCH because you may have not enough quota to write in $HOME.

Identifying large files

Column


Code Block
languagebash
themeDJango
titleTerminal X. find command to search for the number of containing files in subdirectories
$ cd $HOME
$ find . -type f -exec du -h {} + | sort -rh | head -n 10


Then you can decide which files to remove. Note that you could have used the last filter (head -n 10) also in the previous command to avoid a large output of lines, or you could have used here the same final filters as in the previous command in order to save output into a file for a later careful check.

Hidden files 

Insert excerpt
Visual Studio Code for Remote Development
Visual Studio Code for Remote Development
nopaneltrue

Software filesystem

The /software  filesystem is a Lustre file system with much higher throughput than /home. It is intended for software installations and Slurm batch script templates. Each project has an associated directory on the filesystem whose path is /software/projects/<project>. Within a project directory, each project member has his or her own directory whose full path, /software/projects/<project>/<username>, is contained in the MYSOFTWARE environment variable.

...

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 3. Checking the per-user quota
$ lfs quota -u $USER -h /software
Disk quotas for usr user1234 (uid xxxx):
     Filesystem    used   quota   limit   grace   files   quota   limit   grace
      /software  14.16G      0k      0k       -   49053       0  100000       -



Scratch filesystem

The scratch filesystem should be used for working data, which is input and output files actively used by jobs queued or running on the supercomputer.

...

Anchor
ReferenceData
ReferenceData

Reference datasets

Reference data sets are static data required by software for calibrations or testing or as widely used input data. Reference data sets that are used by several project groups will be provided on /scratch by Pawsey to avoid multiple copies existing. These data sets will be contained in subdirectories of /scratch/references.

...

If you would like to request addition of a new reference dataset, please email the Pawsey Helpdesk help@pawsey.org.au 

File permissions and quota

The effect of file permissions and ownership on storage quotas varies depending on which filesystem the data is located. The default behaviour can be summarised as such:

...

The extra tests for the find commands in terminal 7 and terminal 8 speed up the process for many files, by only changing files and directories that need to be changed.

Astronomy filesystem

The Astonomy Filesystem /astro is a Lustre filesystem provided for the scratch space needs of the MWA group who perform computations on the Garrawarla cluster.

...

The expandability of Lustre means that the filesystem can be expanded, without downtime, by adding more OSS's and OST Disk behind them in groups of 2 (for high availability).

Location

The Astronomy filesystem is mounted on all Garrawarla nodes and Setonix data mover nodes as /astro. The top level directory has directories for all the areas that /astro has:

...

The pawsey0001 directory is for Pawsey testing of the system and can be set up in different ways as needed. It will not often be used.

Quotas

At the time of writing MWA have requested that mwaeor, mwavcs, mwaops and mwasci are assigned 370 TB, 580 TB, 20 TB and 600 TB respectively.

...

$ lfs quota -g projectcode /astro

Usage

To check usage you can use the df command to check the entire filesystem. This command gives a breakdown by OST and a summary at the bottom.

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 12. Using the df command to check usage
$ lfs df -h /astro/
UUID                       bytes        Used   Available Use% Mounted on
astrofs-MDT0000_UUID      542.1G       26.0G      479.4G   6% /astro[MDT:0]
astrofs-MDT0001_UUID      542.1G       32.7G      472.7G   7% /astro[MDT:1]
astrofs-OST0000_UUID       57.7T       26.6T       28.1T  49% /astro[OST:0]
astrofs-OST0001_UUID       57.7T       26.4T       28.4T  49% /astro[OST:1]
astrofs-OST0002_UUID       57.7T       25.9T       28.8T  48% /astro[OST:2]
astrofs-OST0003_UUID       57.7T       28.6T       26.2T  53% /astro[OST:3]
astrofs-OST0004_UUID       57.7T       26.9T       27.9T  50% /astro[OST:4]
astrofs-OST0005_UUID       57.7T       26.6T       28.2T  49% /astro[OST:5]
astrofs-OST0006_UUID       57.7T       26.3T       28.4T  49% /astro[OST:6]
astrofs-OST0007_UUID       57.7T       26.8T       28.0T  49% /astro[OST:7]
astrofs-OST0008_UUID       57.7T       26.8T       27.9T  50% /astro[OST:8]
astrofs-OST0009_UUID       57.7T       26.3T       28.5T  48% /astro[OST:9]
astrofs-OST000a_UUID       57.7T       26.6T       28.2T  49% /astro[OST:10]
astrofs-OST000b_UUID       57.7T       26.8T       28.0T  49% /astro[OST:11]
astrofs-OST000c_UUID       57.7T       25.6T       29.2T  47% /astro[OST:12]
astrofs-OST000d_UUID       57.7T       27.1T       27.6T  50% /astro[OST:13]
astrofs-OST000e_UUID       57.7T       27.0T       27.8T  50% /astro[OST:14]
astrofs-OST000f_UUID       57.7T       26.5T       28.3T  49% /astro[OST:15]
astrofs-OST0010_UUID       57.7T       37.3T       17.5T  69% /astro[OST:16]
astrofs-OST0011_UUID       57.7T       38.1T       16.7T  70% /astro[OST:17]
astrofs-OST0012_UUID       57.7T       37.5T       17.3T  69% /astro[OST:18]
astrofs-OST0013_UUID       57.7T       38.0T       16.7T  70% /astro[OST:19]
astrofs-OST0014_UUID       57.7T       38.0T       16.7T  70% /astro[OST:20]
astrofs-OST0015_UUID       57.7T       37.1T       17.6T  68% /astro[OST:21]
astrofs-OST0016_UUID       57.7T       37.1T       17.6T  68% /astro[OST:22]
astrofs-OST0017_UUID       57.7T       38.0T       16.7T  70% /astro[OST:23]
astrofs-OST0018_UUID       57.7T       37.4T       17.4T  69% /astro[OST:24]
astrofs-OST0019_UUID       57.7T       37.7T       17.1T  69% /astro[OST:25]
astrofs-OST001a_UUID       57.7T       38.3T       16.5T  70% /astro[OST:26]
astrofs-OST001b_UUID       57.7T       37.4T       17.4T  69% /astro[OST:27]
astrofs-OST001c_UUID       57.7T       37.5T       17.2T  69% /astro[OST:28]
astrofs-OST001d_UUID       57.7T       37.1T       17.7T  68% /astro[OST:29]
astrofs-OST001e_UUID       57.7T       37.8T       17.0T  70% /astro[OST:30]
astrofs-OST001f_UUID       57.7T       37.8T       16.9T  70% /astro[OST:31]
astrofs-OST0020_UUID       57.6T       34.6T       20.2T  64% /astro[OST:32]
astrofs-OST0021_UUID       57.6T       34.3T       20.5T  63% /astro[OST:33]
astrofs-OST0022_UUID       57.6T       34.9T       19.9T  64% /astro[OST:34]
astrofs-OST0023_UUID       57.6T       33.6T       21.1T  62% /astro[OST:35]
astrofs-OST0024_UUID       57.6T       33.5T       21.2T  62% /astro[OST:36]
astrofs-OST0025_UUID       57.6T       35.0T       19.7T  64% /astro[OST:37]
astrofs-OST0026_UUID       57.6T       33.7T       21.0T  62% /astro[OST:38]
astrofs-OST0027_UUID       57.6T       34.1T       20.6T  63% /astro[OST:39]
astrofs-OST0028_UUID       57.6T       33.5T       21.2T  62% /astro[OST:40]
astrofs-OST0029_UUID       57.6T       33.6T       21.1T  62% /astro[OST:41]
astrofs-OST002a_UUID       57.6T       34.2T       20.5T  63% /astro[OST:42]
astrofs-OST002b_UUID       57.6T       33.8T       21.0T  62% /astro[OST:43]
astrofs-OST002c_UUID       57.6T       34.9T       19.8T  64% /astro[OST:44]
astrofs-OST002d_UUID       57.6T       34.1T       20.7T  63% /astro[OST:45]
astrofs-OST002e_UUID       57.6T       33.8T       20.9T  62% /astro[OST:46]
astrofs-OST002f_UUID       57.6T       34.5T       20.2T  64% /astro[OST:47]
 
filesystem_summary:         2.7P        1.5P        1.0P  60% /astro


Related pages

External links