| Excerpt |
|---|
There are multiple filesystems mounted to each of Pawsey's supercomputers. Each of these filesystems are designed for particular use cases. This page provides a detailed description of these filesystems. |
...
Apart from /home, all are Lustre distributed filesystems. Lustre is an open-source, high performance parallel file system optimised for high throughput.
The filesystems are different in many ways and are designed to facilitate different activities in supercomputing. The intended usage for each of them is explained below. Use outside of these purposes may cause poor performance for a particular activity as well as create detrimental impacts to other users.
...
| Insert excerpt |
|---|
| Visual Studio Code for Remote Development |
|---|
| Visual Studio Code for Remote Development |
|---|
| nopanel | true |
|---|
|
Further explanation about quotas, permissions and copy (cp) vs move (mv) of files and directories is given in the sections below.
Software filesystem
The /software filesystem is a Lustre file system with much higher throughput than /home. It is intended for software installations and Slurm batch script templates. Each project has an associated directory on the filesystem whose path is /software/projects/<project>. Within a project directory, each project member has his or her own directory whose full path, /software/projects/<project>/<username>, is contained in the MYSOFTWARE environment variable.
...
A user is always a member of their own primary group (which is the same as their own username) and can also be a member of more than one project. This is important to know because files created with a group associated to a username rather than a project are limited to a default quota of 1GB and there can be at most 100 of them.
| Column |
|---|
|
| Warning |
|---|
If you encounter a write error, compiler error, or file transfer error on the /scratch or /software filesystems, this is likely because the files are counting against your personal group quota rather than your project's group quota. |
| Tip |
|---|
You should proactively and regularly monitor both file count and quota usage across the filesystems. This practice will reduce your likelihood of hitting the quota limits; whenever this happens, no files can be written until usage is brought back below quota. As regards the /home filesystem, regularly check for and clean unneeded files, which may be generated by software as temporary or cache files. These are often stored in hidden directories (their name starts with a dot). |
|
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 4. List the default file permissions and ownerships for a file created on Setonix /home |
|---|
| $ ls -ld myscript.sh
-rw-r--r-- 1 username username 2 Nov 30 16:33 myscript.sh |
|
...
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal 4X. List the default file permissions for a file created on Setonix /software & /scratch |
|---|
| $ ls -ld otherscript.sh
-rw-r--r-- 1 username projectgroup 2 Nov 30 16:33 otherscript.sh |
|
...
The next two words after the permissions are, respectively, the owner-name and the group-name of the file. In the case of files created in /home, both the owner and the group are assigned to the username by default. And in the case of files created in /software and /scratch, the default for the group-name is the projectgroup for your project. This
| Column |
|---|
|
| Tip |
|---|
It is important to note because the quota limits explained in the section above are really counted on the group-name of the files and not the owner-name. |
|
...
Then, the quota limit of all the files |
|
...
that have groupname = username (added among all our filesytems) is 10k files, which is very limited. This limitation has frequently hit the quota limit for users that do not follow our recommendations for installing software, because some installation tools/commands may override the default group-name of /scratch and software and assign it as username instead of projectgroup. (Check the related pages at the bottom of this page.) |
| Warning |
|---|
For the same reason, when transferring files between filesystems, we do not recommend the use of the mv (move) command, because this command preserves the original group-name. Instead, we recommend the use of the cp command (or other tools that allow the |
|
...
assignment of the default ownerships for the destination filesystem). When using cp, do not use the -a or - |
|
...
p flags. If you want to preserve timestamps, use cp --preserve=timestamps. For example, considering the files in $HOME as above:
|
| Code Block |
|---|
| language | bash |
|---|
| theme | DJango |
|---|
| title | Terminal X. Do not use mv when transferring files to another filesystem |
|---|
| $ ls -ld $HOME/myscript.sh
-rw-r--r-- 1 username username 2 Nov 30 16:33 myscript.sh
$ cp $HOME/myscript.sh $MYSOFTWARE/myscript_cp.sh #good practice
$ mv $HOME/myscript.sh $MYSOFTWARE/myscript_mv.sh #bad practice
$ ls -ld $MYSOFTWARE/myscript*
-rw-r--r-- 1 username projectgroup 6 Dec 10 10:03 /software/projects/project/username/myscript_cp.sh
-rw-r--r-- 1 username username 2 Nov 30 16:33 /software/projects/project/username/myscript_mv.sh |
|
File transfer programs like WinSCP can also cause issues with permissions and groups. You should consult the documentation of your preferred transfer program. rsync users should avoid using the -a and -p flags; these flags will preserve permissions of the source files, which may conflict with the default behaviour on Pawsey systems. Some additional information about file transfer programs is at: Transferring Files in/out Pawsey Filesystems.
...
Related pages
External links
...