File Count Quota Exceeded on /scratch

Problem

Disk quota exceeded error while writing to /scratch

Submitted jobs may also end abruptly after starting. This may be as short as just a few seconds (approx. 1-2 seconds).

SLURM output files may not appear. Slurm job state of FAILED may be given.

Solution

There is a file count limit in place on the /scratch filesystem. The limit is currently set at 1 million inodes, assigned on a per user basis.

Before submitting your job, ensure that you have enough free inodes available under your username on the /scratch filesystem.

To display your currently used /scratch quota, you can run this command:

Terminal 1. Check the quota for /scratch

$ lfs quota /scratch

For more information, see Pawsey File Systems and their Usage.

OpenFOAM users

This issue is particularly common with OpenFOAM users due to the application's default file handling functions. Large uncollated simulations can produce many files and users will quickly find their /scratch quota exhausted. For more information specific to OpenFOAM, please see the Parallel I/O section under OpenFOAM documentation.

If you require assistance in removing your files from /scratch, please contact the Pawsey team, as we have scripts available to automate the process.