If you are using a program that produces many (e.g. millions) small files, you may keep hitting the /scratch file count quota. This practice creates a huge workload on the metadata servers of the filesystem, degrading its performance. This guide will show you how to use Singularity persistent file overlays to avoid hitting the quota by hiding the small files from the underlying Lustre filesystem. The overlay looks like a single file to Setonix, but holds all your many small files inside. This can be useful for applications such a Trinity, Maker, and OpenFoam.
Step-by-step guide
This example will demonstrate how to setup and use the overlay.
| Column |
|---|
|
| Code Block |
|---|
| language | bash |
|---|
| theme | Emacs |
|---|
| title | Terminal 1. Example overlay creation |
|---|
| # Load the singularity module. Note that the version number may change over time.
$ module load singularity/3.6.8-nompi
# Pull your desired container. This is just an example container
$ singularity pull docker://quay.io/biocontainers/maker:2.31.11–pl5262hec0a270_1
# Decide on the max size of the overlay in MB. This example is 200MB. The default is 100MB.
$ export SIZE="200"
# Decide on the name of your overlay. Here we will call it 'my_overlay'
$ export FILE="my_overlay"
# Use Singularity to create the overlay using the specifications you just set
$ singularity overlay create --size $SIZE $FILE
# Create an output directory for your program's files in the overlay root directory (i.e. / ).
# Note that if you put the directory somewhere that's not root, you'll just write to Setonix like normal.
$ singularity exec --overlay my_overlay maker.sif mkdir /maker_out_dir
# Then when you run your program from the container, you specify the output or temp directory to be the one you made. For example:
$ singularity exec --overlay my_overlay maker.sif maker [options] --TMP /maker_out_dir
# To view the files from your run
$ singularity exec --overlay my_overlay maker.sif ls /maker_out_dir
# To copy useful files out to Setonix (copying to the current directory).
#Note how we’re wrapping the copy command within bash -c; this is to defer the evaluation of the * wildcard to when the container runs the command.
$ singularity exec --overlay my_overlay maker.sif bash -c 'cp -p /maker_out_dir/something.fa* ./
|
|
...