...

$ module load openfoam-org-container/8singularity/<VERSION>-mpi
$ export SINGULARITY_CONTAINER=/PathToTheSingularityImage/myopenfoam-8.sif
$ singularity exec $SINGULARITY_CONTAINER printenv | grep FOAM_ETC
FOAM_ETC=/opt/OpenFOAM/OpenFOAM-8/etc
$

(Note that in this example, an imaged owned by the user is being used.)

Or, if the user wishes to open an interactive session within the container:

$ module load openfoam-container/v2012
$ singularity shell $SINGULARITY_CONTAINER
Singularity> echo $FOAM_ETC
/opt/OpenFOAM/OpenFOAM-v2012/etc
Singularity>

(Note that in this example, the image provided by the containerised module is being used.)

And, of course, the singularity command can be used within Slurm batch scripts. So, the execution command in the example script for the solver execution in the OpenFOAM: Example Slurm Batch Scripts page can be modified to explicitly use the singularity command:

#!/bin/bash --login
 
#SBATCH --job-name=[name_of_job]
#SBATCH --partition=work
#SBATCH --ntasks=1152
#SBATCH --ntasks-per-node=128
#SBATCH --cpus-per-task=1
#SBATCH --exclusive
#SBATCH --time=[neededTime]

# --- Load modules and define images:
# -Using the containerised container:
module load openfoam-org-container/7-container/7
# -Using user's own image:
#module load singularity/<VERSION>-mpi  #Adapt <version> to the current provided version of singularity
#SINGULARITY_CONTAINER="/PathToTheSingularityImage/myopenfoam-8.sif"  #Adapt path and name to the correct ones

#--- Specific settings for the cluster you are on
#(Check the specific guide of the cluster for additional settings)

# ---
# Set MPI related environment variables. Not all need to be set
# main variables for multi-node jobs (uncomment for multinode jobs)
export MPICH_OFI_STARTUP_CONNECT=1
export MPICH_OFI_VERBOSE=1
#Ask MPI to provide useful runtime information (uncomment if debugging)
#export MPICH_ENV_DISPLAY=1
#export MPICH_MEMORY_REPORT=1


#--- Automating the list of IORANKS for collated fileHandler
echo "Setting the grouping ratio for collated fileHandling"
nProcs=$SLURM_NTASKS #Number of total processors in decomposition for this case
mGroup=32            #Size of the groups for collated fileHandling (32 is the initial recommendation for Setonix)
of_ioRanks="0"
iC=$mGroup
while [ $iC -le $nProcs ]; do
   of_ioRanks="$of_ioRanks $iC"
   ((iC += $mGroup))
done
export FOAM_IORANKS="("${of_ioRanks}")"
echo "FOAM_IORANKS=$FOAM_IORANKS"

#-- Execute the solver:
srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS -c 1 \
     singularity exec $SINGULARITY_CONTAINER pimpleFoam -parallel

(For the use of their own image, users should comment the line that loads the containerised module and uncomment the lines that load the singularity module and define the SINGULARITY_CONTAINER variable as the real path to their own image. Obviously, the <VERSION> and the real path should be adapted.)

Wrappers of the shell and exec commands

...

Information in the following sections may be useful if you are looking to build a new container with additional tools.

Use of

...

OpenFOAM versions

...

not provided by Pawsey

As mentioned in the main documentation for OpenFOAM, several recent versions of OpenFOAM containers are already available in Setonix through modules. One advantage of these modules is that OpenFOAM tools and solvers are accessible through basic commands without the need of calling singularity explicitly in the command line. The call to singularity is performed under the hood by the tool/solver's wrappers.

If users need a new version of OpenFOAM that is still not installed in our systems, please they can create their own container for that version, or can contact support so that we can make it available to users.

If users are still forced to use an old version of OpenFoam that is not installed in our systems, we recommend to use a container equipped with the needed version.

Pawsey has already built several Docker images with the main intention to serve as examples for users to builid their own images. Pawsey-provided Docker images are available through our registry in Red Hat Quay.io (https://quay.io/pawsey) in directories openfoam & openfoam-org. Other directories with the "-legacy" post-fix contain images built with operating-system versions that do not run properly on Setonix anymore. The corresponding building recipes (Dockerfiles) are at Pawsey GitHub repostory (https://github.com/PawseySC/pawsey-containers). As said, users should use these recipes as examples to build their own.

...

$ mkdir -p $MYSOFTWARE/software/projects/myProjectName/singularity/OpenFOAM
$ cd /software/projects/myProjectName$MYSOFTWARE/singularity/OpenFOAM
$ salloc -n 1 --mem=29600Mb -p copy
salloc: Granted job allocation 67529

$ module load singularity/<version>
$ srun singularity pull openfoam-org_2.4.x_pawsey.sif docker://quay.io/pawsey/openfoam-org:2.4.x
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 11323ed2c653 done
Copying blob 0a71b4dba9db done  
Copying blob 874ae6ae3b00 done  
Copying blob 5d27d6ffcb7e done
...
2022/07/08 10:24:32  info unpack layer: sha256:cf6efca24562cd25028fcc99a4442f6062eeb62a60c7709f1454fc8bd8b0d5ea
2022/07/08 10:24:32  info unpack layer: sha256:072f30ec926604acc07e65868c407e1f0d3a3d98dd46340b04a572e55edcfaa6
2022/07/08 10:24:32  info unpack layer: sha256:9357e5bafecbf1d6af595baf24c54e2a10e182a738133a55c889293dcc7b4d10
2022/07/08 10:24:32  info unpack layer: sha256:7f6dad97028f1961f9b463b213119fea99228cda7f684aeeda213d5848ed125b
...
2022/07/08 10:25:12  info unpack layer: sha256:537ce1385428d12c4ba18c09984440f280cce516bcb65ba3531b4a2a95533cfd
INFO:    Creating SIF file...

$ ls
openfoam-org-2.4.x-pawsey.sif

...