...

Prerequisites

Familiarity with:

• Containers
• Modules
• Slurm Job Scheduler (external site)

Getting container images and initialising Singularity

Check container availability and load the module

Singularity is installed on most Pawsey systems. Use module commands to check availability and the version installed:

...

$ mkdir -p /software/projects/<project-id>/<user-name>/.singularity
$ chown -hR $USER:$PAWSEY_PROJECT /software/projects/<project-id>/<user-name>/.singularity
$ find /software/projects/<project-id>/<user-name>/.singularity -type d -exec chmod g+s {} \;

Pull or build a container image

To provide the image that you want to run, either pull an existing container image or build a container image. 

...

salloc -n 1 -t 4:00:00 -I 

Pull an existing image from a container library

You can pull existing containers from a suitable registry such as Docker Hub, Biocontainers, RedHat Quay or Sylabs Container Library. For most users, this will be the most common way you will use containers. It's a good idea to check what containers are already available before deciding to build your own container. 

...

• The --dir flag specifies the image to be downloaded to a location
• docker://  indicates that you're pulling from the Docker Hub registry 
• user is the hub user 
• image is the image or repository name you're pulling
• tag is the Docker Hub tag that identifies which image to pull

Build a container image

To build a container image, we recommend using Docker, either on a local laptop or workstation or on a cloud virtual machine. For example, the Pawsey Nimbus Cloud has Ubuntu installations that come with both Singularity and Docker pre-installed. You cannot build a container image on Setonix because you will not have admin/sudo privileges. 

...

Then this SIF file can be transferred to Pawsey systems.

Best practices for building and maintaining images 

Building images

Anchor
buildtips buildtips

• Minimize image size
  • Each distinct instruction (such as RUN, CMD, etc) in the Dockerfile generates another layer in the container, increasing its size

  • To minimize image size, use multi-line commands, and clean up package manager caches.

• Avoid software bloat

  • Only install the software you need for a given application into a container.
• Make containers modular
  
  • Creating giant, monolithic containers with every possible application you could need is bad practice. It increases image size, reduces performance, and increases complexity. Containers should only contain a few applications (ideally only one) that you'll use. You can chain together workflows that use multiple containers, meaning if you need to change a particular portion you only need to update a single, small container.

...

# install packages on debian/ubuntu linux os container using apt-get command
RUN apt-get update \
	&& apt-get install -y \
       autoconf \
       automake \
       gcc \
       g++ \
       python \
       python-dev \
    && apt-get clean all \
    && rm -rf /var/lib/apt/lists/*

Managing your Singularity images

Unlike Docker containers, Singularity containers can be managed as simple files. We recommend that projects keep their Singularity containers in a small number of specific directories. For example, each user might store all of their own Singularity container .sif files in a repository directory such as $MYSOFTWARE/singularity/myRepository. For containers that will be used by several users in the group, we recommend that the repository be maintained as a shared directory, such as /scratch/$PAWSEY_PROJECT/singularity/groupRepository.

...

$ singularity cache clean -f

Running jobs with Singularity

Job scripts require minimal modifications to run within a Singularity container. All that is needed is the singularity exec statement followed by the image name and then the name of the command to be run. Listing 2 shows an example script:

...

Bind mounting host directories

The Singularity configuration at Pawsey takes care of always bind mounting the scratch filesystem for you. You can mount additional host directories to the container with the following syntax:

...

Sample use cases

We discuss several common use cases for containers that require some care. Each example shown below highlights the use of particular container features.

Running Python and R

For Singularity containers that have Python or R built-in, use the flag -e  (clean environment) to run the container with an isolated shell environment. This is because both Python and R make extensive use of environment variables and not using a fresh environment can pollute the container environment with pre-existing variables. If you need to read or write from a local directory, you may use the -e flag in conjunction with the -B flag.

...

$ unset $( env | grep ^PYTHON | cut -d = -f 1 | xargs )
$ srun singularity run docker://python:3.8 my_script.py

Using GPUs

Singularity allows users to make use of GPUs within their containers, for both NVIDIA and AMD GPUs. Nimbus uses NVIDIA GPUs, while Setonix uses AMD GPUs. To enable NVIDIA support, add the runtime flag --nv. To use AMD GPUs, add the --rocm flag to your singularity command instead of --nv. 

...

$ sbatch --account=<your-pawsey-project> --partition=gpu gpu.sh

Using MPI

MPI applications can be run within Singularity containers. There are two requirements to do so:

...

#!/bin/bash -l

#SBATCH --account=projectID
#SBATCH --job-name=mpi-OpenFOAMcase
#SBATCH --ntasks=512
#SBATCH --ntasks-per-node=128
#SBATCH --cpus-per-task=1
#SBATCH --exclusive
#SBATCH --time=00:20:00
#SBATCH --partition=work

# ---
# load Singularity
module load singularity/4.1.0-mpi

# ---
# Note we avoid any inadvertent OpenMP threading by setting OMP_NUM_THREADS=1
export OMP_NUM_THREADS=1

# ---
# Set MPI related environment variables for your system (see specific system guides).
export MPICH_OFI_STARTUP_CONNECT=1
export MPICH_OFI_VERBOSE=1

# ---
# Define the container to use
export theRepository=<pathToPersonalImages>
export containerImage=$theRepository/openfoam-v1912.sif

# ---
# cd into the case directory
cd $MYSCRATCH/myOpenFOAMCase

# ---
# execute the simpleFoam parallel solver with MPI
srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS -c $OMP_NUM_THREADS \
     singularity exec $containerImage simpleFoam -parallel -fileHandler collated | tee log.simpleFoam

Singularity on Pawsey Systems

Depending on the cluster, different Singularity modules may be available:

...

• singularity, singularity/VV-mpi: Cray MPI (Setonix) or Intel MPI (others). All ABI compatible with MPICH
• singularity/VV-nompi: For applications that do not require mpi communications (commonly Bioinformatics applications)
• singularity-openmpi: OpenMPI
• singularity-openmpi-gpu: OpenMPI built with CUDA support and any other libraries required by CUDA-aware MPI (for example: gdrcopy)

Features of the modules

These singularity modules set important environment variables to provide a smoother and more efficient user experience. Modules set several key environment variables

...

To ensure that container images are portability, Pawsey provided containers keep host libraries to a minimum. The only case currently supported by Pawsey is mounting of interconnect/MPI libraries, to maximise performance of inter-node communication for MPI and CUDA-aware MPI enabled applications.

Related pages

• Pawsey-Provided Container Images
• Singularity containers failing and reporting issues with libraries

External links

• Singularity Quick Start
• Dockerfile reference
• For specific details about containerised OpenFOAM tools and usage, refer to the OpenFOAM documentation.