On this page:

Familiarity with:

• Modules

• Containers

SHPC allows the installation of software containers in the form of so-called container modules, for transparent usage of containerised applications. An automated process generates a system module for an application, hiding the specificities of the Singularity syntax behind shell functions that take the same name as the corresponding executables.

For containerised applications that are already available in the SHPC registry, installing and using them via SHPC is much simpler than using Singularity itself. For applications that are not yet in the registry, writing a custom container recipe may still be faster than learning how to use Singularity.

SHPC has been configured by Pawsey staff to work out-of-the-box; the following aspects have been set up:

• Directory trees for installed containers and modules

• Default registry for installation recipes

• Use of Singularity as the container runtime

• Naming convention and features of generated module files

• Configuration for MPI and GPU containers

SHPC is used by Pawsey staff to deploy some of the available scientific software, in particular bioinformatics applications.

To check the current installed versions, use the module avail command (current versions may be different from content shown here):

Installing a container for a software included in the SHPC registry requires no knowledge of containers and Singularity: all you need are the software name and version.

The key commands of SHPC are show and install; let's see them in action with an example. Suppose we want to install the bioinformatics package BWA. We can use the shpc show command to browse the SHPC registry of available containers:

The information of interest in this output is the list of available versions (or tags), in this case: 0.7.15 and v0.7.17_cv1. Let's install the former:

That's it!

By default SHPC downloads containers under:

and creates modulefiles under:

You are able to use module avail, module load, and module unload: (as these are system modules, note the slash "/" for the version, instead of the colon ":" above for the tags):

The full list of SHPC commands can be shown by using one of the help commands:

What if a software container is not in the SHPC registry? In this case, you can either write your own container recipe (see terminal 5), or email the Pawsey Helpdesk for help.

Suppose you want to install the bioinformatics tool Velvet. For the sake of this example, we know already that there's a container available for Velvet at quay.io/biocontainers/velvet (external site).

As you can see from the empty output, there's no pre-existing entry in the SHPC Container Registry.

Let's see how to create one; in practice, we need to create a YAML container recipe inside the registry tree of SHPC. First, let's get the location of the registry, and then create an appropriate directory structure using the known container repo quay.io/biocontainers/velvet that was postulated above.

Let's see how a possible recipe for Velvet might look:

Let's comment on the key components of this YAML file:

• docker is the repository path for the container, without version tags

• tags is a list of container tags (versions) with the corresponding SHA message digest (shasum); these need to be manually collected from the repository website, in this case https://quay.io/repository/biocontainers/velvet?tab=tags 

• latest is a copy-paste of the tag from above, to be used as "latest" version

• maintainer is the Github username of the creator (required to contribute the recipe back to the Github repository of SHPC; put any name if you don't have one)

• aliases is a list of command names that will be made available by the SHPC module, with the corresponding commands from inside the container; these need to be manually provided, either by reading through the documentation of the package, or by downloading and inspecting the container

Does this recipe work? Let's give it a go!

You can add advanced features to your recipe using Features. The SHPC authors outline this functionality in their documentation.

Currently (as of 11/8/2023), the following features are supported:

Note that you can use the home feature to allow your container module to bind mount a home of your choosing. This might be needed for some containers that expect to access a certain directory. For example, the aws-cli container expects to have read/write access to your ~/.aws  directory. See the following SHPC recipe as an example which would bind mount the user's home directory:

• Review Software Stack Policies before proceeding with local installations.

• More information on SHPC can be found in the Singularity Registry (HPC) documentation.