SHPC is used by Pawsey staff to deploy some of the available scientific softwares, in particular bioinformatics applications.

Using SHPC

Installing a container for a software included in the SHPC registry requires no knowledge of containers and Singularity: all you need are the software name and version.

The key commands of SHPC are show and install; let's see them in action with an example. Suppose we want to install the bioinformatics package BWA. We can use the shpc show command to browse the SHPC registry of available containers:

...

width	900px

Versions installed in Pawsey systems

To check the current installed versions, use the module avail command (current versions may be different from content shown here):

Column

width	900px

Code Block

language	bash
theme	EmacsDJango
title	Terminal 1. Example SHPC Show commandChecking for installed versions

$ module loadavail shpc/0.0.53  # load SHPC module

$ shpc show -f bwa  # search for a package in SHPC registry (string search)
biocontainers/bwa
ghcr.io/autamus/bwa

$ shpc show biocontainers/bwa  # inspect specific container recipe
docker: biocontainers/bwa
url: https://hub.docker.com/r/biocontainers/bwa
maintainer: '@vsoch'
description: BWA is a software package for mapping low-divergent sequences against
  a large reference genome, such as the human genome.
latest:
  0.7.15: sha256:6f76c11a816b10440fd9d2c64c7183a31cc104a729f31a373c9b2b068138305e
tags:
  0.7.15: sha256:6f76c11a816b10440fd9d2c64c7183a31cc104a729f31a373c9b2b068138305e
  v0.7.17_cv1: sha256:9479b73e108ded3c12cb88bb4e918a5bf720d7861d6d8cdbb46d78a972b6ff1b
aliases:
  bwa: /opt/conda
------------------------------------ /software/setonix/2024.05/pawsey/modules -------------------------------------
   shpc/0.1.28

Using SHPC

Installing a container for a software included in the SHPC registry requires no knowledge of containers and Singularity: all you need are the software name and version.

The key commands of SHPC are show and install; let's see them in action with an example. Suppose we want to install the bioinformatics package BWA. We can use the shpc show command to browse the SHPC registry of available containers:

Column

width	900px

Code Block

language	bash
theme	Emacs
title	Terminal 1. Example SHPC Show command

$ module load shpc/<VERSION>  # load SHPC module with the correct version

$ shpc show -f bwa  # search for a package in SHPC registry (string search)
quay.io/biocontainers/bwa
ghcr.io/autamus/bwa

$ shpc show quay.io/biocontainers/bwa  # inspect specific container recipe
docker: biocontainers/bwa
url: https://biocontainers.pro/tools/bwa
maintainer: '@vsoch'
description: shpc-registry automated BioContainers addition for bwa
latest:
0.7.17--h7132678_10: sha256:f9063141d8c5da87da76392b3a152b927b2913d47373f1874d76f14634b3f684
tags:
0.7.17--h7132678_9: sha256:07822e4293a8c59755b295c448b9541db6c9bdbfdedb010bdbdcc1e1e935370f
0.7.17--h7132678_10: sha256:f9063141d8c5da87da76392b3a152b927b2913d47373f1874d76f14634b3f684
docker: quay.io/biocontainers/bwa
aliases:
bwa: /usr/local/bin/bwa

The information of interest in this output is the list of available versions (or tags), in this case: 0.7.15 and v0.7.17_cv1. Let's install the former:

Column

width	900px

Code Block

language	bash
theme	Emacs
title	Terminal 2. Example SHPC Install command

$ shpc install quay.io/biocontainers/bwa:0.7.1517--h7132678_10
singularity pull --name /software/projects/projectcodepawsey0001/rsrchrbuser/setonix/containers2024.05/containers/sif/quay.io/biocontainers/bwa/0.7.15/17--h7132678_10/quay.io-biocontainers-bwa-0.7.1517--h7132678_10-sha256:6f76c11a816b10440fd9d2c64c7183a31cc104a729f31a373c9b2b068138305ef9063141d8c5da87da76392b3a152b927b2913d47373f1874d76f14634b3f684.sif docker://quay.io/biocontainers/bwa@sha256:6f76c11a816b10440fd9d2c64c7183a31cc104a729f31a373c9b2b068138305ef9063141d8c5da87da76392b3a152b927b2913d47373f1874d76f14634b3f684
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image
source signatures

[..]

INFO:    Creating SIF file...
Module quay.io/software/projects/projects/projectcode/rsrchr/setonix/containers/sif/biocontainers/bwa:0.7.17--h7132678_10 was created.
Creating link $module_base/quay.io/biocontainers/bwa/0.7.15/biocontainers-bwa-0.7.15-sha256:6f76c11a816b10440fd9d2c64c7183a31cc104a729f31a373c9b2b068138305e.sif
Module biocontainers/bwa:0.7.15 was created.17--h7132678_10/module.lua -> $views_base/modules/bwa/0.7.17--h7132678_10.lua

That's it!

By default SHPC downloads containers under:

...

You are able to use module avail, module load, and module unload: (as these are system modules, note the slash "/" for the version, instead of the colon ":" above for the tags):

...

width	900px

...

language	bash
theme	Emacs
title	Terminal 3. Example SHPC module load

...

":" above for the tags):

Column

width	900px

Code Block

language	bash
theme	Emacs
title	Terminal 3. Example SHPC module load

$ module avail bwa  # search module

----------- /software/projects/projectcode/rsrchr/setonix/2024.05/containers/views/modules -------------
   bwa/0.7.17--h7132678_10 

$ module load biocontainers/bwa/0.7.15/module  # load module

$ bwa  # test command

Program: bwa (alignment via Burrows-Wheeler transformation)
Version: 0.7.17-r1188
Contact: Heng Li <lh3@sanger.ac.uk>

Usage: bwa <command> [options]

Command: index index sequences in the FASTA format
mem BWA-MEM algorithm
fastmap identify super-maximal exact matches
pemerge merge overlapping paired ends (EXPERIMENTAL)
aln gapped/ungapped alignment
samse generate alignment (single ended)
sampe generate alignment (paired ended)
bwasw BWA-SW for long queries

shm manage indices in shared memory
fa2pac convert FASTA to PAC format
pac2bwt generate BWT from PAC
pac2bwtgen alternative algorithm for generating BWT

        bwtupdate     update .bwt to the new format
         bwt2sa        generate SA from BWT and Occ

Note: To use BWA, you need to first index the genome with `bwa index'.
      There are three alignment algorithms in BWA: `mem', `bwasw', and
      `aln/samse/sampe'. If you are not sure which to use, try `bwa mem'
      first. Please `man ./bwa.1' for the manual.

Note

title	Loading a module created by SHPC

As of version 0.0.53 of SHPC, modules created using this tool require the suffix /module to be loaded correctly.

For instance:

module load biocontainers/bwa/0.7.15/module

Failing to add the /module will result in an error, and no module will be loaded.

This behaviour is going to be improved in future SHPC versions.

 to use, try `bwa mem'
first. Please `man ./bwa.1' for the manual.

The full list of SHPC commands can be shown by using one of the help commands:

...

Column

width	900px

Code Block

language	bash
theme	Emacs
title	Terminal 4. Velvet not on SHPC registry

$ module load shpc/0.0.53<VERSION>  # load SHPC module

$ shpc show -f velvet
$

...

Currently*, the following features are supported:

Column

width	900px

Name	Description	Default	Options
gpu	If the container technology supports it, add flags to indicate using gpu.	null	nvidia, amd, null
x11	Bind mount ~/.Xauthority or a custom path	null	true (uses default path ~/.Xauthority), false/null (do not enable) or a custom path to an x11 file
home	Specify and bind mount a custom homepath	null	custom path for the home, or false/null

*Correct as of 11/8/2023

Note that you can use the home feature to allow your container module to bind mount a home of your choosing. This might be needed for some containers that expect to access a certain directory. For example, the aws-cli container expects to have read/write access to your ~/.aws directory. See the following SHPC recipe as an example which would bind mount the user's home directory:

...

Column

width	900px

Code Block

language	bash
theme	DJango
title	Listing 2. Example SHPC recipe that demonstrates yaml demonstrating the home feature
linenumbers	true
collapse	true

docker: quay.io/pawsey/hpc-python
url: https://quay.io/repository/pawsey/hpc-python
maintainer: '@marcodelapierre'
description: Base Python images with popular packages for HPC workflows.
latest:
  '2022.03': sha256:962e7c24302b2dc3946bb22326d0cb4385373113a212231488070aa3e43bd1a1
tags:
  '2021.09': sha256:c2f3f585a0be711046583c5861199107c94e047545325834d68d81d2582b7a04
  2021.09-hdf5mpi: sha256:9d34b5908630e028a6a084891af8b6e65f2626c30e57c06e883f8909850c782b
  '2022.03': sha256:962e7c24302b2dc3946bb22326d0cb4385373113a212231488070aa3e43bd1a1
  2022.03-hdf5mpi: sha256:e9a0db88e98c2388d8731a983ed845b46ce0e2d99d4566802b84142ce21e1c23
aliases:
  python: /usr/local/bin/python
  python3: /usr/local/bin/python3
env:
  PYTHONSTARTUP: ''
  PYTHONUSERBASE: ''
features:
  home: true

...

Version	Old Version 7	New Version Current
Changes made by	Sarah Beecroft	Alexis Espinosa
Saved on	Aug 11, 2023	Jan 29, 2025

Versions Compared

Key

Using SHPC

Versions installed in Pawsey systems

Using SHPC

Page Comparison

Versions Compared

Key

Using SHPC

Versions installed in Pawsey systems

Using SHPC