Summary

This page covers information on how to use the new 'Pawsey Bio - Ubuntu 2022.04 - 20212023-1103' image for Nimbus. Instructions on how to choose this image when creating your instance can be found here. This Bio-image is created to cater to bioinformatics users who prefer to have their instances pre-installed with software and tools commonly used in the bioinformatics domain. Some of the software are part of Pawsey's ongoing effort to improve the experience of bioinformatics users at Pawsey.

Before you begin

You may be required to input your SSH public key or the path to your SSH public key on your local machine while using some of these software. Please ensure you have it ready to go. Instructions for how to generate one can be found /wiki/spaces/SUP/pages/55410887., including over 8000 Biocontainer tools.

Pre-installed software

The list of pre-installed software on this image is as follows:

Instructions

On this page, we will only cover instructions for how to use CernVM-FS , Jupyter Notebook, RStudio, and Singularity-HPC. For instructions for other software listed above, please see the software's original documentation page.

CernVM-FS

CernVM-FS is a read-only file system that was developed by another supercomputing centre (Cern). It allows files such as container tools, reference datasets and other shared resources that are commonly used by many researchers to be accessed, added to, and updated in the one location. At Pawsey, we currently mirror the Biocontainer tools and reference genome datasets that are on Galaxy Project's repositoryrepositories . Please note that the datasets these may not be comprehensive, and this service is not meant to replace your current methods for accessing public datasets.

Due to a recent (27th July 2022) change in the CernVM-FS proxy on Pawsey, please ensure to do the following before proceeding:

sudo apt-get autoremove cvmfs
sudo apt-get purge cvmfs
sudo rm -rf /etc/cvmfs/
git clone https://github.com/PawseySC/Pawsey-CernVM-FS.git

Then run the following to set up the repositories from Galaxy and AARNet, respectively:

cd  Pawsey-CernVM-FS

# for Galaxy repos
sudo ./cvmfs-client-setup.sh \
    --stratum-1 cvmfs1-mel0.gvl.org.au \
    --stratum-1 cvmfs1-ufr0.galaxyproject.eu \
    --stratum-1 cvmfs1-tacc0.galaxyproject.org \
    --stratum-1 cvmfs1-iu0.galaxyproject.org \
    --stratum-1 cvmfs1-psu0.galaxyproject.org \
    --proxy cvmfs-cachingproxy.pawsey.org.au \
    pubkeys/cvmfs-config.galaxyproject.org.pub \
    pubkeys/data.galaxyproject.org.pub \
    pubkeys/main.galaxyproject.org.pub \
    pubkeys/sandbox.galaxyproject.org.pub \
    pubkeys/singularity.galaxyproject.org.pub \
    pubkeys/test.galaxyproject.org.pub \
    pubkeys/usegalaxy.galaxyproject.org.pub

# for AARNet repos
sudo ./cvmfs-client-setup.sh \
    --stratum-1 bcws.test.aarnet.edu.au \
    --proxy cvmfs-cachingproxy.pawsey.org.au \
    pubkeys/containers.biocommons.aarnet.edu.au.pub pubkeys/data.biocommons.aarnet.edu.au.pub pubkeys/tools.biocommons.aarnet.edu.au.pub
 

You can then refer to and use the path to the datasets as follows:

The Biocontainer tools are in the format of Singularity containers. To use them, you can skip this step and proceed to Singularity-HPC.

To access and view the list of Biocontainer tools:

Note: It may take a minute or two to load the folders. When you have done it once, it will not take as long to show again.

Jupyter Notebook 

Jupyter Notebooks are very popular way of running bioinformatics analysis due to its interactive nature. We have enabled an automated way of creating such notebooks from a container format. As containers do not store files, all notebooks created from the interactive session are stored on your Nimbus instance under /data.

From the Nimbus dashboard:

Then, to start a Jupyter Notebook, simply run the following:

ansible-playbook /jupyter-on-nimbus/ansible-jupyternotebook.yaml

Notes:

From time to time, you may want to re-clone the jupyter-on-nimbus repo for any future updates. Only essential updates will be notified to Nimbus users. 

git clone https://github.com/PawseySC/jupyter-on-nimbus
sudo rm -rf /jupyter-on-nimbus
sudo mv jupyter-on-nimbus /

RStudio

RStudio is another popular bioinformatics analysis interactive software. Here we have also enabled automation to starting an RStudio server session. As containers do not store files, all R sessions created from the interactive session are stored on your Nimbus instance under /data.

From the Nimbus dashboard:

Navigate to Network > Security Groups:

Click on + Create Security Group, name it 'port 8787' and then select the Create Security Group button:

Then, to start an RStudio server session, simply run the following:

ansible-playbook /rstudio-on-nimbus/ansible-rstudio.yaml -i /rstudio-on-nimbus/vars_list

Notes:

From time to time, you may want to re-clone the rstudio-on-nimbus repo for any future updates. Only essential updates will be notified to Nimbus users. 

ls /cvmfs/singularity.galaxyproject.org

To access the data files:

ls /cvmfs/data.galaxyproject.org

sudo apt-get autoremove cvmfs
sudo apt-get purge cvmfs
sudo rm -rf /etc/cvmfs/
git clone https://github.com/PawseySC/

Pawsey-

CernVM-

FS.git

cd 

Pawsey-CernVM-FS
sudo 

./install-cvmfs.sh install

Singularity-HPC

Singularity-HPC (SHPC) is a software for container modules. If you are familiar with using containers, this is an added bonus to your experience in using containers. If you are not, this is a great way to start using containers. As container syntax can be messy and confusing, being able to use them as modules removes the need for using container syntaxes. Singularity-HPC was created by one of the original developers of Singularity, and the registry includes many bioinformatics containers that can be readily pulled and used. 

Before you begin, ensure to move your containers folder to your storage volume (.i.e. /data), then update the container base path:

we have integrated the use of SHPC seamlessly with CernVM-FS. This means that you can easily access and use over 8000 Biocontainers without needing to understand container syntax.

To see the entire list of containers available on the registry, run the following command:

shpc list 

At Pawsey, we recommend using S-HPC's registry of quay.io/biocontainers containers, as Biocontainers are a reliable source of well-built containers, with versions that are seamlessly matched to BioConda's tools. To narrow down the list to biocontainers, runfor a particular tool e.g, fastqc:

shpc show -f quay.io/biocontainers

To install any of these containers, run:

shpc install quay.io/biocontainers/TOOL_NAMEfastqc

To use installed containers, run the following find and use the tool as you normally would (no container syntax required):

$ module avail fastqc

$ module load quay.io/biocontainers/TOOL_NAME

Notes:

fastqc

$ fastqc --version

To check the list of modules loaded:

$ 

module list