Pawsey provides and maintains a number of prebuilt libraries and applications on Setonix, most of which are installed and managed through Spack. This page outlines how users can use Spack to install additional software or different builds of existing software that are not provided by Pawsey supported modules.  

Before you begin

Read Software Stack Policies before proceeding with any local installations.

Before using Spack to install a package, run the following command to check whether it is already available:

module avail <package-name>

To familiarise yourself with Spack features and concepts, read the Spack documentation (external site).

Motivation for using Spack

Spack provides a simple interface for installing packages. Amongst other benefits, Spack provides human-readable installation commands, is able to automatically build the dependency tree for a large number of packages, and allows different installation configurations to be selected. As a result, Spack lessens the need to know the variety of installation procedures in use, such as automake and cmake. 

Spack at Pawsey

Spack has been configured by Pawsey staff to work out of the box for most usage scenarios. In particular, the following aspects have been tuned:

• Usage of available compilers
• Usage of available performance libraries
• Awareness of system-wide packages
• Directory trees for installed packages and modules
• Naming convention and features of generated modulefiles

Spack is used at Pawsey to deploy a significant fraction of the scientific software stack including HPC applications and HPC libraries, as well as some compilers and utilities.

Using Spack

First, the Spack module has to be loaded:

    $ module load spack/0.17.0

Users can invoke Spack with the spack command. spack accepts a subcommand, or Spack command, as the first argument:

    $ spack <subcommand>

Use the following subcommands in order:

1. Use list to look for package recipes.
2. Use info to inspect available options for a given package.

3. Use spec to test installation and required dependencies for a given package.

   Always test an installation with spec before running install.

4. Finally, proceed with the install subcommand for a given package.

A comprehensive list of Spack commands can be inspected by using variations of the help subcommand.

$ spack help
$ spack help --all
$ spack <subcommand> -h

Example installation process with NAMD

Suppose we want to install the NAMD software for molecular dynamics simulations. First, let's search Spack for an appropriate package.

$ spack list namd
==> 1 packages.
namd

Spack has found one package called namd. It sounds like it's the one we're looking for.

The namd package in Spack has several build time options and dependencies. These options can be explored by using the info subcommand. 

$ spack info namd
MakefilePackage:   namd

Description:
    NAMD is a parallel molecular dynamics code designed for high-performance
    simulation of large biomolecular systems.

Homepage: http://www.ks.uiuc.edu/Research/namd/

Tags:
    None

Preferred version:
    2.14      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.14_Source.tar.gz

Safe versions:
    master    [git] https://charm.cs.illinois.edu/gerrit/namd.git on branch master
    2.15a1    [git] https://charm.cs.illinois.edu/gerrit/namd.git at tag release-2-15-alpha-1
    2.14      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.14_Source.tar.gz
    2.13      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.13_Source.tar.gz
    2.12      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.12_Source.tar.gz

Variants:
    Name [Default]      Allowed values          Description
    ================    ====================    ============================================

    cuda [off]          on, off                 Build with CUDA
    cuda_arch [none]    none, 52, 12, 61,       CUDA architecture
                        37, 60, 30, 11, 10,
                        20, 86, 35, 72, 62,
                        70, 13, 53, 32, 50,
                        75, 80, 21
    fftw [3]            none, 2, 3, mkl,        Enable the use of FFTW/FFTW3/MKL FFT/AMDFFTW
                        amdfftw
    interface [none]    none, tcl, python       Enables TCL and/or python interface

Installation Phases:
    edit    build    install

Build Dependencies:
    amdfftw  charmpp  cuda  fftw  intel-mkl  python  tcl

Link Dependencies:
    amdfftw  charmpp  cuda  fftw  intel-mkl  python  tcl

Run Dependencies:
    None

Virtual Packages:
    None

The info output lists the build time options under Variants, along with the build and link dependencies of the package. 

The dependencies required can be explored by running the spec subcommand:

$ spack spec namd # default installation of namd and the libraries it depends on 
Input spec
--------------------------------
namd

Concretized
--------------------------------
namd@2.14%gcc@11.2.0~cuda cuda_arch=none fftw=3 interface=none patches=cdcbc3846be1dfd6dbf958177c703e15ef5343e1461ed68d02be8fd0512429d5 arch=linux-sles15-zen3
    ^charmpp@6.10.2%gcc@11.2.0~cuda~omp~papi+production~pthreads+shared+smp~syncft~tcp~tracing backend=netlrts build-target=LIBS pmi=none arch=linux-sles15-zen3
    ^fftw@3.3.8.8%gcc@11.2.0+mpi~openmp~pfft_patches precision=double,float arch=linux-sles15-zen3

Spack will produce the default installation, listing for each package that needs to be installed the full compilation options. Focusing on namd, Spack reports the following information:

• The package and version (namd@2.14)
• The compiler and version (%gcc@11.2.0)
• Information related to the variants and the architecture that the package will be built for (in this example, the linux-sles15 operating system using the zen3 CPU architecture)

Variants are listed with the following specifiers:

• +  indicates that the compilation option is enabled
• ~ (no prepending space) or - (with prepending space) indicate that the compilation option is disabled
• Variants with multiple values are listed as option_name=value 

In this case, when namd is compiled cuda is disabled, fftw is set to 3 (use fftw3 library) and no interface has been used. 

Dependencies are preceded by a ^. For instance, here namd  depends on charmpp@6.10.2 and fftw@3.3.8.8, with each dependency listing which compiler and variants it will be built with. 

Explore changes in the build process with spec by adding the variants to the package name.

$ spack spec namd +cuda fftw=mkl interface=python #setting a specific variant 
Input spec
--------------------------------
namd+cuda fftw=mkl interface=python

Concretized
--------------------------------
namd@2.14%gcc@11.2.0+cuda cuda_arch=none fftw=mkl interface=python patches=cdcbc3846be1dfd6dbf958177c703e15ef5343e1461ed68d02be8fd0512429d5 arch=linux-sles15-zen3
    ^charmpp@6.10.2%gcc@11.2.0~cuda~omp~papi+production~pthreads+shared+smp~syncft~tcp~tracing backend=netlrts build-target=LIBS pmi=none arch=linux-sles15-zen3
    ^cuda@11.3.0%gcc@11.2.0~dev arch=linux-sles15-zen3
        ^libxml2@2.9.10%gcc@11.2.0~python arch=linux-sles15-zen3
            ^libiconv@1.16%gcc@11.2.0 arch=linux-sles15-zen3
            ^pkgconf@1.4.2%gcc@11.2.0 arch=linux-sles15-zen3
            ^xz@5.2.4%gcc@11.2.0~pic libs=shared,static arch=linux-sles15-zen3
            ^zlib@1.2.11%gcc@11.2.0+optimize+pic+shared arch=linux-sles15-zen3
    ^intel-mkl@2020.4.304%gcc@11.2.0~ilp64+shared threads=none arch=linux-sles15-zen3
        ^cpio@2.12%gcc@11.2.0 patches=d22633c368b8aedf4c08b23b6fbaa81a52404c8943ab04926404083ac10f1a4b arch=linux-sles15-zen3
    ^python@3.6.13%gcc@11.2.0+bz2+ctypes+dbm~debug+libxml2+lzma+nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib arch=linux-sles15-zen3
    ^tcl@8.6.11%gcc@11.2.0 arch=linux-sles15-zen3

Finally, install the package.

The spack command shown in terminal 5 will try to install and build all of the required packages in the user location configured by Pawsey, which is:

/software/projects/<project-id>/<user-id>/setonix/software/<arch>/<compiler>/<package>-<version>-<unique_hash>

In this example:

• <arch> = linux-sles15-zen3
• <compiler> = gcc-11.2.0
• <package>-<version>-<unique_hash> = namd-2.12-x3jinxvmanf5dm6zujstnpalmlzi4hy5/

$ # Before install, check the spec using specific version of fftw
$ spack spec namd@2.12 ~cuda fftw=3 interface=python ^fftw@3.3.4 +mpi +openmp +pfft_patches ^mpich@3.4.1
Input spec
--------------------------------
namd@2.12~cuda fftw=3 interface=python
    ^fftw@3.3.4+mpi+openmp+pfft_patches

Concretized
--------------------------------
namd@2.12%gcc@11.2.0~cuda cuda_arch=none fftw=3 interface=python arch=linux-sles15-zen3
    ^charmpp@6.7.1%gcc@11.2.0~cuda~omp~papi+production~pthreads+shared+smp~syncft~tcp~tracing backend=netlrts build-target=LIBS patches=92d507d8b72ac01f114f7772510049cc7656f93daeadef46e1f2b00ad8533d2b,e636c3f88a6944c87d910d4c8fdf25146b60f57e09228497a6cc01c7dbae9569 pmi=none arch=linux-sles15-zen3
    ^fftw@3.3.4%gcc@11.2.0+mpi+openmp+pfft_patches patches=4c813199aa39c0d2b23259ee3c3a58c9c96a3c0d09fe3485a33f299b95e0d9e7 precision=double,float arch=linux-sles15-zen3
        ^autoconf@2.69%gcc@11.2.0 arch=linux-sles15-zen3
        ^automake@1.15.1%gcc@11.2.0 arch=linux-sles15-zen3
        ^libtool@2.4.6%gcc@11.2.0 arch=linux-sles15-zen3
        ^mpich@3.4.1%gcc@11.2.0~argobots+fortran+hwloc+hydra+libxml2+pci+romio+slurm~verbs+wrapperrpath device=ch4 netmod=ofi pmi=pmi arch=linux-sles15-zen3
    ^python@3.6.13%gcc@11.2.0+bz2+ctypes+dbm~debug+libxml2+lzma+nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib arch=linux-sles15-zen3
    ^tcl@8.6.11%gcc@11.2.0 arch=linux-sles15-zen3
        ^zlib@1.2.11%gcc@11.2.0+optimize+pic+shared arch=linux-sles15-zen3
$ # the spec looks okay, let's install it
$ sg <projectcode> -c 'spack install namd@2.12 ~cuda fftw=3 interface=python ^fftw@3.3.4 +mpi +openmp +pfft_patches'

$ sg <projectcode> -c 'spack install namd@2.12 ~cuda fftw=3 interface=python ^fftw@3.3.4 +mpi +openmp +pfft_patches'
...
==> Installing namd-2.12-tey6bdfijoegx3dkxfc553nckfecmhku
==> No binary for namd-2.12-tey6bdfijoegx3dkxfc553nckfecmhku found: installing from source
...
==> Error: FetchError: Manual download is required for namd. Refer to http://www.ks.uiuc.edu/Research/namd/ for download instructions.

Install a package for the entire project team

Pawsey introduced a new Spack subcommand, spack project, to perform package installations for the entire project team, rather than just a single user. 

$ spack project spec <arguments>
$ spack project install <arguments>

The key differences compared to spack are the locations where packages are installed and modulefiles are created:

• Software: /software/projects/<project-id>/setonix/<datetag>/software/<arch>/<compiler>/<package>-<version>-<unique_hash>
• Modulefiles: /software/projects/<project-id>/setonix/<datetag>/modules/<cpu-arch>/<compiler-name>/<compiler-version>/<package>/<version>-<unique_hash>.lua

Install a package for different types of hardware nodes

Pawsey systems are made up of different types of hardware nodes; for instance, Setonix includes login, compute, data-mover and visualisation nodes.

If you need to install a package with Spack for usage on a specific node, the best way to proceed is to perform the installation on that specific type of node. For instance, to install applications for your HPC jobs, use a compute node via a Slurm allocation; to install a utility for the data-mover nodes, use a data-mover node.

Using Spack-installed software

Spack-installed software can be executed by any of these methods:

• Calling the executable with the full path
• Using spack load
• Generating a modulefile with Spack and using it (recommended)

Using spack load

The load/unload command provides a simple fashion of updating the PATH environment variable to include software built by Spack. You can investigate the installed software using the find subcommand and then use the load command once you have identified the software to use. If several different versions of the package have been built by Spack, you will need to specify the desired spec:

$ spack find -vd fftw # list the variants and dependencies of fftw
-- linux-sles15-zen3 / gcc@11.2.0 -----------------------------
fftw@3.3.7~mpi~openmp~pfft_patches precision=double,float

fftw@3.3.7~mpi+openmp~pfft_patches precision=double,float

fftw@3.3.8~mpi~openmp~pfft_patches precision=double,float

$ spack load fftw # try loading any fftw
==> Error: fftw matches multiple packages.
  Matching packages:
    m4vsnhq fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
    ax6d2kl fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
    7mlxv4g fftw@3.3.8%gcc@11.2.0 arch=linux-sles15-zen3
  Use a more specific spec.
$ spack load fftw@3.3.7 # try loading a specific version
==> Error: fftw@3.3.7 matches multiple packages.
  Matching packages:
    m4vsnhq fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
    ax6d2kl fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
  Use a more specific spec.
$ spack load fftw@3.3.7+openmp # load the version and variant that uniquely identifies the package

Once a path has been updated you can use the executable.  

Generating and using a modulefile

To generate modulefiles for Spack-built packages use the idiomatic syntax below. You will then be able to see them with module avail and use them with module load/unload.

The spack command shown in Terminal 8 will generate the modulefiles in the user location configured by Pawsey, which is:

/software/projects/<project-id>/<user-id>/setonix/<datetag>/modules/<cpu-arch>/<compiler-name>/<compiler-version>/<package>/<version>-<unique_hash>.lua

In this example:

• <cpu-arch> = zen3
• <compiler-name> = gcc
• <compiler-version> = 11.2.0

$ sg <projectcode> -c 'spack module lmod refresh --delete-tree -y'  # generate LUA modulefiles for Lmod
==> Regenerating lmod module files

$ # list the fftw modules available  
$ module avail fftw
fftw/3.3.7-aormek7
fftw/3.3.7-ibdoa45
fftw/3.3.8-ggtdo27
$ # load a particular fftw library 
$ module load fftw/3.3.7-aormek7
$ # to determine if this module is the correct variant one can examine the module file itself 
$ head -4 /software/projects/pawsey0xxx/user/setonix/modules/zen3/gcc/11.2.0/fftw/3.3.7-aormek7.lua | tail -1 
-- fftw@3.3.7%gcc@11.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-sles15-zen3/aormek7
$ # compare this to other 3.3.7 module 
$ head -4 /software/projects/pawsey000x/user/setonix/modules/zen3/gcc/11.2.0/fftw/3.3.7-ibdoa45.lua | tail -1 
-- fftw@3.3.7%gcc@11.2.0~mpi+openmp+pfft_patches patches=8132c27659f992311dcf3d1500056e0f9400aa22f6824124e3607dbaa8dfe3c0 precision=double,float arch=linux-sles15-zen3/ibdoa45

Installing a software package from a user-modified source code

It is quite common for users to modify or patch the code of popular applications such as LAMMPS and Gromacs. You can use Spack to install the modified code as long as there exists a recipe for it (otherwise you can always create one).  Here is an example on how to install a modified Gromacs source code.

First, we need a Spack recipe that takes a tar file from the current working directory rather than the Internet (note that this is a long command so next line won't display properly in a small window):

$ cp -r /software/setonix/2022.11/spack/var/spack/repos/builtin/packages/gromacs $MYSOFTWARE/setonix/spack_repo/packages

Then open the file

$ vim $MYSOFTWARE/setonix/spack_repo/packages/gromacs/package.py

And replace the url property with the following line

$ url = "file://{0}/gromacs.5.1.2.tar.gz".format(os.getcwd())

Now you can use spack to install your custom version. You will need to tar your code as if it were an official distribution, but name the tar file with a non-official version, for instance gromacs.5.1.2custom.tar.gz. Then run the usual spack install command:

$ spack install gromacs@5.1.2custom

in the same directory of the tar file.

Software built with Spack

Spack makes use of dynamic libraries and rpath when building software. The use of rpath  means that executables contain the paths in which libraries are searched for at runtime. The result of using rpath  is that runtime are reproducible as executables will always point to the same library regardless of environment at runtime. A consequence of this is that if software has been built with not the desired version and build of library, it is not easy to have the executable use a different version. The package must be rebuilt with the different version of the library. 

The use of rpath  has a positive impact on the behaviour of executables provided by modules compared to Pre-Setonix systems (see Modules).

Learning more about Spack

Spack comes with a high-quality and detailed set of documentation and tutorial pages:

• Spack package manager documentation
• Spack package manager tutorial

We recommend referring to these resources for information on topics that are not covered in this Pawsey Spack page, such as:

• Spack configuration and customisation
• Creation and debugging of packages
• Use of Spack to develop an in-house code inside a Spack environment (developer workflow)