...

To familiarise yourself with Spack features and concepts, read the Spack documentation (external site).

Motivation for using Spack

...

    $ module load spack/0.1721.0

Users can invoke Spack with the spack command. spack accepts a subcommand, or Spack command, as the first argument:

...

$ spack help
$ spack help --all
$ spack <subcommand> -h

Example installation process with NAMD

...

$ spack list namd
==> 1 packages.
namd

Spack has found one package called namd. It sounds like it's the one we're looking for.

The namd package in Spack has several build time options and dependencies. These options can be explored by using the info subcommand. 

Example installation process with NAMD

Suppose we want to install the NAMD software for molecular dynamics simulations. First, let's search Spack for an appropriate package.

$ spack list namd
==> 1 packages.
namd

Spack has found one package called namd. It sounds like it's the one we're looking for.

The namd package in Spack has several build time options and dependencies. These options can be explored by using the info subcommand. 

$ spack info namd
MakefilePackage:   namd

Description:
    NAMD is a parallel molecular dynamics code designed for high-performance
    simulation of large biomolecular systems.

Homepage: http://www.ks.uiuc.edu/Research/namd/

Tags:
    None

Preferred version:
    2.14      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.14_Source.tar.gz

Safe versions:
    master    [git] https://charm.cs.illinois.edu/gerrit/namd.git on branch master
    2.15a1    [git] https://charm.cs.illinois.edu/gerrit/namd.git at tag release-2-15-alpha-1
    2.14      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.14_Source.tar.gz
    2.13      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.13_Source.tar.gz
    2.12      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.12_Source.tar.gz

Variants:
    Name [Default]      Allowed values          Description
    ================    ====================    ============================================

    cuda [off]          on, off                 Build with CUDA
    cuda_arch [none]    none, 52, 12, 61,       CUDA architecture
                        37, 60, 30, 11, 10,
                        20, 86, 35, 72, 62,
                        70, 13, 53, 32, 50,
                        75, 80, 21
    fftw [3]            none, 2, 3, mkl,        Enable the use of FFTW/FFTW3/MKL FFT/AMDFFTW
                        amdfftw
    interface [none]    none, tcl, python       Enables TCL and/or python interface

Installation Phases:
    edit    build    install

Build Dependencies:
    amdfftw  charmpp  cuda  fftw  intel-mkl  python  tcl

Link Dependencies:
    amdfftw  charmpp  cuda  fftw  intel-mkl  python  tcl

Run Dependencies:
    None

Virtual Packages:
    None

...

Dependencies are preceded by a ^. For instance, here namd  depends on charm@6charmpp@6.10.2 and fftw@3.3.8.8, with each dependency listing which compiler and variants it will be built with. 

...

$ # Before install, check the spec using specific version of fftw
$ spack spec namd@2.12 ~cuda fftw=3 interface=python ^fftw@3.3.4 +mpi +openmp +pfft_patches ^mpich@3.4.1
Input spec
--------------------------------
namd@2.12~cuda fftw=3 interface=python
    ^fftw@3.3.4+mpi+openmp+pfft_patches

Concretized
--------------------------------
namd@2.12%gcc@11.2.0~cuda cuda_arch=none fftw=3 interface=python arch=linux-sles15-zen3
    ^charmpp@6.7.1%gcc@11.2.0~cuda~omp~papi+production~pthreads+shared+smp~syncft~tcp~tracing backend=netlrts build-target=LIBS patches=92d507d8b72ac01f114f7772510049cc7656f93daeadef46e1f2b00ad8533d2b,e636c3f88a6944c87d910d4c8fdf25146b60f57e09228497a6cc01c7dbae9569 pmi=none arch=linux-sles15-zen3
    ^fftw@3.3.4%gcc@11.2.0+mpi+openmp+pfft_patches patches=4c813199aa39c0d2b23259ee3c3a58c9c96a3c0d09fe3485a33f299b95e0d9e7 precision=double,float arch=linux-sles15-zen3
        ^autoconf@2.69%gcc@11.2.0 arch=linux-sles15-zen3
        ^automake@1.15.1%gcc@11.2.0 arch=linux-sles15-zen3
        ^libtool@2.4.6%gcc@11.2.0 arch=linux-sles15-zen3
        ^mpich@3.4.1%gcc@11.2.0~argobots+fortran+hwloc+hydra+libxml2+pci+romio+slurm~verbs+wrapperrpath device=ch4 netmod=ofi pmi=pmi arch=linux-sles15-zen3
    ^python@3.6.13%gcc@11.2.0+bz2+ctypes+dbm~debug+libxml2+lzma+nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib arch=linux-sles15-zen3
    ^tcl@8.6.11%gcc@11.2.0 arch=linux-sles15-zen3
        ^zlib@1.2.11%gcc@11.2.0+optimize+pic+shared arch=linux-sles15-zen3
$ # the spec looks okay, let's install it
$ sg <projectcode> -c 'spack install namd@2.12 ~cuda fftw=3 interface=python ^fftw@3.3.4 +mpi +openmp +pfft_patches'

$ sg <projectcode> -c 'spack install namd@2.12 ~cuda fftw=3 interface=python ^fftw@3.3.4 +mpi +openmp +pfft_patches'
...
==> Installing namd-2.12-tey6bdfijoegx3dkxfc553nckfecmhku
==> No binary for namd-2.12-tey6bdfijoegx3dkxfc553nckfecmhku found: installing from source
...
==> Error: FetchError: Manual download is required for namd. Refer to http://www.ks.uiuc.edu/Research/namd/ for download instructions.

Install a package for the entire project team

Pawsey provides a utility script, called spack_project.shintroduced a new Spack subcommand, spack project, to perform package installations with Spack that need to be visible to for the entire project team, rather than just a single user. 

This script is a drop-in replacement for the spack command:

$ spack _ project .sh spec <arguments>
$ spack _ project .sh install <arguments>

The key differences compared to spack are the locations where packages are installed and modulefiles are created:

• Software: /software/projects/<project-id>/setonix/<datetag>/software/<arch>/<compiler>/<package>-<version>-<unique_hash>
• Modulefiles: /software/projects/<project-id>/setonix/<datetag>/modules/<cpu-arch>/<compiler-name>/<compiler-version>/<package>/<version>-<unique_hash>.lua

Using Spack-installed software

Spack-installed software can be executed by any of these methods:

• Calling the executable with the full path
• Using spack load
• Generating a modulefile with Spack and using it (recommended)

Using spack load

The load/unload command provides a simple fashion of updating the PATH environment variable to include software built by Spack. If several different versions of the package have been built by Spack, you will need to specify the desired spec. You can investigate the installed software using the find subcommand.

...

...

...

Install a package for different types of hardware nodes

Pawsey systems are made up of different types of hardware nodes; for instance, Setonix includes login, compute, data-mover and visualisation nodes.

If you need to install a package with Spack for usage on a specific node, the best way to proceed is to perform the installation on that specific type of node. For instance, to install applications for your HPC jobs, use a compute node via a Slurm allocation; to install a utility for the data-mover nodes, use a data-mover node.

Using Spack-installed software

Spack-installed software can be executed by any of these methods:

• Calling the executable with the full path
• Using spack load
• Generating a modulefile with Spack and using it (recommended)

Using spack load

The load/unload command provides a simple fashion of updating the PATH environment variable to include software built by Spack. You can investigate the installed software using the find subcommand and then use the load command once you have identified the software to use. If several different versions of the package have been built by Spack, you will need to specify the desired spec:

$ spack find -vd fftw # list the variants and dependencies of fftw
-- linux-sles15-zen3 / gcc@11.2.0 -----------------------------
fftw@3.3.7~mpi~openmp~pfft_patches precision=double,float

fftw@3.3.7~mpi+openmp~pfft_patches precision=double,float

fftw@3.3.8~mpi~openmp~pfft_patches precision=double,float

$ spack load --sh fftw # try loading any fftw
==> Error: fftw matches multiple packages.
  Matching packages:
    m4vsnhq fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
    ax6d2kl fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
    7mlxv4g fftw@3.3.8%gcc@11.2.0 arch=linux-sles15-zen3
  Use a more specific spec.
$ spack load --sh fftw@3.3.7 # try loading a specific version
==> Error: fftw@3.3.7 matches multiple packages.
  Matching packages:
    m4vsnhq fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
    ax6d2kl fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
  Use a more specific spec.
$ spack load --sh fftw@3.3.7+openmp # load the version and variant that uniquely identifies the package

...

/software/projects/<project-id>/<user-id>/setonix/<datetag>/modules/<cpu-arch>/<compiler-name>/<compiler-version>/<package>/<version>-<unique_hash>.lua

...

$ sg <projectcode> -c 'spack module lmod refresh --delete-tree -y'  # generate LUA modulefiles for Lmod
==> Regenerating lmod module files

$ # list the fftw modules available  
$ module avail fftw
fftw/3.3.7-aormek7
fftw/3.3.7-ibdoa45
fftw/3.3.8-ggtdo27
$ # load a particular fftw library 
$ module load fftw/3.3.7-aormek7
$ # to determine if this module is the correct variant one can examine the module file itself 
$ head -4 /software/projects/pawsey0xxx/user/setonix/modules/zen3/gcc/11.2.0/fftw/3.3.7-aormek7.lua | tail -1 
-- fftw@3.3.7%gcc@11.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-sles15-zen3/aormek7
$ # compare this to other 3.3.7 module 
$ head -4 /software/projects/pawsey000x/user/setonix/modules/zen3/gcc/11.2.0/fftw/3.3.7-ibdoa45.lua | tail -1 
-- fftw@3.3.7%gcc@11.2.0~mpi+openmp+pfft_patches patches=8132c27659f992311dcf3d1500056e0f9400aa22f6824124e3607dbaa8dfe3c0 precision=double,float arch=linux-sles15-zen3/ibdoa45

gcc/11.2.0/fftw/3.3.7-ibdoa45.lua | tail -1 
-- fftw@3.3.7%gcc@11.2.0~mpi+openmp+pfft_patches patches=8132c27659f992311dcf3d1500056e0f9400aa22f6824124e3607dbaa8dfe3c0 precision=double,float arch=linux-sles15-zen3/ibdoa45

.

Installing a software package from a user-modified source code

It is quite common for users to modify or patch the code of popular applications such as LAMMPS and Gromacs. You can use Spack to install the modified code as long as there exists a recipe for it (otherwise you can always create one).  Here is an example on how to install a modified Gromacs source code.

First, we need a Spack recipe that takes a tar file from the current working directory rather than the Internet (note that this is a long command so next line won't display properly in a small window):

$ cp -r /software/setonix/2024.05/spack/var/spack/repos/builtin/packages/gromacs $MYSOFTWARE/setonix/spack_repo/packages

Then open the file

$ vim $MYSOFTWARE/setonix/spack_repo/packages/gromacs/package.py

And replace the url property with the following line

$ url = "file://{0}/gromacs.5.1.2.tar.gz".format(os.getcwd())

Now you can use spack to install your custom version. You will need to tar your code as if it were an official distribution, but name the tar file with a non-official version, for instance gromacs.5.1.2custom.tar.gz. Then run the usual spack install command:

$ spack install gromacs@5.1.2custom

in the same directory of the tar file.

Uninstalling software built with Spack

To remove the modules created with Spack use the command:

$ spack module lmod rm

and answer questions from the promt.

To remove the installed software with Spack use the command:

$ spack uninstall <package>@<version>

and answer questions from the promt.

Software built with Spack

Spack makes use of dynamic libraries and rpath when building software. The use of rpath  means that executables contain the paths in which libraries are search searched for at runtime. The result of using rpath  is that runtime are reproducible as executables will always point to the same library regardless of environment at runtime. A consequence of this is that if software has been built with not the desired version and build of library, it is not easy to have the executable use a different version. The package must be rebuilt with the different version of the library. 

...