The Module system is used to provide users with access to software on HPC systems. This page explains the basics concepts of modules and how to use them.

On this Page

Introduction

Our HPC systems have several tens of different installed software packages, each one requiring its own set of environment variables to run properly. A module system is a simple and efficient way to keep the shell environment clean, by collecting variables related to each package in a separate unit, or module, that can be loaded only when required. Typical environment variables that are updated when loading a module include PATH, LD_LIBRARY_PATH and MANPATH; each package usually defines other specific variables, too.

There are two different implementations of the module system: Environment Modules and Lmod. Pawsey adopts LMOD for its supercomputing systems.

Software installed using Spack does not need modules of its dependencies loaded because paths to those are hardcoded in the executables and libraries using the RPATH header.

Common tasks handled by modules

There are three types of operations commonly executed by modules when loaded:

Variable definition: see table 1 for a list of the most common variables and their use.
Prerequisite module loading: these are scientific libraries and other packages that must be loaded for the application to work.
Declaration of conflicts: this is typically used to declare that a given package has not been compiled with a certain compiler or for a certain CPU architecture.

Table 1. List of environment variables that a module usually modifies when it is loaded

Variable	Definition	Scope
`PATH`	Path to the executable	Runtime of the current package
`LD_LIBRARY_PATH`	Path to the dynamically loaded shared library	Runtime of another package that links to the current library package
`LIBRARY_PATH`	Path to the shared library	Compile time of another package that links to current library package
`CPATH`	Path to the header files	Compile time of another package that links to current library package
`MANPATH`	Path to documentation files loaded by "man"	Anytime
Module-specific path	Some packages for instance need to know the path of their root installation folder	Specific to each package

How to use modules

Modules are configured and ready to use upon login on all Pawsey HPC systems. Any supported software has its corresponding module. Here we cover key commands; for a complete list, see the Lmod documentation (external site) or execute module help. The basic module commands are listed in table 2.

Table 2. List of modules commands

Command	What it does
`module spider`	list all the modules that are available on the machine (loaded and not loaded)
`module avail`	list the modules that are available AND can be loaded without conflicts on that machine (loaded and not loaded) based on currently loaded modules
`module list`	list all of the loaded modules in your current shell environment
`module load`	load the specified module to your environment
`module unload`	unload the specified module from your environment
`module swap <modA> <modB>`	swap modules (unloading modA and loading modB)
`module display`	output the content of the modulefile without loading it
`module show`	like module display
`module whatis`	output summary information about the specified module without loading it

Typical output of a `module` command

Logging into a system will typically load a default set of modules. The numbered list of these modules can be seen by running module list.

When logging into Pawsey systems, some modules are loaded by default. In Cray systems this includes many cray related modules. Often the Slurm scheduling system is a module.

Cray module lists

Terminal 1. module list for Setonix

$ module list   
  1) craype-x86-milan                        6) pawseyenv/2024.05  11) cray-libsci/23.09.1.1
  2) libfabric/1.15.2.0                      7) gcc/12.2.0         12) PrgEnv-gnu/8.4.0
  3) craype-network-ofi                      8) craype/2.7.23      13) pawsey
  4) perftools-base/23.09.0                  9) cray-dsmml/0.2.2   14) pawseytools
  5) xpmem/2.5.2-2.4_3.47__gd0f7936.shasta  10) cray-mpich/8.1.27  15) slurm/22.05.2

Non-Cray systems will typically have fewer modules loaded by default, often providing the GCC compiler module and the Slurm module.

Terminal 2. module list for non-cray systems

$ module list

Currently Loaded Modules:
  1) cascadelake/1.0   2) pawseytools/1.29   3) slurm/20.02.3   4) gcc/8.3.0

Note

The pawseytools module is also loaded by default on all Pawsey systems. This module contains a collection of useful utilities provided by Pawsey staff.

slurm is also loaded by default as it provides the Slurm job scheduler.

Loading, unloading, and swapping modules

On Setonix, when loading a package built with Spack both the module name and version must be provided. Providing only the module name results in an error:

Terminal 3. Example module commands

$ module load python
Lmod has detected the following error:  Default module versions are disabled by your systems
administrator.

	Please load this module as <name>/<version>.

While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    python/3.11.6    /software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages/python/3.11.6.lua

On the other hand, Cray modules can still be loaded by providing the module name only; providing the version is optional, but recommended for reproducibility.

The following example loads the python/3.9.15 module, then swaps python version to 3.11.6, finally unloads the python module.

Terminal 4. Example module commands

$ module load python/3.9.15
$ module swap python/3.9.15 python/3.11.6
$ module unload python

The module swap command is particularly useful in the context of switching between different programming environments on Cray system:

Terminal 5. Swapping programming environments

$ module swap PrgEnv-gnu PrgEnv-aocc 
$ module swap PrgEng-aocc PrgEnv-cray
$ module swap PrgEnv-cray PrgEnv-gnu

or compilers:

Terminal 6. Swapping compilers

$ module swap gcc aocc/3.2.0
$ module swap aocc cce/16.0.1
$ module swap cce gcc/12.2.0

Displaying information on modules

Further information of a descriptive nature can be obtained by using the commands module whatis and module display. These commands show the paths to executables, libraries, any other modules that are loaded, conflicts, and importantly, may define variables needed by the application at runtime.

Terminal 7. Example module whatis

$ module whatis cray-hdf5
cray-hdf5/1.12.2.7  : The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.

Terminal 8. Example module display

$  module display cray-hdf5
----------------------------------------------------------------------------
   /opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0/cray-hdf5/1.12.1.1.lua:
----------------------------------------------------------------------------
family("hdf5")
conflict("PrgEnv-pathscale")
conflict("cray-hdf5")
conflict("cray-hdf5-parallel")
help([[Release info:  /opt/cray/pe/hdf5/1.12.2.7/release_info]])
whatis("The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.")
prepend_path("PATH","/opt/cray/pe/hdf5/1.12.2.7/bin")
prepend_path("PKG_CONFIG_PATH","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1/lib/pkgconfig")
prepend_path("PE_PKGCONFIG_LIBS","hdf5_hl:hdf5")
setenv("PE_HDF5_PKGCONFIG_LIBS","hdf5_hl:hdf5")
prepend_path("PE_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
setenv("PE_HDF5_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
prepend_path("PE_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("PE_HDF5_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("CRAY_HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7")
setenv("PE_HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7")
setenv("CRAY_HDF5_VERSION","1.12.2.7")
setenv("CRAY_HDF5_PREFIX","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
setenv("HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
setenv("HDF5_ROOT","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
prepend_path("CRAY_LD_LIBRARY_PATH","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1/lib")
prepend_path("MODULEPATH","/opt/cray/pe/lmod/modulefiles/hdf5/gnu/8.0/cray-hdf5/1.12.2")

Using modules in Slurm scripts

The most typical use of modules would be for loading applications to be run using the SLURM scheduler. An example of running the GCC version of the lammps package is provided in listing 1.

Listing 1. Sample Slurm script

#!/bin/bash -l
#SBATCH --account=<project>
#SBATCH --partition=work
#SBATCH --time=24:00:00
#SBATCH --nodes=2
#SBATCH --job-name=lammps-job

module swap PrgEnv-cray PrgEnv-gnu
module load lammps/20230802.3

srun --export=all -n 48 lmp_mpi -in lammps.inp -log lammps.log

Do not add module load commands in the shell initialisation scripts (~/.bashrc, ~/.tcshrc and so on). This practice makes the user environment less reproducible by colleagues and by Pawsey staff.

System-wide, group and user modules

The module environment looks for module files in a sequence of directories stored in the MODULEPATH environment variable.

Terminal 9. Example module path

$ echo $MODULEPATH 
/opt/cray/pe/lmod/modulefiles/mpi/gnu/8.0/ofi/1.0/cray-mpich/8.0:/software/projects/pawsey0001/setonix/2024.05/modules/zen3/gcc/12.2.0:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/astro-applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/bio-applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/libraries:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/utilities:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/visualisation:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/python-packages:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/benchmarking:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/developer-tools:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/dependencies:/software/setonix/2024.05/custom/modules/zen3/gcc/12.2.0/custom:/opt/cray/pe/lmod/modulefiles/comnet/gnu/8.0/ofi/1.0:/opt/cray/pe/lmod/modulefiles/mix_compilers:/opt/cray/pe/lmod/modulefiles/compiler/gnu/8.0:/software/setonix/2024.05/containers/views/modules:/software/setonix/2024.05/pawsey/modules:/software/projects/pawsey0001/espinosa/setonix/2024.05/containers/views/modules:/opt/cray/pe/lmod/modulefiles/perftools/23.09.0:/opt/cray/pe/lmod/modulefiles/net/ofi/1.0:/opt/cray/pe/lmod/modulefiles/cpu/x86-milan/1.0:/opt/cray/pe/modulefiles/Linux:/opt/cray/pe/modulefiles/Core:/opt/cray/pe/lmod/lmod/modulefiles/Core:/opt/cray/pe/lmod/modulefiles/core:/opt/cray/pe/lmod/modulefiles/craype-targets/default:/opt/pawsey/modulefiles:/software/pawsey/modulefiles:/opt/cray/modulefiles

These module files can be stored at user, group or system level, with this precedence. As a result, if a user installs their own custom version of a system-wide supported software, their module will be selected instead of the system-wide one when running module load.

Modules for user-based Spack installations of software

Users can install their own software using the Spack package manager. Spack also provides commands to produce a module file for this installation (see Spack for details of how to install software). The resulting module files follow a different naming convention to that of the system modules, producing modules with <name>/<version>-<unique_hash>.

Terminal 10. Example spack installation and module production

$ module load spack/0.21.0 # load spack module
$ sg <PROJECT_NAME> -c 'spack install abyss' # install abyss, de novo, parallel, paired-end sequence assembler
$ sg <PROJECT_NAME> -c 'spack module lmod refresh abyss' # produce module file 
$ module use <PATH-TO-MODULES> # if necessary, update the module path
$ module avail abyss
----------------------------------------------------------------------------
/software/projects/<project-id>/<user-name>/setonix/<DATE_TAG>/modules/zen3/gcc/12.2.0/abyss/2.3.1-gb3u5yw.lua
----------------------------------------------------------------------------
  abyss/2.3.1-gb3u5yw
$ module whatis abyss/2.3.1-gb3u5yw
byss/2.3.1-gb3u5yw : Name : abyss
abyss/2.3.1-gb3u5yw : Short description : ABySS is a de novo, parallel, paired-end sequence assembler that is designed for short reads. The single-processor version is useful for assembling genomes up to 100 Mbases in size.
abyss/2.3.1-gb3u5yw : Version : 2.3.1
abyss/2.3.1-gb3u5yw : Compiler : gcc@12.2.0
abyss/2.3.1-gb3u5yw : Flags :  cflags="-O3" cppflags="-O3" cxxflags="-O3" fflags="-O3 -fallow-argument-mismatch"
abyss/2.3.1-gb3u5yw : Target : zen3
abyss/2.3.1-gb3u5yw : Build date : Tue Jun 14 09:02:28 2024
abyss/2.3.1-gb3u5yw : Spack configuration :  maxk=128
abyss/2.3.1-gb3u5yw : Path : /software/projects/pawsey0000/matilda/setonix/<DATE-TAG>/software/cray-sles15-zen3/gcc-12.2.0/abyss-2.3.1-gb3u5ywgkvuni2vw5pvx23rg6nwkx7t6
abyss/2.3.1-gb3u5yw : Configure options : --with-boost=/software/setonix/<DATE_TAG>/cray-sles15-zen3/gcc-12.2.0/boost-1.83.0-fzeiorblgokslk6yvksmsbfahga336cl --with-sqlite=/usr --with-mpi=/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1 --enable-maxk=128

Creating a module file

If a group or user wants to create their own module file for a custom installed application, they should locate and name it according to the rules of LMOD. For more information, check the Lmod documentation.

These new modules residing in custom directories can be added to the module path with the module use command.

Terminal 11. Adding directories to the module path

# if path is <PATH-TO-MODULES>/<software>/<version>
$ module use <PATH-TO-MODULES>
$ module load <software>/<version>

Module cache

The Lmod module environment utilises a module cache to speed up module commands. You can check if the module environment is based on Lua using --version .

Terminal 12. Getting the date of the module cache

$ module --version

Modules based on Lua: Version 8.7.19  2023-01-17 10:45 -06:00
    by Robert McLay mclay@tacc.utexas.edu

To force the cache to update, remove the cache directory and run module avail.

Terminal 13. Cleaning the module cache

rm -rf ~/.cache/lmod
rm -rf ~/.lmod.d/.cache
module avail

Modules and reproducibility

Software on Setonix is built using the Spack package manager. Spack makes use of dynamic linking and rpath. The result of this is that modules providing domain software are self-contained and do not require other modules to be loaded (except for Python dependencies). The executables provided by the modules themselves contain the paths to the libraries to be used at run time, which improves reproducibility.

This behaviour differs from modules on pre-Setonix systems.

Module loading behaviour on pre-Setonix systems: click here to expand...

On pre-Setonix systems, all executables made extensive use of the LD_LIBRARY_PATH environment variable to include libraries at run time. Consequently, the order of how modules are loaded could affect the runtime.

Consider the following example: module A depends on functionality that can be provided by either library B or C . Module A by default loads B .
Four possible scenarios are:

Scenario 1

Scenario 2

Scenario 3

Scenario 4

Scenario 5

module load A

module load B

module load A

module load A

module load C

module load A

module load C

module load B

module load C

module load A

Result: A+B Result: A+B Result: A+C Result: A+B Result: A+B

On Setonix the order does not matter as A contains links to B and hence the result will always be A+B.

Related pages

Spack package manager
See Compiling for discussion of dynamic versus static linking and use of rpath

External links

Browser not supported