Modules
The Module system is used to provide users with access to software on HPC systems. This page explains the basics concepts of modules and how to use them.
Introduction
Our HPC systems have several tens of different installed software packages, each one requiring its own set of environment variables to run properly. A module system is a simple and efficient way to keep the shell environment clean, by collecting variables related to each package in a separate unit, or module, that can be loaded only when required. Typical environment variables that are updated when loading a module include PATH, LD_LIBRARY_PATH and MANPATH; each package usually defines other specific variables, too.
There are two different implementations of the module system: Environment Modules and Lmod. Pawsey adopts LMOD for its supercomputing systems.
Software installed using Spack does not need modules of its dependencies loaded because paths to those are hardcoded in the executables and libraries using the RPATH header.
Common tasks handled by modules
There are three types of operations commonly executed by modules when loaded:
- Variable definition: see table 1 for a list of the most common variables and their use.
- Prerequisite module loading: these are scientific libraries and other packages that must be loaded for the application to work.
- Declaration of conflicts: this is typically used to declare that a given package has not been compiled with a certain compiler or for a certain CPU architecture.
Table 1. List of environment variables that a module usually modifies when it is loaded
| Variable | Definition | Scope |
|---|---|---|
PATH | Path to the executable | Runtime of the current package |
LD_LIBRARY_PATH | Path to the dynamically loaded shared library | Runtime of another package that links to the current library package |
LIBRARY_PATH | Path to the shared library | Compile time of another package that links to current library package |
CPATH | Path to the header files | Compile time of another package that links to current library package |
MANPATH | Path to documentation files loaded by "man" | Anytime |
Module-specific path | Some packages for instance need to know the path of their root installation folder | Specific to each package |
How to use modules
Modules are configured and ready to use upon login on all Pawsey HPC systems. Any supported software has its corresponding module. Here we cover key commands; for a complete list, see the Lmod documentation (external site) or execute module help. The basic module commands are listed in table 2.
Table 2. List of modules commands
| Command | What it does |
|---|---|
module spider | list all the modules that are available on the machine (loaded and not loaded) |
module avail | list the modules that are available AND can be loaded without conflicts on that machine (loaded and not loaded) based on currently loaded modules |
module list | list all of the loaded modules in your current shell environment |
module load | load the specified module to your environment |
module unload | unload the specified module from your environment |
module swap <modA> <modB> | swap modules (unloading modA and loading modB) |
module display | output the content of the modulefile without loading it |
module show | like module display |
module whatis | output summary information about the specified module without loading it |
Typical output of a module command
Logging into a system will typically load a default set of modules. The numbered list of these modules can be seen by running module list.
When logging into Cray systems, the PrgEnv-cray module is loaded by default, providing the Cray compiler suite, cce, the MPI module, cray-mpich, and others. Often the Slurm scheduling system is a module.
$ module list 1) craype-x86-milan 5) xpmem/2.2.40-2.1_2.56__g3cf3325.shasta 9) cray-mpich/8.1.14 2) libfabric/1.11.0.4.75 6) gcc/11.2.0 10) cray-libsci/21.08.1.2 3) craype-network-ofi 7) craype/2.7.14 11) PrgEnv-gnu/8.3.2 4) perftools-base/21.12.0 8) cray-dsmml/0.2.2 12) pawsey
Non-Cray systems will typically have fewer modules loaded by default, often providing the GCC compiler module and the Slurm module.
$ module list Currently Loaded Modules: 1) cascadelake/1.0 2) pawseytools/1.29 3) slurm/20.02.3 4) gcc/8.3.0
Note
The pawseytools module is also loaded by default on all Pawsey systems. This module contains a collection of useful utilities provided by Pawsey staff.
slurm is also loaded by default as it provides the Slurm job scheduler.
Loading, unloading, and swapping modules
On Setonix, when loading a package built with Spack both the module name and version must be provided. Providing only the module name results in an error:
$ module load python
Lmod has detected the following error: Default module versions are disabled by your systems administrator.
Please load this module as <name>/<version>.
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
python/3.10.4 /software/setonix/current/modules/zen3/gcc/11.2.0/programming-languages/python/3.10.4.lua
On the other hand, Cray modules can still be loaded by providing the module name only; providing the version is optional, but recommended for reproducibility.
The following example loads the python/3.9.7 module, then swaps python version to 3.10.4, finally unloads the python module.
$ module load python/3.9.7 $ module swap python/3.9.7 python/3.10.4 $ module unload python
The module swap command is particularly useful in the context of switching between different programming environments on Cray system:
$ module swap PrgEnv-gnu PrgEnv-aocc $ module swap PrgEng-aocc PrgEnv-cray $ module swap PrgEnv-cray PrgEnv-gnu
or compilers:
$ module swap gcc aocc/3.2.0 $ module swap aocc cce/13.0.2 $ module swap cce gcc/11.2.0
Displaying information on modules
Further information of a descriptive nature can be obtained by using the commands module whatis and module display. These commands show the paths to executables, libraries, any other modules that are loaded, conflicts, and importantly, may define variables needed by the application at runtime.
Using modules in Slurm scripts
The most typical use of modules would be for loading applications to be run using the SLURM scheduler. An example of running the GCC version of the lammps package is provided in listing 1.
#!/bin/bash -l #SBATCH --account=<project> #SBATCH --partition=work #SBATCH --time=24:00:00 #SBATCH --nodes=2 #SBATCH --job-name=lammps-job module swap PrgEnv-cray PrgEnv-gnu module load lammps/20210929.3 srun --export=all -n 48 lmp_mpi -in lammps.inp -log lammps.log
module load commands in the shell initialisation scripts (~/.bashrc, ~/.tcshrc and so on). This practice makes the user environment less reproducible by colleagues and by Pawsey staff.System-wide, group and user modules
The module environment looks for module files in a sequence of directories stored in the MODULEPATH environment variable.
$ echo $MODULEPATH /opt/cray/pe/lmod/modulefiles/mpi/crayclang/10.0/ofi/1.0/cray-mpich/8.0:/opt/cray/pe/lmod/modulefiles/comnet/crayclang/10.0/ofi/1.0:/opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0:/opt/cray/pe/lmod/modulefiles/mix_compilers:/opt/cray/pe/lmod/modulefiles/perftools/21.12.0
These module files can be stored at user, group or system level, with this precedence. As a result, if a user installs their own custom version of a system-wide supported software, their module will be selected instead of the system-wide one when running module load.
Modules for user-based Spack installations of software
Users can install their own software using the Spack package manager. Spack also provides commands to produce a module file for this installation (see Spack for details of how to install software). The resulting module files follow a different naming convention to that of the system modules, producing modules with <name>/<version>-<unique_hash>.
Creating a module file
If a group or user wants to create their own module file for a custom installed application, they should locate and name it according to the rules of LMOD. For more information, check the Lmod documentation.
These new modules residing in custom directories can be added to the module path with the module use command.
# if path is <PATH-TO-MODULES>/<software>/<version> $ module use <PATH-TO-MODULES> $ module load <software>/<version>
Module cache
The Lmod module environment utilises a module cache to speed up module commands. You can check if the module environment is based on Lua using --version .
$ module --version
Modules based on Lua: Version 8.3.1 2020-02-16 19:46 :z
by Robert McLay mclay@tacc.utexas.edu
To force the cache to update, remove the cache directory and run module avail.
rm -rf ~/.lmod.d/.cache module avail
Modules and reproducibility
Software on Setonix is built using the Spack package manager. Spack makes use of dynamic linking and rpath. The result of this is that modules providing domain software are self-contained and do not require other modules to be loaded (except for Python dependencies). The executables provided by the modules themselves contain the paths to the libraries to be used at run time, which improves reproducibility.
This behaviour differs from modules on pre-Setonix systems.
Related pages
External links
- Lmod documentation
- Lmod home page
- Environment Modules documentation
- Modules homepage
- Modulefile examples from simple to complex