Modules
The Module system is used to provide users with access to software on HPC systems. This page explains the basics concepts of modules and how to use them.
On this page:
- 1 Introduction
- 2 How to use modules
- 3 System-wide, group and user modules
- 4 Modules and reproducibility
- 5 Related pages
- 6 External links
Introduction
Our HPC systems have several tens of different installed software packages, each one requiring its own set of environment variables to run properly. A module system is a simple and efficient way to keep the shell environment clean, by collecting variables related to each package in a separate unit, or module, that can be loaded only when required. Typical environment variables that are updated when loading a module include PATH, LD_LIBRARY_PATH and MANPATH; each package usually defines other specific variables, too.
There are two different implementations of the module system: Environment Modules and Lmod. Pawsey adopts LMOD for its supercomputing systems.
Software installed using Spack does not need modules of its dependencies loaded because paths to those are hardcoded in the executables and libraries using the RPATH header.
Common tasks handled by modules
There are three types of operations commonly executed by modules when loaded:
Variable definition: see table 1 for a list of the most common variables and their use.
Prerequisite module loading: these are scientific libraries and other packages that must be loaded for the application to work.
Declaration of conflicts: this is typically used to declare that a given package has not been compiled with a certain compiler or for a certain CPU architecture.
Table 1. List of environment variables that a module usually modifies when it is loaded
Variable | Definition | Scope |
|---|---|---|
| Path to the executable | Runtime of the current package |
| Path to the dynamically loaded shared library | Runtime of another package that links to the current library package |
| Path to the shared library | Compile time of another package that links to current library package |
| Path to the header files | Compile time of another package that links to current library package |
| Path to documentation files loaded by "man" | Anytime |
Module-specific path | Some packages for instance need to know the path of their root installation folder | Specific to each package |
How to use modules
Modules are configured and ready to use upon login on all Pawsey HPC systems. Any supported software has its corresponding module. Here we cover key commands; for a complete list, see the Lmod documentation (external site) or execute module help. The basic module commands are listed in Table 2.
Table 2. List of module commands
Command | What it does |
|
|---|---|---|
| list all the modules that are available on the machine (loaded and not loaded) |
|
| list the modules that are available AND can be loaded without conflicts on that machine (loaded and not loaded) based on currently loaded modules |
|
| list all of the loaded modules in your current shell environment |
|
| load the specified module to your environment |
|
| unload the specified module from your environment |
|
| swap modules (unloading modA and loading modB) |
|
| output the content of the modulefile without loading it |
|
| like module display |
|
| output summary information about the specified module without loading it |
|
Typical output of a module command
Logging into a system will typically load a default set of modules. The numbered list of these modules can be seen by running module list.
When logging into Pawsey systems, some modules are loaded by default. In Cray systems this includes many cray related modules. Often the Slurm scheduling system is a module.
Terminal 1. Module list for Setonix
$ module list
1) craype-x86-milan 6) pawseyenv/2024.05 11) cray-libsci/23.09.1.1
2) libfabric/1.15.2.0 7) gcc/12.2.0 12) PrgEnv-gnu/8.4.0
3) craype-network-ofi 8) craype/2.7.23 13) pawsey
4) perftools-base/23.09.0 9) cray-dsmml/0.2.2 14) pawseytools
5) xpmem/2.5.2-2.4_3.47__gd0f7936.shasta 10) cray-mpich/8.1.27 15) slurm/22.05.2Non-Cray systems will typically have fewer modules loaded by default, often providing the GCC compiler module and the Slurm module.
Terminal 2. Module list for non-Cray systems
$ module list
Currently Loaded Modules:
1) cascadelake/1.0 2) pawseytools/1.29 3) slurm/20.02.3 4) gcc/8.3.0Note
The pawseytools module is also loaded by default on all Pawsey systems. This module contains a collection of useful utilities provided by Pawsey staff.
slurm is also loaded by default as it provides the Slurm job scheduler.
Loading, unloading, and swapping modules
On Setonix, when loading a package built with Spack both the module name and version must be provided. Providing only the module name results in an error:
Terminal 3. Example module commands
$ module load python
Lmod has detected the following error: Default module versions are disabled by your systems
administrator.
Please load this module as <name>/<version>.
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
python/3.11.6 /software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages/python/3.11.6.luaOn the other hand, Cray modules can still be loaded by providing the module name only; providing the version is optional, but recommended for reproducibility.
The following example loads the python/3.9.15 module, then swaps python version to 3.11.6, finally unloads the python module.
Terminal 4. Example module commands
$ module load python/3.9.15
$ module swap python/3.9.15 python/3.11.6
$ module unload pythonThe module swap command is particularly useful in the context of switching between different programming environments on Cray system:
Terminal 5. Swapping programming environments
$ module swap PrgEnv-gnu PrgEnv-aocc
$ module swap PrgEng-aocc PrgEnv-cray
$ module swap PrgEnv-cray PrgEnv-gnuor compilers:
Terminal 6. Swapping compilers
$ module swap gcc aocc/3.2.0
$ module swap aocc cce/16.0.1
$ module swap cce gcc/12.2.0Displaying information on modules
Further information of a descriptive nature can be obtained by using the commands module whatis and module display. These commands show the paths to executables, libraries, any other modules that are loaded, conflicts, and importantly, may define variables needed by the application at runtime.
Terminal 7. Example module whatis
$ module whatis cray-hdf5
cray-hdf5/1.12.2.7 : The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.Terminal 8. Example module display
$ module display cray-hdf5
----------------------------------------------------------------------------
/opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0/cray-hdf5/1.12.1.1.lua:
----------------------------------------------------------------------------
family("hdf5")
conflict("PrgEnv-pathscale")
conflict("cray-hdf5")
conflict("cray-hdf5-parallel")
help([[Release info: /opt/cray/pe/hdf5/1.12.2.7/release_info]])
whatis("The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.")
prepend_path("PATH","/opt/cray/pe/hdf5/1.12.2.7/bin")
prepend_path("PKG_CONFIG_PATH","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1/lib/pkgconfig")
prepend_path("PE_PKGCONFIG_LIBS","hdf5_hl:hdf5")
setenv("PE_HDF5_PKGCONFIG_LIBS","hdf5_hl:hdf5")
prepend_path("PE_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
setenv("PE_HDF5_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
prepend_path("PE_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("PE_HDF5_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("CRAY_HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7")
setenv("PE_HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7")
setenv("CRAY_HDF5_VERSION","1.12.2.7")
setenv("CRAY_HDF5_PREFIX","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
setenv("HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
setenv("HDF5_ROOT","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
prepend_path("CRAY_LD_LIBRARY_PATH","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1/lib")
prepend_path("MODULEPATH","/opt/cray/pe/lmod/modulefiles/hdf5/gnu/8.0/cray-hdf5/1.12.2")Using modules in Slurm scripts
The most typical use of modules would be for loading applications to be run using the SLURM scheduler. An example of running the GCC version of the lammps package is provided in Listing 1.
Listing 1. Sample Slurm script
#!/bin/bash -l
#SBATCH --account=<project>
#SBATCH --partition=work
#SBATCH --time=24:00:00
#SBATCH --nodes=2
#SBATCH --job-name=lammps-job
module swap PrgEnv-cray PrgEnv-gnu
module load lammps/20230802.3
srun --export=all -n 48 lmp_mpi -in lammps.inp -log lammps.logDo not add module load commands in the shell initialisation scripts (~/.bashrc, ~/.tcshrc and so on). This practice makes the user environment less reproducible by colleagues and by Pawsey staff.
System-wide, group and user modules
The module environment looks for module files in a sequence of directories stored in the MODULEPATH environment variable.
Terminal 8. Example module path
$ echo $MODULEPATH
/opt/cray/pe/lmod/modulefiles/mpi/gnu/8.0/ofi/1.0/cray-mpich/8.0:/software/projects/pawsey0001/setonix/2024.05/modules/zen3/gcc/12.2.0:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/astro-applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/bio-applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/libraries:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/utilities:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/visualisation:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/python-packages:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/benchmarking:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/developer-tools:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/dependencies:/software/setonix/2024.05/custom/modules/zen3/gcc/12.2.0/custom:/opt/cray/pe/lmod/modulefiles/comnet/gnu/8.0/ofi/1.0:/opt/cray/pe/lmod/modulefiles/mix_compilers:/opt/cray/pe/lmod/modulefiles/compiler/gnu/8.0:/software/setonix/2024.05/containers/views/modules:/software/setonix/2024.05/pawsey/modules:/software/projects/pawsey0001/espinosa/setonix/2024.05/containers/views/modules:/opt/cray/pe/lmod/modulefiles/perftools/23.09.0:/opt/cray/pe/lmod/modulefiles/net/ofi/1.0:/opt/cray/pe/lmod/modulefiles/cpu/x86-milan/1.0:/opt/cray/pe/modulefiles/Linux:/opt/cray/pe/modulefiles/Core:/opt/cray/pe/lmod/lmod/modulefiles/Core:/opt/cray/pe/lmod/modulefiles/core:/opt/cray/pe/lmod/modulefiles/craype-targets/default:/opt/pawsey/modulefiles:/software/pawsey/modulefiles:/opt/cray/modulefilesThese module files can be stored at user, group or system level, with this precedence. As a result, if a user installs their own custom version of a system-wide supported software, their module will be selected instead of the system-wide one when running module load.
Modules for user-based Spack installations of software
Users can install their own software using the Spack package manager. Spack also provides commands to produce a module file for this installation (see Spack for details of how to install software). The resulting module files follow a different naming convention to that of the system modules, producing modules with <name>/<version>-<unique_hash>.
Terminal 10. Example spack installation and module production
$ module load spack/0.21.0 # load spack module
$ sg <PROJECT_NAME> -c 'spack install abyss' # install abyss, de novo, parallel, paired-end sequence assembler
$ sg <PROJECT_NAME> -c 'spack module lmod refresh abyss' # produce module file
$ module use <PATH-TO-MODULES> # if necessary, update the module path
$ module avail abyss
----------------------------------------------------------------------------
/software/projects/<project-id>/<user-name>/setonix/<DATE_TAG>/modules/zen3/gcc/12.2.0/abyss/2.3.1-gb3u5yw.lua
----------------------------------------------------------------------------
abyss/2.3.1-gb3u5yw
$ module whatis abyss/2.3.1-gb3u5yw
byss/2.3.1-gb3u5yw : Name : abyss
abyss/2.3.1-gb3u5yw : Short description : ABySS is a de novo, parallel, paired-end sequence assembler that is designed for short reads. The single-processor version is useful for assembling genomes up to 100 Mbases in size.
abyss/2.3.1-gb3u5yw : Version : 2.3.1
abyss/2.3.1-gb3u5yw : Compiler : gcc@12.2.0
abyss/2.3.1-gb3u5yw : Flags : cflags="-O3" cppflags="-O3" cxxflags="-O3" fflags="-O3 -fallow-argument-mismatch"
abyss/2.3.1-gb3u5yw : Target : zen3
abyss/2.3.1-gb3u5yw : Build date : Tue Jun 14 09:02:28 2024
abyss/2.3.1-gb3u5yw : Spack configuration : maxk=128
abyss/2.3.1-gb3u5yw : Path : /software/projects/pawsey0000/matilda/setonix/<DATE-TAG>/software/cray-sles15-zen3/gcc-12.2.0/abyss-2.3.1-gb3u5ywgkvuni2vw5pvx23rg6nwkx7t6
abyss/2.3.1-gb3u5yw : Configure options : --with-boost=/software/setonix/<DATE_TAG>/cray-sles15-zen3/gcc-12.2.0/boost-1.83.0-fzeiorblgokslk6yvksmsbfahga336cl --with-sqlite=/usr --with-mpi=/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1 --enable-maxk=128Creating a module file
If a group or user wants to create their own module file for a custom installed application, they should locate and name it according to the rules of LMOD. For more information, check the Lmod documentation.
These new modules residing in custom directories can be added to the module path with the module use command.
Terminal 11. Adding directories to the module path
# if path is <PATH-TO-MODULES>/<software>/<version>
$ module use <PATH-TO-MODULES>
$ module load <software>/<version>Module cache
The Lmod module environment utilises a module cache to speed up module commands. You can check if the module environment is based on Lua using --version.
Terminal 12. Getting the date of the module cache
$ module --version
Modules based on Lua: Version 8.7.19 2023-01-17 10:45 -06:00
by Robert McLay mclay@tacc.utexas.eduTo force the cache to update, remove the cache directory and run module avail.
Terminal 13. Cleaning the module cache
rm -rf ~/.cache/lmod
rm -rf ~/.lmod.d/.cache
module availModules and reproducibility
Software on Setonix is built using the Spack package manager. Spack makes use of dynamic linking and rpath. The result of this is that modules providing domain software are self-contained and do not require other modules to be loaded (except for Python dependencies). The executables provided by the modules themselves contain the paths to the libraries to be used at run time, which improves reproducibility.
This behaviour differs from modules on pre-Setonix systems.
Related pages
External links