Excerpt |
---|
The Module system is used to provide users with access to software on HPC systems. This page explains the basics concepts of modules and how to use them. |
Introduction
Our HPC systems have several tens of different installed software packages, each one requiring its own set of environment variables to run properly. A module system is a simple and efficient way to keep the shell environment clean, by collecting variables related to each package in a separate unit, or module, that can be loaded only when required. Typical environment variables that are updated when loading a module include PATH
, LD_LIBRARY_PATH
and MANPATH
; each package usually defines other specific variables, too.
There are two different implementations of the module system: Environment Modules and Lmod. Pawsey adopts LMOD for its supercomputing systems.
Column |
---|
|
Note |
---|
Software installed using Spack does not need modules of its dependencies loaded because paths to those are hardcoded in the executables and libraries using the RPATH header. |
|
Common tasks handled by modules
There are three types of operations commonly executed by modules when loaded:
- Variable definition: see table 1 for a list of the most common variables and their use.
- Prerequisite module loading: these are scientific libraries and other packages that must be loaded for the application to work.
- Declaration of conflicts: this is typically used to declare that a given package has not been compiled with a certain compiler or for a certain CPU architecture.
Column |
---|
Table 1. List of environment variables that a module usually modifies when it is loaded Variable | Definition | Scope |
---|
PATH | Path to the executable | Runtime of the current package | LD_LIBRARY_PATH | Path to the dynamically loaded shared library | Runtime of another package that links to the current library package | LIBRARY_PATH | Path to the shared library | Compile time of another package that links to current library package | CPATH | Path to the header files | Compile time of another package that links to current library package | MANPATH | Path to documentation files loaded by "man" | Anytime | Module-specific path | Some packages for instance need to know the path of their root installation folder | Specific to each package |
|
How to use modules
Modules are configured and ready to use upon login on all Pawsey HPC systems. Any supported software has its corresponding module. Here we cover key commands; for a complete list, see the Lmod documentation (external site) or execute module help
. The basic module commands are listed in table 2.
Column |
---|
Table 2. List of modules commands Command | What it does |
---|
module spider | list all the modules that are available on the machine (loaded and not loaded) | module avail | list the modules that are available AND can be loaded without conflicts on that machine (loaded and not loaded) based on currently loaded modules | module list | list all of the loaded modules in your current shell environment | module load | load the specified module to your environment | module unload | unload the specified module from your environment | module swap <modA> <modB> | swap modules (unloading modA and loading modB) | module display | output the content of the modulefile without loading it | module show | like module display | module whatis | output summary information about the specified module without loading it |
|
Typical output of a module
command
Logging into a system will typically load a default set of modules. The numbered list of these modules can be seen by running module list.
When logging into Cray systems, the PrgEnv-cray
module is loaded by default, providing the Cray compiler suite, cce, the MPI module, cray-mpich, and others. Often the Slurm scheduling system is a module.
Panel |
---|
|
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 1. module list for Setonix |
---|
| $ module list
1) craype-x86-milan 5) xpmem/2.2.40-2.1_2.56__g3cf3325.shasta 9) cray-mpich/8.1.14
2) libfabric/1.11.0.4.75 6) gcc/11.2.0 10) cray-libsci/21.08.1.2
3) craype-network-ofi 7) craype/2.7.14 11) PrgEnv-gnu/8.3.2
4) perftools-base/21.12.0 8) cray-dsmml/0.2.2 12) pawsey |
Expand |
---|
title | Click here to expand for other examples of default modules ... |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 1b. module list for Magnus |
---|
| $ module list
Currently Loaded Modulefiles:
1) slurm/17.02.9 5) craype-network-aries 9) ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari 13) xpmem/2.2.4-6.0.5.1_8.13__g35d5e73.ari 17) rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari 21) cray-mpich/7.7.0
2) pawseytools/1.20 6) craype/2.5.13 10) pmi/5.0.13 14) job/2.2.2-6.0.5.0_8.47__g3c644b5.ari 18) atp/2.1.1 22) craype-haswell
3) modules/3.2.10.6 7) cray-libsci/17.12.1 11) dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari 15) dvs/2.7_2.2.53-6.0.5.2_8.6__ge13250e 19) perftools-base/7.0.0
4) cce/8.6.5 8) udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari 12) gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari 16) alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari 20) PrgEnv-cray/6.0.4 |
|
|
|
Non-Cray systems will typically have fewer modules loaded by default, often providing the GCC compiler module and the Slurm module.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 2. module list for non-cray systems |
---|
| $ module list
Currently Loaded Modules:
1) cascadelake/1.0 2) pawseytools/1.29 3) slurm/20.02.3 4) gcc/8.3.0 |
Note |
---|
| The pawseytools module is also loaded by default on all Pawsey systems. This module contains a collection of useful utilities provided by Pawsey staff. slurm is also loaded by default as it provides the Slurm job scheduler.
|
|
Loading, unloading, and swapping modules
On Setonix, when loading a package built with Spack both the module name and version must be provided. Providing only the module name results in an error:
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 3. Example module commands |
---|
| $ module load python
Lmod has detected the following error: Default module versions are disabled by your systems administrator.
Please load this module as <name>/<version>.
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
python/3.10.4 /software/setonix/current/modules/zen3/gcc/11.2.0/programming-languages/python/3.10.4.lua |
|
On the other hand, Cray modules can still be loaded by providing the module name only; providing the version is optional, but recommended for reproducibility.
The following example loads the python/3.9.7
module, then swaps python version to 3.10.4
, finally unloads the python module.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 4. Example module commands |
---|
| $ module load python/3.9.7
$ module swap python/3.9.7 python/3.10.4
$ module unload python |
|
The module swap
command is particularly useful in the context of switching between different programming environments on Cray system:
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 5. Swapping programming environments |
---|
| $ module swap PrgEnv-gnu PrgEnv-aocc
$ module swap PrgEng-aocc PrgEnv-cray
$ module swap PrgEnv-cray PrgEnv-gnu |
|
or compilers:
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 6. Swapping compilers |
---|
| $ module swap gcc aocc/3.2.0
$ module swap aocc cce/13.0.2
$ module swap cce gcc/11.2.0 |
|
Further information of a descriptive nature can be obtained by using the commands module whatis
and module display
. These commands show the paths to executables, libraries, any other modules that are loaded, conflicts, and importantly, may define variables needed by the application at runtime.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 7. Example module whatis |
---|
collapse | true |
---|
| $ module whatis cray-hdf5
cray-hdf5/1.12.1.1 : The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format. |
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 8. Example module display |
---|
collapse | true |
---|
| $ module display cray-hdf5
----------------------------------------------------------------------------
/opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0/cray-hdf5/1.12.1.1.lua:
----------------------------------------------------------------------------
family("hdf5")
conflict("PrgEnv-pathscale")
help([[Release info: /opt/cray/pe/hdf5/1.12.1.1/release_info]])
whatis("The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.")
prepend_path("PATH","/opt/cray/pe/hdf5/1.12.1.1/bin")
prepend_path("PKG_CONFIG_PATH","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0/lib/pkgconfig")
prepend_path("PE_PKGCONFIG_LIBS","hdf5_hl:hdf5")
setenv("PE_HDF5_PKGCONFIG_LIBS","hdf5_hl:hdf5")
prepend_path("PE_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
setenv("PE_HDF5_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
prepend_path("PE_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("PE_HDF5_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("CRAY_HDF5_DIR","/opt/cray/pe/hdf5/1.12.1.1")
setenv("PE_HDF5_DIR","/opt/cray/pe/hdf5/1.12.1.1")
setenv("CRAY_HDF5_VERSION","1.12.1.1")
setenv("CRAY_HDF5_PREFIX","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0")
setenv("HDF5_DIR","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0")
setenv("HDF5_ROOT","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0") |
|
Using modules in Slurm scripts
The most typical use of modules would be for loading applications to be run using the SLURM scheduler. An example of running the GCC version of the lammps
package is provided in listing 1.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | Emacs |
---|
title | Listing 1. Sample Slurm script |
---|
linenumbers | true |
---|
| #!/bin/bash -l
#SBATCH --account=<project>
#SBATCH --partition=work
#SBATCH --time=24:00:00
#SBATCH --nodes=2
#SBATCH --job-name=lammps-job
module swap PrgEnv-cray PrgEnv-gnu
module load lammps/20210929.3
srun --export=all -n 48 lmp_mpi -in lammps.inp -log lammps.log |
Warning |
---|
Do not add module load commands in the shell initialisation scripts (~/.bashrc , ~/.tcshrc and so on). This practice makes the user environment less reproducible by colleagues and by Pawsey staff. |
|
System-wide, group and user modules
The module environment looks for module files in a sequence of directories stored in the MODULEPATH
environment variable.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 9. Example module path |
---|
| $ echo $MODULEPATH
/opt/cray/pe/lmod/modulefiles/mpi/crayclang/10.0/ofi/1.0/cray-mpich/8.0:/opt/cray/pe/lmod/modulefiles/comnet/crayclang/10.0/ofi/1.0:/opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0:/opt/cray/pe/lmod/modulefiles/mix_compilers:/opt/cray/pe/lmod/modulefiles/perftools/21.12.0 |
|
These module files can be stored at user, group or system level, with this precedence. As a result, if a user installs their own custom version of a system-wide supported software, their module will be selected instead of the system-wide one when running module load.
Modules for user-based Spack installations of software
Users can install their own software using the Spack package manager. Spack also provides commands to produce a module file for this installation (see Spack for details of how to install software). The resulting module files follow a different naming convention to that of the system modules, producing modules with <name>/<version>-<unique_hash>.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 10. Example spack installation and module production |
---|
collapse | true |
---|
| $ module load spack/0.17.0 # load spack module
$ spack install abyss # install abyss, de novo, parallel, paired-end sequence assembler
$ spack module lmod refresh abyss # produce module file
$ module use <PATH-TO-MODULES> # if necessary, update the module path
$ module avail abyss
----------------------------------------------------------------------------
/software/projects/<project-id>/<user-name>/setonix/modules/zen3/gcc/11.2.0/abyss/2.3.1-gb3u5yw.lua
----------------------------------------------------------------------------
abyss/2.3.1-gb3u5yw
$ module whatis abyss/2.3.1-gb3u5yw
byss/2.3.1-gb3u5yw : Name : abyss
abyss/2.3.1-gb3u5yw : Short description : ABySS is a de novo, parallel, paired-end sequence assembler that is designed for short reads. The single-processor version is useful for assembling genomes up to 100 Mbases in size.
abyss/2.3.1-gb3u5yw : Version : 2.3.1
abyss/2.3.1-gb3u5yw : Compiler : gcc@11.2.0
abyss/2.3.1-gb3u5yw : Flags : cflags="-O3" cppflags="-O3" cxxflags="-O3" fflags="-O3 -fallow-argument-mismatch"
abyss/2.3.1-gb3u5yw : Target : zen3
abyss/2.3.1-gb3u5yw : Build date : Tue Jun 14 09:02:28 2022
abyss/2.3.1-gb3u5yw : Spack configuration : maxk=128
abyss/2.3.1-gb3u5yw : Path : /software/projects/pawsey0001/pelahi/setonix/software/cray-sles15-zen3/gcc-11.2.0/abyss-2.3.1-gb3u5ywgkvuni2vw5pvx23rg6nwkx7t6
abyss/2.3.1-gb3u5yw : Configure options : --with-boost=/software/setonix/current/cray-sles15-zen3/gcc-11.2.0/boost-1.76.0-fzeiorblgokslk6yvksmsbfahga336cl --with-sqlite=/usr --with-mpi=/opt/cray/pe/mpich/8.1.14/ofi/gnu/9.1 --enable-maxk=128
|
|
Creating a module file
If a group or user wants to create their own module file for a custom installed application, they should locate and name it according to the rules of LMOD. For more information, check the Lmod documentation.
These new modules residing in custom directories can be added to the module path with the module use
command.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 11. Adding directories to the module path |
---|
| # if path is <PATH-TO-MODULES>/<software>/<version>
$ module use <PATH-TO-MODULES>
$ module load <software>/<version> |
|
Module cache
The Lmod module environment utilises a module cache to speed up module commands. You can check if the module environment is based on Lua using --version
.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 12. Getting the date of the module cache |
---|
| $ module --version
Modules based on Lua: Version 8.3.1 2020-02-16 19:46 :z
by Robert McLay mclay@tacc.utexas.edu |
|
To force the cache to update, remove the cache directory and run module avail.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 13. Cleaning the module cache |
---|
| rm -rf ~/.lmod.d/.cache
module avail |
|
Modules and reproducibility
Software on Setonix is built using the Spack package manager. Spack makes use of dynamic linking and rpath
. The result of this is that modules providing domain software are self-contained and do not require other modules to be loaded (except for Python dependencies). The executables provided by the modules themselves contain the paths to the libraries to be used at run time, which improves reproducibility.
This behaviour differs from modules on pre-Setonix systems.
Expand |
---|
title | Module loading behaviour on pre-Setonix systems: click here to expand... |
---|
|
Info |
---|
On pre-Setonix systems, all executables made extensive use of the LD_LIBRARY_PATH environment variable to include libraries at run time. Consequently, the order of how modules are loaded could affect the runtime. Consider the following example: module A depends on functionality that can be provided by either library B or C . Module A by default loads B . Four possible scenarios are: Scenario 1 | Scenario 2 | Scenario 3 | Scenario 4 | Scenario 5 |
---|
module load A
| module load B
module load A
| module load A
module load C
| module load A
module load C
module load B
| module load C
module load A
| Result: A+B | Result: A+B | Result: A+C | Result: A+B | Result: A+B |
On Setonix the order does not matter as A contains links to B and hence the result will always be A+B. |
|
Related pages
- Spack package manager
- See Compiling for discussion of dynamic versus static linking and use of
rpath
External links