...
Logging into a system will typically load a default set of modules. The numbered list of these modules can be seen by running module list.
When logging into Cray Pawsey systems, the PrgEnv-cray
module is some modules are loaded by default, providing the Cray compiler suite, cce, the MPI module, cray-mpich, and others. In Cray systems this includes many cray related modules. Often the Slurm scheduling system is a module.
On the other hand, Cray modules can still be loaded by providing the module name only; providing the version is optional, but recommended for reproducibility.
The following example loads the python/3.9.7
module, then swaps python version to 3.10.4
, finally unloads the python module.
Column |
---|
|
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 1. module list for Setonix |
---|
| $ module list
1) craype-x86-milan 5) xpmem/2.2.40-2.1_2.56__g3cf3325.shasta 9) cray-mpich/8.1.14 2) libfabric/1.11.0.4.75 6) gcc/11 6) pawseyenv/2024.05 11) cray-libsci/23.09.1.1
2) libfabric/1.15.2.0 7) gcc/12.2.0 1012) crayPrgEnv-libscignu/218.084.1.20
3) craype-network-ofi 7) craype/2.7.14 8) craype/2.7.23 11) PrgEnv-gnu/8.3.213) pawsey
4) perftools-base/2123.1209.0 8) cray-dsmml/0.2.2 9) cray-dsmml/0.2.2 14) pawseytools
12) pawsey | Expand |
---|
title | Click here to expand for other examples of default modules ... |
---|
| Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 1b. module list for Magnus |
---|
| $ module list
Currently Loaded Modulefiles:
1) slurm/17.02.9 5) craype-network-aries 9) ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari 13) xpmem/2.2.4-6.0.5.1_8.13__g35d5e73.ari 17) rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari 21) cray-mpich/7.7.0
2) pawseytools/1.20 6) craype/2.5.13 10) pmi/5.0.13 14) job/2.2.2-6.0.5.0_8.47__g3c644b5.ari 18) atp/2.1.1 22) craype-haswell
3) modules/3.2.10.6 7) cray-libsci/17.12.1 11) dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari 15) dvs/2.7_2.2.53-6.0.5.2_8.6__ge13250e 19) perftools-base/7.0.0
4) cce/8.6.5 8) udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari 12) gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari 16) alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari 20) PrgEnv-cray/6.0.4 |
|
|
Non-Cray systems will typically have fewer modules loaded by default, often providing the GCC compiler module and the Slurm module.
...
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 2. module list for non-cray systems |
---|
|
$ module list
Currently Loaded Modules:
1) cascadelake/1.0 2) pawseytools/1.29 3) slurm/20.02.3 4) gcc/8.3.0 |
Note |
---|
|
The pawseytools module is also loaded by default on all Pawsey systems. This module contains a collection of useful utilities provided by Pawsey staff. slurm is also loaded by default as it provides the Slurm job scheduler.
|
Loading, unloading, and swapping modules
On Setonix, when loading a package built with Spack both the module name and version must be provided. Providing only the module name results in an error:
...
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 3. Example module commands |
---|
|
$ module load python
Lmod has detected the following error: Default module versions are disabled by your systems administrator.
Please load this module as <name>/<version>.
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
python/3.10.4 /software/setonix/current/modules/zen3/gcc/11.2.0/programming-languages/python/3.10.4.lua |
5) xpmem/2.5.2-2.4_3.47__gd0f7936.shasta 10) cray-mpich/8.1.27 15) slurm/22.05.2 |
|
|
Non-Cray systems will typically have fewer modules loaded by default, often providing the GCC compiler module and the Slurm module.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 2. module list for non-cray systems |
---|
| $ module list
Currently Loaded Modules:
1) cascadelake/1.0 2) pawseytools/1.29 3) slurm/20.02.3 4) gcc/8.3.0 |
Note |
---|
| The pawseytools module is also loaded by default on all Pawsey systems. This module contains a collection of useful utilities provided by Pawsey staff. slurm is also loaded by default as it provides the Slurm job scheduler.
|
|
Loading, unloading, and swapping modules
On Setonix, when loading a package built with Spack both the module name and version must be provided. Providing only the module name results in an error:
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 3. Example module commands |
---|
| $ module load python
Lmod has detected the following error: Default module versions are disabled by your systems
administrator.
Please load this module as <name>/<version>.
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
python/3.11.6 /software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages/python/3.11.6.lua |
|
On the other hand, Cray modules can still be loaded by providing the module name only; providing the version is optional, but recommended for reproducibility.
The following example loads the python/3.9.15
module, then swaps python version to 3.11.6
, finally unloads the python module.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 4. Example module commands |
---|
| $ module load python/3.9.15
$ module swap python/3.9.15 python/3.11.6
$ module unload python |
|
The module swap
command is particularly useful in the context of switching between different programming environments on Cray system:
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 5. Swapping programming environments |
---|
| $ module swap PrgEnv-gnu PrgEnv-aocc
$ module swap PrgEng-aocc PrgEnv-cray
$ module swap PrgEnv-cray PrgEnv-gnu |
|
or compilers:
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 6. Swapping compilers |
---|
| $ module swap gcc aocc/3.2.0
$ module swap aocc cce/16.0.1
$ module swap cce gcc/12.2.0 |
|
Displaying information on modules
Further information of a descriptive nature can be obtained by using the commands module whatis
and module display
. These commands show the paths to executables, libraries, any other modules that are loaded, conflicts, and importantly, may define variables needed by the application at runtime.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 7. Example module whatis |
---|
collapse | true |
---|
| $ module whatis cray-hdf5
cray-hdf5/1.12.2.7 : The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format. |
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 8. Example module display |
---|
collapse | true |
---|
| $ module display cray-hdf5
----------------------------------------------------------------------------
/opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0/cray-hdf5/1.12.1.1.lua:
----------------------------------------------------------------------------
family("hdf5")
conflict("PrgEnv-pathscale")
conflict("cray-hdf5")
conflict("cray-hdf5-parallel")
help([[Release info: /opt/cray/pe/hdf5/1.12.2.7/release_info]])
whatis("The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.")
prepend_path("PATH","/opt/cray/pe/hdf5/1.12.2.7/bin")
prepend_path("PKG_CONFIG_PATH","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1/lib/pkgconfig")
prepend_path("PE_PKGCONFIG_LIBS","hdf5_hl:hdf5")
setenv("PE_HDF5_PKGCONFIG_LIBS","hdf5_hl:hdf5")
prepend_path("PE_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
setenv("PE_HDF5_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
prepend_path("PE_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("PE_HDF5_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("CRAY_HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7")
setenv("PE_HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7")
setenv("CRAY_HDF5_VERSION","1.12.2.7")
setenv("CRAY_HDF5_PREFIX","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
setenv("HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
setenv("HDF5_ROOT","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
prepend_path("CRAY_LD_LIBRARY_PATH","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1/lib")
prepend_path("MODULEPATH","/opt/cray/pe/lmod/modulefiles/hdf5/gnu/8.0/cray-hdf5/1.12.2") |
|
Using modules in Slurm scripts
The most typical use of modules would be for loading applications to be run using the SLURM scheduler. An example of running the GCC version of the lammps
package is provided in listing 1.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | Emacs |
---|
title | Listing 1. Sample Slurm script |
---|
linenumbers | true |
---|
| #!/bin/bash -l
#SBATCH --account=<project>
#SBATCH --partition=work
#SBATCH --time=24:00:00
#SBATCH --nodes=2
#SBATCH --job-name=lammps-job
module swap PrgEnv-cray PrgEnv-gnu
module load lammps/20230802.3
srun --export=all -n 48 lmp_mpi -in lammps.inp -log lammps.log |
Warning |
---|
Do not add module load commands in the shell initialisation scripts (~/.bashrc , ~/.tcshrc and so on). This practice makes the user environment less reproducible by colleagues and by Pawsey staff. |
|
System-wide, group and user modules
The module environment looks for module files in a sequence of directories stored in the MODULEPATH
environment variable.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 49. Example module commandspath |
---|
| $ moduleecho $MODULEPATH load python/3.9.7
$ module swap python/3.9.7 python/3.10.4
$ module unload python |
|
The module swap
command is particularly useful in the context of switching between different programming environments on Cray system:
...
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 5. Swapping programming environments |
---|
|
$ module swap PrgEnv-gnu PrgEnv-aocc
$ module swap PrgEng-aocc PrgEnv-cray
$ module swap PrgEnv-cray PrgEnv-gnu |
or compilers:
...
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 6. Swapping compilers |
---|
|
$ module swap gcc aocc/3.2.0
$ module swap aocc cce/13.0.2
$ module swap cce gcc/11.2.0 |
Displaying information on modules
Further information of a descriptive nature can be obtained by using the commands module whatis
and module display
. These commands show the paths to executables, libraries, any other modules that are loaded, conflicts, and importantly, may define variables needed by the application at runtime.
...
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 7. Example module whatis |
---|
collapse | true |
---|
|
$ module whatis cray-hdf5
cray-hdf5/1.12.1.1 : The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format. |
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 8. Example module display |
---|
collapse | true |
---|
|
$ module display cray-hdf5
----------------------------------------------------------------------------
/opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0/cray-hdf5/1.12.1.1.lua:
----------------------------------------------------------------------------
family("hdf5")
conflict("PrgEnv-pathscale")
help([[Release info: /opt/cray/pe/hdf5/1.12.1.1/release_info]])
whatis("The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.")
prepend_path("PATH","/opt/cray/pe/hdf5/1.12.1.1/bin")
prepend_path("PKG_CONFIG_PATH","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0/lib/pkgconfig")
prepend_path("PE_PKGCONFIG_LIBS","hdf5_hl:hdf5")
setenv("PE_HDF5_PKGCONFIG_LIBS","hdf5_hl:hdf5")
prepend_path("PE_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
setenv("PE_HDF5_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
prepend_path("PE_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("PE_HDF5_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("CRAY_HDF5_DIR","/opt/cray/pe/hdf5/1.12.1.1")
setenv("PE_HDF5_DIR","/opt/cray/pe/hdf5/1.12.1.1")
setenv("CRAY_HDF5_VERSION","1.12.1.1")
setenv("CRAY_HDF5_PREFIX","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0")
setenv("HDF5_DIR","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0")
setenv("HDF5_ROOT","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0") |
Using modules in Slurm scripts
The most typical use of modules would be for loading applications to be run using the SLURM scheduler. An example of running the GCC version of the lammps
package is provided in listing 1.
...
Code Block |
---|
language | bash |
---|
theme | Emacs |
---|
title | Listing 1. Sample Slurm script |
---|
linenumbers | true |
---|
|
#!/bin/bash -l
#SBATCH --account=<project>
#SBATCH --partition=work
#SBATCH --time=24:00:00
#SBATCH --nodes=2
#SBATCH --job-name=lammps-job
module swap PrgEnv-cray PrgEnv-gnu
module load lammps/20210929.3
srun --export=all -n 48 lmp_mpi -in lammps.inp -log lammps.log |
Warning |
---|
Do not add module load commands in the shell initialisation scripts (~/.bashrc , ~/.tcshrc and so on). This practice makes the user environment less reproducible by colleagues and by Pawsey staff. |
System-wide, group and user modules
The module environment looks for module files in a sequence of directories stored in the MODULEPATH
environment variable.
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 9. Example module path |
---|
| $ echo $MODULEPATH
/opt/cray/pe/lmod/modulefiles/mpi/gnu/8.0/ofi/1.0/cray-mpich/8.0:/software/projects/pawsey0001/setonix/2024.05/modules/zen3/gcc/12.2.0:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/astro-applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/bio-applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/libraries:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/utilities:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/visualisation:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/python-packages:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/benchmarking:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/developer-tools:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/dependencies:/software/setonix/2024.05/custom/modules/zen3/gcc/12.2.0/custom:/opt/cray/pe/lmod/modulefiles/comnet/gnu/8.0/ofi/1.0:/opt/cray/pe/lmod/modulefiles/mix_compilers:/opt/cray/pe/lmod/modulefiles/compiler/gnu/8.0:/software/setonix/2024.05/containers/views/modules:/software/setonix/2024.05/pawsey/modules:/software/projects/pawsey0001/espinosa/setonix/2024.05/containers/views/modules:/opt/cray/pe/lmod/modulefiles/perftools/23.09.0:/opt/cray/pe/lmod/modulefiles/mpinet/crayclangofi/101.0/ofi:/opt/cray/pe/lmod/modulefiles/cpu/x86-milan/1.0/cray-mpich/8.0:/opt/cray/pe/modulefiles/Linux:/opt/cray/pe/modulefiles/Core:/opt/cray/pe/lmod/lmod/modulefiles/comnet/crayclang/10.0/ofi/1.0Core:/opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0core:/opt/cray/pe/lmod/modulefiles/mix_compilerscraype-targets/default:/opt/cray/pe/lmod/modulefiles/perftools/21.12.0pawsey/modulefiles:/software/pawsey/modulefiles:/opt/cray/modulefiles |
|
These module files can be stored at user, group or system level, with this precedence. As a result, if a user installs their own custom version of a system-wide supported software, their module will be selected instead of the system-wide one when running module load.
...
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 10. Example spack installation and module production |
---|
collapse | true |
---|
| $ module load spack/0.1721.0 # load spack module
$ sg <PROJECT_NAME> -c 'spack install abyss' # install abyss, de novo, parallel, paired-end sequence assembler
$ sg <PROJECT_NAME> -c 'spack module lmod refresh abyss' # produce module file
$ module use <PATH-TO-MODULES> # if necessary, update the module path
$ module avail abyss
----------------------------------------------------------------------------
/software/projects/<project-id>/<user-name>/setonix/<DATE_TAG>/modules/zen3/gcc/1112.2.0/abyss/2.3.1-gb3u5yw.lua
----------------------------------------------------------------------------
abyss/2.3.1-gb3u5yw
$ module whatis abyss/2.3.1-gb3u5yw
byss/2.3.1-gb3u5yw : Name : abyss
abyss/2.3.1-gb3u5yw : Short description : ABySS is a de novo, parallel, paired-end sequence assembler that is designed for short reads. The single-processor version is useful for assembling genomes up to 100 Mbases in size.
abyss/2.3.1-gb3u5yw : Version : 2.3.1
abyss/2.3.1-gb3u5yw : Compiler : gcc@11gcc@12.2.0
abyss/2.3.1-gb3u5yw : Flags : cflags="-O3" cppflags="-O3" cxxflags="-O3" fflags="-O3 -fallow-argument-mismatch"
abyss/2.3.1-gb3u5yw : Target : zen3
abyss/2.3.1-gb3u5yw : Build date : Tue Jun 14 09:02:28 20222024
abyss/2.3.1-gb3u5yw : Spack configuration : maxk=128
abyss/2.3.1-gb3u5yw : Path : /software/projects/pawsey0001pawsey0000/pelahimatilda/setonix/<DATE-TAG>/software/cray-sles15-zen3/gcc-1112.2.0/abyss-2.3.1-gb3u5ywgkvuni2vw5pvx23rg6nwkx7t6
abyss/2.3.1-gb3u5yw : Configure options : --with-boost=/software/setonix/current<DATE_TAG>/cray-sles15-zen3/gcc-1112.2.0/boost-1.7683.0-fzeiorblgokslk6yvksmsbfahga336cl --with-sqlite=/usr --with-mpi=/opt/cray/pe/mpich/8.1.1425/ofi/gnu/9.1 --enable-maxk=128
|
|
...
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 12. Getting the date of the module cache |
---|
| $ module --version
Modules based on Lua: Version 8.37.119 20202023-0201-1617 1910:4645 -06:z00
by Robert McLay mclay@tacc.utexas.edu |
|
...
Column |
---|
|
Code Block |
---|
language | bash |
---|
theme | DJango |
---|
title | Terminal 13. Cleaning the module cache |
---|
| rm -rf ~/.cache/lmod
rm -rf ~/.lmod.d/.cache
module avail |
|
...