Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Logging into a system will typically load a default set of modules. The numbered list of these modules can be seen by running module list.

When logging into Cray Pawsey systems, the  PrgEnv-cray module is some modules are loaded by default, providing the Cray compiler suite, cce, the MPI module, cray-mpich, and others. In Cray systems this includes many cray related modules. Often the Slurm scheduling system is a module.

Panel
titleCray module lists

On the other hand, Cray modules can still be loaded by providing the module name only; providing the version is optional, but recommended for reproducibility.

The following example loads the python/3.9.7 module, then swaps python version to 3.10.4, finally unloads the python module.

Column
width900px


Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 1. module list for Setonix
$ module list   
  1) craype-x86-milan         5) xpmem/2.2.40-2.1_2.56__g3cf3325.shasta     9) cray-mpich/8.1.14   2) libfabric/1.11.0.4.75    6) gcc/11      6) pawseyenv/2024.05  11) cray-libsci/23.09.1.1
  2) libfabric/1.15.2.0                      7) gcc/12.2.0         1012) crayPrgEnv-libscignu/218.084.1.20
  3) craype-network-ofi       7) craype/2.7.14              8) craype/2.7.23            11) PrgEnv-gnu/8.3.213) pawsey
  4) perftools-base/2123.1209.0   8) cray-dsmml/0.2.2              9) cray-dsmml/0.2.2   14) pawseytools
    12) pawsey
Expand
titleClick here to expand for other examples of default modules ...
Code Block
languagebash
themeDJango
titleTerminal 1b. module list for Magnus
$ module list
Currently Loaded Modulefiles:
  1) slurm/17.02.9                                   5) craype-network-aries                            9) ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari         13) xpmem/2.2.4-6.0.5.1_8.13__g35d5e73.ari         17) rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari         21) cray-mpich/7.7.0
  2) pawseytools/1.20                                6) craype/2.5.13                                  10) pmi/5.0.13                                     14) job/2.2.2-6.0.5.0_8.47__g3c644b5.ari           18) atp/2.1.1                                      22) craype-haswell
  3) modules/3.2.10.6                                7) cray-libsci/17.12.1                            11) dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari         15) dvs/2.7_2.2.53-6.0.5.2_8.6__ge13250e           19) perftools-base/7.0.0
  4) cce/8.6.5                                       8) udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari        12) gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari  16) alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari         20) PrgEnv-cray/6.0.4

  

Non-Cray systems will typically have fewer modules loaded by default, often providing the GCC compiler module and the Slurm module.

...

width900px
Code Block
languagebash
themeDJango
titleTerminal 2. module list for non-cray systems
$ module list

Currently Loaded Modules:
  1) cascadelake/1.0   2) pawseytools/1.29   3) slurm/20.02.3   4) gcc/8.3.0
Note
titleNote

The pawseytools module is also loaded by default on all Pawsey systems. This module contains a collection of useful utilities provided by Pawsey staff.

slurm  is also loaded by default as it provides the Slurm job scheduler.

Loading, unloading, and swapping modules

On Setonix, when loading a package built with Spack both the module name and version must be provided. Providing only the module name results in an error:

...

width900px
Code Block
languagebash
themeDJango
titleTerminal 3. Example module commands
$ module load python
Lmod has detected the following error:  Default module versions are disabled by your systems administrator.

	Please load this module as <name>/<version>.

While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    python/3.10.4    /software/setonix/current/modules/zen3/gcc/11.2.0/programming-languages/python/3.10.4.lua
5) xpmem/2.5.2-2.4_3.47__gd0f7936.shasta  10) cray-mpich/8.1.27  15) slurm/22.05.2



  

Non-Cray systems will typically have fewer modules loaded by default, often providing the GCC compiler module and the Slurm module.

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 2. module list for non-cray systems
$ module list

Currently Loaded Modules:
  1) cascadelake/1.0   2) pawseytools/1.29   3) slurm/20.02.3   4) gcc/8.3.0


Note
titleNote

The pawseytools module is also loaded by default on all Pawsey systems. This module contains a collection of useful utilities provided by Pawsey staff.

slurm  is also loaded by default as it provides the Slurm job scheduler.


Loading, unloading, and swapping modules

On Setonix, when loading a package built with Spack both the module name and version must be provided. Providing only the module name results in an error:

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 3. Example module commands
$ module load python
Lmod has detected the following error:  Default module versions are disabled by your systems
administrator.

	Please load this module as <name>/<version>.

While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    python/3.11.6    /software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages/python/3.11.6.lua


On the other hand, Cray modules can still be loaded by providing the module name only; providing the version is optional, but recommended for reproducibility.

The following example loads the python/3.9.15 module, then swaps python version to 3.11.6, finally unloads the python module.

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 4. Example module commands
$ module load python/3.9.15
$ module swap python/3.9.15 python/3.11.6
$ module unload python


The module swap  command is particularly useful in the context of switching between different programming environments on Cray system: 

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 5. Swapping programming environments
$ module swap PrgEnv-gnu PrgEnv-aocc 
$ module swap PrgEng-aocc PrgEnv-cray
$ module swap PrgEnv-cray PrgEnv-gnu


or compilers:

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 6. Swapping compilers
$ module swap gcc aocc/3.2.0
$ module swap aocc cce/16.0.1
$ module swap cce gcc/12.2.0


Displaying information on modules

Further information of a descriptive nature can be obtained by using the commands module whatis and module display. These commands show the paths to executables, libraries, any other modules that are loaded, conflicts, and importantly, may define variables needed by the application at runtime. 

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 7. Example module whatis
collapsetrue
$ module whatis cray-hdf5
cray-hdf5/1.12.2.7  : The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.


Code Block
languagebash
themeDJango
titleTerminal 8. Example module display
collapsetrue
$  module display cray-hdf5
----------------------------------------------------------------------------
   /opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0/cray-hdf5/1.12.1.1.lua:
----------------------------------------------------------------------------
family("hdf5")
conflict("PrgEnv-pathscale")
conflict("cray-hdf5")
conflict("cray-hdf5-parallel")
help([[Release info:  /opt/cray/pe/hdf5/1.12.2.7/release_info]])
whatis("The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.")
prepend_path("PATH","/opt/cray/pe/hdf5/1.12.2.7/bin")
prepend_path("PKG_CONFIG_PATH","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1/lib/pkgconfig")
prepend_path("PE_PKGCONFIG_LIBS","hdf5_hl:hdf5")
setenv("PE_HDF5_PKGCONFIG_LIBS","hdf5_hl:hdf5")
prepend_path("PE_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
setenv("PE_HDF5_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
prepend_path("PE_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("PE_HDF5_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("CRAY_HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7")
setenv("PE_HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7")
setenv("CRAY_HDF5_VERSION","1.12.2.7")
setenv("CRAY_HDF5_PREFIX","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
setenv("HDF5_DIR","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
setenv("HDF5_ROOT","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1")
prepend_path("CRAY_LD_LIBRARY_PATH","/opt/cray/pe/hdf5/1.12.2.7/gnu/9.1/lib")
prepend_path("MODULEPATH","/opt/cray/pe/lmod/modulefiles/hdf5/gnu/8.0/cray-hdf5/1.12.2")


Using modules in Slurm scripts

The most typical use of modules would be for loading applications to be run using the SLURM scheduler. An example of running the GCC version of the lammps package is provided in listing 1.

Column
width900px


Code Block
languagebash
themeEmacs
titleListing 1. Sample Slurm script
linenumberstrue
#!/bin/bash -l
#SBATCH --account=<project>
#SBATCH --partition=work
#SBATCH --time=24:00:00
#SBATCH --nodes=2
#SBATCH --job-name=lammps-job

module swap PrgEnv-cray PrgEnv-gnu
module load lammps/20230802.3

srun --export=all -n 48 lmp_mpi -in lammps.inp -log lammps.log


Warning
Do not add module load commands in the shell initialisation scripts (~/.bashrc, ~/.tcshrc and so on). This practice makes the user environment less reproducible by colleagues and by Pawsey staff.


System-wide, group and user modules

The module environment looks for module files in a sequence of directories stored in the MODULEPATH environment variable.

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 49. Example module commandspath
$ moduleecho $MODULEPATH load python/3.9.7
$ module swap python/3.9.7 python/3.10.4
$ module unload python

The module swap  command is particularly useful in the context of switching between different programming environments on Cray system: 

...

width900px
Code Block
languagebash
themeDJango
titleTerminal 5. Swapping programming environments
$ module swap PrgEnv-gnu PrgEnv-aocc 
$ module swap PrgEng-aocc PrgEnv-cray
$ module swap PrgEnv-cray PrgEnv-gnu

or compilers:

...

width900px
Code Block
languagebash
themeDJango
titleTerminal 6. Swapping compilers
$ module swap gcc aocc/3.2.0
$ module swap aocc cce/13.0.2
$ module swap cce gcc/11.2.0

Displaying information on modules

Further information of a descriptive nature can be obtained by using the commands module whatis and module display. These commands show the paths to executables, libraries, any other modules that are loaded, conflicts, and importantly, may define variables needed by the application at runtime. 

...

width900px
Code Block
languagebash
themeDJango
titleTerminal 7. Example module whatis
collapsetrue
$ module whatis cray-hdf5
cray-hdf5/1.12.1.1  : The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.
Code Block
languagebash
themeDJango
titleTerminal 8. Example module display
collapsetrue
$  module display cray-hdf5
----------------------------------------------------------------------------
   /opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0/cray-hdf5/1.12.1.1.lua:
----------------------------------------------------------------------------
family("hdf5")
conflict("PrgEnv-pathscale")
help([[Release info:  /opt/cray/pe/hdf5/1.12.1.1/release_info]])
whatis("The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.")
prepend_path("PATH","/opt/cray/pe/hdf5/1.12.1.1/bin")
prepend_path("PKG_CONFIG_PATH","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0/lib/pkgconfig")
prepend_path("PE_PKGCONFIG_LIBS","hdf5_hl:hdf5")
setenv("PE_HDF5_PKGCONFIG_LIBS","hdf5_hl:hdf5")
prepend_path("PE_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
setenv("PE_HDF5_FORTRAN_PKGCONFIG_LIBS","hdf5hl_fortran:hdf5_fortran")
prepend_path("PE_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("PE_HDF5_CXX_PKGCONFIG_LIBS","hdf5_hl_cpp:hdf5_cpp")
setenv("CRAY_HDF5_DIR","/opt/cray/pe/hdf5/1.12.1.1")
setenv("PE_HDF5_DIR","/opt/cray/pe/hdf5/1.12.1.1")
setenv("CRAY_HDF5_VERSION","1.12.1.1")
setenv("CRAY_HDF5_PREFIX","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0")
setenv("HDF5_DIR","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0")
setenv("HDF5_ROOT","/opt/cray/pe/hdf5/1.12.1.1/crayclang/10.0")

Using modules in Slurm scripts

The most typical use of modules would be for loading applications to be run using the SLURM scheduler. An example of running the GCC version of the lammps package is provided in listing 1.

...

width900px
Code Block
languagebash
themeEmacs
titleListing 1. Sample Slurm script
linenumberstrue
#!/bin/bash -l
#SBATCH --account=<project>
#SBATCH --partition=work
#SBATCH --time=24:00:00
#SBATCH --nodes=2
#SBATCH --job-name=lammps-job

module swap PrgEnv-cray PrgEnv-gnu
module load lammps/20210929.3

srun --export=all -n 48 lmp_mpi -in lammps.inp -log lammps.log
Warning
Do not add module load commands in the shell initialisation scripts (~/.bashrc, ~/.tcshrc and so on). This practice makes the user environment less reproducible by colleagues and by Pawsey staff.

System-wide, group and user modules

The module environment looks for module files in a sequence of directories stored in the MODULEPATH environment variable.

Column
width900px
Code Block
languagebash
themeDJango
titleTerminal 9. Example module path
$ echo $MODULEPATH 

/opt/cray/pe/lmod/modulefiles/mpi/gnu/8.0/ofi/1.0/cray-mpich/8.0:/software/projects/pawsey0001/setonix/2024.05/modules/zen3/gcc/12.2.0:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/astro-applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/bio-applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/applications:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/libraries:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/programming-languages:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/utilities:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/visualisation:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/python-packages:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/benchmarking:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/developer-tools:/software/setonix/2024.05/modules/zen3/gcc/12.2.0/dependencies:/software/setonix/2024.05/custom/modules/zen3/gcc/12.2.0/custom:/opt/cray/pe/lmod/modulefiles/comnet/gnu/8.0/ofi/1.0:/opt/cray/pe/lmod/modulefiles/mix_compilers:/opt/cray/pe/lmod/modulefiles/compiler/gnu/8.0:/software/setonix/2024.05/containers/views/modules:/software/setonix/2024.05/pawsey/modules:/software/projects/pawsey0001/espinosa/setonix/2024.05/containers/views/modules:/opt/cray/pe/lmod/modulefiles/perftools/23.09.0:/opt/cray/pe/lmod/modulefiles/mpinet/crayclangofi/101.0/ofi:/opt/cray/pe/lmod/modulefiles/cpu/x86-milan/1.0/cray-mpich/8.0:/opt/cray/pe/modulefiles/Linux:/opt/cray/pe/modulefiles/Core:/opt/cray/pe/lmod/lmod/modulefiles/comnet/crayclang/10.0/ofi/1.0Core:/opt/cray/pe/lmod/modulefiles/compiler/crayclang/10.0core:/opt/cray/pe/lmod/modulefiles/mix_compilerscraype-targets/default:/opt/cray/pe/lmod/modulefiles/perftools/21.12.0pawsey/modulefiles:/software/pawsey/modulefiles:/opt/cray/modulefiles


These module files can be stored at user, group or system level, with this precedence. As a result, if a user installs their own custom version of a system-wide supported software, their module will be selected instead of the system-wide one when running module load. 

...

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 10. Example spack installation and module production
collapsetrue
$ module load spack/0.1721.0 # load spack module
$ sg <PROJECT_NAME> -c 'spack install abyss' # install abyss, de novo, parallel, paired-end sequence assembler
$ sg <PROJECT_NAME> -c 'spack module lmod refresh abyss' # produce module file 
$ module use <PATH-TO-MODULES> # if necessary, update the module path
$ module avail abyss
----------------------------------------------------------------------------
/software/projects/<project-id>/<user-name>/setonix/<DATE_TAG>/modules/zen3/gcc/1112.2.0/abyss/2.3.1-gb3u5yw.lua
----------------------------------------------------------------------------
  abyss/2.3.1-gb3u5yw
$ module whatis abyss/2.3.1-gb3u5yw
byss/2.3.1-gb3u5yw : Name : abyss
abyss/2.3.1-gb3u5yw : Short description : ABySS is a de novo, parallel, paired-end sequence assembler that is designed for short reads. The single-processor version is useful for assembling genomes up to 100 Mbases in size.
abyss/2.3.1-gb3u5yw : Version : 2.3.1
abyss/2.3.1-gb3u5yw : Compiler : gcc@11gcc@12.2.0
abyss/2.3.1-gb3u5yw : Flags :  cflags="-O3" cppflags="-O3" cxxflags="-O3" fflags="-O3 -fallow-argument-mismatch"
abyss/2.3.1-gb3u5yw : Target : zen3
abyss/2.3.1-gb3u5yw : Build date : Tue Jun 14 09:02:28 20222024
abyss/2.3.1-gb3u5yw : Spack configuration :  maxk=128
abyss/2.3.1-gb3u5yw : Path : /software/projects/pawsey0001pawsey0000/pelahimatilda/setonix/<DATE-TAG>/software/cray-sles15-zen3/gcc-1112.2.0/abyss-2.3.1-gb3u5ywgkvuni2vw5pvx23rg6nwkx7t6
abyss/2.3.1-gb3u5yw : Configure options : --with-boost=/software/setonix/current<DATE_TAG>/cray-sles15-zen3/gcc-1112.2.0/boost-1.7683.0-fzeiorblgokslk6yvksmsbfahga336cl --with-sqlite=/usr --with-mpi=/opt/cray/pe/mpich/8.1.1425/ofi/gnu/9.1 --enable-maxk=128


...

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 12. Getting the date of the module cache
$ module --version

Modules based on Lua: Version 8.37.119  20202023-0201-1617 1910:4645 -06:z00
    by Robert McLay mclay@tacc.utexas.edu


...

Column
width900px


Code Block
languagebash
themeDJango
titleTerminal 13. Cleaning the module cache
rm -rf ~/.cache/lmod
rm -rf ~/.lmod.d/.cache
module avail


...