The Cray (HPE), GNU, and AMD compilation environments are available on Setonix.
A CPE provides compiler wrappers, shown in table 6, for both the Cray Compiling Environment (CCE) and third-party compiler drivers. When using the wrappers, the actual compiler invoked is determined by the Programming Environment (PrgEnv-cray, PrgEnv-aocc or PrgEnv-gnu), as loaded through the module system. These compiler wrappers handle common tasks like linking adding MPI and numerical libraries such as BLAS/LAPACK, and cross-compilation (discussed below). Compiler wrappers compile both serial and parallel code; there is no separate MPI compiler (e.g. mpicc, mpicxx, mpif90). These wrappers also work in Makefiles and build scripts, without the need to modify them.
gcc). Use the appropriate compiler wrapper in conjunction with the correct choice of the programming environment.Table 1. Compiler wrappers that are available for every programming environment on an HPE Cray supercomputer
| Language | Wrapper |
|---|---|
| C | cc |
| C++ | CC |
| Fortran | ftn |
The Fortran compiler coming with the Cray Programming Environment is developed entirely by Cray and supports the Fortran 2018 standard (ISO/IEC 1539:2018), with some exceptions and deferred features. The C/C++ compiler is instead based on Clang/LLVM, with some Cray enhancements. For instance, the OpenMP implementation is HPE Cray proprietary. CCE compilers are documented through their man pages.
The CCE C/C++ compiler supports Unified Parallel C (UPC), an extension of the C programming language designed for high-performance computing on large-scale parallel machines.
Furthermore, the following third-party programming languages are bundled with the Programming Environment: Python 3.8.x through the module cray-python, and R 4.0 through the module cray-R.
In order to build code optimised for the compute nodes, three ways are available:
- through a Slurm interactive session on the compute nodes (suggested for small codes),
- through a Slurm batch job on the compute nodes, or
- interactively on the login node using the compute node-specific modules and compiler flags (again, for small codes).
We suggest always compiling code on the nodes it will run on, hence the compute nodes.
Compiling MPI code
As mentioned above, wrappers are able to compile both serial and parallel code. Then, regardless of the selected programming environment, users compile MPI code choosing from the same wrappers shown in table 6, according to the programming language used.
Compiling OpenMP code
Users must use specific flags to compile code that makes use of OpenMP for multithreading, with a different syntax depending on the selected programming environment.
Table 2. Flags enabling OpenMP compilation for various compilers.
| PrgEnv-cray | PrgEnv-aocc | PrgEnv-gnu | |||
|---|---|---|---|---|---|
| Language | Command | Language | Command | Language | Command |
| C | cc -fopenmp hello_omp.c | C | cc -fopenmp hello_omp.c | C | cc -fopenmp hello_omp.c |
| C++ | CC -fopenmp hello_omp.cpp | C++ | CC -fopenmp hello_omp.cpp | C++ | CC -fopenmp hello_omp.cpp |
| Fortran | ftn -h omp hello_omp.f90 | Fortran | ftn -fopenmp hello_omp.f90 | Fortran | ftn -fopenmp hello_omp.f90 |
To execute OpenMP programs, set the OMP_NUM_THREADS environment variable with the number of threads to be created, and request the same number of cores using the -c, --cpus-per-task option of srun.
Compiling OpenACC code
OpenACC is only supported by the Cray Fortran compiler and GCC for C and C++.
Compiling HIP code or HIP_MPI code
$ module load rocm/version craype-accel-amd-gfx90a
$ hipcc foo.cu
for MPI enabled codes
$ module load rocm/version craype-accel-amd-gfx90a
$ export CPATH=${CPATH}:${CRAY_MPICH_DIR}/include
$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${CRAY_MPICH_DIR}/lib
$ hpicc -L${CRAY_MPICH_DIR}/lib -lmpi foo.cu
* foo.cu is a ported file from cuda code
Compiler manual pages
Executing man cc, man CC or man ftn within will open the manual pages of the wrapper. Manual pages for every compiler are also available.