Compiling Code on Setonix

The Cray (HPE), GNU, and AMD compilation environments are available on Setonix.

The Cray Programming Environment (CPE) provides compiler wrappers, shown in Table 1, for both the Cray Compiling Environment (CCE) and third-party compiler drivers. When using the wrappers, the actual compiler invoked is determined by the Programming Environment (PrgEnv-cray, PrgEnv-aocc or PrgEnv-gnu) as loaded through the module system. These compiler wrappers handle common tasks like linking MPI and numerical libraries such as BLAS/LAPACK, and cross-compilation (discussed below). Compiler wrappers compile both serial and parallel code; there is no separate, MPI-specific compiler wrapper (e.g. mpicc, mpicxx, mpif90). These wrappers also work in Makefiles and build scripts, without the need to modify them.

Users should not attempt to explicitly invoke specific compilers (for example, gcc). Use the appropriate compiler wrapper in conjunction with the correct choice of the programming environment.

Table 1. Compiler wrappers that are available for every programming environment on an HPE Cray supercomputer

Language	Wrapper

Language	Wrapper
C	`cc`
C++	`CC`
Fortran	`ftn`

The Fortran compiler coming with the Cray Programming Environment is developed entirely by Cray and supports the Fortran 2018 standard (ISO/IEC 1539:2018), with some exceptions and deferred features. The C/C++ compiler is instead based on Clang/LLVM, with some Cray enhancements. For instance, the OpenMP implementation is HPE Cray proprietary. CCE compilers are documented through their man pages.

The CCE C/C++ compiler supports Unified Parallel C (UPC), an extension of the C programming language designed for high-performance computing on large-scale parallel machines.

Furthermore, the following third-party programming languages are bundled with the Programming Environment: Python 3.8.x through the module cray-python, and R 4.0 through the module cray-R.

In order to build code optimised for the compute nodes, three ways are available:

through a Slurm interactive session on the compute nodes (suggested for small codes),
through a Slurm batch job on the compute nodes, or
interactively on the login node using the compute node-specific modules and compiler flags (again, for small codes).

We suggest always compiling code on the nodes it will run on, hence the compute nodes.

Compiling MPI code

As mentioned above, wrappers are able to compile both serial and parallel code. Then, regardless of the selected programming environment, users compile MPI code choosing from the same wrappers shown in Table 1, according to the programming language used.

Compiling OpenMP code

Users must use specific flags to compile code that makes use of OpenMP for multithreading, with a different syntax depending on the selected programming environment.

In previous versions of the Cray compiler, OpenMP compilation was enabled by default.

Table 2. Flags enabling OpenMP compilation for various compilers.

PrgEnv-cray		PrgEnv-aocc		PrgEnv-gnu

PrgEnv-cray		PrgEnv-aocc		PrgEnv-gnu
Language	Command	Language	Command	Language	Command
C	`cc -fopenmp hello_omp.c`	C	`cc -fopenmp hello_omp.c`	C	`cc -fopenmp hello_omp.c`
C++	`CC -fopenmp hello_omp.cpp`	C++	`CC -fopenmp hello_omp.cpp`	C++	`CC -fopenmp hello_omp.cpp`
Fortran	`ftn -h omp hello_omp.f90`	Fortran	`ftn -fopenmp hello_omp.f90`	Fortran	`ftn -fopenmp hello_omp.f90`

To execute OpenMP programs, set the OMP_NUM_THREADS environment variable with the number of threads to be created, and request the same number of cores using the -c, --cpus-per-task option of srun.

Compiling OpenACC code

OpenACC is only supported by the Cray Fortran compiler and GCC for C and C++.

Before trying to compile FORTRAN OpenACC codes with the Cray compiler, you should read BOTH of the manual pages for the compiler driver, ftn

$ man ftn
 * ftn (1)
   ftn (1+1)
Man: What manual page do you want?
Man: 1
.. read the info in the man page ..
$ man ftn
 * ftn (1)
   ftn (1+1)
Man: What manual page do you want?
Man: 1+1
.. read the info in the man page ..
$

as both contain useful, albeit different, info not found elsewhere: for example

 -target-accel=accelerator
              Set  accelerator  target  to either a specific GPU or host. This
              setting overrides the default setting defined by the users' mod-
              ule  environment. To see which accelerator targets are available
              on your system, enter this command:
                 > module avail craype-accel

Once you have determined which “accelerator target” you should be using, you will need to load that module, ahead of the compilation.

You will also need to invoke the ftn complier driver with a flag to indicate that the compiler should enable recognition of OpenACC accelerator directives (This is documented in the “1+1” man page).

$ ftn -h acc <remainder of yoor command line arguments>

Compiling HIP code or HIP_MPI code

For some CUDA code ported to HIP in foo.cu:

$ module load rocm/<VERSION> craype-accel-amd-gfx90a
$ hipcc foo.cu

For MPI-enabled codes:

$ module load rocm/<VERSION> craype-accel-amd-gfx90a
$ export CPATH=${CPATH}:${CRAY_MPICH_DIR}/include
$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${CRAY_MPICH_DIR}/lib
$ hipcc -L${CRAY_MPICH_DIR}/lib -lmpi foo.cu

Compiler manual pages

Executing man cc, man CC or man ftn within will open the manual pages of the wrapper. Manual pages for every compiler are also available.