Profiling with Linaro MAP
Linaro MAP is a commercial profiling tool, and the recommended method of parallel profiling on Pawsey supercomputing systems. It provides a graphical user interface and remote client for analysing profiling information.
Prerequisite knowledge
You should be familiar with writing, compiling and running parallel codes to start profiling with Linaro MAP.
The Linaro Forge license supports a total number of 1024 running processes (tasks) at a time. For instance, the licence won't allow any other user to run a debugging job if user A is debugging a 512 task job and users B and C are profiling a 256 task job each.
Introduction to MAP
Linaro MAP gathers detailed information regarding the performance of parallel code using a graphical user interface. Use Linaro MAP for profiling serial, MPI, OpenMP and mixed mode executables.
There are two usage modes in Linaro MAP:
Linaro MAP Remote Client can be executed on the local machine (laptop or desktop)
In this mode the Remote Client can connect to the compute system's login node and read the profiling data provided on it. Data can be then analysed with the use of local platform.Linaro MAP GUI can be executed directly on the login node
In this mode profiling session can be executed directly on the login node of the system.
Best Practice
Use the remote client mode when profiling with Linaro MAP.
Profiling steps
The following is an overview of the process for using Linaro MAP to profile your program:
Generate the profile data for the application
Use the Linaro Map Remote Client to analyse the profile data.
Step-by-Step Example
In this section we will provide a step-by-step introduction to Linaro MAP.
Step 1: Get the source code
This example profiles a MPI program which calculates the value of pi.
Create a file called darts-mpi.c with the following source code:
Listing 1. Computing PI using MPI.
/* Compute pi using the six basic MPI functions */
#include <mpi.h>
#include <stdio.h>
static long num_trials = 1000000;
static long MULTIPLIER = 1366;
static long ADDEND = 150889;
static long PMOD = 714025;
long random_last = 0;
double lcgrandom() {
long random_next;
random_next = (MULTIPLIER * random_last + ADDEND)%PMOD;
random_last = random_next;
return ((double)random_next/(double)PMOD);
}
int main(int argc, char **argv) {
long i;
long Ncirc = 0;
double pi, x, y;
double r = 1.0; // radius of circle
double r2 = r*r;
int rank, size, manager = 0;
MPI_Status status;
long my_trials, temp;
int j;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
my_trials = num_trials/size;
if (num_trials%(long)size > (long)rank) my_trials++;
random_last = rank;
for (i = 0; i < my_trials; i++) {
x = lcgrandom();
y = lcgrandom();
if ((x*x + y*y) <= r2)
Ncirc++;
}
if (rank == manager) {
for (j = 1; j < size; j++) {
MPI_Recv(&temp, 1, MPI_LONG, j, j, MPI_COMM_WORLD, &status);
Ncirc += temp;
}
pi = 4.0 * ((double)Ncirc)/((double)num_trials);
printf("\n \t Computing pi using six basic MPI functions: \n");
printf("\t For %ld trials, pi = %f\n", num_trials, pi);
printf("\n");
} else {
MPI_Send(&Ncirc, 1, MPI_LONG, manager, rank, MPI_COMM_WORLD);
}
MPI_Finalize();
return 0;
}Step 2: Generate a MAP MPI wrapper library
This step needs to be performed once only for a given MPI implementation. Therefore, this should be done separately for each Pawsey systems.
For example, on Setonix use "map-libs-setonix" in place of "map-libs" in the commands below.
Replace "projectname" and "username" with your Pawsey project code and username, and issue the following commands:
Terminal 1. Generating wrapper libraries.
$ mkdir /software/projects/projectname/username/map-libs
$ cd /software/projects/projectname/username/map-libs
$ module load linaro-forge
$ make-profiler-libraries
Creating Cray shared libraries in /software/projects/projectname/username/map-libs
Created the libraries:
libmap-sampler.so (and .so.1, .so.1.0, .so.1.0.0)
libmap-sampler-pmpi.so (and .so.1, .so.1.0, .so.1.0.0)
To instrument a program, add these compiler options:
compilation for use with MAP - not required for Performance Reports:
-g (or '-G2' for native Cray Fortran) (and -O3 etc.)
linking (both MAP and Performance Reports):
-dynamic -L/software/projects/projectname/username/map-libs -lmap-sampler-pmpi -lmap-sampler -Wl,--eh-frame-hdr -Wl,-rpath=/software/projects/projectname/username/map-libs
Note: These libraries must be on the same NFS/Lustre/GPFS filesystem as your program.
Before running your program (interactively or from a queue), set
LD_LIBRARY_PATH:
export LD_LIBRARY_PATH=/software/projects/projectname/username/map-libs:$LD_LIBRARY_PATH
map ...
or add -Wl,-rpath=/software/projects/projectname/username/map-libs when linking your program.
The instructions output to the terminal describe the appropriate link stage arguments required to compile your code. Make a record of the output. The command will produce profiling library files in the working directory:
Terminal 2. Displaying the location of the libraries.
$ ls /software/projects/projectname/username/location
libmap-sampler-pmpi.so libmap-sampler-pmpi.so.1 libmap-sampler-pmpi.so.1.0
libmap-sampler-pmpi.so.1.0.0 libmap-sampler.so
By default make-profiler-libraries will generate shared libraries. Use the following command to generate static profiler libraries:
$ module load linaro-forge
$ make-profiler-libraries --lib-type=static
Please note that separate profiler libraries needs to be generated for each supercomputer you want to run the profiler on.
Step 3: Compile and link your MPI application
Generate the executable by following instructions given below:
use the -g compile option to retain symbolic information for compilation,
link your application with the link arguments generated via the make-profiler-libraries command.
This process is illustrated below for Setonix, in the directory containing the mpi-darts.c file:
Terminal 3. Compiling the code.
$ cc -g -c darts-mpi.c
$ cc -dynamic -L/software/projects/projectname/username/map-libs -lmap-sampler-pmpi -lmap-sampler -Wl,--eh-frame-hdr -Wl,-rpath=/software/projects/projectname/username/map-libs -o darts-mpi darts-mpi.o
Please note -G2 can be used instead of -g in PrgEnv-cray to allow a higher level of optimisation. (The MAP client will display a meaningful warning message if insufficient debugging information is available from the executable.)
Step 4: Execute the code to generate profiling information
You can execute your profiling job in two ways (choose one of the methods described below):
Option 1: Submit the job to the SLURM scheduling system
Write a Slurm batch script:
Listing 2. The Slurm batch script.
#!/bin/bash --login
#SBATCH --account=<project>
#SBATCH --partition=debug
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
#SBATCH --threads-per-core=1
#SBATCH --time=0:15:00
module load Linaro-forge/25.1.3
map --profile srun -n 4 ./darts-mpiThe above script describes a 15 minute single node job which executes 4 processes in the debug partition on Setonix.
Note that we are using --profile option for map. This will cause the profiling data to be generated without using Linaro MAP's GUI.
You can now submit the job to the scheduler:
$ sbatch job.slurm
Option 2: Use the interactive session
Allocate a profiling session in the debug partition by running:
$ salloc --nodes=1 --tasks-per-node=4 --partition=debug --account=<project> --time=0:15:00
salloc: Granted job allocation 2509285You can now run the profiling job:
$ module load Linaro-forge/25.1.3
$ map --profile srun -n 4 ./profilemeA successful execution will produce a file with .map extension in the working directory. This file contains all profiling information. The full name of the file contains the name of the executable, number of processes, nodes, threads and the timestamp, e.g.:
$ ls *.map
profileme_4p_1n_1t_2022-03-04_10-30.mapStep 5: Download and install Linaro Forge Remote Client
Visit the Linaro Linaro Forge download page and download the Linaro Forge Remote Client (available for Windows, OS/X and Linux).
Note that the version of the Remote Client needs to be compatible with the Linaro Forge version available on the Pawsey's system you are planning to use for debugging.
You may need to navigate to the 'older versions of Linaro Forge' button to download the version that is compatible with the Setonix Linaro-Forge version.
Run the module avail Linaro-forge command to check which versions are available on the particular system, e.g.:
$ module avail linaro-forge
--------------------------------------------------------------------- /software/pawsey/modulefiles ---------------------------------------------------------------------
linaro-forge/24.0.3 linaro-forge/25.1.3 (D)
Install the correct/compatible version of the Linaro Forge Remote Client by following instructions in the installer.
On running the client on your local machine, select the "Linaro Map" tab on the left, then the"Configure" option from the "Remote Launch" menu. Choose "Add" and configure the remote launch settings. Settings for Setonix are shown on the screenshots below:
Choose "OK" to save the configuration.
Note that the correct Remote Installation Directory needs to be specified. The directory name might change with different OS and Linaro Forge versions.
The optional remote script entry should be left blank. One can then try the "Test Remote Launch" for which you will need to enter your password.
Note that "username" needs to be replaced with your Pawsey username.
Step 6: Execute the Remote Client and connect
Now start the Linaro Forge Remote Client on your local machine and connect to the Pawsey's system (select the correct option from the "Remote Launch" menu). When connected select the "Load Profile Data File" and choose the appropriate Linaro MAP profile file.
Step 7: Analyse profiling information
The main Linaro MAP profiler window should appear. You can navigate through the code and analyse the performance of the code from different angles. The screenshot below presents the profile information for the example program. We can see that ~90% runtime is spent in the random number generator library call, mostly in memory accesses.
Next steps
Linaro MAP provides a great deal of functionality to analyse different aspects of the application's performance. Not all of the available functionality is described in the example above. For further information, consult the Linaro MAP user guide.
Note Linaro MAP does not support AMD GPUs, see Profiling GPU-Enabled codes for alternative tools.