Slurm Job Cancelled Due to Time Limit

Problem

A job finishes with a message saying slurmstepd error, with the job cancelled due to time limit. The error message will be in the Slurm output file. The slurmstepd error message may be preceded with a message from srun that the job step was aborted.

slurmstepd: error: *** JOB 3501970 ON nid00161 CANCELLED AT 2018-01-08T15:17:49 DUE TO TIME LIMIT ***

Solution

The job may have reached the maximum time it was allowed to run.

Slurm allocates resources to a job for a fixed amount of time. This time limit is either specified in the job request, or if none is specified then it is the default limit. There are maximum limits on all Slurm partitions, so if you have not requested the maximum then try increasing the time limit in the request with the --time= flag to sbatch or sallo.

#SBATCH --time=12:00:00

To see the maximum and default time limits, use sinfo:

$ sinfo -o "%.10P %.5a %.10l %.15L %.6D %.6t" -p workq
 PARTITION AVAIL  TIMELIMIT     DEFAULTTIME  NODES  STATE
    workq*    up 1-00:00:00         1:00:00      1  drain
    workq*    up 1-00:00:00         1:00:00     20   resv
    workq*    up 1-00:00:00         1:00:00     34    mix
    workq*    up 1-00:00:00         1:00:00     25  alloc

Related articles