QOSMaxJobs Appearing for Pending Slurm Jobs

Problem

There are available resources but my jobs are not running, with QOSMaxJobs given as the reason in squeue output.

Solution

There is a limit on the number of jobs that can any user can run concurrently. If you have submitted jobs that go beyond this limit, they will be marked with QOSMaxJobs. Jobs marked with QOSMaxJobs will be launched when you have completed some of your currently running jobs.

For projects who have used more than their allocation there are stricter limits in place for the running of jobs. This is in addition to the lowering of priority when projects use up 100% of their allocation.