Wiss. Rechnen » SLURM

Caution: this page contains information about the OMNI cluster! The corresponding page for the HoRUS cluster can be found here.

The job scheduler SLURM, version 20.02.5, is installed on the OMNI cluster. The purpose of SLURM is to distribute jobs over the compute nodes in such a way that the available resources are used efficiently and wait times are minimized. More information on queuing jobs can be found here. The SLURM documentation is here.

Tip: SLURM is only available if the SLURM module is loaded (module load slurm). By default, the SLURM module is always loaded upon login, but it can be unloaded intentionally or by accident (e.g. with module purge). If you do not have the SLURM commands, type module list to check that the SLURM module is actually loaded. More information on modules here.

A SLURM job gets its settings from several sources. These are, in descending order of priority:

  • Command line parameters of the command used to queue the job.
  • Environment variables, if specified.
  • The parameters at the beginning of the job script, if specified.
  • The default settings of SLURM.

That means, for example, that you can specify in a job script which settings are to be used, and if you need to deviate from these for a single run you can simply specify a command line option.

SLURM terminology

SLURM knows and mirrors the division of the cluster into nodes with several cores. When queuing jobs, there are several ways of requesting resources and it is important to know which term means what in SLURM. Here are some basic SLURM terms:

  • A job is a self-contained computation that may encompass multiple tasks and is given specific resources like individual CPUs, a specific amount of RAM or entire nodes. These resources are said to have been allocated for the job.

  • A task is a single run of a single process. By default, one task is run per node and one CPU is assigned per task.

  • A partition (usually called queue outside SLURM) is a waiting line in which jobs are put by users.

Commands and options

Here are the most common SLURM commands you need as a user:

Command Function
squeue List jobs.
sinfo Show informationen about allocated and free nodes.
sbatch Queues a batch job.
srun Outside a job: queues a job with a single Linux command. Within a job: runs a Linux command once per task in parallel.
spartition Display partition information. ZIMT addition.
scancel Delete a job.
sview Launch graphical user interface.
scontrol More detailed informationen (not all information is available to normal users).

Many of these commands have options (command line arguments) which you can specify when calling them. You will find a full list by entering man <command name> or <command name> --help as well as in the SLURM documentation of sbatch. The most important ones are summarized here:

Option Function
--partition, -p Specifies on which partition job is to be queued. If omitted, defq is used.
--nodes, -N Sets the number of nodes on which job is to be run.
--ntasks, -n Sets the number of tasks for the jobs.
--ntasks-per-node Sets the maximum number of tasks per node. Usually important for MPI programs.
--cpus-per-task Sets the number of cores per task. Usually important for OpenMP programs.
--mem Sets the RAM limit per node. The job will be terminated if the limit is exceeded. The number is in kilobytes.
--time, -t Sets the time limit for the job. If the time limit is exceeded, the job will be terminated. Format: D-HH:MM:SS
--output=<Dateiname> For the sbatch command this specifies the log file into which the stdout stream is to be directed. By default, this will be a file named slurm-<JobID>.out which will be created in the same directory from which sbatch was run.
--error=<Dateiname> For the sbatch command this specifies the log file into which the stderr stream is to be directed. By default, stderr will be redirected into stdout, see above
--mail-type Sets the events for which an e-mail is to be sent to the address specified by --mail-user. Possible options are BEGIN, END, FAIL, REQUEUE, ALL.
--mail-user=<Adresse> Specifies recipient e-mail address for job notifications.

Environment variables

SLURM uses a number of environment variables. Some can be used to set SLURM settings (unless they are superseded by command line arguments when the job is queued), for sbatch they are listed in the documentation under job input variables. Other variables are set by SLURM when queuing the job. These can be used to obtain information about the job from within the job script. Here are only a few examples, a more complete list is in the SLURM documentation under job output variables.

Variable Function
SLURM_JOB_ID ID number of the job.
SLURM_JOB_NAME Name of the job, for sbatch this is the name of the job script by default.
SLURM_JOB_NUM_NODES  Number of nodes allocated for the job.
SLURM_JOB_PARTITION Partition in which the job runs.
SLURM_NTASKS Number of tasks
SLURM_TASK_PID Linux process ID of the corresponding task.

Aktualisiert um 16:33 am 8. February 2021 von Jan Steiner