BLAS and LAPACK

On the OMNI cluster, both BLAS and LAPACK are available in multiple forms.

OpenBLAS is a widely used library for calculating vector and matrix operations. It implements both the BLAS interface (BLAS = Basic Linear Algebra Subprograms) and the LAPACK interface (LAPACK = Linear Algebra PACKage). On the OMNI cluster, OpenBLAS is installed in the version 0.3.7 (compiled with GCC 9.3.0) as part of OpenHPC.

Additionally, SCALAPACK version 2.1.0 is installed on OMNI. SCALAPACK includes a subset of the LAPACK routines in a parallelized form.

On this page, we describe how to use the corresponding environment modules and how to link to the libraries when compiling your own software. The general BLAS documentation can be found here, the LAPACK documentation is here. The documentation specific to the OpenBLAS implementation is here, that for SCALAPACK is here.

The Intel MKL, which also implements the BLAS and LAPACK interfaces, is installed on the cluster as well.

Environment modules

OpenBLAS environment module

In order to use OpenBLAS, the corresponding environment module needs to be loaded:

module load openblas

Note: Due to OpenHPC’s module structure, not all modules are automatically displayed when you type module avail. OpenBLAS is only shown with a module avail when the corresponding GCC module is loaded (module load gnu9). By default, this module is already loaded, however you might have unloaded the module, either intentionally or by accident. In that case you can display the OpenBLAS module with

module spider openblas

SCALAPACK environment module

The SCALAPACK module can be loaded with

module load scalapack

Note: Since SCALAPACK, like OpenBLAS, comes from OpenHPC, a similar rule about module avail applies. The SCALAPACK module will only be displayed when a compiler module and an MPI module are already loaded. By default, the gnu9 and openmpi4 modules are loaded.

Like OpenBLAS, the SCALAPACK module can always be displayed with

module spider scalapack

Note also that SCALAPACK depends on OpenBLAS. That means the OpenBLAS module will always be loaded as a dependency.

In addition to the default combination of GCC 9.3.0 and OpenMPI 4, other variants of SCALAPCK are available which are linked against other compilers and MPI libraries. You can show the complete list with module spider scalapack as in the following example

$ module spider scalapack
(...)
    Description:
      A subset of LAPACK routines redesigned for heterogenous computing
      
    You will need to load all module(s) on any one of the lines below before the "scalapack/2.1.0" module is available to load.

      gnu9/9.3.0  impi/2019.5
      gnu9/9.3.0  openmpi4/4.0.4
   (...)
      intel/19.1.2.100006_cm9.0_ae8d786e90  impi/2020.2.100006_cm9.0_ae8d786e90
      intel/19.1.2.100006_cm9.0_ae8d786e90  openmpi4/4.0.4

As you can see in this shortened list, there are multiple combinations of compilers (both Intel and GCC) and MPI libraries with which SCALAPACK can be used. As it says in the output, all modules on one line need to be loaded (e.g. gnu9/9.3.0 impi/2019.5) if you want to use SCALAPACK.

Compiling and linking

Compiling programs that use OpenBLAS libraries

When you load the OpenBLAS module, the following environment variables are set:

• OPENBLAS_DIR: OpenBLAS installation directory
• OPENBLAS_LIB: OpenBLAS library directory
• OPENBLAS_INC: OpenBLAS include directory

If you would like to compile a program that uses OpenBLAS, you need to specify the corresponding directories when compiling and linking your program.

Additionally, some OpenBLAS directories are added to the PATH, LD_LIBRARY_PATH and other directories when you load the module. You can display the specifics of this with module show openblas.

Caution: OpenBLAS uses dynamically loaded libraries (shared objects). If you link your program against OpenBLAS, then the OpenBLAS module needs to be loaded during execution as well (or at least the OpenBLAS directory needs to be in LD_LIBRARY_PATH).

Compiling with SCALAPACK

Analogously to OpenBLAS, the SCALAPACK module will set a number of environment variables:

• SCALAPACK_DIR: the SCALAPACK installation directory
• SCALAPACK_LIB: the SCALAPACK library directory

You can display the whole list with module show scalapack. Like OpenBLAS, linking against SCALAPACK means that the module must be loaded during execution of your program (or at least the SCALAPACK directory needs to be in LD_LIBRARY_PATH).