Caution: this page contains information about the OMNI cluster! The corresponding page for the HoRUS cluster can be found here.
On the OMNI cluster, both BLAS and LAPACK are available in multiple forms.
OpenBLAS is a widely used library for calculating vector and matrix operations. It implements both the BLAS interface (BLAS = Basic Linear Algebra Subprograms) and the LAPACK interface (LAPACK = Linear Algebra PACKage). On the OMNI cluster, OpenBLAS is installed in the version 0.3.7 (compiled with GCC 9.3.0) as part of OpenHPC.
Additionally, SCALAPACK version 2.1.0 is installed on OMNI. SCALAPACK includes a subset of the LAPACK routines in a parallelized form.
On this page, we describe how to use the corresponding environment modules and how to link to the libraries when compiling your own software. The general BLAS documentation can be found here, the LAPACK documentation is here. The documentation specific to the OpenBLAS implementation is here, that for SCALAPACK is here.
The Intel MKL, which also implements the BLAS and LAPACK interfaces, is installed on the cluster as well.
OpenBLAS environment module
In order to use OpenBLAS, the corresponding environment module needs to be loaded:
module load openblas
Note: due to OpenHPC’s module structure, not all modules are automatically displayed when you type
module avail. OpenBLAS is only shown with a
module avail when the corresponding GCC module is loaded (
module load gnu9). By default, this module is already loaded, however you might have unloaded the module, either intentionally or by accident. In that case you can display the OpenBLAS module with
module spider openblas
SCALAPACK environment module
The SCALAPACK module can be loaded with
module load scalapack
Note: since SCALAPACK, like OpenBLAS, comes from OpenHPC, a similar rule about
module avail applies. The SCALAPACK module will only be displayed when a compiler module and an MPI module are already loaded. By default, the
openmpi4 modules are loaded.
Like OpenBLAS, the SCALAPACK module can always be displayed with
module spider scalapack
Note also that SCALAPACK depends on OpenBLAS. That means the OpenBLAS module will always be loaded as a dependency.
In addition to the default combination of GCC 9.3.0 and OpenMPI 4, other variants of SCALAPCK are available which are linked against other compilers and MPI libraries. You can show the complete list with
module spider scalapack as in the following example
$ module spider scalapack (...) Description: A subset of LAPACK routines redesigned for heterogenous computing You will need to load all module(s) on any one of the lines below before the "scalapack/2.1.0" module is available to load. gnu9/9.3.0 impi/2019.5 gnu9/9.3.0 openmpi4/4.0.4 (...) intel/188.8.131.52006_cm9.0_ae8d786e90 impi/2020.2.100006_cm9.0_ae8d786e90 intel/184.108.40.206006_cm9.0_ae8d786e90 openmpi4/4.0.4
As you can see in this shortened list, there are multiple combinations of compilers (both Intel and GCC) and MPI libraries with which SCALAPACK can be used. As it says in the output, all modules on one line need to be loaded (e.g.
gnu9/9.3.0 impi/2019.5) if you want to use SCALAPACK.
Compiling programs that use OpenBLAS libraries
When you load the OpenBLAS module, the following environment variables are set:
OPENBLAS_DIR: OpenBLAS installation directory
OPENBLAS_LIB: OpenBLAS library directory
OPENBLAS_INC: OpenBLAS include directory
If you would like to compile a program that uses OpenBLAS, you need to specify the corresponding directories when compiling and linking your program.
Additionally, some OpenBLAS directories are added to the
LD_LIBRARY_PATH and other directories when you load the module. You can display the specifics of this with
module show openblas.
Caution: OpenBLAS uses dynamically loaded libraries (shared objects). If you link your program against OpenBLAS, then the OpenBLAS module needs to be loaded during execution as well (or at least the OpenBLAS directory needs to be in
Compiling with SCALAPACK
Analogously to OpenBLAS, the SCALAPACK module will set a number of environment variables:
SCALAPACK_DIR: the SCALAPACK installation directory
SCALAPACK_LIB: the SCALAPACK library directory
You can display the whole list with
module show scalapack. Like OpenBLAS, linking against SCALAPACK means that the module must be loaded during execution of your program (or at least the SCALAPACK directory needs to be in