Python

Default Version

Python 3.6.8 is installed on the cluster as the default version. In order to use it, you don’t need to load any modules. Specifically, for this Python version, some packages like numpy, scipy, and mpi4py are available if the corresponding modules are loaded as listed below:

Newer Version

The python version 3.11.2 can be used by loading the module python. Note: This module python has preinstalled compatible packages like numpy, scipy and mpi4py, so you don’t need to load the modules of these packages separately, unlike the default version. Use this command to list pre-installed packages.

You can load python module with the following command:

$ module load python
Loading Python 3.11.2 Environment

You can check the Python version with the following command:

$ python --version
Python 3.11.2

You can also check the installation directory with the following command:

$ which python
/cm/shared/omni/apps/python/3.11.2/bin/python

Caution: For compatibility reasons, 2.7.18 is still available as part of the operating system. Python 2 has been deprecated since early 2020 and should not be used! All following explanations cover only Python 3 and may not be applicable to Python 2.

A good starting point for beginner Python programmers is for example the tutorials section of the Python website.

Note: Example commands starting with $ refer to the Linux console, while commands starting with >>> refer to the Python console. These characters need to be omitted when typing in a command.

There are additional Python installations that come as part of Miniconda and Jupyter (and Miniconda allows you to install additional Python versions). Use of those Python installations is explained on the Miniconda and Jupyter pages.

Calling Python

You can call the Python interactive console with the command python:

$ python
Python 3.6.8 (default, Apr 16 2020, 01:36:27)
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

The command python2 will call the Python2 console.

To execute a Python script, enter

$ python <scriptname>

You can also execute individual Python commands using the option -c

$ python -c "print('Hello world.'); print('Use semicolons to enter multiple commands.')"

Note that the entire command or sequence of commands is enclosed by quotation marks (either single or double quotes are possible).

The commands described here can of course also be used in shell script (e.g. job scripts).

Python packages

Python code is organized into packages which contain modules. Each file with the suffix .py is a module, each directory that contains a file named __init__.py is a package. In the central Python Package Index (PyPI) there are a large number of packages available. Some of these are already installed on the cluster.

Pre-installed packages

You can display all installed packages by opening the Python console and entering

>>> help("modules")

Some packages that are commonly used in scientific programming are available by loading additional modules. Specifically, those are numpy, scipy und mpi4py. You can display these packages with

module spider py3

Installing packages youself

Note: The package manager Conda, which can install a large number of Python packages, is installed on the OMNI cluster as well. If you use Conda or the Anaconda distribution already, you should use that instead. Anaconda on the OMNI cluster is described here.

If you need a package that is not installed, you can use the program pip3 to install it yourself. To do that, enter

$ pip3 install --user <package name>

The package and any packages it depends on will then be installed in your home directory, by default in

/home/<YourUsername>/.local/lib/python<version>/site-packages

Packages in this directory will be found by Python immediately. Note that packages that were installed in this way are only available to you, not to other users. You can use a Python module or package within Python by importing it

>>> import examplemodule

or alternatively

>>> from examplepackage import examplemodule

Custom software and environment variables

If you want to run computations on the cluster with software developed by you (or manually downloaded by you through means other than Pip), you need to make sure that Python will find your software. A Python script that imports your software module

runthis.py
import mysoftware

# Do stuff with mysoftware…

will crash with the following error message:

>>> import mysoftware
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'mysoftware'

That is because Python, when importing modules (or packages), only searches for the module name in predefined paths. You can see the list of these paths by entering in the Python console

>>>import sys; print(sys.path)

You can add a directory inside Python

>>> sys.path.append(/home/<YourUsername>/mysoftware)

but it is more useful to use the environment variable PYTHONPATH by setting and exporting it at the beginning of your job script

$ export PYTHONPATH=$PYTHONPATH:/home/<YourUsername>/mysoftware

In both cases the addition to the search path is only temporary, namely as long as Python (in the case of sys.path) or your terminal (in the case of PYTHONPATH) is open. You can add paths permanently to sys.path by adding a text file that ends in .pth in a directory that is already on the path. In that file, you can add the new directories (one per line).

By default, the addition to the path is not recursive, i.e. even if you add a directory to sys.path or PYTHONPATH you cannot reach any Python modules in its subdirectories. However, if a subdirectory contains a file named __init__.py, Python will treat the subdirectory as a package (see above) and you can import either the package or individual modules from it.