Miniconda

The Anaconda distribution is a collection of software for scientific purposes. It contains a Python installation, an R installation and the package manager Conda, which can be used to install additional Anaconda packages. Since the complete Anaconda distribution with all packages takes up a lot of storage space, there is a variant called Miniconda, which only contains Python, Conda and a few basic packages. Both variants are completely free and open source.

Miniconda is available on the OMNI cluster. It is based on Python 3.8.5 and Conda 4.9.2. In order to use Miniconda, the module miniconda3 needs to be loaded.

module load miniconda3

You can find the Conda documentation here. A complete list of available first-party Anaconda packages can be found here. Note that Conda also can also install third-party packages and can manage multiple sources (called “channels” in Conda). The most important community channel is conda-forge.

Note for Tensorflow and Pytorch users: There are preinstalled versions of both Tensorflow and Pytorch, both with GPU support, in our GPU stack.

Setting up Conda

You typically use Conda with the command conda. For example, the command:

conda help

can display an overview of possible Conda commands. You can find a complete list in the Conda documentation here.

When you use Conda for the first time, you need to initialize it with conda init, and then close and reopen the shell (for example by disconnecting and reconnecting to the cluster). Here is an example output for the command:

[demo_user@hpc-login01 ~]$ conda init
no change     /cm/shared/omni/apps/miniconda3/condabin/conda
no change     /cm/shared/omni/apps/miniconda3/bin/conda
no change     /cm/shared/omni/apps/miniconda3/bin/conda-env
no change     /cm/shared/omni/apps/miniconda3/bin/activate
no change     /cm/shared/omni/apps/miniconda3/bin/deactivate
no change     /cm/shared/omni/apps/miniconda3/etc/profile.d/conda.sh
no change     /cm/shared/omni/apps/miniconda3/etc/fish/conf.d/conda.fish
no change     /cm/shared/omni/apps/miniconda3/shell/condabin/Conda.psm1
no change     /cm/shared/omni/apps/miniconda3/shell/condabin/conda-hook.ps1
no change     /cm/shared/omni/apps/miniconda3/lib/python3.8/site-packages/xontrib/conda.xsh
no change     /cm/shared/omni/apps/miniconda3/etc/profile.d/conda.csh
modified      /home/js056352/.bashrc

==> For changes to take effect, close and re-open your current shell. <==

(After logout and subsequent login...)

(base) [demo_user@hpc-login01 ~]$

You can see two things: First, conda init has modified the user’s .bashrc file (you can open the file with a text editor you can see that the PATH variable has been modified). Second, the command line now shows (base) on the left. This is the name of the active Conda environment. We will get into Conda environments in the next section.

Caution: There is a danger of unintended side effects if you load or unload the miniconda3 module, especially after a conda init, because the order of directories in your PATH variable will be switched. We recommend that you have the miniconda3 module permanently active whenever you use Miniconda. This can be accomplished for example by adding module load miniconda3 to your .bashrc before the conda init (see also Linux Basics).

Note also that the commands python and python3 will direct to Conda-’s own Python as soon as Conda is active. This Python installation is distinct from the regular system Python on the cluster. Additionally, for each Conda environment you create (see next section), another Python instance will be installed.

You have other options for configuring Conda, those are explained here.

Conda Environments

Conda Environments, much like Python’s Virtual Environments, are designed to separate software for different projects. Conda Environments have the advantage of not being limited to Python. Caution: You must create at least one custom Conda Environment because the default environment (base) does not allow you to install packages. You can show all environments with conda env list:

$ conda env list

# conda environments:
#
base                  *  /cm/shared/omni/apps/miniconda3
test                     /home/demo_user/.conda/envs/test
                         /home/demo_user/miniconda3

As you can see in this example, the base environment is in the Conda installation directory where normal users do not have write permissions. Your custom environments on the other hand are in your home directory.

Creating a new environment

You can create your own environment with the command:

conda create --name <Envname>

You can also already install packages at this point. If you want to specify a particular Python version (Python is treated like any other package by Conda), you can tell Conda that with the following command:

conda create --name <Envname> python=3.8

This syntax (<packagename>=<version>) works for any Conda package. However, we recommend to only do this if you absolutely need it.

Using an environment

You need to activate an environment before you can use it. It is also helpful to deactivate the environment after use. The commands are

conda activate <Envname>

and

conda deactivate

The name of the active environment is shown in brackets on the left of the command line. If you do not want any Conda Environment active, you have to type conda deactivate a second time.

More information about Conda Environments can be found here.

Installing packages

You can install Conda packages with the command:

conda install <package name>

As mentioned before, Conda packages will be installed in a subfolder of your home directory. If you get an error message about missing permissions, make sure you are not in the base environment.

Caution: If you want to install Python packages, you should use conda install over pip install whenever possible, since mixing the two may lead to conflicts.

We would also like to remind you that you can also install packages from third-party sources. Conda, like most package managers, has an option to add additional channels. An important unofficial channel is conda-forgewhich contains community packages. Installing unofficial packages is done at your own risk and should be done as little as possible.

More details on Conda packages can be found in the Conda documentation here, more about Conda channels here. A complete list of available Anaconda packages is here.

Conda and Slurm jobs

Some of our users have reported problems that result from the complex interaction between the following factors:

• The loading of the Conda module
• The activation of the Conda environment
• The execution of the commands in the user’s .bashrc file
• The Slurm script options (#SBATCH ...) that govern which environment variables are passed on to the job

There is no solution to this problem that works in all cases, because much depends on the modules used and the order of above mentioned steps.

A job script of the following form has proved useful to a number of our users with this type of problem:

#!/bin/bash
#SBATCH (...)

source ~/.bashrc
conda deactivate
conda activate <Your env name>

<Your other commands>