The Anaconda distribution is a collection of software for scientific purposes. It contains a Python installation, an R installation and the package manager Conda, which can be used to install additional Anaconda packages. Since the complete Anaconda distribution with all packages takes up a lot of storage space, there is a variant called Miniconda, which only contains Python, Conda and a few basic packages. Both variants are completely free and open source.
Miniconda is available on the OMNI cluster. It is based on Python 3.8.5 and Conda 4.9.2. In order to use Miniconda, the module
miniconda3 needs to be loaded.
module load miniconda3
You can find the Conda documentation here. A complete list of available first-party Anaconda packages can be found here. Note that Conda also can also install third-party packages and can manage multiple sources (called “channels” in Conda). The most important community channel is conda-forge.
Note for Tensorflow and Pytorch users: There are preinstalled versions of both Tensorflow and Pytorch, both with GPU support, in our GPU stack.
Setting up Conda
You typically use Conda with the command
conda. For example, the command:
can display an overview of possible Conda commands. You can find a complete list in the Conda documentation here.
When you use Conda for the first time, you need to initialize it with
conda init, and then close and reopen the shell (for example by disconnecting and reconnecting to the cluster). Here is an example output for the command:
[demo_user@hpc-login01 ~]$ conda init no change /cm/shared/omni/apps/miniconda3/condabin/conda no change /cm/shared/omni/apps/miniconda3/bin/conda no change /cm/shared/omni/apps/miniconda3/bin/conda-env no change /cm/shared/omni/apps/miniconda3/bin/activate no change /cm/shared/omni/apps/miniconda3/bin/deactivate no change /cm/shared/omni/apps/miniconda3/etc/profile.d/conda.sh no change /cm/shared/omni/apps/miniconda3/etc/fish/conf.d/conda.fish no change /cm/shared/omni/apps/miniconda3/shell/condabin/Conda.psm1 no change /cm/shared/omni/apps/miniconda3/shell/condabin/conda-hook.ps1 no change /cm/shared/omni/apps/miniconda3/lib/python3.8/site-packages/xontrib/conda.xsh no change /cm/shared/omni/apps/miniconda3/etc/profile.d/conda.csh modified /home/js056352/.bashrc ==> For changes to take effect, close and re-open your current shell. <== (After logout and subsequent login...) (base) [demo_user@hpc-login01 ~]$
You can see two things: First,
conda init has modified the user’s
.bashrc file (you can open the file with a text editor you can see that the
PATH variable has been modified). Second, the command line now shows
(base) on the left. This is the name of the active Conda environment. We will get into Conda environments in the next section.
Caution: There is a danger of unintended side effects if you load or unload the
miniconda3 module, especially after a
conda init, because the order of directories in your
PATH variable will be switched. We recommend that you have the
miniconda3 module permanently active whenever you use Miniconda. This can be accomplished for example by adding
module load miniconda3 to your
.bashrc before the
conda init (see also Linux Basics).
Note also that the commands
python3 will direct to Conda-’s own Python as soon as Conda is active. This Python installation is distinct from the regular system Python on the cluster. Additionally, for each Conda environment you create (see next section), another Python instance will be installed.
You have other options for configuring Conda, those are explained here.
Conda Environments, much like Python’s Virtual Environments, are designed to separate software for different projects. Conda Environments have the advantage of not being limited to Python. Caution: You must create at least one custom Conda Environment because the default environment (
base) does not allow you to install packages. You can show all environments with
conda env list:
$ conda env list # conda environments: # base * /cm/shared/omni/apps/miniconda3 test /home/demo_user/.conda/envs/test /home/demo_user/miniconda3
As you can see in this example, the
base environment is in the Conda installation directory where normal users do not have write permissions. Your custom environments on the other hand are in your home directory.
Creating a new environment
You can create your own environment with the command:
conda create --name <Envname>
You can also already install packages at this point. If you want to specify a particular Python version (Python is treated like any other package by Conda), you can tell Conda that with the following command:
conda create --name <Envname> python=3.8
This syntax (
<packagename>=<version>) works for any Conda package. However, we recommend to only do this if you absolutely need it.
Using an environment
You need to activate an environment before you can use it. It is also helpful to deactivate the environment after use. The commands are
conda activate <Envname>
The name of the active environment is shown in brackets on the left of the command line. If you do not want any Conda Environment active, you have to type
conda deactivate a second time.
More information about Conda Environments can be found here.
You can install Conda packages with the command:
conda install <package name>
As mentioned before, Conda packages will be installed in a subfolder of your home directory. If you get an error message about missing permissions, make sure you are not in the
Caution: If you want to install Python packages, you should use
conda install over
pip install whenever possible, since mixing the two may lead to conflicts.
We would also like to remind you that you can also install packages from third-party sources. Conda, like most package managers, has an option to add additional channels. An important unofficial channel is
conda-forgewhich contains community packages. Installing unofficial packages is done at your own risk and should be done as little as possible.
Conda and Slurm jobs
Some of our users have reported problems that result from the complex interaction between the following factors:
- The loading of the Conda module
- The activation of the Conda environment
- The execution of the commands in the user’s
- The Slurm script options (
#SBATCH ...) that govern which environment variables are passed on to the job
There is no solution to this problem that works in all cases, because much depends on the modules used and the order of above mentioned steps.
A job script of the following form has proved useful to a number of our users with this type of problem:
#!/bin/bash #SBATCH (...) source ~/.bashrc conda deactivate conda activate <Your env name> <Your other commands>