Caution: this page contains information about the OMNI cluster! The corresponding page for the HoRUS cluster can be found here.
The cluster has multiple file systems which serve different purposes. For normal users, this means your data is in three different irectories (and their subdirectories). Those are:
- You home directory in
/home/<YourUsername>. This is the default directory for all your data. Home directories are limited to 100 GB per user. You can find more about home directories below.
- The workspace directory in
/work. Workspaces are for short-term storage of large amounts of data. Every time you create a workspace, a subdirectory is created here for you. That subdircetory is called your workspace. Workspaces are not limited in size, but are limited in time. The total disk space available on the cluster is 500 TB. More on this below in the Workspaces section.
- The burst buffer directory
/fast. This directory is physically located on a partition with solid state disks (SSDs). It is intended for computations where large amounts of data need to be moved quickly. The burst buffer is only 32 TB in size. More on using the burst buffer below in the Burst Buffer section.
The file systems are the same from every login and compute node.
Any user automatically gets a home directory, into which you can put your data. This directory has the path
/home/<YourUsername>. The size of your home directory is limited to 100 GB.
Home directory snapshots
Daily snapshots are made of all home directories, these snapshots are deleted after 30 days. If you lose files in your home directory, you can change to the directory
/home/.snapshot. In that directory you can find the daily snapshots, each in its subfolder. Caution: these snspshots should not be regarded as reliable backups. We recommend you backup your data yourself on another computer in addition to the snapshots.
You can simply copy files that you want to restore back to your normal home directory:
cd ./home/.snapshot/daily_<Datum>_0010/<Your Username> cp <File or files> <Your home directory>
cd /home/.snapshot/daily.2020-08-04_0010/demo_user/ cp file1 file2 /home/demo_user
To copy entire folders with their content (recursively) use the option
-r. Please note that existing data at the target location may potentially be overwritten. The
cp command offers options like e.g.
-i to give you more control over this. You can show the help with
Example: Copy folder exampledir back to home and confirm overwriting for every file:
cd /home/.snapshot/daily.2020-08-04_0010/demo_user/ cp -i -r exampledir /home/demo_user
Example: Restore all deleted files in folder exampledir without overwriting any existing files
cd /home/.snapshot/daily.2020-08-04_0010/demo_user/ cp -n -r exampledir /home/demo_user
Caution: on the HoRUS cluster it was possible to read other peoples’ home directories. This is no longer the case on OMNI. You can however continue to make files and directories available for other users with the
chmod command (see also Linux basics).
For your compute jobs it is recommended not to use the home directory but rather to create a so-called workspace. This has two advantages: first, there is no size limit for workspaces, and second, the workspaces are located physically on another hard drive with a faster connection to the compute nodes. Workspaces have a limited duration: after the workspace expires, it is deleted. You can extend this duration up to three times.
Caution: there is no automatic backup for workspaces!
Create and extend workspaces
You can create a new workspace with the command
ws_allocate <WS name> <duration>
where the duration has to be given in days. The maximum possible duration without extensions is 30 days.
Caution: if you leave out the duration, the workspace will only be allocated for one day only.
The workspace will be created in a subdirectory of
/work/ws-tmp/ and its name consists of your username and the workspace name specified by you. The workspace is available like any other folder with the
cd command. In the following example:
$ ws_allocate test1 4 Info: creating workspace. /work/ws-tmp/demo_user-test1 remaining extensions : 3 remaining time in days: 4
you can see that a workspace named
test1 with an initial duration of 4 days has been created and is available via
If you want to extend an existing workspace, you need to enter
ws_extend <WS name> <duration>
with the name of an existing workspace and a new duration. You can extend the duration three times, by a maximum of 30 days each time. If you enter the name of a workspace that does not exist, it will be created as if you had used
ws_allocate command also has some additional features which you can see with
Selecting the filesystem manually
Unlike HoRUS, you can select the filesystem for your workspace on OMNI. The regular workspace filesystem is called
work, the burst buffer described below is called
fast. You can use the command
ws_list -l to display the available file systems.
When doing a
ws_extend, you can specify the filesystem with the
-F option. If you do not use
-F , the default (
work) will be used:
ws_allocate -F [work|fast] <WS name> <duration> ws_extend -F [work|fast] <WS name> <duration>
Caution: when you extend a workspace, you need to specify the same file system as for the original
ws_allocate. That means, if, for example, you allocated a workspace with
ws_allocate -F fast, you also need to extend it with
ws_extend -F fast. If the workspace was originally created without the
-F option, then the default (
work) was used and you do not need to specify it again.
The workspace mechanism can send you an e-mail before a workspace expires.
We recommend that you always use this function to avoid data losses.
The corresponding command is then:
ws_allocate <WS name> <duration> -r <number of days> -m <your e-mail address>
With the option
-m you can specify the e-mail address and with
-r you specify how many days before expiration you want to be warned. If you do not want to re-enter your e-mail address every time you can put a text file named
.ws_user.conf in your home directory. In that file you write your address according to the following example:
Note that there needs to be a space after the colon (YAML syntax).
You can also create a calender entry with
ws_send_ical <WS name> <e-mail address>
List your workspaces
You can list your existing workspaces by entering
Release (delete) a workspace
If you do not need a workspace any more, you can release it. Caution: all data in this workspace will be unavailable from that point on.
To do that you can use the command
ws_release <Workspace name>
As mentioned above, expired workspaces are not available any more, but the data are not deleted immediately. The data inside a workspace will be kept for 10 days, even after the workspace expired or was released, before being deleted completely. Therefore it is possible to restore the data if a workspace expired accidentally. To do that, follow these steps:
You can list your expired workspaces via:
$ ws_restore -l <user>-<old-workspace>-<number> unavailable since Tue Jun 12 09:30:01 2018
Create a new workspace:
ws_allocate <new-workspace> <duration>
Restore the expired workspace with the command
ws_restoreinside the new workspace. For that, you need the complete name of the old workspace (which includes your user name and an ID number), which you can get via
ws_restore <user>-<old-workspace>-<number> <new-workspace>
The new workspace will contain the old one in a subdirectory.
Type the displayed text. This serves to make automatic workspace restoration impossible.
The OMNI cluster has a so-called burst buffer, meaning a fast storage partition. It consists of SSDs and has a size of 32 TB.
There are two things you need to be aware of in relation to the burst buffer
- The burst buffer is less stable than the other filesystems, therefore you should move the data to a regular workspace as soon as your computation is complete.
- The burst buffer with its 32 TB size is not very large and is shared betwen all cluster users. Please only use it if you really need the faster speed.
Creating a burst buffer workspace
Functionally, your directories on the burst buffer are also workspaces. Most commands therefore work identically to the previous section. To create a workspace in the burst buffer you need to use
ws_allocate as usual, but you need to specify that you want to use the
ws_allocate -F fast <WS name> <duration>
Note that you also need to specify the
-F fast option for a
ws_extend on the burst buffer.
You can use the command
ws_list -l to display a list of all file systems where you can create workspaces:
$ ws_list -l available filesystems: fast work (default)