Wiss. Rechnen » File Systems
 

Caution: this page contains information about the OMNI cluster! The corresponding page for the HoRUS cluster can be found here.


The cluster has multiple file systems which serve different purposes. For normal users, this means your data is in three different irectories (and their subdirectories). Those are:

  • You home directory in /home/<YourUsername>. This is the default directory for all your data. Home directories are limited to 100 GB per user. You can find more about home directories below.
  • The workspace directory in /work. Workspaces are for short-term storage of large amounts of data. Every time you create a workspace, a subdirectory is created here for you. That subdircetory is called your workspace. Workspaces are not limited in size, but are limited in time. The total disk space available on the cluster is 500 TB. More on this below in the Workspaces section.
  • The burst buffer directory /fast. This directory is physically located on a partition with solid state disks (SSDs). It is intended for computations where large amounts of data need to be moved quickly. The burst buffer is only 32 TB in size. More on using the burst buffer below in the Burst Buffer section.

The file systems are the same from every login and compute node.

Home directory

Any user automatically gets a home directory, into which you can put your data. This directory has the path /home/<YourUsername>. The size of your home directory is limited to 100 GB.

Home directory snapshots

Daily snapshots are made of all home directories, these snapshots are deleted after 30 days. If you lose files in your home directory, you can change to the directory /home/.snapshot. In that directory you can find the daily snapshots, each in its subfolder. Caution: these snspshots should not be regarded as reliable backups. We recommend you backup your data yourself on another computer in addition to the snapshots.

You can simply copy files that you want to restore back to your normal home directory:

cd ./home/.snapshot/daily_<Datum>_0010/<Your Username>
cp <File or files> <Your home directory>

Example:

cd /home/.snapshot/daily.2020-08-04_0010/demo_user/
cp file1 file2 /home/demo_user

To copy entire folders with their content (recursively) use the option -r. Please note that existing data at the target location may potentially be overwritten. The cp command offers options like e.g. -i to give you more control over this. You can show the help with man cp.

Example: Copy folder exampledir back to home and confirm overwriting for every file:

cd /home/.snapshot/daily.2020-08-04_0010/demo_user/
cp -i -r exampledir /home/demo_user

Example: Restore all deleted files in folder exampledir without overwriting any existing files

cd /home/.snapshot/daily.2020-08-04_0010/demo_user/
cp -n -r exampledir /home/demo_user

Caution: on the HoRUS cluster it was possible to read other peoples’ home directories. This is no longer the case on OMNI. You can however continue to make files and directories available for other users with the chmod command (see also Linux basics).

Workspaces

For your compute jobs it is recommended not to use the home directory but rather to create a so-called workspace. This has two advantages: first, there is no size limit for workspaces, and second, the workspaces are located physically on another hard drive with a faster connection to the compute nodes. Workspaces have a limited duration: after the workspace expires, it is deleted. You can extend this duration up to three times.

Caution: there is no automatic backup for workspaces!

Create and extend workspaces

You can create a new workspace with the command

ws_allocate <WS name> <duration>

where the duration has to be given in days. The maximum possible duration without extensions is 30 days.

Caution: if you leave out the duration, the workspace will only be allocated for one day only.

The workspace will be created in a subdirectory of /work/ws-tmp/ and its name consists of your username and the workspace name specified by you. The workspace is available like any other folder with the cd command. In the following example:

$ ws_allocate test1 4
Info: creating workspace.
/work/ws-tmp/demo_user-test1
remaining extensions  : 3
remaining time in days: 4

you can see that a workspace named test1 with an initial duration of 4 days has been created and is available via cd /work/ws-tmp/demo_user-test1.

If you want to extend an existing workspace, you need to enter

ws_extend <WS name> <duration>

with the name of an existing workspace and a new duration. You can extend the duration three times, by a maximum of 30 days each time. If you enter the name of a workspace that does not exist, it will be created as if you had used ws_allocate.

The ws_allocate command also has some additional features which you can see with man ws_allocate.

Selecting the filesystem manually

Unlike HoRUS, you can select the filesystem for your workspace on OMNI. The regular workspace filesystem is called work, the burst buffer described below is called fast. You can use the command ws_list -l to display the available file systems.

When doing a ws_allocate or ws_extend, you can specify the filesystem with the -F option. If you do not use -F , the default (work) will be used:

ws_allocate -F [work|fast] <WS name>  <duration>
ws_extend -F [work|fast] <WS name>  <duration>

Caution: when you extend a workspace, you need to specify the same file system as for the original ws_allocate. That means, if, for example, you allocated a workspace with ws_allocate -F fast, you also need to extend it with ws_extend -F fast. If the workspace was originally created without the -F option, then the default (work) was used and you do not need to specify it again.

E-Mail notifications

The workspace mechanism can send you an e-mail before a workspace expires.

We recommend that you always use this function to avoid data losses.

The corresponding command is then:

ws_allocate <WS name> <duration> -r <number of days> -m <your e-mail address>

With the option -m you can specify the e-mail address and with -r you specify how many days before expiration you want to be warned. If you do not want to re-enter your e-mail address every time you can put a text file named .ws_user.conf in your home directory. In that file you write your address according to the following example:

mail: demo_user@uni-siegen.de

Note that there needs to be a space after the colon (YAML syntax).

You can also create a calender entry with

ws_send_ical <WS name> <e-mail address>

List your workspaces

You can list your existing workspaces by entering

ws_list

Release (delete) a workspace

If you do not need a workspace any more, you can release it. Caution: all data in this workspace will be unavailable from that point on.

To do that you can use the command

ws_release <Workspace name>

Restore a deleted workspace

As mentioned above, expired workspaces are not available any more, but the data are not deleted immediately. The data inside a workspace will be kept for 10 days, even after the workspace expired or was released, before being deleted completely. Therefore it is possible to restore the data if a workspace expired accidentally. To do that, follow these steps:

  1. You can list your expired workspaces via:

    $ ws_restore -l 
    <user>-<old-workspace>-<number>
            unavailable since Tue Jun 12 09:30:01 2018
  2. Create a new workspace:

    ws_allocate <new-workspace> <duration>
  3. Restore the expired workspace with the command ws_restore inside the new workspace. For that, you need the complete name of the old workspace (which includes your user name and an ID number), which you can get via ws_restore -l:

    ws_restore <user>-<old-workspace>-<number> <new-workspace>

    The new workspace will contain the old one in a subdirectory.

  4. Type the displayed text. This serves to make automatic workspace restoration impossible.

Burst Buffer

The OMNI cluster has a so-called burst buffer, meaning a fast storage partition. It consists of SSDs and has a size of 32 TB.

There are two things you need to be aware of in relation to the burst buffer

  • The burst buffer is less stable than the other filesystems, therefore you should move the data to a regular workspace as soon as your computation is complete.
  • The burst buffer with its 32 TB size is not very large and is shared betwen all cluster users. Please only use it if you really need the faster speed.

Creating a burst buffer workspace

Functionally, your directories on the burst buffer are also workspaces. Most commands therefore work identically to the previous section. To create a workspace in the burst buffer you need to use ws_allocate as usual, but you need to specify that you want to use the /fast filesystem.

ws_allocate -F fast <WS name> <duration>

Note that you also need to specify the -F fast option for a ws_extend on the burst buffer.

You can use the command ws_list -l to display a list of all file systems where you can create workspaces:

$ ws_list -l
available filesystems:
fast
work (default)

Aktualisiert um 17:03 am 8. February 2021 von Jan Steiner