FAIR Storage Service

Lorentz Institute offers its members a data storage option that complies with the FAIR Guiding Principles for scientific data management and stewardship.¹⁾ The goal is to offer a standard platform that enables, facilitates, and promotes the implementation of robust data-management plans complying with the most stringent professional requirements (scientific publications, research grants applications, etc..). Our FAIR storage server is powered by the iRODS Data Management Software²⁾, the de facto standard in scientific data management. Below you will find brief instructions on how to interact (store/view/retrieve data) with our storage, nonetheless you are strongly advised to read the official iRODS documentation before operating on our server.

The IL FAIR storage system is fault tolerant to prevent any catastrophic data loss. Data-loss protection occurs on two levels: hardware and software.

The first protection layer against data loss is provided by ad hoc enterprise-grade storage servers that constitute the backend of the iRODS system. Our main storage system is a TrueNAS Enterprise R50 with disks arranged in a RAIDZ2 array with a fault tolerance of two disks.

The second layer of protection against data loss is provided by iRODS replicas (copies of data across multiple storage backends). We offer a storage resource in which two copies of each data object are kept at any time across two different enterprise-graded storage systems essentially increasing fault tolerance.

Research data older than 10 years could be moved to different storage systems.

Writing access to this server must be requested by sending an email to support@lorentz.leidenuniv.nl.

Once access has been granted, you can interact with the server via any of the supported iRODS clients. For the sake of simplicity, we have set up all the IL GNU/Linux workstations and the xmaris cluster with the following iRODS clients so that you can start using our storage system immediately

• iCommands
• DavRODS

Additionally our remote desktop system is also able to access our iRODS server directly from its OpenOnDemand files app using https://rclone.org/ as described here.

iCommands is the most flexible and powerful client, but it might require a basic prior knowledge of the GNU/Linux command line.

At the Lorentz Institute we also offer several automated data-ingestion rules that post-process submitted data to our storage system using iCommands, for instance

• automatic storage of metadata of jupyter notebooks
• automatic storage of EXIF metadata of JPG, PNG images
• automatic addition of metadata to any data object from simple templates
• custom ingestion rules for your data? Contact support@lorentz.leidenuniv.nl

Please note that the automatic post-processing functionality has currently been tested only with iCommands and DavRODS.

Create an iRODS configuration file at ${HOME}/.irods/irods_environment.json. If the directory ${HOME}/.irods does not exist, then create it

mkdir -p ${HOME}/.irods

Create/edit in the directory mentioned above a file named irods_environment.json starting from the following simple template

{
    "irods_default_number_of_transfer_threads": 4,
    "irods_host": "<HOSTNAME>",
    "irods_maximum_size_for_single_buffer_in_megabytes": 32,
    "irods_port": <PORT>,
    "irods_default_resource": "<RESOURCE_NAME>",
    "irods_transfer_buffer_size_for_parallel_transfer_in_megabytes": 4,
    "irods_user_name": "<USERNAME>",
    "irods_zone_name": "<ZONE_NAME>",
    "irods_authentication_scheme": "PAM_password"
}

Edit the lines above at your convenience by looking at the available options and their meaning in the docs. The advice here is to keep your conf file simple at the beginning by just copying the template above to the specified location and by appropriately replacing the tags denoted by < > with the desired values. To connect to the IL FAIR storage you have the following possible options

Please not that even if you specify a default resource name in your iRODS configuration file, it is always possible to overwrite the resource destination during iCommands operations by specifying the option -R.

In a terminal window of your GNU/Linux workstation or the xmaris cluster, load the iCommands module

module load icommands

and initiate a connection to our storage server by typing iinit

remote [878] $ iinit 
Enter your current PAM password:
remote [879] $ 

Upon a successful authentication you will be able to interact with our FAIR storage server. Many iRODS commands have the same names and use of common GNU/Linux terminal commands with the only difference that they are prefixed by the letter i. For example to list the contents of your home collection (that is your home directory on the storage server), just type ils

remote [879] $ ils 
/ilZone/home/bongo:
  irodsfs_amd64_linux_v0.7.6.tar
  QT_7b.mp4
  test1
  test2
  test3
  C- /ilZone/home/bongo/TEST1

A more detailed output is obtained by using the options -l (long output) or -L (super long output) and an overview of the available options is obtained by passing the option -h.

A list of available commands is given by typing ihelp or by browsing to the iRODS online documentation.

In what follows iRODS objects are either data objects (e.g. a file) or collection objects (e.g. a directory).

Metadata are one of the most important features of iRODS that makes this software compliant with the FAIR principles. We advice you spend some time familiarize yourself with the concept of metadata and you get used to the idea that data without metadata is in practice incomplete data.

iRODS metadata are defined by Attribute-Value-Unit (AVU) triplets, for instance Length 10 meters

iRODS lets you easily share your data with external collaborators (users unknown to the IL systems). The only requirement is that they have access to iCommands or have a web browser. See the examples below

iCommands: Tickets

iRODS tickets constitute a powerful and flexible way to share your data with external collaborators who have access to iCommands at their institutions. In the example session below an IL user creates a read-only access ticket for a data object called results.dat and shares this unique alphanumeric code with his collaborator who will use it to gain access to the data.

# IL user iRODS session
iticket create read results.dat
ticket:68CK4jheDK924Jz
 
# Take note of the ticket id and pass it to your collaborator with 
# the full pat to your data object
 
iticket ls 68CK4jheDK924Jz
...
ticket type: read
obj type: data
owner name: bongo
owner zone: ilZone
...
expire time: none
data-object name: results.dat
data collection: /ilZone/home/bongo
No host restrictions
No user restrictions
No group restrictions

# External collaborator iRODS session
# Must first login as anonymous on the IL iRODS system (see above)
iget -t 68CK4jheDK924Jz /ilZone/home/bongo/results.dat

# IL user iRODS session
# delete the ticket if no longer needed
iticket delete 68CK4jheDK924Jz

If your collaborator has no access to icommands, then place the object you would like to share in the folder /ilZone/home/anonymous and ichmod to give the user anonymous reading access to your object which will be shared using the web browser at https://access.lorentz.leidenuniv.nl/anon/anonymous .

Web Browsers: webdav

Without creating a ticket for a data object, you could login to our FAIR storage system place any object you would like to share in a special collection called /ilZone/home/anonymous. Any external collaborator can then access it (READ ONLY) by browsing to https://access.lorentz.leidenuniv.nl/anon/anonymous in a similar fashion to the example here.

Access via the ticket system using a web browser will follow the scheme

https://access.lorentz.leidenuniv.nl/anon/<YOUR_HOME_PATH_TO_DATA_TO_BE_SHARED>?ticket=<TICKET_ID>
# example 
https://access.lorentz.leidenuniv.nl/anon/home/bongo/my/Data4You?ticket=hT6Yji111

To facilitate the uploading of objects metadata – an important component of any FAIR storage system – the IL storage system is programmed to automatically add the most basic piece of metadata upon each data object upload making sure that each stored data object has a minimal metadata consinstency. The metadata added to each data object is summarised in the table below

Further, the uploading of .ipynb, .png and .jpg files will automatically add to the storage metadata information found in any jupyter notebook and in the EXIF of .png and .jpg images.

Another powerful data ingestion rule allows you to upload metadata to an arbitrary data object by uploading a file template whose name follows a certain convention. Let us suppose you have just uploaded a data object whose iRODS path is /zoneName/home/username/collection_name/results.dat, then to add metadata to this object create locally a template file with name results.dat.metadata where each line has the format Attribute Name = Attribute Value

#cat results.dat.metadata
cluster = xmaris
node = maris048
cmd = script input1 input2
date = Aug 2022

and upload it to the IL storage server via DevRODS or iCommands, for instance

iput results.dat.metadata /zoneName/home/username/collection_name/results.dat.metadata

A suggested (but by no means exhaustive) list of metadata attribute names that you should add to any of your data files is given below

This method lets you access the IL FAIR Storage with the comfort of a GUI. Note though that differently than when using iCommands, it is not possible to select a different destination resource than ilorentzNoReplicaResource if you use WebDAV.

Anonymous access

Menu → Places → Connect to Server⁴⁾

Finder → Go → Connect to Server

Specify your IL credentials when prompted and click on Connect.

This access method works only on our remote desktop system

Open a terminal

$ rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> iRODS
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, and Tencent COS
   \ "s3"
...
36 / Union merges the contents of several upstream fs
   \ "union"
37 / Webdav
   \ "webdav"
38 / Yandex Disk
   \ "yandex"
39 / Zoho
   \ "zoho"
40 / http Connection
   \ "http"
41 / premiumize.me
   \ "premiumizeme"
42 / seafile
   \ "seafile"
Storage> 37
 
** See help for webdav backend at: https://rclone.org/webdav/ **
 
URL of http host to connect to
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Connect to example.com
   \ "https://example.com"
url> https://access.lorentz.leidenuniv.nl/davrods
Name of the Webdav site/service/software you are using
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Nextcloud
   \ "nextcloud"
 2 / Owncloud
   \ "owncloud"
 3 / Sharepoint Online, authenticated by Microsoft account.
   \ "sharepoint"
 4 / Sharepoint with NTLM authentication. Usually self-hosted or on-premises.
   \ "sharepoint-ntlm"
 5 / Other site/service or software
   \ "other"
vendor> iRODS
User name. In case NTLM authentication is used, the username should be in the format 'Domain\User'.
Enter a string value. Press Enter for the default ("").
user> YOUR_IL_USERNAME
Password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> y
Enter the password:
password:
Confirm the password:
password:
Bearer token instead of user/pass (e.g. a Macaroon)
Enter a string value. Press Enter for the default ("").
bearer_token> 
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
[iRODS]
type = webdav
url = https://access.lorentz.leidenuniv.nl/davrods
vendor = iRODS
user = YOUR_IL_USERNAME
pass = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
 
Name                 Type
====                 ====
iRODS                webdav
 
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
 
$

The navigate to https://remote.lorentz.leidenuniv.nl/pun/sys/dashboard/files/iRODS to interact with our iRODS storage via a web browser.