Skip to main content

Rucio

Rucio is a service developed by CERN that enables centralized management of large volumes of data backed by many heterogeneous storage backends. We are using it in NDIP so that jobs can access data located in different places transparently for user. To implement this we have developed a Rucio object store plugin for Galaxy and configured Galaxy to use Rucio object store in our instances.

Deployment​

Rucio runs inside a Docker container on a dedicated VM in the ORC cloud. It is deployed via a CI/CD job using Ansible.

note

SSH key for the VM can be found in Gitlab CI/CD Variables. You might need to request for access to this page.

Host​

The Rucio VM is provisioned through our infrastructure monorepo (see the infrastructure section for more details). The VM's IP address can be found in the Ansible inventory.

Deployment Details​

Rucio is deployed via Docker Compose along with PostgreSQL and a db-init container that initializes the database.

The deployment is managed through the rucio Ansible role. This role includes:

  • Configuration templates for the config files and Docker Compose setup.
  • Tasks to copy config files, start Docker containers, and create/update Rucio RSEs.

Configuration​

Configuration values are defined in Ansible variable files.

The most important file is the one with the RSE configuration. Use it to add or update RSEs (Rucio Storage Elements).