Skip to main content

Remote Data Broker

Remote Data Broker is a low-level service within the NDIP data management solution responsible for serving files to other NDIP components using the HTTP protocol.

Code​

The service is written in Go, and its source code, along with a more detailed description, including build, configuration, and usage instructions, can be found here.

Deployment​

Servers​

Remote Data Broker servers run on calvera and calvera-test. They are deployed as system services as a part of Galaxy deployment via a CI/CD job using Ansible.

Clients​

Remote Data Broker clients can be installed together with Pulsar via a CI/CD job using Ansible on cloud machines and the SNS GPU machine. In the Slate and Spin clusters, they are installed manually (due to two-factor authentication) using a Helm chart.

Hosts​

As stated above, RDB servers are running on calvera.ornl.gov and calvera-test.ornl.gov - virtual machines that are provisioned through our infrastructure monorepo (see the infrastructure section for more details).

RDB clients run on Pulsar worked nodes that have no access to shared storage. These are:

See corresponding links for more information.

note

SSH key for the SNS GPU VM can be found in Gitlab CI/CD Variables under PULSAR_SSH_KEY. You might need to request for access to this page.

Deployment Details​

  • Calvera Nodes: Managed via the ornl.tuning Ansible task. Configuration values are defined in the Ansible variable file.

  • Pulsar Nodes (excluding Kubernetes clusters): Managed via the ornl.tuning.pulsar Ansible task. Configuration values are defined in the Ansible variable file.

  • Slate: Deployed manually via a Helm chart. See the README for details.

  • Spin: Deployed manually via a Helm chart. See the deployment script for more details.