Skip to main content

Use OLCF HPC resources

In this document, we describe configuration steps to allow executing jobs on ORNL Ascent and Summit HPC clusters. See architecture for deployment design.

So we need:

  1. Deploy Pulsar on Onyx/Marble Kubernetes clusters using Helm Chart to accept jobs from Galaxy.
  2. Configure Calvera (and/or Calvera-test) to send jobs to the new Pulsar. You need to change job_conf.xml file for that and restart Galaxy. We do that in our GitLab repo. So, you add a job plugin
        <plugin id="pulsar_mq_onyx_gen169" type="runner" load="galaxy.jobs.runners.pulsar:PulsarMQJobRunner">
            <param id="galaxy_url">https://{{ inventory_hostname }}</param>
            <param id="amqp_url">amqp://{{ rabbitmq_user }}:{{ rabbitmq_password }}@{{ rabbitmq_url }}</param>
            <param id="manager">{{ pulsar_manager_onyx_gen169 }}</param>
        </plugin>

and a job destination

        <destination id="pulsar_onyx_gen169" runner="pulsar_mq_onyx_gen169">
                    <param id="jobs_directory">/ccsopen/proj/gen169/staging</param>
        </destination>

For Marble, you do the same, just use css in jobs_directory parameter:

        <plugin id="pulsar_mq_marble_cph102" type="runner" load="galaxy.jobs.runners.pulsar:PulsarMQJobRunner">
            <param id="galaxy_url">https://{{ inventory_hostname }}</param>
            <param id="amqp_url">amqp://{{ rabbitmq_user }}:{{ rabbitmq_password }}@{{ rabbitmq_url }}</param>
            <param id="manager">{{ pulsar_manager_marble_cph102 }}</param>
        </plugin>
        <destination id="pulsar_marble_cph102" runner="pulsar_mq_marble_cph102">
                    <param id="jobs_directory">/ccs/proj/cph102/staging</param>
        </destination>

Also we added new variables - pulsar_manager_onyx_gen169 (for project gen169) and to Ansible pulsar_manager_marble_cph102 for (project cph102) config (both production and test environments)

note

It is important to create one deployment per project. So, one needs to use the correct project for RabbitMQ queue names/Pulsar managers and also use the corresponding OpenShift project to deploy to. We'll be looking at options to (semi-)automate the process in the future.