Skip to main content

Dynamic Tool Destinations

Galaxy needs to know where (on which resources) to run tools. This is defined in the job_conf.xml file, where you can:

  • Manually define a destination for a specific tool, or
  • Use dynamic destinations that apply rules to determine the final destination.

The list of available destinations is here, and the dynamic rules are defined here.

By default, tools use the dynamic_resource_orchestration destination. The rules for dynamic selection are configured in resource_orchestration.yaml based on tool requirements and available resources.

This file consists of three sections:

Tools​

  • Defines resource requirements such as CPU, memory, and GPU.

  • May include features to send tools to destinations that support for special functionality:

    • oncat for ONCAT-enabled tools.
    • thinlinc, novnc, or x11 for interactive tools.
    • root for tools that start containers as root.

This setup allows Galaxy to automatically select destinations that meet a tool’s requirements. Alternatively, tools can also restrict execution to specific allowed destinations if desired.

Machines​

  • Defines the physical nodes available for jobs.

  • Each machine specifies:

    • Number of CPU cores
    • Memory
    • Optional GPUs

Examples include standard CPU nodes or GPU nodes for GPU-enabled tools.

Destinations​

  • Maps tools to specific machines and optionally enforce required features.

  • Ensures jobs run only on nodes that meet their requirements.

  • Destinations may specify:

    • The machine to run on
    • Features required (x11, novnc, oncat, root)

This ensures proper resource allocation, access control, and support for interactive workflows.

Examples:​

Adding a tool to Galaxy​

If you want to add a tool to Galaxy. You don't have to add it to the job_conf.xml file anymore. By default, Galaxy will use the dynamic_resource_orchestration destination.

In file resource_orchestration.yaml add the tool to the tools section and specify resources for the tool and the features it requires:

  example_tool_id_that_needs_gpu_and_oncat:
    gpu: 0.1
    features:
      oncat: true
  example_tool_id_that_needs_x11:
    features:
      x11: true
  example_tool_id_that_needs_to_be_sent_to_a_specific_destination:
    allowed_destinations:
      - pulsar_reg_to_sns

info

If you specify nothing, the tool will assume it needs one CPU core and 1 MB of memory and will be sent to the first available destination.

info

Some features are mutually exclusive. For example, root=true and oncat=true cannot be used together because no destination supports both. Some features depend on others. For instance, thinlinc=true requires root=true, since the only destination that supports thilinc also requires root=true (default is root=false)

Adding a new Machine​

Machines define the physical resources available to run tools. To add a machine, include it in the machines section of resource_orchestration.yaml and specify its CPU, memory, and optional GPUs:

sns_node01:
  cpu: 64
  memory: 1024

sns_gpu_node01:
  cpu: 64
  memory: 1024
  gpu: 4
  • cpu – Number of CPU cores available.
  • memory – Memory in GB.
  • gpu – Number of GPUs available (optional).

Destinations​

Destinations in this file specify machine (and thus available resources) and available features. To configure a destination, add it to the destinations section of resource_orchestration.yaml:

pulsar_sns_gpu_oncat:
  machine: sns_gpu_node01
  features:
    oncat: true
info

The actual destinations and their configuration are defined in the job_conf.xml file. There you can see which destinations are available and how they are configured. Destination names in resource_orchestration.yaml file must correspond to the names in the job_conf.xml file.