Skip to main content

Developer Release Notes - version 23.08

This is a developer facing release info - list of all merge requests related to this release for the projects relevent for the platform. See user facing changelog here

Development​

Galaxy Tools, v23.08.0​

Galaxy, v23.1.dev0.ornl​

  • Run client tests automatically (thanks to Sergey Yakubov). Merge Request 51
  • Fix parcel version (thanks to Sergey Yakubov). Merge Request 56
  • Fix jobinformation client tests (thanks to Sergey Yakubov). Merge Request 55
  • Delete datasets in object store (thanks to Sergey Yakubov). Merge Request 54
  • Make stdout monitoring variable names consistent (thanks to Gregory Cage). Merge Request 53
  • Add config option to hide the sign out link in user menu (thanks to Gregory Cage). Merge Request 52
  • Update to galaxy 23.1 (thanks to Sergey Yakubov). Merge Request 50
  • Check for none instead of empty string in pulsar job finish (thanks to Sergey Yakubov). Merge Request 48
  • Load stdout when requesting job status (thanks to Sergey Yakubov). Merge Request 42
  • Add oidc_provider to rucio plugin dictionary (thanks to Sergey Yakubov). Merge Request 47
  • Make stdout code display scrollable and auto scroll (thanks to Sergey Yakubov). Merge Request 46
  • Add registertoolaction (thanks to Sergey Yakubov). Merge Request 45
  • Pass oidc token to object store (thanks to Sergey Yakubov). Merge Request 44
  • Add option to ingest files to rucio object store (thanks to Sergey Yakubov). Merge Request 41
  • Change tokens to sync with code in pr (thanks to Sergey Yakubov). Merge Request 40
  • Update rucio (thanks to Sergey Yakubov). Merge Request 39
  • Add rucio scope as object store parameter (thanks to Sergey Yakubov). Merge Request 38
  • Cleanup rucio object store (thanks to Sergey Yakubov). Merge Request 37
  • Add an api endpoint to return oidc id token (thanks to Sergey Yakubov). Merge Request 36
  • Cherry-pick changes related to oidc tokens done for pr to galaxy upstream (thanks to Sergey Yakubov). Merge Request 35
  • Fix h5web viz (thanks to Sergey Yakubov). Merge Request 34
  • Remove strictmode (thanks to Gregory Watson). Merge Request 33
  • Prevent text editor visualization showing for binary data (thanks to Gregory Watson). Merge Request 32
  • Adding path element to avoid conflict with dataset api. (thanks to Gregory Watson). Merge Request 31
  • Use h5grove to improve error handling (thanks to Gregory Watson). Merge Request 30
  • Remove ajax call to avoid dataset download. (thanks to Gregory Watson). Merge Request 29
  • Run a job in docker container as a user from token (thanks to Sergey Yakubov). Merge Request 28
  • Clean up sshfs oidc code (thanks to Sergey Yakubov). Merge Request 27
  • Get rid of remote data broker (thanks to Sergey Yakubov). Merge Request 26
  • Rucio plugin for object store (thanks to Sergey Yakubov). Merge Request 25
  • An option to separate command and metadata command for pulsar (thanks to Sergey Yakubov). Merge Request 23
  • Do not auto download output files from object store (thanks to Sergey Yakubov). Merge Request 22
  • Ask for confirmation to view large files (thanks to Sergey Yakubov). Merge Request 19
  • Add galaxy testing (thanks to Sergey Yakubov). Merge Request 21
  • Update gravity to 0.13.4 and document setting env vars on dynamic (thanks to Sergey Yakubov). Merge Request 20
  • Get oidc token from dynamical config of the sshoidc file source (thanks to Sergey Yakubov). Merge Request 18
  • Access sns storage using oidc tokens (thanks to Sergey Yakubov). Merge Request 17
  • Refresh oidc tokens in galaxy (thanks to Sergey Yakubov). Merge Request 16
  • Implement object store for remote data broker (thanks to Sergey Yakubov). Merge Request 14
  • Added .dat type (thanks to Robert Smith). Merge Request 15

Pulsar, 0.15.3.dev3.ornl​

  • Update stdout file location in job manager (thanks to Gregory Cage). Merge Request 23
  • Update pulsar to 0.15.3 (thanks to Sergey Yakubov). Merge Request 22
  • Update to work with galaxy 23.1 (thanks to Sergey Yakubov). Merge Request 21
  • Send none instead of removing stdout in status message (thanks to Sergey Yakubov). Merge Request 20
  • Update dependencies in docker image (thanks to Sergey Yakubov). Merge Request 18
  • Don't send stdout in job postprocess if live updating stdout (thanks to Sergey Yakubov). Merge Request 17
  • Update implementation of stdout update in pulsar (thanks to Gregory Cage). Merge Request 15
  • Allow object store configuration per manager (thanks to Sergey Yakubov). Merge Request 16
  • Set meta as user (thanks to Sergey Yakubov). Merge Request 14
  • Rollback warn about missing file in cache, skip when using object store (thanks to Sergey Yakubov). Merge Request 13
  • Do not check local cache for remote file transfer action (thanks to Sergey Yakubov). Merge Request 12
  • Build and push pulsar dist to repository (thanks to Sergey Yakubov). Merge Request 11
  • Fix job params (thanks to Sergey Yakubov). Merge Request 10
  • Add job submit parameters to ones set by pulsar config (thanks to Sergey Yakubov). Merge Request 9
  • Get oidc tokens from galaxy (thanks to Sergey Yakubov). Merge Request 8
  • Add live updating of output files to galaxy (thanks to Gregory Cage). Merge Request 7
  • Add an option to ignore umask while creating working directories (thanks to Sergey Yakubov). Merge Request 6
  • Remove remote data broker in pulsar (thanks to Sergey Yakubov). Merge Request 5
  • Send out data to object store from pulsar (thanks to Sergey Yakubov). Merge Request 4

Remote Data Broker, v0.3.0​

  • Add delete endpoint, refactor (thanks to Sergey Yakubov). Merge Request 10
  • Get file using outbound connections only (thanks to Sergey Yakubov). Merge Request 9
  • Read file as user and return it via rest api (thanks to Sergey Yakubov). Merge Request 8
  • Adjust api to work with galaxy object store (thanks to Sergey Yakubov). Merge Request 7
  • Added logging, use filename, not folder, for output path (thanks to Sergey Yakubov). Merge Request 6
  • Data manager with local storage backend (thanks to Sergey Yakubov). Merge Request 5
  • Merge to see coverage and badges work (thanks to Sergey Yakubov). Merge Request 4

OIDC PAM module, v0.2.0​

Deployment​

Calvera, v23.08.0​

  • Add a link to release notes (thanks to Sergey Yakubov). Merge Request 79
  • Add topaz tools to calvera (thanks to Sergey Yakubov). Merge Request 78
  • Add large cpu vm to job config (thanks to Sergey Yakubov). Merge Request 77
  • Update calvera configuration (thanks to Sergey Yakubov). Merge Request 75
  • Hide logout button (thanks to Sergey Yakubov). Merge Request 74
  • Update calvera configuration (thanks to Sergey Yakubov). Merge Request 73
  • Configure galaxy-container-monitor (thanks to Sergey Yakubov). Merge Request 72
  • Add acknowledgment statement (thanks to Sergey Yakubov). Merge Request 70
  • Update pulsar version (thanks to Sergey Yakubov). Merge Request 68
  • Update galaxy to 23.1 (thanks to Sergey Yakubov). Merge Request 67
  • Mount ceph volumes to docker containers (thanks to Sergey Yakubov). Merge Request 64
  • Update rdb version (thanks to Sergey Yakubov). Merge Request 62
  • Update dependencies and config (thanks to Sergey Yakubov). Merge Request 61
  • Add export tool and its destination (thanks to Sergey Yakubov). Merge Request 60
  • Update calvera.ornl.gov certificate (thanks to Sergey Yakubov). Merge Request 58
  • Deploy remote data broker (thanks to Sergey Yakubov). Merge Request 59
  • Update calvera for register tool and its destination (thanks to Sergey Yakubov). Merge Request 55
  • Copy section_filters.py to proper destination (thanks to Gregory Cage). Merge Request 54
  • Add filters for all current tool sections on calvera (thanks to Gregory Cage). Merge Request 47
  • Configure calvera to use rucio object store (thanks to Sergey Yakubov). Merge Request 50
  • Update readme and remove cert to confirm correct certs (thanks to Mark Abraham). Merge Request 48
  • Update wildcard with as root/intermediate(s) only, pem encoded and test (thanks to Randall Petras). Merge Request 45
  • Add gzi admin back (thanks to Sergey Yakubov). Merge Request 44
  • Temporarily remove gzi from admin list for ndip workshop (thanks to Sergey Yakubov). Merge Request 43
  • Adds asrp refinement tools to tool configuration (thanks to Sergey Yakubov). Merge Request 42
  • Update readme.md, images/1.png (thanks to Sergey Yakubov). Merge Request 41
  • Add tools to calvera (thanks to Sergey Yakubov). Merge Request 40
  • Dev (thanks to Sergey Yakubov). Merge Request 39
  • Set destination for rpa_plot_chirpa (thanks to Sergey Yakubov). Merge Request 38
  • Dev (thanks to Sergey Yakubov). Merge Request 37
  • Update db backup location (thanks to Sergey Yakubov). Merge Request 36
  • Configure calvera to use new pulsars (thanks to Sergey Yakubov). Merge Request 35
  • Update nginx config to use /training instead of /training-material for training site (thanks to Sergey Yakubov). Merge Request 34
  • Configure calvera to use new pulsars (thanks to Sergey Yakubov). Merge Request 33
  • Add rpa plot tool (thanks to Gregory Watson). Merge Request 32
  • Update hpc resource configuration (thanks to Sergey Yakubov). Merge Request 31
  • Update calvera certificates (thanks to Sergey Yakubov). Merge Request 29
  • Add training material redirect (thanks to Sergey Yakubov). Merge Request 30
  • Update calvera certificates (thanks to Sergey Yakubov). Merge Request 28
  • Mega merge (thanks to Gregory Watson). Merge Request 27
  • Add mrpapp to tool config (thanks to Gregory Watson). Merge Request 26
  • Configure dynamic destinations for summit/ascent (thanks to Sergey Yakubov). Merge Request 25
  • Add a role to install extra packages (thanks to Mark Abraham). Merge Request 24
  • Add job runner and destination for marble project cph102 (thanks to Sergey Yakubov). Merge Request 23
  • Confgure runners for project gen169 (thanks to Sergey Yakubov). Merge Request 22
  • Refactor ansible and gitlab scripts to deploy calvera (thanks to Sergey Yakubov). Merge Request 21
  • Add usans and gpsans tools to calvera (thanks to Sergey Yakubov). Merge Request 20
  • Add pattern matching tool (thanks to Marshall McDonnell). Merge Request 19
  • Deploy dca tools to calvera (thanks to Sergey Yakubov). Merge Request 18
  • Close issue #62 - deploying calvera-test instance (thanks to Sergey Yakubov). Merge Request 17
  • Configure calvera to access sns storage using oidc tokens (thanks to Sergey Yakubov). Merge Request 16
  • Add and rearrange tools, get rid of neutrons/neutrons, add categories (thanks to Sergey Yakubov). Merge Request 15
  • Update calvera for incident beamline workflows (thanks to Sergey Yakubov). Merge Request 14
  • Add .dat type (thanks to Robert Smith). Merge Request 13
  • Add incident beamline tools (thanks to Sergey Yakubov). Merge Request 12
  • Updates the calvera infrastructure table readme (thanks to Gregory Watson). Merge Request 11
  • Add datatypes conf file (thanks to Robert Smith). Merge Request 10
  • Add ssh file source to analysis.sns.gov and user preferences to authenticate to it (thanks to Gregory Cage). Merge Request 9
  • Switch to gitlab jupyter image for interactive tool (thanks to Gregory Cage). Merge Request 7
  • Gzi oidc config (thanks to Gregory Cage). Merge Request 6
  • Change source of interactive tool image to docker hub (thanks to Gregory Cage). Merge Request 5
  • Switch interactive notebook jupyter image to ornl image (thanks to Gregory Cage). Merge Request 4
  • Add galaxy file source config (thanks to Gregory Watson). Merge Request 3
  • Adds static port information about calvera in readme (thanks to Marshall McDonnell). Merge Request 2
  • Adds story + task gitlab issue templates (thanks to Marshall McDonnell). Merge Request 1

Pulsar on Cades, v23.08.0​

Pulsar on Slate, v23.08.0​

Remote Data Broker on Slate, v23.08.0​

Rucio, v23.08.0​

  • Adds healthcheck for database (thanks to Marshall McDonnell). Merge Request 1
  • Add ci yaml (thanks to Marshall McDonnell). Merge Request 2
  • Updates readme for cert_dir_org + rucio ui note (thanks to Marshall McDonnell). Merge Request 1

Infrastructure​

Calvera VM​

CEPH storage​

Pulsar CPU VM​

Pulsar GPU VM​

  • Mount ceph storage to pulsar gpu (thanks to Sergey Yakubov). Merge Request 1

Pulsar Export VM​

  • Update mount options (thanks to Sergey Yakubov). Merge Request 2
  • Create a vm with sns/hfir mounted with write access (thanks to Sergey Yakubov). Merge Request 1

Pulsar Large VM​

  • Update file file-systems.yaml to enable sns-hfir mounts (thanks to Sergey Yakubov). Merge Request 1

Rucio​

  • Updating certs (thanks to Sergey Yakubov). Merge Request 1
  • Add installing docker-compose via user_data (thanks to Marshall McDonnell). Merge Request 18
  • Bump orc runner v0.10 (thanks to Marshall McDonnell). Merge Request 17
  • Move to tf module (thanks to Marshall McDonnell). Merge Request 16
  • Refactor out deployment (thanks to Marshall McDonnell). Merge Request 15
  • Refactor the gitlab-ci yaml + remove dockerfile (thanks to Marshall McDonnell). Merge Request 14
  • Adds input security group with default chenda-rucio-sg (thanks to Marshall McDonnell). Merge Request 13
  • Update to the orc_ubuntu-20.04_v20211117_1 image (thanks to Marshall McDonnell). Merge Request 12
  • Refactoring for xrootd noauth setup, bump in rucio versions, and fixes to test script (thanks to Marshall McDonnell). Merge Request 11
  • Updates to the rucio client for new certs layout (thanks to Marshall McDonnell). Merge Request 9
  • Fix name of certificate on host mounts for docker-compose (thanks to Marshall McDonnell). Merge Request 8
  • Switch to the correct pki dir for chmod (thanks to Marshall McDonnell). Merge Request 7
  • Change to not create local self-signed certs (thanks to Marshall McDonnell). Merge Request 6
  • Refactor to use ornl issued certificates and also self-signed certs for backend (thanks to Marshall McDonnell). Merge Request 5
  • Switch to rucio dev (thanks to Marshall McDonnell). Merge Request 3
  • Switches to the x509 for auth type in client script (thanks to Marshall McDonnell). Merge Request 2
  • Adds rucio cluster, client, and certs dirs (thanks to Marshall McDonnell). Merge Request 1

Kubernetes Cluster​