Managing the Galaxy Codebase

Due to needs specific to the NDIP project, Calvera runs a customized fork of Galaxy with specific features that have not yet been pushed upstream (if they will even be pushed up at all).

Altered Files

Below is a list of the files that differ from upstream, as well as a small description of the changes either in that file or the general feature for which the file was changed.(can check git blame or commit history for details, not all changes may be listed).

client/src/api/schema/schema.ts

• This file is auto-generated by running make update-client-api-schema command from the Makefile in the top level directory. Should not be manually edited.

client/src/components/Form/Elements/FormData/FormData.vue

• Edited so that a button can be used to add single dataset inputs on tools that support it. (uses singleDatasetInput property)
• Disabled batch mode for repeated input forms.

client/src/components/Form/FormElement.vue

• Added singleDatasetInput property.

client/src/components/History/Content/ContentItem.vue

• Sets the dataset to the stopped state if it has been stopped.

client/src/components/History/Content/Dataset/DatasetActions.vue

• Adds a stop button next to the other elements in a History entry while the associated job is running. Clicking on it lets you stop the job, while still getting results.

client/src/components/History/Content/model/states.ts

• Added stopped state which appears similar to a finished job in the history.

client/src/components/History/model/queries.ts

• Adds function that sends a request to the stopJob endpoint used by the history content.

client/src/components/JobInformation/JobInformation.vue

• Displays stopped as the status in the job details if appropriate.

client/src/components/JobInformation/JobInformation.test.js

• Adjusts tests for the custom JobInformation component.

client/src/components/Login/LoginForm.vue

• Add ability to disable internal login if disableInternalLogin property is set.

client/src/components/Login/LoginIndex.vue

• Add ability to disable internal login if disableInternalLogin property is set.

client/src/component/Upload/DefaultBoxORNL.vue

• Streamlines collection uploading by removing the extra modal and build button, but just allows you to just press the start button and have it automatally create the collection.
• Implements the ability to name the collection using a text box in the Box.
• Can now select whole directories to upload for both datasets and collections.

client/src/component/Upload/UploadContainerORNL.vue

• Uses DefaultBoxORNL instead of DefaultBox.

client/src/component/Upload/UploadModal.vue

• Uses the UploadContainerORNL component instead of UploadContainer.

client/src/components/Upload/UploadModal.test.js

• Adds tests for some custom components.

client/src/entry/analysis/modules/Login.vue

• Pass disable internal login property to login index.

client/src/style/scss/base.scss

• Add style for stopped state.

client/src/style/scss/dataset.scss

• Add style and icon for stopped datasets.

client/src/style/scss/theme/blue.scss

• Add style for stopped state.

config/plugins/visualizations/editor/config/editor.xml

• Add not_isinstance test for visualization data source.

config/plugins/visualizations/fits_graph_viewer/static/fits_graph_viewer.js

• A fix to the viewer for start up issue.

doc/source/admin/galaxy_options.rst

• added parameter for disabling batch mode inputs
• added parameter for pingfed login

dockerfiles/Dockerfile

• Added this dockerfile to facilitate our deployments.

dockerfiles/Dockerfile.full

• Added this dockerfile to facilitate our deployments.

lib/galaxy/authnz/init.py

• Adjusted token refreshing to take in a new threshold for when to refresh.

lib/galaxy/authnz/custos_authnz.py

• Properly decodes a token and checks for the proper user id (some providers have different key names for the subject)

lib/galaxy/authnz/managers.py

• Allows users to set the required scopes for OIDC providers in the oidc_backends_configuration.xml config file.
• Allows users to set the used well-known oidc config url for OIDC providers in the oidc_backends_configuration.xml config file.

lib/galaxy/authnz/psa_authnz.py

• Decodes user tokens by using the well known OIDC config
• Verifies Microsoft Azure token by querying the user info ednpoint due to their being a nonce in the header.

lib/galaxy/authnz/xsd/oidc_backends_config.xsd

• Added required_scope parameter to oidc backends configuration. (Lets you set a scope that will need to be in token in order to verify properly)

lib/galaxy/config/sample/galaxy.yml.sample

• Added sample config for features added including: hiding sign out, setting token refresh threshold.

lib/galaxy/config/sample/job_conf.xml.sample_advanced

• Example config for allowing user with oidc token to run inside of a docker container.

lib/galaxy/config/schemas/config_schema.yml

• Example config for disabling batch mode inputs for tools.

lib/galaxy/datatypes/binary.py

• Add user to get_filename arguments.

lib/galaxy/datatypes/blast.py

• Pull file to cache for other datatypes.

lib/galaxy/datatypes/data.py

• Object store support
• Refactoring filename sync.

lib/galaxy/datatypes/isa.py

• Refactoring filename sync.

lib/galaxy/datatypes/protocols.py

• Use get_filename function.

lib/galaxy/datatypes/spaln.py

• Use get_filename function.

lib/galaxy/datatypes/tabular.py

• Use get_filename function.

lib/galaxy/datatypes/text.py

• Use get_filename function.

lib/galaxy/dependencies/init.py

• Add sshoidc dependency.

lib/galaxy/dependencies/conditional-requirements.txt

• Add Rucio as conditional dependency.

lib/galaxy/files/sources/sshoidc.py

• Added to allow us to ssh into SNS/HFIR filesystems using an OIDC token.

lib/galaxy/job_execution/setup.py

• Use get_filename function.

lib/galaxy/jobs/init.py

• Use get_filename function.
• Set dataset update_time.
• Pass user to dataset delete function.

lib/galaxy/jobs/command_factory.py

• Add option to separate job command and metadata command.

lib/galaxy/jobs/handler.py

• Use get_filename function

lib/galaxy/jobs/runners/init.py

• Get OIDC user from multiple providers.
• Inject user as environment variable in tools

lib/galaxy/managers/configuration.py

• Added several configuration options.

lib/galaxy/managers/datasets.py

• Use get_filename function.
• Pass user to delete function.

lib/galaxy/managers/hdas.py

• Add stopped field
• Pass user to purge dataset function.

lib/galaxy/managers/hdcas.py

• Pass user to object store.

lib/galaxy/metadata/init.py

• Use get_filename function.
• Fix metadata for interactive tools.

lib/galaxy/metadata/set_metadata.py

• Exit set_metadata with error.
• Use get_filename function.

lib/galaxy/jobs/runners/pulsar.py

• Added a soft kill option used for stopping jobs while still getting results (as opposed to canceling and deleting everything.)

lib/galaxy/job_execution/container_monitor.py

• Properly raises an exception when a port is not found for a container.

lib/galaxy/managers/jobs.py

• Added ability to stop job early and still get results.

lib/galaxy/model/__init__.py

• Added stopped state, for when jobs are being stopped.

lib/galaxy/model/none_like.py

• Use get_filename.

lib/galaxy/objectstore/init.py

• Add user to delete call.

lib/galaxy/objectstore/_caching_base.py

• Add kwargs to_construct_path

lib/galaxy/objectstore/rucio.py

• Rucio related fixes (some may have been already committed upstream.)`

lib/galaxy/schema/schema.py

• Added stopped to JobState.

lib/galaxy/tool_util/deps/container_classes.py

• Get user to run inside container from OIDC token.

lib/galaxy/tool_util/deps/docker_util.py

• Get user to run inside container from OIDC token.

lib/galaxy/tool_util/parser/xml.py

• Auto-refresh OIDC tokens.

lib/galaxy/tools/actions/upload.py

• Set file extension instead of relying on sniffer.
• Add registerToolAction

lib/galaxy/util/json.py

• Add HTML escape for options in mako templates.

lib/galaxy/webapps/base/webapp.py

• Refactoring token refreshing.

lib/galaxy/webapps/galaxy/api/jobs.py

• An endpoint for stopping jobs.

lib/galaxy/webapps/galaxy/api/job_tokens.py

• Refactoring token refreshing.

lib/galaxy/webapps/galaxy/controllers/authnz.py

• Fix redirect URL after logout.

lib/galaxy/webapps/galaxy/controllers/history.py

• Pass user to HDA delete call.

lib/galaxy/webapps/galaxy/services/authenticate.py

• Add functionality to get API key with OIDC token.

lib/galaxy/webapps/galaxy/services/datasets.py

• Use get_filename function.

lib/galaxy/webapps/galaxy/services/history_contents.py

• Use get_filename function.
• Passes user instead of GalaxyWebTransaction to object store.

lib/galaxy_test/api/test_jobs.py

• Tests for stopping jobs.

templates/js-app.mako

• Apply HTML escape for options in mako template.

templates/webapps/galaxy/dataset/large_file_download_warning.mako

• Preview for binary files and warning on large file download.

The following tests as well:

test/unit/app/jobs/dynamic_tool_destination/mockGalaxy.py
test/unit/app/tools/test_wrappers.py
test/unit/data/datatypes/test_connectivity_table.py
test/unit/data/datatypes/test_validation.py
test/unit/data/datatypes/util.py
test/unit/tool_util/data/test_tool_data_bundles.py
test/unit/tool_util/test_cwl.py

Upgrading Galaxy

The first step to updating the Galaxy codebase to the latest upstream version is to set the upstream Galaxy repository, https://github.com/galaxyproject/galaxy, (you can also use a fork of the repo that you maintain) as a remote.

If using an IDE, look up instructions on how to do so for your particular IDE. Instructions for the terminal are given below:

1. Navigate to the Galaxy code pulled down from https://code.ornl.gov/ndip/galaxy in your terminal. If you do not have the Galaxy codebase locally, then you'll need to clone the repo.

git clone git@code.ornl.gov:ndip/galaxy.git # Only if you do not already have the code cloned locally.
cd galaxy

2. Add the upstream repository as a remote.

git remote add "upstream" git@github.com:galaxyproject/galaxy.git

You can check your current remotes with:

git remote -v

3. Create a new branch for the upgrade from the dev branch.

git checkout dev
git checkout -b upgrade-galaxy-to-version-2x.x

4. Start the merge process:

git pull upstream release_2x.x

5. After you run that command, Git will tell you which files have conflicts. In general, if a file is not in the above Altered Files list, then you should just accept the changes from the upstream repo. For the custom altered files, for the most part, you'll want to keep the custom code where possible. If you can confirm that upstream version is preferable than the custom file, then update this document's Altered File list. In case you are unfamiliar with merge conflicts, then this guide walks you through the process: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts/resolving-a-merge-conflict-using-the-command-line

6. After fixing merge conflicts, commit the rest of the files.

git add .
git commit -m"Commit message about updating"

or

git add *
git commit -m"Commit message about updating"

7. Run Galaxy locally and make sure everything is running properly.

8. Push your branch

git push --set-upstream origin upgrade-galaxy-to-version-2x.x

9. Create a merge request in the Gitlab UI and assign a reviewer.

10. If the CI passes, and the code changes look good, merge your branch through the UI.

11. Deploy Galaxy by referencing the deployment guide on this docs site.