Github CI Maintenance

Introduction

CI stands for continuous integration. The CI repo contains code that handles a number of tasks related to ensuring new code merged into the Offline and Production repos passes the necessary tests; specifically, it handles all tasks that involve communication with GitHub, such as checking if a new comment contains a test command or posting the test results on the PR. It does not contain the tests themselves; those are in the codetools repo. Code in the CI repo is written in Python.

CI repo code is used in our Jenkins jobs that handle build tests for Offline and Production. Each Jenkins job makes a fresh clone of the CI repo and downloads any new dependencies or dependencies with updated required versions. This means any changes made to the CI repo take effect immediately, and will be seen the next time someone runs the Offline or Production build tests.

Managing the CI Github repo

automated tools

Three main automated tools work on the CI repo: dependabot, pre-commit-hooks, and tests.

dependabot

Dependabot manages updates to the dependencies listed in CI/requirements.txt. It checks weekly for updates, and makes a PR to CI to change requirements.txt if it finds them. Dependabot actions are controlled by the config file at CI/.github/dependabot.yml. Dependabot runs on GitHub infrastructure.

pre-commit-hooks

pre-commit-hooks ensures uniform style and formatting. When you make a PR to the CI repo, it checks for whitespace errors, python warnings, etc, and then adds its own commit to your PR fixing these issues. These actions are controlled by the config file at CI/.pre-commit-config.yaml. When the pre-commit-hooks related packages have updates, pre-commit-hooks makes a PR with the versions in .pre-commit-config.yaml updated. pre-commit-hooks runs on GitHub infrastructure.

tests

The tests are run when new code is pushed to a branch of the Mu2e/CI repo. These are NOT the build tests for Offline and Production; these are only for changes to the CI repo. Both dependabot and pre-commit-hooks make their own branches for their PRs, so the these tests always run on those. The tests are controlled by the config file at CI/.github/workflows/tests.yml. Right now they only check that all the dependencies install correctly in the python versions we want. The tests run on GitHub infrastructure -- referred to elsewhere on this page as the "GitHub test runner".

Python version issues

The OS default python version on the Jenkins machines is Python 3.6. This is the version Jenkins uses when it creates the virtual environment where the CI repo code runs, so all code in the CI repo needs to work properly with that version. Some dependencies have stopped supporting Python 3.6 in their newest versions. Additionally, the GitHub test runner has dropped Python 3.6 from its latest version. This means we need to avoid problems caused by unwanted updates.

If a dependency in CI/requirements.txt is updated to a version that doesn't support Python 3.6, the next Jenkins job will stop downloading dependencies when it reaches the unsupported one, and will continue with whatever dependencies it happens to already have. If a dependency listed later in requirements.txt also received an update, the updated version will not be used. The way to avoid this problem is to always check that the automatic tests run on GitHub all pass, especially the Python 3.6 test, before merging any updates from depdendabot. If they don't pass, simply close the PR. To ignore updates for a dependency in CI/requirements.txt, add it to the "ignore" section of CI/.github/dependabot.yml. If, on the other hand, the Jenkins version of python is upgraded, it may be necessary to remove items from the "ignore" section so they can be updated. As an example of an "ignore" section:

#requirements.txt
updates:
  - package-ecosystem: "pip"
    ... other things ...
    ignore:
      - dependency-name: "PyGithub"
      - dependency-name: "requests"

If the GitHub test runner is set to use the latest version, it will fail to set up Python with the error message "Version 3.6 with arch x64 not found". This will result in PRs meant to update the CI repo failing the tests, even if the PR itself would cause no problems. The way to avoid this problem is to ensure that the "runs-on" item in CI/.github/workflows/tests.yml is set to ubuntu-20.04 and not to ubuntu-latest. If Jenkins is later updated to a new version of Python, this can be changed.

Important note: The python version required for the CI repo is not synchronized with the version required for Offline and Production. They use the version of python required by art, which is regularly updated. This gets set up when muse setup is run. Unfortunately, the build test workflow needs to use the CI repo before the muse setup can happen, so can't just piggyback off Offline to get the same Python version -- hence using the OS default python. This also means that, in order to keep the different dependencies separated, if a Jenkins job calls a script from the CI repo after running muse setup, it should use a subshell.