Jenkins

Introduction

Jenkins is a freeware java code build system that the Scientific Computing Division is maintaining for the experiments. It runs on several powerful build machines where the code can be built using significant parallelism. It also has many useful management features. You can browse the Jenkins system. and docs for code managers. Jenkins manages and runs build scripts, triggers, archiving and reports, but the actual compilation with scons is black box to Jenkins.

Anyone can view the Jenkins pages, but to edit the pages you need to put in a ticket to get edit permissions in Jenkins, then authorize yourself to the site with a certificate. To edit, the Jenkins page should have your username at the top right. If it says "anonymous", try deleting your "buildmaster" cookies, restarting the browser, check that your cert is loaded in the browser, and try again.

Running Projects

Uses Jenkins for several purposes

1. for Continuous Integration builds. These are triggered automatically
   1. Github PR Tests for reacting to github commits with a build test and few short validation tests
   2. mu2e_muse_ci_build for reacting to a github merge by building Offline and writing it to cvmfs in a special area which can be accessed by "muse setup HEAD".
2. validation.
   1. mu2e-offline-geant-int builds every night to check that the future version of geant is still integrated
   2. mu2e_muse_test checks many aspects of Muse functionality - run by hand when the head of the Muse repo needs testing
3. building code for UPS packages. When a new tag is available these can be run by choosing "run with parameters," entering the tag and options in the project fields and building. The result is a link to a tarball which can be fetched and installed on cvmfs.
   1. mu2e-ADC-build artdaq_code
   2. mu2e-ADCM-build artdaq_core_mu2e
   3. mu2e-BTrk-build
   4. mu2e-KinKal-build
   5. mu2e-pcie-build
   6. mu2e-scons-build
   7. mu2e-TRACE-build
4. building Musings which are sets of code repos built to work together.
   1. mu2e-musing-build see MuseMaintenance.

Most are driven by a set of scripts kept in the codetools repository and UPS product. Below is more information on a few projects.

Continuous Integration Project

The computing division has written a set of scripts to run inside Jenkins and give additions features, such as timelines, to the build steps. This is called the Continuous Integration Project. We do not use this service at this time.

Trigger by URL

Projects can be kicked off by POST'ing a url with curl.

curl  -X POST "https://buildmaster.fnal.gov/buildmaster/view/mu2e/job/mu2e-BTrk-build/buildWithParameters?token=02345&PACKAGE_VERSION=v1_02_04&COMPILER=e14"

You can get the token (a kind of password) from the configure page. The parameters are also defined on the configure page.

Pushing a cert

To retrieve a file using a cert. Export your CILogon cert as a .p12 file and note the password. Then

openssl pkcs12 -in cilogon.p12 -out file.key.pem -nocerts -nodes
openssl pkcs12 -in cilogon.p12 -out file.crt.pem -clcerts -nokeys    
curl -O -E ./file.crt.pem --key ./file.key.pem \
https://buildmaster.fnal.gov/buildmaster/view/Art/job/build-mu-dist/BUILDTYPE=debug,label1=swarm,label2=SLF6/lastSuccessfulBuild/artifact/copyBack/cetlib-3.01.03-slf6-x86_64-e15-debug.tar.bz2

Push a cert with a url that triggers a project, so the jenkins project can use it.

# I assume that X509_USER_PROXY points to an existing proxy path
# in the 'mu2e' Jenkins project used in the link below you need
# to define a "Password Parameter" variable named: MU2E_CREDENTIAL


# on your local node:

~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~
credbits="MU2E_CREDENTIAL@$X509_USER_PROXY"

val2="value2"

curl -f \
--data-urlencode "token=1234" \
--data-urlencode "var1=value1" \
--data-urlencode "var2=$val2" \
--data-urlencode "$credbits" \
https://buildmaster.fnal.gov/job/mu2e/buildWithParameters
~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~

# I assume your Jenkins project is named "mu2e", replace that bit for different Jenkins projects
# var1 and var2 need to be defined in the Jenkins configuration to be able to retrieve their value
within a Jenkins job

# In your Jenkins job script to retrieve the proxy you can do the following:
~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~

export X509_USER_PROXY=/tmp/$$.proxy

rm -fv $X509_USER_PROXY
touch $X509_USER_PROXY
chmod 0600  $X509_USER_PROXY
echo "${MU2E_CREDENTIAL}" > $X509_USER_PROXY

voms-proxy-info -all

~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~

Github PR Tests

When a pull request (PR) is made or updated in the Offline or Production respositories, or certain comments are entered into an existing PR (see GitHubWorkflow#GitHub_Pull_Request_Procedures_and_FNALbuild), this triggers a series of Jenkins jobs which run tests on the PR code before it can be merged. The process for doing this is described below. Much of the code used by these jobs is described at Github_CI_Maintenance.

Diagram of the interactions between GitHub and Jenkins jobs when a PR is opened in the Offline repository.

receive-gh-event and receive-gh-event-production

These are the jobs that receive the initial notification from GitHub.com, and launch the other jobs.

The jobs receive-gh-event and receive-gh-event-production represent the first step in the PR testing flow for the Offline and Production repositories, respectively. Their job configurations are kept in the GitHubPRTests directory on Jenkins. They are triggered by GitHub webhooks -- a POST request from GitHub to Jenkins. This is set up in the Github repository settings (more info about GitHub webhooks here). Each Jenkins job corresponds to a webhook from one repository: Offline pull requests trigger receive-gh-event, and Production pull requests trigger receive-gh-event-production. Once either of these jobs begins, it checks the values of a few environment variables. Some of these comes from the GitHub Pull Request Builder plugin. After this step, it launches the next job in the sequence, mu2e-github-bot.

mu2e-github-bot

This is the job that makes all the decisions about what tests need to get run. All the pull request tests get launched from here.

This job configuration is kept in the GitHubPRTests directory on Jenkins. It is launched by either receive-gh-event or receive-gh-event-production. It clones the latest updates from the Mu2e/CI repository, then sets up a virtual environment and installs the CI code's dependencies (listed in the file requirements.txt). It then runs the CI script process-pull-request, which in turn calls another script called process_pr. This script is described in more detail here.

These scripts do all the checks to ensure that the tests can and should be run on this PR, including but not limited to: checking that the issue really is a PR and that it's still open, and that the author of the PR is in the Mu2e organization. If the PR is not new, they check to see that either the base branch's HEAD has changed, or that a comment has been posted that requests the tests be run.

They also select which tests, if any, to run on this PR. Currently, the requirements for running tests are: 1) the build tests get run on new PR's 2) the build tests get run when someone has posted a comment to run them (see here) and there are not already tests running After these steps are done, the script posts a comment on the PR indicating that the tests were triggered. The last step in this job is to trigger the downstream job mu2e-offline-build-test or mu2e-production-build-test.

mu2e-offline-build-test

This is the job that builds the code and runs the tests for Offline PRs.

This job configuration is kept in the GitHubPRTests directory on Jenkins. It is launched by the Jenkins job mu2e-github-bot. mu2e-offline-build-test clones the Mu2e/codetools and Mu2e/CI repositories. The codetools repo contains the code to run the tests; the CI repo contains the code to communicate with GitHub (more info about the CI and codetools repos at Github_CI_Maintenance). It then launches the codetools script gh_pr_bootstrap.sh, which in turn runs the job.sh script in the appropriate test folder. This script does the appropriate cloning and merging of Offline and Production, and contains the list of test fcl files to run.

If the merge fails, it reports the failure in a comment on the PR and exits. If it succeeds, it runs build.sh (which builds the code with muse and runs the tests fcl). It reports test results to the PR as they finish, and then posts a comment with a summary of all the test results, and links to log files. To access these log files, you need to be on the Fermilab VPN or logged into one of the mu2egpvm machines.

The following repeats the last paragraph with more detail. For CI launched from a PR to the Mu2e GitHub Offline repo, gh_pr_bootstrap.sh will source bin/github/jenkins_tests/mu2e-offline-build-test/job.sh at this line: [1]

Note that it is run in sub-shell so environment modifications are contained within the sub-shell. This script will do the following:

1. all of the clones and merges specified by the CI command
2. the whitespace and TODO/FIXME checking
   • Note that TODO/FIXME only looks for all upper case
3. It sources bin/github/jenkins_tests/mu2e-offline-build-test/build.sh, again in a subshell, at this line:[2], which
   • builds the code
   • runs all of the tests of the form "mu2e -c xxx.fcl".
   • runs the root based overlap checker.
4. On return from build.sh, it runs clang-tidy.
   • Because build.sh is run in a sub-shell, job.sh must spack load llvm or the UPS equivalent:
   • The AL9 build machines in Jenkins have /usr/bin/clang-tidy installed.
   • Therefore testing to see if clang-tidy is already defined is not a good way to tell if you need to spack load llvm.
   • THe system clang-tidy is 17.0.6 while our llvm clang-tidy is older, 14.0.6
5. It sends the final report to GitHub.

mu2e-production-build-test

This job configuration is kept in the GitHubPRTests directory on Jenkins. It does essentially the same things as mu2e-offline-build-test, but is launched when the initial PR is in Production. The code run by this job is essentially identical to the Offline build tests, but is a separate set of files in another directory to allow it to be changed independently of the Offline tests if necessary.