Jenkins: Difference between revisions
(10 intermediate revisions by 2 users not shown) | |||
Line 8: | Line 8: | ||
==Running Projects== | ==Running Projects== | ||
mu2e | Uses Jenkins for several purposes | ||
# for Continuous Integration builds. These are triggered automatically | |||
## '''Github PR Tests''' for reacting to github commits with a build test and few short validation tests | |||
## '''mu2e_muse_ci_build''' for reacting to a github merge by building Offline and writing it to cvmfs in a special area which can be accessed by "muse setup HEAD". | |||
# validation. | |||
## '''mu2e-offline-geant-int''' builds every night to check that the future version of geant is still integrated | |||
## '''mu2e_muse_test''' checks many aspects of Muse functionality - run by hand when the head of the Muse repo needs testing | |||
# building code for [[UPS]] packages. When a new tag is available these can be run by choosing "run with parameters," entering the tag and options in the project fields and building. The result is a link to a tarball which can be fetched and installed on [[Cvmfs|cvmfs]]. | |||
##'''mu2e-ADC-build''' artdaq_code | |||
##'''mu2e-ADCM-build''' artdaq_core_mu2e | |||
##'''mu2e-BTrk-build''' | |||
##'''mu2e-KinKal-build''' | |||
##'''mu2e-pcie-build''' | |||
##'''mu2e-scons-build''' | |||
##'''mu2e-TRACE-build''' | |||
# building [[Muse#Musings_.28published_muse_builds.29|Musings]] which are sets of code repos built to work together. | |||
## '''mu2e-musing-build''' see [[MuseMaintenance]]. | |||
Most are driven by a set of scripts kept in the codetools repository and UPS product. Below is more information on a few projects. | |||
Most are driven by a set of scripts kept in the codetools repository and UPS product. | |||
==Continuous Integration Project== | ==Continuous Integration Project== | ||
Line 98: | Line 88: | ||
~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~ | ~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
</pre> | </pre> | ||
==Github PR Tests== | |||
When a pull request (PR) is made or updated in the Offline or Production respositories, or certain comments are entered into an existing PR (see [[GitHubWorkflow#GitHub_Pull_Request_Procedures_and_FNALbuild]]), this triggers a series of Jenkins jobs which run tests on the PR code before it can be merged. The process for doing this is described below. Much of the code used by these jobs is described at [[Github_CI_Maintenance]]. | |||
[[File:OfflineBuildTestFlow.png|450px|thumb|left|Diagram of the interactions between GitHub and Jenkins jobs when a PR is opened in the Offline repository.]] | |||
'''receive-gh-event''' and '''receive-gh-event-production''' | |||
These are the jobs that receive the initial notification from GitHub.com, and launch the other jobs. | |||
The jobs receive-gh-event and receive-gh-event-production represent the first step in the PR testing flow for the Offline and Production repositories, respectively. Their job configurations are kept in the GitHubPRTests directory on Jenkins. They are triggered by GitHub webhooks -- a POST request from GitHub to Jenkins. This is set up in the Github repository settings (more info about GitHub webhooks [https://docs.github.com/en/developers/webhooks-and-events/webhooks/about-webhooks here]). Each Jenkins job corresponds to a webhook from one repository: Offline pull requests trigger receive-gh-event, and Production pull requests trigger receive-gh-event-production. | |||
Once either of these jobs begins, it checks the values of a few environment variables. Some of these comes from the GitHub Pull Request Builder [https://plugins.jenkins.io/ghprb/ plugin]. After this step, it launches the next job in the sequence, mu2e-github-bot. | |||
'''mu2e-github-bot''' | |||
This is the job that makes all the decisions about what tests need to get run. All the pull request tests get launched from here. | |||
This job configuration is kept in the GitHubPRTests directory on Jenkins. It is launched by either receive-gh-event or receive-gh-event-production. It clones the latest updates from the [https://github.com/Mu2e/CI Mu2e/CI] repository, then sets up a virtual environment and installs the CI code's dependencies (listed in the file requirements.txt). It then runs the CI script process-pull-request, which in turn calls another script called process_pr. This script is described in more detail [[Github_CI_Maintenance#Code_for_controlling_whether_build_tests_are_run|here]]. | |||
These scripts do all the checks to ensure that the tests can and should be run on this PR, including but not limited to: checking that the issue really is a PR and that it's still open, and that the author of the PR is in the Mu2e organization. If the PR is not new, they check to see that either the base branch's HEAD has changed, or that a comment has been posted that requests the tests be run. | |||
They also select which tests, if any, to run on this PR. Currently, the requirements for running tests are: | |||
1) the build tests get run on new PR's | |||
2) the build tests get run when someone has posted a comment to run them ([https://mu2ewiki.fnal.gov/wiki/GitHubWorkflow#GitHub_Pull_Request_Procedures_and_FNALbuild see here]) and there are not already tests running | |||
After these steps are done, the script posts a comment on the PR indicating that the tests were triggered. The last step in this job is to trigger the downstream job mu2e-offline-build-test or mu2e-production-build-test. | |||
'''mu2e-offline-build-test''' | |||
This is the job that builds the code and runs the tests for Offline PRs. | |||
This job configuration is kept in the GitHubPRTests directory on Jenkins. It is launched by the Jenkins job mu2e-github-bot. mu2e-offline-build-test clones the [https://github.com/Mu2e/codetools Mu2e/codetools] and [https://github.com/Mu2e/CI Mu2e/CI] repositories. The codetools repo contains the code to run the tests; the CI repo contains the code to communicate with GitHub (more info about the CI and codetools repos at [[Github_CI_Maintenance]]). It then launches the codetools script gh_pr_bootstrap.sh, which in turn runs the job.sh script in the appropriate test folder. This script does the appropriate cloning and merging of Offline and Production, and contains the list of test fcl files to run. | |||
If the merge fails, it reports the failure in a comment on the PR and exits. If it succeeds, it runs build.sh (which builds the code with muse and runs the tests fcl). It reports test results to the PR as they finish, and then posts a comment with a summary of all the test results, and links to log files. To access these log files, you need to be on the Fermilab VPN or logged into one of the mu2egpvm machines. | |||
The following repeats the last paragraph with more detail. For CI launched from a PR to the Mu2e GitHub Offline repo, gh_pr_bootstrap.sh will source bin/github/jenkins_tests/mu2e-offline-build-test/job.sh at this line: [https://github.com/Mu2e/codetools/blob/ef4bd046b9008c35a13de97ff115484a2151db70/bin/gh_pr_bootstrap.sh#L214] | |||
Note that it is run in sub-shell so environment modifications are contained within the sub-shell. This script will do the following: | |||
# all of the clones and merges specified by the CI command | |||
# the whitespace and TODO/FIXME checking | |||
#* Note that TODO/FIXME only looks for all upper case | |||
# It sources bin/github/jenkins_tests/mu2e-offline-build-test/build.sh, again in a subshell, at this line:[https://github.com/Mu2e/codetools/blob/ef4bd04b9008c35a13de97ff115484a2151db70/bin/github/jenkins_tests/mu2e-offline-build-test/job.sh#L175], which | |||
#* builds the code | |||
#* runs all of the tests of the form "mu2e -c xxx.fcl". | |||
#* runs the root based overlap checker. | |||
# On return from build.sh, it runs clang-tidy. | |||
#* Because build.sh is run in a sub-shell, job.sh must spack load llvm or the UPS equivalent: | |||
#* The AL9 build machines in Jenkins have /usr/bin/clang-tidy installed. | |||
#* Therefore testing to see if clang-tidy is already defined is not a good way to tell if you need to spack load llvm. | |||
#* THe system clang-tidy is 17.0.6 while our llvm clang-tidy is older, 14.0.6 | |||
# It sends the final report to GitHub. | |||
'''mu2e-production-build-test''' | |||
This job configuration is kept in the GitHubPRTests directory on Jenkins. It does essentially the same things as mu2e-offline-build-test, but is launched when the initial PR is in Production. The code run by this job is essentially identical to the Offline build tests, but is a separate set of files in another directory to allow it to be changed independently of the Offline tests if necessary. | |||
[[Category:Computing]] | [[Category:Computing]] | ||
[[Category:Code]] | [[Category:Code]] | ||
[[Category:CodeManagement]] | [[Category:CodeManagement]] |
Latest revision as of 22:18, 14 June 2024
Introduction
Jenkins is a freeware java code build system that the Scientific Computing Division is maintaining for the experiments. It runs on several powerful build machines where the code can be built using significant parallelism. It also has many useful management features. You can browse the Jenkins system. and docs for code managers. Jenkins manages and runs build scripts, triggers, archiving and reports, but the actual compilation with scons is black box to Jenkins.
Anyone can view the Jenkins pages, but to edit the pages you need to put in a ticket to get edit permissions in Jenkins, then authorize yourself to the site with a certificate. To edit, the Jenkins page should have your username at the top right. If it says "anonymous", try deleting your "buildmaster" cookies, restarting the browser, check that your cert is loaded in the browser, and try again.
Running Projects
Uses Jenkins for several purposes
- for Continuous Integration builds. These are triggered automatically
- Github PR Tests for reacting to github commits with a build test and few short validation tests
- mu2e_muse_ci_build for reacting to a github merge by building Offline and writing it to cvmfs in a special area which can be accessed by "muse setup HEAD".
- validation.
- mu2e-offline-geant-int builds every night to check that the future version of geant is still integrated
- mu2e_muse_test checks many aspects of Muse functionality - run by hand when the head of the Muse repo needs testing
- building code for UPS packages. When a new tag is available these can be run by choosing "run with parameters," entering the tag and options in the project fields and building. The result is a link to a tarball which can be fetched and installed on cvmfs.
- mu2e-ADC-build artdaq_code
- mu2e-ADCM-build artdaq_core_mu2e
- mu2e-BTrk-build
- mu2e-KinKal-build
- mu2e-pcie-build
- mu2e-scons-build
- mu2e-TRACE-build
- building Musings which are sets of code repos built to work together.
- mu2e-musing-build see MuseMaintenance.
Most are driven by a set of scripts kept in the codetools repository and UPS product. Below is more information on a few projects.
Continuous Integration Project
The computing division has written a set of scripts to run inside Jenkins and give additions features, such as timelines, to the build steps. This is called the Continuous Integration Project. We do not use this service at this time.
Trigger by URL
Projects can be kicked off by POST'ing a url with curl.
curl -X POST "https://buildmaster.fnal.gov/buildmaster/view/mu2e/job/mu2e-BTrk-build/buildWithParameters?token=02345&PACKAGE_VERSION=v1_02_04&COMPILER=e14"
You can get the token (a kind of password) from the configure page. The parameters are also defined on the configure page.
Pushing a cert
To retrieve a file using a cert. Export your CILogon cert as a .p12 file and note the password. Then
openssl pkcs12 -in cilogon.p12 -out file.key.pem -nocerts -nodes openssl pkcs12 -in cilogon.p12 -out file.crt.pem -clcerts -nokeys curl -O -E ./file.crt.pem --key ./file.key.pem \ https://buildmaster.fnal.gov/buildmaster/view/Art/job/build-mu-dist/BUILDTYPE=debug,label1=swarm,label2=SLF6/lastSuccessfulBuild/artifact/copyBack/cetlib-3.01.03-slf6-x86_64-e15-debug.tar.bz2
Push a cert with a url that triggers a project, so the jenkins project can use it.
# I assume that X509_USER_PROXY points to an existing proxy path # in the 'mu2e' Jenkins project used in the link below you need # to define a "Password Parameter" variable named: MU2E_CREDENTIAL # on your local node: ~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~ credbits="MU2E_CREDENTIAL@$X509_USER_PROXY" val2="value2" curl -f \ --data-urlencode "token=1234" \ --data-urlencode "var1=value1" \ --data-urlencode "var2=$val2" \ --data-urlencode "$credbits" \ https://buildmaster.fnal.gov/job/mu2e/buildWithParameters ~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~ # I assume your Jenkins project is named "mu2e", replace that bit for different Jenkins projects # var1 and var2 need to be defined in the Jenkins configuration to be able to retrieve their value within a Jenkins job # In your Jenkins job script to retrieve the proxy you can do the following: ~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~ export X509_USER_PROXY=/tmp/$$.proxy rm -fv $X509_USER_PROXY touch $X509_USER_PROXY chmod 0600 $X509_USER_PROXY echo "${MU2E_CREDENTIAL}" > $X509_USER_PROXY voms-proxy-info -all ~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~
Github PR Tests
When a pull request (PR) is made or updated in the Offline or Production respositories, or certain comments are entered into an existing PR (see GitHubWorkflow#GitHub_Pull_Request_Procedures_and_FNALbuild), this triggers a series of Jenkins jobs which run tests on the PR code before it can be merged. The process for doing this is described below. Much of the code used by these jobs is described at Github_CI_Maintenance.
receive-gh-event and receive-gh-event-production
These are the jobs that receive the initial notification from GitHub.com, and launch the other jobs.
The jobs receive-gh-event and receive-gh-event-production represent the first step in the PR testing flow for the Offline and Production repositories, respectively. Their job configurations are kept in the GitHubPRTests directory on Jenkins. They are triggered by GitHub webhooks -- a POST request from GitHub to Jenkins. This is set up in the Github repository settings (more info about GitHub webhooks here). Each Jenkins job corresponds to a webhook from one repository: Offline pull requests trigger receive-gh-event, and Production pull requests trigger receive-gh-event-production. Once either of these jobs begins, it checks the values of a few environment variables. Some of these comes from the GitHub Pull Request Builder plugin. After this step, it launches the next job in the sequence, mu2e-github-bot.
mu2e-github-bot
This is the job that makes all the decisions about what tests need to get run. All the pull request tests get launched from here.
This job configuration is kept in the GitHubPRTests directory on Jenkins. It is launched by either receive-gh-event or receive-gh-event-production. It clones the latest updates from the Mu2e/CI repository, then sets up a virtual environment and installs the CI code's dependencies (listed in the file requirements.txt). It then runs the CI script process-pull-request, which in turn calls another script called process_pr. This script is described in more detail here.
These scripts do all the checks to ensure that the tests can and should be run on this PR, including but not limited to: checking that the issue really is a PR and that it's still open, and that the author of the PR is in the Mu2e organization. If the PR is not new, they check to see that either the base branch's HEAD has changed, or that a comment has been posted that requests the tests be run.
They also select which tests, if any, to run on this PR. Currently, the requirements for running tests are: 1) the build tests get run on new PR's 2) the build tests get run when someone has posted a comment to run them (see here) and there are not already tests running After these steps are done, the script posts a comment on the PR indicating that the tests were triggered. The last step in this job is to trigger the downstream job mu2e-offline-build-test or mu2e-production-build-test.
mu2e-offline-build-test
This is the job that builds the code and runs the tests for Offline PRs.
This job configuration is kept in the GitHubPRTests directory on Jenkins. It is launched by the Jenkins job mu2e-github-bot. mu2e-offline-build-test clones the Mu2e/codetools and Mu2e/CI repositories. The codetools repo contains the code to run the tests; the CI repo contains the code to communicate with GitHub (more info about the CI and codetools repos at Github_CI_Maintenance). It then launches the codetools script gh_pr_bootstrap.sh, which in turn runs the job.sh script in the appropriate test folder. This script does the appropriate cloning and merging of Offline and Production, and contains the list of test fcl files to run.
If the merge fails, it reports the failure in a comment on the PR and exits. If it succeeds, it runs build.sh (which builds the code with muse and runs the tests fcl). It reports test results to the PR as they finish, and then posts a comment with a summary of all the test results, and links to log files. To access these log files, you need to be on the Fermilab VPN or logged into one of the mu2egpvm machines.
The following repeats the last paragraph with more detail. For CI launched from a PR to the Mu2e GitHub Offline repo, gh_pr_bootstrap.sh will source bin/github/jenkins_tests/mu2e-offline-build-test/job.sh at this line: [1]
Note that it is run in sub-shell so environment modifications are contained within the sub-shell. This script will do the following:
- all of the clones and merges specified by the CI command
- the whitespace and TODO/FIXME checking
- Note that TODO/FIXME only looks for all upper case
- It sources bin/github/jenkins_tests/mu2e-offline-build-test/build.sh, again in a subshell, at this line:[2], which
- builds the code
- runs all of the tests of the form "mu2e -c xxx.fcl".
- runs the root based overlap checker.
- On return from build.sh, it runs clang-tidy.
- Because build.sh is run in a sub-shell, job.sh must spack load llvm or the UPS equivalent:
- The AL9 build machines in Jenkins have /usr/bin/clang-tidy installed.
- Therefore testing to see if clang-tidy is already defined is not a good way to tell if you need to spack load llvm.
- THe system clang-tidy is 17.0.6 while our llvm clang-tidy is older, 14.0.6
- It sends the final report to GitHub.
mu2e-production-build-test
This job configuration is kept in the GitHubPRTests directory on Jenkins. It does essentially the same things as mu2e-offline-build-test, but is launched when the initial PR is in Production. The code run by this job is essentially identical to the Offline build tests, but is a separate set of files in another directory to allow it to be changed independently of the Offline tests if necessary.