Jenkins

Introduction

Jenkins is a freeware java code build system that the Scientific Computing Division is maintaining for the experiments. It runs on several powerful build machines where the code can be built using significant parallelism. It also has many useful management features. You can browse the Jenkins system. and docs for code managers. Jenkins manages and runs build scripts, triggers, archiving and reports, but the actual compilation with scons is black box to Jenkins.

Anyone can view the Jenkins pages, but to edit the pages you need to put in a ticket to get edit permissions in Jenkins, then authorize yourself to the site with a certificate. To edit, the Jenkins page should have your username at the top right. If it says "anonymous", try deleting your "buildmaster" cookies, restarting the browser, check that your cert is loaded in the browser, and try again.

Running Projects

mu2e has these Jenkins projects:

1. mu2e-offline-nightly builds every night and checks several jobs, geant volume overlaps, and validation. It produces a report which is mailed to a few people and archived here.
2. mu2e-offline-build for building a tag of the offline code and producing tarballs suitable for moving the build to a code disk.
3. mu2e-offline-ci runs every hour and if there was a recent git commit, then run a build which the developers can check for errors.
4. mu2e-offline-geant-int builds every night to check that the future version of geant is still integrated

Most are driven by a set of scripts kept in the codetools repository and UPS product.

Nightly build

The mu2e-offline-nightly project checks out and builds the head every night at about 3AM. It then runs a series of checks:

• 10 standard fcl jobs and checks that they have a good return code
• a geometry overlap check
• generates 5,000 events in genReco and produces a set of validation histograms
• copies in a standard file and runs CutAndCount on it

The logs and histogram files are saved. The histograms for validation and CutAndCount are compared to the previous day's histograms and report is made. The results are posted on a web page and mailed to a few people. This report step is run in the mu2epro account in $HOME/cron/val.

Release build

The release build is triggered by hand when a new tag is ready.

1. log onto Jenkins system. To do this procedure, you will need to be identified to Jenkins by a cert loaded in your browser before you visit the site. The Jenkins page should have your username at the top right. If it says "anonymous", you are not authenticated.
2. browse to mu2e tab then mu2e-offline-build project
3. click on "configure," go the bottom, in "execute shell" box. Edit the tag name like:

   export MU2E_RELEASE_TAG=v5_2_1

   or branch name like

   export MU2E_BRANCH=V5_2_1_test_branch

   at the top of the script. Click "save" at the bottom of this page. You should be on the project top page.

4. Click "build now". It will take up to 1.5 hours if there are lots of other builds running on the build machines. When the status balls stop pulsing, it is done.
5. Check logs by clicking on one of the status balls, then "last build" then "console output".
6. Pull the releases to the code disk using CVMFS instructions
7. If the tarball is needed for remote distribution, that can be added to the procedure, or it can be retrieved by hand from Jenkins.

Integration build

Once an hour, jenkin will check if there has been a new commit to our main git repository and if there was, it will run the mu2e-offline-ci project. This project builds prof and debug from the head. If the build fails, mail will be sent to the person who made the commit. You can also check the results by browsing the Jenkins system. Go to the "mu2e" tab, the "mu2e-offline-ci" project. If the balls are pulsing, the build is still running. If green, the build was clean. If red, then the build failed. Click on one of the builds, then "last build", then "console output" to see the log file.

Continuous Integration Project

The computing division has written a set of scripts to run inside Jenkins and give additions features, such as timelines, to the build steps. This is called the Continuous Integration Project

Trigger by URL

Projects can be kicked off by POST'ing a url with curl.

curl  -X POST "https://buildmaster.fnal.gov//view/mu2e/job/mu2e-BTrk-build/buildWithParameters?token=02345&PACKAGE_VERSION=v1_02_04&COMPILER=e14"

You can get the token (a kind of password) from the configure page. The parameters are also defined on the configure page.

Pushing a cert

To retrieve a file using a cert. Export your CILogin cert as a .p12 file and note the password. Then

openssl pkcs12 -in cilogin.p12 -out file.key.pem -nocerts -nodes
openssl pkcs12 -in cilogin.p12 -out file.crt.pem -clcerts -nokeys    
curl -O -E ./file.crt.pem --key ./file.key.pem \
https://buildmaster.fnal.gov/view/Art/job/build-mu-dist/BUILDTYPE=debug,label1=swarm,label2=SLF6/lastSuccessfulBuild/artifact/copyBack/cetlib-3.01.03-slf6-x86_64-e15-debug.tar.bz2

Push a cert with a url that triggers a project, so the jenkins project can use it.

# I assume that X509_USER_PROXY points to an existing proxy path
# in the 'mu2e' Jenkins project used in the link below you need
# to define a "Password Parameter" variable named: MU2E_CREDENTIAL


# on your local node:

~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~
credbits="MU2E_CREDENTIAL@$X509_USER_PROXY"

val2="value2"

curl -f \
--data-urlencode "token=1234" \
--data-urlencode "var1=value1" \
--data-urlencode "var2=$val2" \
--data-urlencode "$credbits" \
https://buildmaster.fnal.gov/job/mu2e/buildWithParameters
~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~

# I assume your Jenkins project is named "mu2e", replace that bit for different Jenkins projects
# var1 and var2 need to be defined in the Jenkins configuration to be able to retrieve their value
within a Jenkins job

# In your Jenkins job script to retrieve the proxy you can do the following:
~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~

export X509_USER_PROXY=/tmp/$$.proxy

rm -fv $X509_USER_PROXY
touch $X509_USER_PROXY
chmod 0600  $X509_USER_PROXY
echo "${MU2E_CREDENTIAL}" > $X509_USER_PROXY

voms-proxy-info -all

~~~8<~~~~~~~~~~~~~~~~~~~~~~~~~