Difference between revisions of "MuseMaintenance"

From Mu2eWiki
Jump to navigation Jump to search
 
(14 intermediate revisions by 3 users not shown)
Line 1: Line 1:
  
==Publishing Offline tags==
+
==Publishing Musings==
  
When Offline is tagged, it needs to be build and published on cvmfs in Muse formatA [[Jenkins|jenkins]] build project is used to build prof and debug in parallelIf the build fails, the project will return an error.  The output is tarballs that can then be installed on cvmfs.
+
[[Muse#Musings_.28published_muse_builds.29|Musings]] are various sets of [[Muse]]-ready repos built together and published on [[Cvmfs|cvmfs]]There they can then be accessed by "muse setup".
 +
This [[Jenkins]] musing build is used to build code with the [[Muse]] system, and produce a build tarball, ready to be installed in the [[Cvmfs|cvmfs]] <code>/cvmfs/mu2e.opensciencegrid.org/Musings</code> areaIn this build, Jenkins is configured to build prof and debug in parallel, and produce a tarball for each.
  
* go to [[Jenkins|jenkins]] [https://buildmaster.fnal.gov/buildmaster/view/mu2e/job/mu2e-offline-build-muse/ mu2e-offline-build-muse] project. (Requires an account and a cert in the browser.)
+
Some Musings we are currently (2022) supporting:
* select "Build with Parameters", enter the ''tag'' string, and hit "Build".  takes about 45 min.
+
* Offline - contains only a build of a tag of the Offline repo
* When done, select "BUILDTYPE=prof"
+
* SimJob - contains Production repo and has a backing link to an Offline Musing
* if anything went wrong, select "last build" and look at the console output and the log file
+
* TrkAna - contains the TrkAna repo and has a backing link to a SimJob
* under artifacts, right click on the tarball and "copy link address"
+
 
* log onto the [[Cvmfs|cvmfs]] machine, and start a transaction
+
Here is the recipe for building and installing a musing.
* cd to /cvmfs/mu2e.opensciencegrid.org/Musings/, wget the tarball, extract it (bzipped, use "-j")
+
# locally, before running on Jenkins
* remove tarball
+
## checkout the <code>Mu2e/MuseConfig</code> repo
* repeat for the debug tarball
+
## read the instructions in <code>MuseConfig/musings/README.mk</code>
* if needed, repoint the "Offline/current" link to this new tag
+
## modify the appropriate file in <code>musings</code> directory
** cd /cvmfs/mu2e.opensciencegrid.org/Musings/Offline
+
## commit and merge (no tags needed, build is from the head)
** rm current
+
# on Jenkins
** ln -s vxx_xx_xx current
+
##log onto Jenkins system. You will need to be identified to Jenkins by a cert loaded in your browser before you visit the site. The Jenkins page should have your username at the top right. If it says "anonymous", you are not authenticated.
* cd ~ and publish the transaction
+
## browse to mu2e tab then '''mu2e-musing-build''' project
* once cvmfs is updated, test with  
+
## click in "build with parameters"
muse setup ''tag''
+
## enter the musing name and tag
 +
## click "build"
 +
## check logs by clicking on one of the status balls, then "last build" then "console output".
 +
## when jenkins indicates done (solid status icons), click into the build and there will be a link to the tarball as the "build artifacts" from the build process
 +
## capture the link to the tarball with a right click
 +
# log onto the [[Cvmfs|cvmfs]] machine, and start a transaction
 +
## cd to /cvmfs/mu2e.opensciencegrid.org/Musings/, wget the tarball, and extract it in this dir
 +
## remove the tarball
 +
## repeat for the debug tarball
 +
##if needed, repoint the "Offline/current" link to this new tag
 +
### cd into the directory for the musing you are installing
 +
### rm current
 +
### ln -s tag_name current
 +
## cd ~ and publish the transaction
 +
## once cvmfs is updated, test with <code>muse setup ''tag''</code>
  
 
The script that this project runs is in the '''codetools''' repo,  
 
The script that this project runs is in the '''codetools''' repo,  
  codetools/bin/jenkinsMuseBuild.sh
+
  codetools/bin/jenkinsMusingBuild.sh
When the jenkins project is run, it checks out the head of codetools to find the script.
+
When the Jenkins project is run, it checks out the head of codetools to find the script.
 
 
==Publishing Production tags==
 
 
 
Production contains most of the top-level fcl to use Offline, so is often updated and published at the same time as Offline.  To "build" and publish a tag of Production with tag name ''tag'', in a scratch area.  '''Note'''  since Production has no default envset, you want to be sure it builds with the same envset (and other qualifiers) as the version of Offline it will be used with.  This is necessary because as muse sets up, it checks that the builds being incorporated through links all have the build being setup.
 
 
 
setup mu2e
 
setup muse
 
git clone https://github.com/Mu2e/Production
 
git -C Production checkout -b temp ''tag''
 
muse setup
 
muse build
 
muse tarball -r Production/''tag''
 
 
 
You will get an output tarball.  In a second window, in the same directory
 
setup mu2e
 
setup muse
 
muse setup -q debug
 
muse build
 
muse tarball -r Production/''tag''
 
You will get another tarball.  Even though Production doesn't produce anything in the build area, it is necessary create the build directories so the setup command doesn't give warnings about "Production wasn't built with this setup".  Muse doesn't have a concept of a repo that build with no output, yet.
 
 
 
Then on the cvmfs installation machine
 
* log onto the [[Cvmfs|cvmfs]] machine, and start a transaction
 
* cd to /cvmfs/mu2e.opensciencegrid.org/Musings/, wget the tarball, extract it (bzipped, use "-j")
 
* remove tarball
 
* repeat for the debug tarball
 
* if needed, repoint the "Production/current" link to this new tag
 
* cd ~ and publish the transaction
 
 
 
==Publishing ProdJob==
 
 
 
ProdJob is a published Muse build area that contains both Offline and Production.  It can serve as the basis for submitting production jobs.  Currently, the plan is to publish Offline and Production as separate Musings, then make a ProdJob linking to these two.  We expect that Production will be updated more frequently than Offline, and ProdJob allows for this case efficiently.
 
 
 
In a scratch area, assuming an Offline tag ''otag'', a Production tag ''ptag'' and a targeted ProdJob tag ''tag'',
 
 
 
setup mu2e
 
setup muse
 
muse link Offline ''otag''
 
muse link Production ''ptag''
 
muse setup
 
muse build
 
muse tarball -r Prodjob/''tag''
 
And, as described in previous sections, also setup, build, and tar a "debug" version in a second process. Upload the resulting tarballs to cvmfs Musings area.
 
 
 
  
 
==Publishing Muse itself==
 
==Publishing Muse itself==
 
To develop Muse, create a fork in your github account, check it out locally
 
To develop Muse, create a fork in your github account, check it out locally
  setup mu2e
+
  mu2einit
 
  git clone git@github.com:<your GitHub username>/Muse
 
  git clone git@github.com:<your GitHub username>/Muse
 
and to use this local code instead of "setup muse" do
 
and to use this local code instead of "setup muse" do
Line 87: Line 58:
 
==Testing Muse==
 
==Testing Muse==
  
Muse contains a series of tests than can be run by hand or as a [[Jenkins]] project (mu2e_muse_test).  The script can be accessed with  
+
Muse contains a series of tests than can be run by hand or as a [[Jenkins]] project ([https://buildmaster.fnal.gov/buildmaster/view/mu2e/job/mu2e_muse_test/ mu2e_muse_test]).  The script can be run locally with  
  setup muse
+
  mu2einit
 
  mu2eTest.sh -h
 
  mu2eTest.sh -h
  

Latest revision as of 22:10, 19 July 2024

Publishing Musings

Musings are various sets of Muse-ready repos built together and published on cvmfs. There they can then be accessed by "muse setup". This Jenkins musing build is used to build code with the Muse system, and produce a build tarball, ready to be installed in the cvmfs /cvmfs/mu2e.opensciencegrid.org/Musings area. In this build, Jenkins is configured to build prof and debug in parallel, and produce a tarball for each.

Some Musings we are currently (2022) supporting:

  • Offline - contains only a build of a tag of the Offline repo
  • SimJob - contains Production repo and has a backing link to an Offline Musing
  • TrkAna - contains the TrkAna repo and has a backing link to a SimJob

Here is the recipe for building and installing a musing.

  1. locally, before running on Jenkins
    1. checkout the Mu2e/MuseConfig repo
    2. read the instructions in MuseConfig/musings/README.mk
    3. modify the appropriate file in musings directory
    4. commit and merge (no tags needed, build is from the head)
  2. on Jenkins
    1. log onto Jenkins system. You will need to be identified to Jenkins by a cert loaded in your browser before you visit the site. The Jenkins page should have your username at the top right. If it says "anonymous", you are not authenticated.
    2. browse to mu2e tab then mu2e-musing-build project
    3. click in "build with parameters"
    4. enter the musing name and tag
    5. click "build"
    6. check logs by clicking on one of the status balls, then "last build" then "console output".
    7. when jenkins indicates done (solid status icons), click into the build and there will be a link to the tarball as the "build artifacts" from the build process
    8. capture the link to the tarball with a right click
  3. log onto the cvmfs machine, and start a transaction
    1. cd to /cvmfs/mu2e.opensciencegrid.org/Musings/, wget the tarball, and extract it in this dir
    2. remove the tarball
    3. repeat for the debug tarball
    4. if needed, repoint the "Offline/current" link to this new tag
      1. cd into the directory for the musing you are installing
      2. rm current
      3. ln -s tag_name current
    5. cd ~ and publish the transaction
    6. once cvmfs is updated, test with muse setup tag

The script that this project runs is in the codetools repo,

codetools/bin/jenkinsMusingBuild.sh

When the Jenkins project is run, it checks out the head of codetools to find the script.

Publishing Muse itself

To develop Muse, create a fork in your github account, check it out locally

mu2einit
git clone git@github.com:<your GitHub username>/Muse

and to use this local code instead of "setup muse" do

export MUSE_DIR=$PWD/Muse
export PATH=$PWD/Muse/bin:$PATH
export MUSE_ENVSET_DIR=$PWD/Muse/config
alias muse="source muse"

now you can add envsets and test them directly (as opposed to the muse/uNNN mechanism). All the scripts should work like a normal Muse UPS setup. Develop and commit, pull and tag changes as in any other github repo. To publish tag,

mkdir products
museInstall.sh tag ./products

the products directory will contain an UPS-installed version tag, and a tarball suitable to uploading to cvmfs artproducts area. This script does not checkout or attempt to confirm that the tag is correct, it is assuming the code is actually equivalent to the tagged code. To make the new version UPS-current, edit the version number into the file under the Muse/current.chain directory on cvmfs. You don't need a debug version because this is UPS, which has a NULL product qualifier. For some reason I started using the version number pattern vN_NN_NN, maybe following art.

Testing Muse

Muse contains a series of tests than can be run by hand or as a Jenkins project (mu2e_muse_test). The script can be run locally with

mu2einit
mu2eTest.sh -h