Difference between revisions of "Muse"
| Line 4: | Line 4: | ||
As Mu2e approaches data-taking there will be many more ntuples, calibration procedures, and analysis code developed. Including this user code in the main Offline repo will be unwieldy and hard to maintain. The main Offline repo should remain minimal, including only what is needed for simulation and reconstruction jobs, for simple maintenance and faster builds. Code for calibration, ntuple, and analysis code, maintained and used by individuals and small groups should be stored in a smaller repos controlled by the appropriate group. These smaller repos could be standalone contain modules and utilities that depend on code and functionality in other small repos, or in Offline. To make this work, we need a system to build the smaller repos against an existing Offline build, or together with the Offline, sharing include files, fcl, and cross-linking. The system should support common use cases and be flexible. The Muse system is a set of scripts developed inside Mu2e that provides this functionality. It is a UPS product containing a set of scripts and support for scons code build at the core. | As Mu2e approaches data-taking there will be many more ntuples, calibration procedures, and analysis code developed. Including this user code in the main Offline repo will be unwieldy and hard to maintain. The main Offline repo should remain minimal, including only what is needed for simulation and reconstruction jobs, for simple maintenance and faster builds. Code for calibration, ntuple, and analysis code, maintained and used by individuals and small groups should be stored in a smaller repos controlled by the appropriate group. These smaller repos could be standalone contain modules and utilities that depend on code and functionality in other small repos, or in Offline. To make this work, we need a system to build the smaller repos against an existing Offline build, or together with the Offline, sharing include files, fcl, and cross-linking. The system should support common use cases and be flexible. The Muse system is a set of scripts developed inside Mu2e that provides this functionality. It is a UPS product containing a set of scripts and support for scons code build at the core. | ||
| − | In this page we will use Ana1 and Ana2 as examples of smaller, non-Offline code repos. WorkDir will represent the path to a user project area on /mu2e/app, or possible other disk where you can build code. AnaDir will represent an area where analysis files are kept : scripts, root scripts, ntuples, histograms. | + | In this page we will use Ana1 and Ana2 as examples of smaller, non-Offline code repos. WorkDir will represent the path to a user project area on /mu2e/app, or possible other disk where you can build code. AnaDir will represent an area where analysis files are kept : scripts, root scripts, ntuples, histograms, and notes. If the user's work is mostly concerned with code development and commits, then it may be conveneient to use WorkDir as AnaDir, or, if the works is mostly using a statis build, then they may be different. Or there may be several versions of AnaDir, or AnaDir's with different goals, all using the same WorkDir. |
| + | |||
==Quick Start== | ==Quick Start== | ||
Always required: | Always required: | ||
Revision as of 18:50, 30 March 2021
Introduction
As Mu2e approaches data-taking there will be many more ntuples, calibration procedures, and analysis code developed. Including this user code in the main Offline repo will be unwieldy and hard to maintain. The main Offline repo should remain minimal, including only what is needed for simulation and reconstruction jobs, for simple maintenance and faster builds. Code for calibration, ntuple, and analysis code, maintained and used by individuals and small groups should be stored in a smaller repos controlled by the appropriate group. These smaller repos could be standalone contain modules and utilities that depend on code and functionality in other small repos, or in Offline. To make this work, we need a system to build the smaller repos against an existing Offline build, or together with the Offline, sharing include files, fcl, and cross-linking. The system should support common use cases and be flexible. The Muse system is a set of scripts developed inside Mu2e that provides this functionality. It is a UPS product containing a set of scripts and support for scons code build at the core.
In this page we will use Ana1 and Ana2 as examples of smaller, non-Offline code repos. WorkDir will represent the path to a user project area on /mu2e/app, or possible other disk where you can build code. AnaDir will represent an area where analysis files are kept : scripts, root scripts, ntuples, histograms, and notes. If the user's work is mostly concerned with code development and commits, then it may be conveneient to use WorkDir as AnaDir, or, if the works is mostly using a statis build, then they may be different. Or there may be several versions of AnaDir, or AnaDir's with different goals, all using the same WorkDir.
Quick Start
Always required:
setup mu2e setup muse
When building on mu2egpvm01-06, use "-j 4", when building on mu2ebuild01, use "-j 20", see Machines. See GitHubWorkflow for more details on using git. See #mgit for more information on using partial checkout (building only parts of Offline).
Build Offline
Not expecting to commit files back to the Offline repo:
cd WorkDir git clone https://github.com/Mu2e/Offline or muse setup muse build -j 20
Build Offline developer
Expecting to commit files back to the Offline repo:
cd WorkDir git clone https://github.com/Mu2e/Offline
git clone git@github.com:<your GitHub username>/Offline
muse setup
muse build -j 20
Returning later
To re-setup, later, in another process
cd WorkDir muse setup
or
cd AnaDir muse setup WorkDir
Setup options
git clone https://github.com/<their GitHub user name>/Offline cd Offline
The following is a recipe for testing building Offline with the Tutorial repo, as an example.
setup mu2e
setup muse
< cd to a working dir >
git clone https://github.com/Mu2e/Offline
git clone https://github.com/Mu2e/Tutorial
muse -h
muse setup
muse build -j 20 >& b_00.log
mu2e -n 10 -c Offline/Validation/fcl/ceSimReco.fcl
mu2e -s mcs.owner.val-ceSimReco.dsconf.seq.art \
-c Tutorial/DataExploration/fcl/Ex01.fcl
< in the same working dir, but in a second window>
muse setup -q debug
muse status
muse build -j 20 >& d_00.log
mu2e -s mcs.owner.val-ceSimReco.dsconf.seq.art \
-c Tutorial/DataExploration/fcl/Ex02.fcl
An example of a backing release
setup mu2e
setup muse
< cd to a working dir that will be the backing build >
git clone https://github.com/Mu2e/Offline
muse setup
muse build -j 20 >& b_00.log
< *in a new process* cd to a working dir that will link to the backing build >
setup mu2e
setup muse
git clone https://github.com/Mu2e/Tutorial
muse link /path/backing_working_dir/Offline
muse setup
muse build -j 20 >& b_00.log
mu2e -n 10 -c Offline/Validation/fcl/ceSimReco.fcl
mu2e -s mcs.owner.val-ceSimReco.dsconf.seq.art \
-c Tutorial/DataExploration/fcl/Ex02.fcl
Example of a partial Offline build using "mgit" the muse version of pgit
setup mu2e
setup muse
cd <a working dir>
... establish a backing build
muse link
Recent published releases:
Recent CI builds
2021-03-12 21:45 master/43ebbdfe
2021-03-12 12:44 master/c2409d93
muse link master/43ebbdfe
... establish partial code checkout
include your github username to setup an origin remote (requires authentication)
mgit init rlcee
cd Offline
mgit add HelloWorld
sed -i 's/From/From '$USER'/' HelloWorld/src/HelloWorld_module.cc
git remote -v
git status
cd ..
muse setup
muse build -j 20 --mu2eCompactPrint
mu2e -c HelloWorld/test/hello.fcl
Example of jobs submission
.. in any Muse build muse tarball Tarball: /mu2e/data/users/rlc/museTarball/tmp.BqvJOks3xI/Code.tar.bz2 mu2eprodsys ... \ --code /mu2e/data/users/rlc/museTarball/tmp.BqvJOks3xI/Code.tar.bz2