MuseBuildCodeTutorial: Difference between revisions

From Mu2eWiki
Jump to navigation Jump to search
No edit summary
 
(37 intermediate revisions by 3 users not shown)
Line 1: Line 1:
==Stuff==
As Mu2e approaches data-taking there will be many more ntuples, calibration procedures, and analysis code developed.  Muse is a UPS product containing a set of scripts and scons commands to enable building Offline and other Muse-ready repos together.  Currently it could be described as pre-beta, so not completely stable. 
The following is a recipe for testing building Offline with the Tutorial repo, as an example.
<pre>
setup mu2e
setup muse
  < cd to a working dir >
git clone https://github.com/Mu2e/Offline
git clone https://github.com/Mu2e/Tutorial
muse -h
muse setup
muse build -j 20 >& b_00.log
mu2e -n 10 -c Offline/Validation/fcl/ceSimReco.fcl
mu2e -s mcs.owner.val-ceSimReco.dsconf.seq.art \
    -c Tutorial/DataExploration/fcl/Ex01.fcl
  < in the same working dir, but in a second window>
muse setup -q debug
muse status
muse build -j 20 >& d_00.log
mu2e -s mcs.owner.val-ceSimReco.dsconf.seq.art \
    -c Tutorial/DataExploration/fcl/Ex02.fcl
</pre>
An example of a backing release
<pre>
setup mu2e
setup muse
  < cd to a working dir that will be the backing build >
git clone https://github.com/Mu2e/Offline
muse setup
muse build -j 20 >& b_00.log
  < *in a new process* cd to a working dir that will link to the backing build >
setup mu2e
setup muse
git clone https://github.com/Mu2e/Tutorial
muse link /path/backing_working_dir/Offline
muse setup
muse build -j 20 >& b_00.log
mu2e -n 10 -c Offline/Validation/fcl/ceSimReco.fcl
mu2e -s mcs.owner.val-ceSimReco.dsconf.seq.art \
    -c Tutorial/DataExploration/fcl/Ex02.fcl
</pre>
Example of a partial Offline build using "mgit" the muse version of pgit
<pre>
setup mu2e
setup muse
cd <a working dir>
  ... establish a backing build
muse link
Recent published releases:
Recent CI builds
2021-03-12 21:45 master/43ebbdfe
2021-03-12 12:44 master/c2409d93
muse link master/43ebbdfe
  ... establish partial code checkout
        include your github username to setup an origin remote (requires authentication)
mgit init rlcee
cd Offline
mgit add HelloWorld
sed -i 's/From/From '$USER'/' HelloWorld/src/HelloWorld_module.cc
git remote -v
git status
cd ..
muse setup
muse build -j 20 --mu2eCompactPrint
mu2e -c HelloWorld/test/hello.fcl
</pre>
Example of jobs submission
<pre>
  .. in any Muse build
muse tarball
Tarball: /mu2e/data/users/rlc/museTarball/tmp.BqvJOks3xI/Code.tar.bz2
mu2eprodsys ... \
  --code /mu2e/data/users/rlc/museTarball/tmp.BqvJOks3xI/Code.tar.bz2
</pre>
[[Category:Computing]]
[[Category:Code]]
[[Category:CodeManagement]]




==Introduction==
==Introduction==


In the case of [[DetectorIntro|Mu2e]], an [[ComputingStart|event]] records the detector response during one microbunch, consisting of a short burst of millions of protons hitting the production target, thousands of muons stopping in the aluminum stopping target, and those muons interacting or decaying.  This happens nominally every 1695 ns and each of these burst forms a Mu2e event.
In the case of [[DetectorIntro|Mu2e]], an [[Computing Concepts#art Events|event]] records the detector response during one microbunch, consisting of a short burst of millions of protons hitting the production target, thousands of muons stopping in the aluminum stopping target, and those muons interacting or decaying.  This happens nominally every 1695 ns and each of these burst forms a Mu2e event.


When we read and process events to simulate the detector, reconstruct raw data, or analyze data events we will need a framework to organize our executables, and way to control the behavior of the executable, and a format to write the events into.  These functions are provided by the ''art''' software package.
When we read and process events to simulate the detector, reconstruct raw data, or analyze data events we will need a framework to organize our executables, and way to control the behavior of the executable, and a format to write the events into.  These functions are provided by the ''art'' software package.


When we want to operate on the data, we write c++ code.  We track code files using a code management tool called '''git''', which writes files to a commercial site, [http://www.github.com github].  We have a main repository (or '''repo''') called '''Offline''' and other smaller special-purpose repos.  We compile and link the code using a set of Mu2e scripts called [[Muse]].  Inside those scripts, we use a code build system called '''scons'''.  Finally, inside scons, we use linux '''gcc''' to compile and link the code.
When we want to operate on the data, we write c++ code.  We track code source files using a code management tool called '''git''', which writes files to a commercial site, [http://www.github.com github].  We have a main repository (or '''repo''') called '''Offline''' and other smaller special-purpose repos.  We compile and link the code using a set of Mu2e scripts called [[Muse]].  Inside those scripts, we use a code build system called '''scons'''.  Finally, inside scons, we use linux '''gcc''' to compile and link the code.


==Art==
==Art==
Line 132: Line 27:


==Code==
==Code==
Please be sure your environment is prepared by consulting [[CodeEnvironment]].


We will start with a tour of the Mu2e code base.  We keep our code organized using a piece of software called '''git'''.  This freeware package stores our code in a compact format and keep track of lots of useful things:
We will start with a tour of the Mu2e code base.  We keep our code organized using a piece of software called '''git'''.  This freeware package stores our code in a compact format and keep track of lots of useful things:
Line 138: Line 34:
* tracking parallel development by many people, and merging all that work back together when ready.
* tracking parallel development by many people, and merging all that work back together when ready.


So lets look at what is stored in our git repository. If you are in the tutorial docker container,
So lets look at what is stored in our git repository. If you are on the interactive central servers (mu2egpvm or mu2ebuild)
mkdir -p /home/tutorial_code
cd /home/tutorial_code
source /setupmu2e-art.sh
or if you are on the interactive central servers (mu2egpvm)
  mkdir -p /mu2e/app/users/$USER/tutorial_code
  mkdir -p /mu2e/app/users/$USER/tutorial_code
  cd mu2e/app/users/$USER/tutorial_code
  cd /mu2e/app/users/$USER/tutorial_code
  source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh
  mu2einit


Since we will be doing several builds, create some subdirectories:
Since we will be doing several builds, create some subdirectories:
Line 159: Line 51:
Here are few git commands to get you started.
Here are few git commands to get you started.
  git status
  git status
tells you which branch you are on.  When multiple people are working on code development, they will probably be working on their own branches - a version of the code they can control.  When the changes are tested and approved, the personal, working branch is merged back into the main code branch, called "master". See the active branches:
tells you which branch you are on.  When multiple people are working on code development, they will probably be working on their own branches - a version of the code they can control.  When the changes are tested and approved, the personal, working branch is merged back into the main code branch, called "main". See the active branches:
  git branch -al
  git branch -al
You can switch to a different active working branch:
You can switch to a different active working branch:
Line 207: Line 99:
event.put(std::move(chCol));
event.put(std::move(chCol));
This adds a newly created data product to the event.  All of these concepts will be covered in more detail later.
This adds a newly created data product to the event.  All of these concepts will be covered in more detail later.
Now cd back out of the Offline directory
cd..


==Configuring and Building Code==
==Configuring and Building Code==


Code management can be complex and so tools are developed to do most of the work, and simplify the user's commands.  The art package, scons, Muse, and gcc are all provides in a system written and maintained by the lab computing division, called [[UPS]].  At this point you have already told the UPS system to make certain packages and tools, available in your path.  For example, if you ask ups to show you what is setup:
You should now be in the directory tutorial_code/off, which contains the Offline subdirectory.
 
Code management can be complex and so tools are developed to do most of the work, and simplify the user's commands.  The art package, scons, Muse, and gcc are all provided in a system written and maintained by the lab computing division, in a system called [[UPS]].  At this point you have already told the UPS system to make certain packages and tools available in your path.  For example, if you ask UPS to show you what it has added to your interactive process:
  ups active
  ups active
and you should get an answer
and you should get an answer similar to this
  Active ups products:
  Active ups products:
  git              v2_30_1        -f Linux64bit+3.10-2.17                    -z /cvmfs/mu2e.opensciencegrid.org/artexternals
  git              v2_30_1        -f Linux64bit+3.10-2.17                    -z /cvmfs/mu2e.opensciencegrid.org/artexternals
  ups              v6_0_8          -f Linux64bit+3.10-2.17                    -z /cvmfs/mu2e.opensciencegrid.org/artexternals
  ups              v6_0_8          -f Linux64bit+3.10-2.17                    -z /cvmfs/mu2e.opensciencegrid.org/artexternals
So git is one of the tools you have already put in your path (when you sourced setupmu2e-art.sh). Now ask what versions of the Muse package are available:
  muse              v2_07_00        -f NULL                                    -z /cvmfs/mu2e.opensciencegrid.org/artexternals
ups list -aK+ muse
You should see many versions available.  One should be marked "current" - this is the version you get when you don't specify a version.  To tell UPS to add Muse tools to the path, we use the UPS "setup" command
setup muse
If you nw ask what's active ("ups active") you should see there is a version of Muse there. You can see that the tools are added by looking for the main muse command:
which muse
which gives a location of the tool like:
/cvmfs/mu2e.opensciencegrid.org/artexternals/muse/v1_10_0/bin/muse


We will use UPS to get art tools and libraries, and much more, added to our paths, but first we need to discuss


We will use UPS to get art tools and libraries, and much more, added to our paths, but first we need to discuss a few choices.  We can build code for speed, we call this option "prof", or we can tell the compiler to keep the symbols organized, which runs slower, but is useful for debugging, so we call this second option "debug". There are other options for details of how to run simulations, or visualization.  The build defaults to "prof", so we will not use any switches, and ask Muse to to use UPS to add art to our paths:
  muse setup
Note that "setup muse" is causing UPS to add muse tools to our path, but "muse setup" is asking Muse scripts to add art and everything else to your paths.  To prepare for a debug build we would have used "muse setup -q debug".  You can only run "muse setup" once in a process so to change anything, you have to start new process and setup with your alternative choices. Try these commands to see where you are now:
ups active
muse status
muse -v status


The basic configuration of the package's versions and build options is now done.


In a typical use pattern, you would modify some of the code, or add code, then you would want to compile it so you can run it.  This is the job of the build system. In our case we use a set of scripts called [[Muse]], which call a system named [[Scons|scons]], which in turn calls gcc to compile code. The build system has to look over the code, decide what needs to be compiled, do the compilation according to a recipe, then see what needs to be linked and do that linking.  The first time you run a build, it will compile and link everything.  After that, it should only do the steps which are effected by how you changed or added to the code.
In a typical use pattern, you would modify some of the code, or add code, then you would want to compile it so you can run it.  This is the job of the build system. In our case, we continue to use the set of scripts called [[Muse]], which call a package named [[Scons|scons]], which in turn calls gcc to compile code. The build system has to look over the code, decide what needs to be compiled, do the compilation according to a recipe, then see what needs to be linked and do that linking.  The first time you run a build, it will compile and link everything.  After that, it should only do the steps which are effected by how you changed or added to the code.


scons action is driven off the <code>SConstruct</code> file, which is written in python and tells scons how to do its job.  You can take a look at it and see lots of stuff related to code building.  One of the things it does is to look for <code>SConscript</code> files in the directory tree, such as
cat TrkHitReco/src/SConscript
these files, mostly in src subdirectories, can customize the scons behavior for operating on the code in the directory where it sits.


When you build the Mu2e code, you have some options. Your primary choice is whether to build prof (optimized) or debug (not optimized - easier to debug).  You can see your options with
Now it is finally time to build:
./buildopts
muse build -j 4
You can switch between prof and debug with
the "-j 4" starts the build running with 4 threads, which is appropriate if you are logged on a laptop or a mu2egpvm machineFor mu2ebuild02, where you would usually build code, you can use "-j 32" since it has many more cores.
./buildopts --build=debug
or the reverse.  The other options are beyond the scope of this page. Once you start building you have to stick with your choices until you start over (delete all the built files and setup in a new process). There is a help.
  ./buildopts --help




There are many software packages written by non-Mu2e parties (like art and root).  We need to provide the build system pointers to those packages. We do this by
Once it starts compiling, you might as well kill it with ''ctrl-c'', since it will take long time (4 min on mu2ebuild02) to run.  
source setup.sh
You can see everything you now point to:
ups active
observe your environment now points to particular version of art, root and many other packages.  (You have to finish with any buildopts choices before running setup.sh, since the latter locks in the choices)


'''ups''' is a Fermilab [[UPS|software package]] that allows you to organize these third party packages and their versions, operating system flavors and inter-dependencies.  You can see your OS flavor:
If you log off and want to return to work here, you would
ups flavor
  mu2einit
or see what versions of a package, like root, are available:
  cd mu2e/app/users/$USER/tutorial_code/off
ups list -aK+ root
  muse setup
 
and you are ready to build and runcd'ing to the working directory first is important, so Muse knows this is where your code is.
 
 
Once you're done with buildopts and setups, you can build:
<pre>
> scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o Analyses/src/BkgRates_module.os -c
...
</pre>
once it starts compiling, you might as well kill it with ''ctrl-c'', since it will take long time to run.  In proactice you will want to build on mnode mu2ebuild01 which has 16 cores and you can build it with 20 threads.
  scons -j 20
 
You can now remove any new files that were written by the build.
scons -c
This leaves all the source code alone.  It also doesn't effect the setup choices you made with buildopts and setup.sh, those are still in effect and fixed in your process.


==Releases==
==Releases==


At appropriate times, we mark the code with a tag such as "v7_4_1".  This marks a state of the code we want to save.  Maybe we are recording the state of code at a point where we make a production run for simulation, or maybe we are marking major changes in the code. Take look at the [[ReleaseList| release list]] and the [[OfflineNotesV7#v7_4_1 | notes]] for a recent release.
At appropriate times, we mark the code with a tag such as "v09_12_00".  This marks a state of the code we want to save.  Maybe we are recording the state at a point where we make a production run for simulation, or maybe we are marking major changes in the code. Take look at the [https://github.com/Mu2e/Offline/releases release list] on github.


After tagging, we build the code and save it so anyone can use it - this is "release".
After tagging, we build the code and save it so anyone can use it, without building it themselves, - this is a "release".


If you are in the tutorial container,
Starting a new process (not the one from the last exercise) make a new test area:
  ls -l /Offline
  mkdir -p /mu2e/app/users/$USER/tutorial_code/two
If you are on a mu2egpvm machines:
cd mu2e/app/users/$USER/tutorial_code/two
  ls -l /cvmfs/mu2e.opensciencegrid.org/Offline | tail
mu2einit
Now list what releases are available
  muse backing
You will see two sets of Offline builds, one labeled "published releases" and one labeled "CI builds". We will use the former. You should see some release versions. To select a version, and link it to this area:
  muse backing Offline v10_05_00
you will now see a subdirectory created, with a link to the tagged release code.


If you wanted to setup this fixed release. 
Now to up the game a bit, lets add a small repo to this area:
  source /setupmu2e-art.sh
  git clone https://github.com/Mu2e/Tutorial
source /Offline/v7_4_1/SLF6/prof/Offline/setup.sh
This repo is some example code, but in practice you can add repos providing ntuples, other tools, or analysisNow that this code directory is organized, we can setup and build it.
If you are on a mu2egpvm machines:
  muse setup
source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh
muse build -j 4
  source /cvmfs/mu2e.opensciencegrid.org/Offline/v7_4_1/SLF6/prof/Offline/setup.sh
If successful, you see that a subdirectory "build" appeared.  You can explore this director to find object files and shared-object libraries.  You path will include these build products, so you could run them.
Since we setup the code above we can't also setup these versions of the codeIn order to do that, we need to start a new process, so either start a new tutorial container, or login into the mu2egpvm machines again.


==Partial Builds==
==Partial Builds==


When we started building the code in the exercise above, we killed it because it would take too long.  On a single node it will take an 45 min, and on mu2ebuild01 with its 16 cores, it will take 10minThis is a common problem, so we have developed two ways to build part of the code.  Note that this is for a full build form scratch - following incremental builds are usually much faster.
When we started building the code in the exercise above, we killed it because it would take too long.  On a mu2egpvm node it will take an 45 min, and on mu2ebuild02 with its 32 cores, it will take 4 min(Note that these times are for a full build from scratch - any following incremental builds are usually much faster.) These long times are a common problem, so we have developed a way to build part of the code.   


In both methods we need a fully built "base release" that already exists, such as the ones we just looked at.  We then make a local directory with just a small piece of the code.  We build this small piece, and then take all the rest of the compiled code from the "base release".  Because of this, we almost
We need a fully built "base release" that already exists, such as the ones we just looked at.  We then make a local directory with just a small piece of the code.  We build this small piece, and then take all the rest of the compiled code from the "base release".   


In both methods you must be aware of the big possible pitfall.  If any header file that is compiled into your local small piece of code is different than the same header file compiled into the base release, then the code will have corrupt memory and is likely to fail in unpredictable ways.   
'''Warning''' In this method you must be aware of a big possible pitfall.  If any header file that is compiled into your local small piece of code is different than the same header file compiled into the base release, then the code will have corrupt memory and is likely to fail horribly in unpredictable ways.  It is best to use this partial build technique only when building a small part of the code base, and only changing cc files.  It works great to do simple things like add a debugging print statement, or test a new module, or modify some fcl.  If you have to modify header files, or do extensive code development, it is best to clone the whole Offline repo.


The first method is called [[SatelliteRelease|Satellite]] releases. You can create one now.  You will need to setup the Offline code in different way, so you have to get away from your current setup.  If you are in the container, exit and restart. If you are on mu2egpvm, start a new process.
We will need to change the build directory organization, so we will have to start a new process.
mkdir -p /mu2e/app/users/$USER/tutorial_code/part
cd mu2e/app/users/$USER/tutorial_code/part
mu2einit
muse link


Then, if you are in the container,
Now organize the code in this area.  First, take a look at the output of "muse link".  Recall there were two sections, "Recent published Offline releases" which we used in the previous exercise, and the second list, the "Recent Offline CI builds""CI" stand for continuous integration, which means every time the Offline repo changes, we build it and add it to this list. So you have a complete build of the most recent Offline code at your fingertipsYou'll see the top line is the most recent, now cut and past that name to tell muse to use it as a backing build:
source /setupmu2e-art.sh
  muse link master/be905d45
  mkdir -p /home/sat
Of course the latest you see will be different from this example. The hex code is a "git hash" and labels one exact state of the Offline repo. Now we have indicated to Muse to use this build for Offline libraries.
cd /home/sat
/Offline/v7_4_1/SLF6/prof/Offline/bin/createSatelliteRelease
  cd Satellite
and if you are on mu2egpvm
source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh
  mkdir -p /mu2e/app/users/$USER/sat
  cd /mu2e/app/users/$USER/sat
  /cvmfs/mu2e.opensciencegrid.org/Offline/v7_4_1/SLF6/prof/Offline/bin/createSatelliteRelease
cd Satellite


You made your buildopts choices (prof/debug) when you chose which directory that createSatelliteRelease script is in. Take a look at what is in this directory.  You should see a link to the SConstruct file in the base release and special setup.sh file.  You can now
Now we will introduce a small piece of Offline code. This piece of code is compiled and provided in the backing build we just linked, but if we also build it locally, Muse will prefer to use the local version, and take all the rest of Offline from the backing build.
source setup.sh


Lets add some code:
To start the local subset of Offline code:
  cp -r $MU2E_BASE_RELEASE/HelloWorld .
  mgit init
and you can build it - it is small
You should see an "Offline" directory appear. Now lets add the HellowWorld subdirectory from Offline to this local area
  scons
cd Offline
When scons completes successfully the output ends with the line
mgit add HelloWorld
<pre>
  ls -l
scons: done building targets.
  cd ..
</pre>
You should see that the HelloWorld subdirectory of Offline has been added locally. Now that the code is organized, tell Muse to finish setting up the paths:
You can see your new libraries in lib, for example
  muse setup
  ls -l lib/libmu2e_HelloWorld_HelloProducer_module.so
and build
which is now in the path before, and therefore replaces, the same in the base release:
  muse build -j 4
  ls -l $MU2E_BASE_RELEASE/lib/libmu2e_HelloWorld_HelloProducer_module.so
 
You can modify code and recompile - remember the big pitfall!  This method has best used for project that has small scope and you know you are not going to commit back to the main git repo.
 
The second method of compiling a small part of the code is called [[GitPartialCheckout|pgit]].  This also gives you a local directory with a small part of the code you can compile quickly, and it is also backed by a fully pre-build base release. It also has the same potential big pitfall - losing sync of header files. The advantage of pgit is that you still have access to the git repo when you are working on the local code and you can pull and push code back to the main git repo while you are working in the local area.  We are in the process of commissioning this method and establishing it can fully replace satellite releases and is not confusing to users.  We will not work an example here, but you can [[GitPartialCheckout|try it]] if you want.  At the moment, it will not work in the tutorial container.


== Exploring The Partial Build==
== Exploring The Partial Build==
Line 345: Line 207:
         << endl;
         << endl;
</pre>
</pre>
The Mu2e wiki has a short writeup about [[ComputingStart#EventIDs | EventIDs]]
The Mu2e wiki has a short writeup about [[Computing Concepts#art, EventIDs, Runs and Subruns| EventIDs]].
<li> Run the HelloWorld module:
<li> Run the HelloWorld module:
<pre>
<pre>
   mu2e -c HelloWorld/test/hello.fcl  
   mu2e -c Offline/HelloWorld/test/hello.fcl  
</pre>
</pre>
Identify the printout from the module:
Identify the printout from the module:
Line 356: Line 218:
Hello, world.  From analyze: run: 1 subRun: 0 event: 3
Hello, world.  From analyze: run: 1 subRun: 0 event: 3
</pre>
</pre>
This will also print two warning messages:
This will also print some warning messages, which will be fixed.
<pre>
INFO: using default process_name of "DUMMY".
%MSG-e duplicateDestination:


============================================================================
    Duplicate name for a MessageLogger Statistics destination: "cout"
    Only original configuration instructions are used.
============================================================================
%MSG
</pre>
Ignore them.  We will fix them.
The next tutorial has a lot of information about using fcl files so this tutorial will not discuss them much.
The next tutorial has a lot of information about using fcl files so this tutorial will not discuss them much.
<li> You can also run a larger number of events:
<li> You can also run a larger number of events:
Line 376: Line 225:
   mu2e -c HelloWorld/test/hello.fcl -n 20
   mu2e -c HelloWorld/test/hello.fcl -n 20
</pre>
</pre>
<li> Edit the file HelloWorld/src/HelloWorld_module.cc and change the printout.  Then rebuild it:
<li> Edit the file Offline/HelloWorld/src/HelloWorld_module.cc and change the printout.  Then rebuild it:
<pre>
<pre>
scons
muse build
mu2e -c HelloWorld/test/hello.fcl
mu2e -c Offline/HelloWorld/test/hello.fcl
</pre>
</pre>
Observe that the printout made by the module has changed.
Observe that the printout made by the module has changed.
Line 386: Line 235:
<li> Make a copy of HelloWorld/src/HelloWorld_module.cc  
<li> Make a copy of HelloWorld/src/HelloWorld_module.cc  
<pre>
<pre>
cp HelloWorld/src/HelloWorld_module.cc HelloWorld/src/Bonjour_module.cc
cp Offline/HelloWorld/src/HelloWorld_module.cc Offline/HelloWorld/src/Bonjour_module.cc
cp HelloWorld/test/hello.fcl HelloWorld/test/bonjour.fcl
cp Offline/HelloWorld/test/hello.fcl Offline/HelloWorld/test/bonjour.fcl
</pre>
</pre>
<li>
<li>
Line 401: Line 250:
<li> Build and run the new module
<li> Build and run the new module
<pre>
<pre>
scons
muse build
mu2e -c HelloWorld/test/bonjour.fcl
mu2e -c Offline/HelloWorld/test/bonjour.fcl
</pre>
</pre>
Observe the changed printout.  If you have questions about why you did not also change the module label, you can re-read the wiki page on [[Modules]].
Observe the changed printout.  If you have questions about why you did not also change the module label, you can re-read the wiki page on [[Modules]].
Line 408: Line 257:
<li> Make a new .fcl file that runs both modules in one job:
<li> Make a new .fcl file that runs both modules in one job:
<pre>
<pre>
cp HelloWorld/test/hello.fcl HelloWorld/test/both.fcl
cp Offline/HelloWorld/test/hello.fcl Offline/HelloWorld/test/both.fcl
</pre>
</pre>
Edit both.fcl so it looks like:
Edit both.fcl so it looks like:
Line 429: Line 278:
</pre>
</pre>
Note that you do not need to rebuild the libraries because you did not change the source code.
Note that you do not need to rebuild the libraries because you did not change the source code.
<li> Rebuild the code even though there is nothing to do.  You should see the following;
<li> Rebuild the code (muse build) even though there is nothing to do.  You should see the following;
<pre>
<pre>
scons: Reading SConscript files ...
scons: Reading SConscript files ...
Line 439: Line 288:
<li> In both.fcl, swap the order of "hi" and "hello" in the definition of "e1".  Rerun.  Notice that the order of the printout does not change!  ''art'' reserves the right to run analyzer modules in an arbitrary order and we should never write code that depends on a particular order.  ''art'' does respect user specified order for producer and filter modules.
<li> In both.fcl, swap the order of "hi" and "hello" in the definition of "e1".  Rerun.  Notice that the order of the printout does not change!  ''art'' reserves the right to run analyzer modules in an arbitrary order and we should never write code that depends on a particular order.  ''art'' does respect user specified order for producer and filter modules.
</ol>
</ol>
You can also tell muse to make a tarball of your build area:
muse tarball
this is useful for when you want to run on the grid to generate or process large datasets using your the code in your build area.


[[Category:Computing]]
[[Category:Computing]]
[[Category:Tutorial]]
[[Category:Tutorial]]

Latest revision as of 21:26, 19 July 2024


Introduction

In the case of Mu2e, an event records the detector response during one microbunch, consisting of a short burst of millions of protons hitting the production target, thousands of muons stopping in the aluminum stopping target, and those muons interacting or decaying. This happens nominally every 1695 ns and each of these burst forms a Mu2e event.

When we read and process events to simulate the detector, reconstruct raw data, or analyze data events we will need a framework to organize our executables, and way to control the behavior of the executable, and a format to write the events into. These functions are provided by the art software package.

When we want to operate on the data, we write c++ code. We track code source files using a code management tool called git, which writes files to a commercial site, github. We have a main repository (or repo) called Offline and other smaller special-purpose repos. We compile and link the code using a set of Mu2e scripts called Muse. Inside those scripts, we use a code build system called scons. Finally, inside scons, we use linux gcc to compile and link the code.

Art

The art package provides:

  • a framework for creating executables
  • the event format (we define the contents)
  • a control language called "fhicl" or "fcl" (pronounced fickle)
  • services such as random numbers

When we write code, typically to simulate the detector, reconstruct the data, or analyze the data, we write our code in c++ modules. We then run the framework, and tell it to call the code in our module. We also tell it what input files to use, if any. The framework opens the input files, reads data, priovides services, calls our modules with the data, event by event, and then handles any filtering of events, and writing output.

When we run executables in the art framework, we have to tell the framework which modules to run and how to configure the modules. A module configuration might include how to find the input data inside the file, and what parameters and cuts to use in performing its task. This function is provided by a fcl file - you always provide a fcl file when you start up an art executable.

art executables read and write files in the art format. This is a highly-structured format which contains the event data, processing history, and other information. Under the covers, art uses the root package (used widely in HEP) to read and write the event files. Each event contains pieces which are referred to as products. For example, in each event, there may be a product which contains raw hits, one that contains reconstructed hits, and one that contains reconstructed tracks. In simulation, there are products with truth information.

All of these concepts will be fleshed out in subsequent tutorials.


Code

Please be sure your environment is prepared by consulting CodeEnvironment.

We will start with a tour of the Mu2e code base. We keep our code organized using a piece of software called git. This freeware package stores our code in a compact format and keep track of lots of useful things:

  • all the history of each file as it is developed
  • coherent versions of the whole set of code (tags)
  • tracking parallel development by many people, and merging all that work back together when ready.

So lets look at what is stored in our git repository. If you are on the interactive central servers (mu2egpvm or mu2ebuild)

mkdir -p /mu2e/app/users/$USER/tutorial_code
cd /mu2e/app/users/$USER/tutorial_code
mu2einit

Since we will be doing several builds, create some subdirectories:

 mkdir off
 cd off

Retrieve the code base from the git repository:

git clone https://github.com/Mu2e/Offline

after a few seconds you should see a directory Offline. Go into that dir

cd Offline

and look around. Note the directory .git. This is where git keeps information such code history and tags. Most of the directories you see here contain our working code. Each directory contains a relatively small, logically-grouped set of code.

Here are few git commands to get you started.

git status

tells you which branch you are on. When multiple people are working on code development, they will probably be working on their own branches - a version of the code they can control. When the changes are tested and approved, the personal, working branch is merged back into the main code branch, called "main". See the active branches:

git branch -al

You can switch to a different active working branch:

git checkout MDC2018
git status

or to a fixed release:

git checkout v09_12_00

all that output text is saying this fixed point of the code and you shouldn't be trying to modify it, which is fine, makes sense.

git status

see the files which are different from the previous release:

git diff --numstat v09_11_00 | head

and then see the code that changed in that first file listed:

git diff v09_11_00 Analyses/src/CaloCalib_module.cc

and then the file history:

git log v09_11_00 Analyses/src/CaloCalib_module.cc

In the git status you should also see "working branch clean" this means your area is still the same as the central repository you just cloned. If there were new or changed files, it would tell you. You can read much more about git and how we use it, including making your own branches and committing code modifications back to the central repo.

Let's look at the code structure. For example, under

ls -l TrkHitReco

you can see directories: fcl, inc, src, test and data. Most of the top-level directories follow this pattern. The c++ includes related to the code are kept under inc, the c++ source code in src, the fcl scripts that configure modules are in fcl, and sometimes scripts are under test or text files used by the code are kept under data. Many directories don't need test or data - see the TrkReco directory for example. Take a look at what in these directories.

Recall that we write modules, which are then run by the framework to act on the event data. Any module we write is compiled into a shared object library which is loaded by the framework if our fcl tells it to. For example,

TrkHitReco/src/StrawHitReco_module.cc

would be compiled into

../build/<some string>/Offline/lib/libmu2e_TrkHitReco_StrawHitReco_module.so

which we can then ask the framework to load and run (or not). The shared object is not there, because we haven't done the compiling, which we will get to in a minute.

Just as a quick peak, open the cc file:

emacs (or vi, cat) TrkHitReco/src/StrawHitReco_module.cc

Find the line

class StrawHitReco : public art::EDProducer

Since the module code inherits from a framework base class (EDproducer), the framework can manipulate it.

Find the line

StrawHitReco::beginJob

this method is called once at the beginning of the job. Similarly

StrawHitReco::beginRun

is called when the input events change run number.

StrawHitReco::produce

is called for every event. It is called "produce" because it typically produces data to insert into the event, in this case, the reconstructed tracker hits. Other option are "filter" if this module can select a subset of events to write out, and "analyze" if the module is not writing any new data to the event.

Find the line

event.getValidHandle(_sdtoken)

this is retrieving the raw hits from the event. Find the line event.put(std::move(chCol)); This adds a newly created data product to the event. All of these concepts will be covered in more detail later.

Now cd back out of the Offline directory

cd..

Configuring and Building Code

You should now be in the directory tutorial_code/off, which contains the Offline subdirectory.

Code management can be complex and so tools are developed to do most of the work, and simplify the user's commands. The art package, scons, Muse, and gcc are all provided in a system written and maintained by the lab computing division, in a system called UPS. At this point you have already told the UPS system to make certain packages and tools available in your path. For example, if you ask UPS to show you what it has added to your interactive process:

ups active

and you should get an answer similar to this

Active ups products:
git               v2_30_1         -f Linux64bit+3.10-2.17                    -z /cvmfs/mu2e.opensciencegrid.org/artexternals
ups               v6_0_8          -f Linux64bit+3.10-2.17                    -z /cvmfs/mu2e.opensciencegrid.org/artexternals
muse              v2_07_00        -f NULL                                    -z /cvmfs/mu2e.opensciencegrid.org/artexternals


We will use UPS to get art tools and libraries, and much more, added to our paths, but first we need to discuss a few choices. We can build code for speed, we call this option "prof", or we can tell the compiler to keep the symbols organized, which runs slower, but is useful for debugging, so we call this second option "debug". There are other options for details of how to run simulations, or visualization. The build defaults to "prof", so we will not use any switches, and ask Muse to to use UPS to add art to our paths:

 muse setup

Note that "setup muse" is causing UPS to add muse tools to our path, but "muse setup" is asking Muse scripts to add art and everything else to your paths. To prepare for a debug build we would have used "muse setup -q debug". You can only run "muse setup" once in a process so to change anything, you have to start new process and setup with your alternative choices. Try these commands to see where you are now:

ups active
muse status
muse -v status

The basic configuration of the package's versions and build options is now done.

In a typical use pattern, you would modify some of the code, or add code, then you would want to compile it so you can run it. This is the job of the build system. In our case, we continue to use the set of scripts called Muse, which call a package named scons, which in turn calls gcc to compile code. The build system has to look over the code, decide what needs to be compiled, do the compilation according to a recipe, then see what needs to be linked and do that linking. The first time you run a build, it will compile and link everything. After that, it should only do the steps which are effected by how you changed or added to the code.


Now it is finally time to build: muse build -j 4 the "-j 4" starts the build running with 4 threads, which is appropriate if you are logged on a laptop or a mu2egpvm machine. For mu2ebuild02, where you would usually build code, you can use "-j 32" since it has many more cores.


Once it starts compiling, you might as well kill it with ctrl-c, since it will take long time (4 min on mu2ebuild02) to run.

If you log off and want to return to work here, you would

 mu2einit
 cd mu2e/app/users/$USER/tutorial_code/off
 muse setup

and you are ready to build and run. cd'ing to the working directory first is important, so Muse knows this is where your code is.

Releases

At appropriate times, we mark the code with a tag such as "v09_12_00". This marks a state of the code we want to save. Maybe we are recording the state at a point where we make a production run for simulation, or maybe we are marking major changes in the code. Take look at the release list on github.

After tagging, we build the code and save it so anyone can use it, without building it themselves, - this is a "release".

Starting a new process (not the one from the last exercise) make a new test area:

mkdir -p /mu2e/app/users/$USER/tutorial_code/two
cd mu2e/app/users/$USER/tutorial_code/two
mu2einit

Now list what releases are available

 muse backing

You will see two sets of Offline builds, one labeled "published releases" and one labeled "CI builds". We will use the former. You should see some release versions. To select a version, and link it to this area:

 muse backing Offline v10_05_00

you will now see a subdirectory created, with a link to the tagged release code.

Now to up the game a bit, lets add a small repo to this area:

git clone https://github.com/Mu2e/Tutorial

This repo is some example code, but in practice you can add repos providing ntuples, other tools, or analysis. Now that this code directory is organized, we can setup and build it.

muse setup
muse build -j 4

If successful, you see that a subdirectory "build" appeared. You can explore this director to find object files and shared-object libraries. You path will include these build products, so you could run them.

Partial Builds

When we started building the code in the exercise above, we killed it because it would take too long. On a mu2egpvm node it will take an 45 min, and on mu2ebuild02 with its 32 cores, it will take 4 min. (Note that these times are for a full build from scratch - any following incremental builds are usually much faster.) These long times are a common problem, so we have developed a way to build part of the code.

We need a fully built "base release" that already exists, such as the ones we just looked at. We then make a local directory with just a small piece of the code. We build this small piece, and then take all the rest of the compiled code from the "base release".

Warning In this method you must be aware of a big possible pitfall. If any header file that is compiled into your local small piece of code is different than the same header file compiled into the base release, then the code will have corrupt memory and is likely to fail horribly in unpredictable ways. It is best to use this partial build technique only when building a small part of the code base, and only changing cc files. It works great to do simple things like add a debugging print statement, or test a new module, or modify some fcl. If you have to modify header files, or do extensive code development, it is best to clone the whole Offline repo.

We will need to change the build directory organization, so we will have to start a new process.

mkdir -p /mu2e/app/users/$USER/tutorial_code/part
cd mu2e/app/users/$USER/tutorial_code/part
mu2einit
muse link

Now organize the code in this area. First, take a look at the output of "muse link". Recall there were two sections, "Recent published Offline releases" which we used in the previous exercise, and the second list, the "Recent Offline CI builds". "CI" stand for continuous integration, which means every time the Offline repo changes, we build it and add it to this list. So you have a complete build of the most recent Offline code at your fingertips. You'll see the top line is the most recent, now cut and past that name to tell muse to use it as a backing build:

muse link master/be905d45

Of course the latest you see will be different from this example. The hex code is a "git hash" and labels one exact state of the Offline repo. Now we have indicated to Muse to use this build for Offline libraries.

Now we will introduce a small piece of Offline code. This piece of code is compiled and provided in the backing build we just linked, but if we also build it locally, Muse will prefer to use the local version, and take all the rest of Offline from the backing build.

To start the local subset of Offline code:

mgit init

You should see an "Offline" directory appear. Now lets add the HellowWorld subdirectory from Offline to this local area

cd Offline
mgit add HelloWorld
ls -l
cd ..

You should see that the HelloWorld subdirectory of Offline has been added locally. Now that the code is organized, tell Muse to finish setting up the paths:

 muse setup

and build

 muse build -j 4

Exploring The Partial Build

Here are some suggested exercises.

  1. Look at the source code for the module HelloWorld/src/HelloWorld_module.cc . This file has a lot of structure that you can read about on the wiki page for Modules. The "payload" of the module is the two lines that printout the EventID for each event:
        cerr << "Hello, world with changes.  From analyze: "
             << event.id()
             << endl;
    

    The Mu2e wiki has a short writeup about EventIDs.

  2. Run the HelloWorld module:
      mu2e -c Offline/HelloWorld/test/hello.fcl 
    

    Identify the printout from the module:

    Hello, world.  From analyze: run: 1 subRun: 0 event: 1
    Hello, world.  From analyze: run: 1 subRun: 0 event: 2
    Hello, world.  From analyze: run: 1 subRun: 0 event: 3
    

    This will also print some warning messages, which will be fixed.

    The next tutorial has a lot of information about using fcl files so this tutorial will not discuss them much.

  3. You can also run a larger number of events:
      mu2e -c HelloWorld/test/hello.fcl -n 20
    
  4. Edit the file Offline/HelloWorld/src/HelloWorld_module.cc and change the printout. Then rebuild it:
     muse build
     mu2e -c Offline/HelloWorld/test/hello.fcl
    

    Observe that the printout made by the module has changed.

  5. Make a new module by making a copy of a module and modifying the copy:
    1. Make a copy of HelloWorld/src/HelloWorld_module.cc
      cp Offline/HelloWorld/src/HelloWorld_module.cc Offline/HelloWorld/src/Bonjour_module.cc
      cp Offline/HelloWorld/test/hello.fcl Offline/HelloWorld/test/bonjour.fcl
      
    2. Edit the new .cc file and make a global change of HelloWorld to Bonjour. Change the printout again so that you can distinguish this module from the original.
    3. Edit the .fcl file to replace
       module_type : HelloWorld
      

      with

       module_type : Bonjour
      
    4. Build and run the new module
      muse build
      mu2e -c Offline/HelloWorld/test/bonjour.fcl
      

      Observe the changed printout. If you have questions about why you did not also change the module label, you can re-read the wiki page on Modules.

  6. Make a new .fcl file that runs both modules in one job:
    cp Offline/HelloWorld/test/hello.fcl Offline/HelloWorld/test/both.fcl
    

    Edit both.fcl so it looks like:

      analyzers: {
        hello: {
          module_type : HelloWorld
        }
        hi : {
          module_type : Bonjour
        }
      }
    
      p1 : [ ]
      e1 : [hi, hello]
    

    Now run both modules in one job:

    mu2e -c HelloWorld/test/both.fcl
    

    Note that you do not need to rebuild the libraries because you did not change the source code.

  7. Rebuild the code (muse build) even though there is nothing to do. You should see the following;
    scons: Reading SConscript files ...
    scons: done reading SConscript files.
    scons: Building targets ...
    scons: `.' is up to date.
    scons: done building targets.
    
  8. In both.fcl, swap the order of "hi" and "hello" in the definition of "e1". Rerun. Notice that the order of the printout does not change! art reserves the right to run analyzer modules in an arbitrary order and we should never write code that depends on a particular order. art does respect user specified order for producer and filter modules.

You can also tell muse to make a tarball of your build area:

muse tarball

this is useful for when you want to run on the grid to generate or process large datasets using your the code in your build area.