MuseBuildCodeTutorial

Introduction

In the case of Mu2e, an event records the detector response during one microbunch, consisting of a short burst of millions of protons hitting the production target, thousands of muons stopping in the aluminum stopping target, and those muons interacting or decaying. This happens nominally every 1695 ns and each of these burst forms a Mu2e event.

When we read and process events to simulate the detector, reconstruct raw data, or analyze data events we will need a framework to organize our executables, and way to control the behavior of the executable, and a format to write the events into. These functions are provided by the art software package.

When we want to operate on the data, we write c++ code. We track code source files using a code management tool called git, which writes files to a commercial site, github. We have a main repository (or repo) called Offline and other smaller special-purpose repos. We compile and link the code using a set of Mu2e scripts called Muse. Inside those scripts, we use a code build system called scons. Finally, inside scons, we use linux gcc to compile and link the code.

Art

The art package provides:

• a framework for creating executables
• the event format (we define the contents)
• a control language called "fhicl" or "fcl" (pronounced fickle)
• services such as random numbers

When we write code, typically to simulate the detector, reconstruct the data, or analyze the data, we write our code in c++ modules. We then run the framework, and tell it to call the code in our module. We also tell it what input files to use, if any. The framework opens the input files, reads data, priovides services, calls our modules with the data, event by event, and then handles any filtering of events, and writing output.

When we run executables in the art framework, we have to tell the framework which modules to run and how to configure the modules. A module configuration might include how to find the input data inside the file, and what parameters and cuts to use in performing its task. This function is provided by a fcl file - you always provide a fcl file when you start up an art executable.

art executables read and write files in the art format. This is a highly-structured format which contains the event data, processing history, and other information. Under the covers, art uses the root package (used widely in HEP) to read and write the event files. Each event contains pieces which are referred to as products. For example, in each event, there may be a product which contains raw hits, one that contains reconstructed hits, and one that contains reconstructed tracks. In simulation, there are products with truth information.

All of these concepts will be fleshed out in subsequent tutorials.

Code

We will start with a tour of the Mu2e code base. We keep our code organized using a piece of software called git. This freeware package stores our code in a compact format and keep track of lots of useful things:

• all the history of each file as it is developed
• coherent versions of the whole set of code (tags)
• tracking parallel development by many people, and merging all that work back together when ready.

So lets look at what is stored in our git repository. If you are on the interactive central servers (mu2egpvm or mu2ebuild)

mkdir -p /mu2e/app/users/$USER/tutorial_code
cd mu2e/app/users/$USER/tutorial_code
source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh 

Since we will be doing several builds, create some subdirectories:

 mkdir off
 cd off

Retrieve the code base from the git repository:

git clone https://github.com/Mu2e/Offline

after a few seconds you should see a directory Offline. Go into that dir

cd Offline

and look around. Note the directory .git. This is where git keeps information such code history and tags. Most of the directories you see here contain our working code. Each directory contains a relatively small, logically-grouped set of code.

Here are few git commands to get you started.

git status

tells you which branch you are on. When multiple people are working on code development, they will probably be working on their own branches - a version of the code they can control. When the changes are tested and approved, the personal, working branch is merged back into the main code branch, called "master". See the active branches:

git branch -al

You can switch to a different active working branch:

git checkout MDC2018
git status

or to a fixed release:

git checkout v09_12_00

all that output text is saying this fixed point of the code and you shouldn't be trying to modify it, which is fine, makes sense.

git status

see the files which are different from the previous release:

git diff --numstat v09_11_00 | head

and then see the code that changed in that first file listed:

git diff v09_11_00 Analyses/src/CaloCalib_module.cc

and then the file history:

git log v09_11_00 Analyses/src/CaloCalib_module.cc

In the git status you should also see "working branch clean" this means your area is still the same as the central repository you just cloned. If there were new or changed files, it would tell you. You can read much more about git and how we use it, including making your own branches and committing code modifications back to the central repo.

Let's look at the code structure. For example, under

ls -l TrkHitReco

you can see directories: fcl, inc, src, test and data. Most of the top-level directories follow this pattern. The c++ includes related to the code are kept under inc, the c++ source code in src, the fcl scripts that configure modules are in fcl, and sometimes scripts are under test or text files used by the code are kept under data. Many directories don't need test or data - see the TrkReco directory for example. Take a look at what in these directories.

Recall that we write modules, which are then run by the framework to act on the event data. Any module we write is compiled into a shared object library which is loaded by the framework if our fcl tells it to. For example,

TrkHitReco/src/StrawHitReco_module.cc

would be compiled into

../build/<some string>/Offline/lib/libmu2e_TrkHitReco_StrawHitReco_module.so

which we can then ask the framework to load and run (or not). The shared object is not there, because we haven't done the compiling, which we will get to in a minute.

Just as a quick peak, open the cc file:

emacs (or vi, cat) TrkHitReco/src/StrawHitReco_module.cc

Find the line

class StrawHitReco : public art::EDProducer

Since the module code inherits from a framework base class (EDproducer), the framework can manipulate it.

Find the line

StrawHitReco::beginJob

this method is called once at the beginning of the job. Similarly

StrawHitReco::beginRun

is called when the input events change run number.

StrawHitReco::produce

is called for every event. It is called "produce" because it typically produces data to insert into the event, in this case, the reconstructed tracker hits. Other option are "filter" if this module can select a subset of events to write out, and "analyze" if the module is not writing any new data to the event.

Find the line

event.getValidHandle(_sdtoken)

this is retrieving the raw hits from the event. Find the line event.put(std::move(chCol)); This adds a newly created data product to the event. All of these concepts will be covered in more detail later.

Now cd back out of the Offline directory

cd..

Configuring and Building Code

You should now be in the directory tutorial_code/off, which contains the Offline subdirectory.

Code management can be complex and so tools are developed to do most of the work, and simplify the user's commands. The art package, scons, Muse, and gcc are all provided in a system written and maintained by the lab computing division, in a system called UPS. At this point you have already told the UPS system to make certain packages and tools available in your path. For example, if you ask UPS to show you what it has added to your interactive process:

ups active

and you should get an answer

Active ups products:
git               v2_30_1         -f Linux64bit+3.10-2.17                    -z /cvmfs/mu2e.opensciencegrid.org/artexternals
ups               v6_0_8          -f Linux64bit+3.10-2.17                    -z /cvmfs/mu2e.opensciencegrid.org/artexternals

So git is one of the tools you have already put in your path (when you sourced setupmu2e-art.sh). Now ask what versions of the Muse package are available to us:

ups list -aK+ muse

You should see many versions available. One should be marked "current" - this is the version you get when you don't ask for a specific a version. To tell UPS to add Muse tools to the path, we use the UPS "setup" command

setup muse

If you now ask what's active ("ups active") you should see there is a version of Muse there. The "muse" command is now in your path.

We will use UPS to get art tools and libraries, and much more, added to our paths, but first we need to discuss a few choices. We can build code for speed, we call this option "prof", or we can tell the compiler to keep the symbols organized, which runs slower, but is useful for debugging, so we call this second option "debug". There are other options for details of how to run simulations, or visualization. The build defaults to "prof", so we will not use any switches, and ask Muse to to use UPS to add art to our paths:

 muse setup

Note that "setup muse" is causing UPS to add muse tools to our path, but "muse setup" is asking Muse scripts to add art and everything else to your paths. To prepare for a debug build we would have used "muse setup -q debug". You can only run "muse setup" once in a process so to change anything, you have to start new process and setup with your alternative choices. Try these commands to see where you are now:

ups active
muse status
muse -v status

The basic configuration of the package's versions and build options is now done.

In a typical use pattern, you would modify some of the code, or add code, then you would want to compile it so you can run it. This is the job of the build system. In our case, we continue to use the set of scripts called Muse, which call a package named scons, which in turn calls gcc to compile code. The build system has to look over the code, decide what needs to be compiled, do the compilation according to a recipe, then see what needs to be linked and do that linking. The first time you run a build, it will compile and link everything. After that, it should only do the steps which are effected by how you changed or added to the code.

Now it is finally time to build: muse build -j 4 the "-j 4" starts the build running with 4 threads, which is appropriate if you are logged on a laptop or a mu2egpvm machine. For mu2ebuild01, where you would usually build code, you can use "-j 20" since it has many more cores.

Once it starts compiling, you might as well kill it with ctrl-c, since it will take long time (10 min on mu2ebuild01) to run.

If you log off and want to return to work here, yo would

 source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh
 cd mu2e/app/users/$USER/tutorial_code/off
 setup muse
 muse setup

and you are ready to build and run

Releases

At appropriate times, we mark the code with a tag such as "v09_12_00". This marks a state of the code we want to save. Maybe we are recording the state at a point where we make a production run for simulation, or maybe we are marking major changes in the code. Take look at the release list on github.

After tagging, we build the code and save it so anyone can use it, without building it themselves, - this is a "release".

Starting a new process (not the one from the last exercise) make a new test area:

mkdir -p /mu2e/app/users/$USER/tutorial_code/two
cd mu2e/app/users/$USER/tutorial_code/two
source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh 

and access Muse tools

 setup muse

Now list what release are available

 muse list

You will see two sets of Offline builds, one labeled "published releases" and one labeled "CI builds". We will use the former. You should see some release versions. To select a version, and link it to this area:

 muse link v09_12_00

you will now see a subdirectory created, with a link to the tagged release code.

Now to up the game a bit, lets add a small repo to this area:

git clone https://github.com/Mu2e/Tutorial

This repo is some example code, but in practice you can add repos providing ntuples, other tools, or analysis. Now that this code directory is organized, we can setup and build it.

muse setup
muse build -j 4

If successful, you see that a subdirectory "build" appeared. You can explore this director to find object files and shared-object libraries. You path will include these build products, so you could run them.

This section is not applicable yet, since there are no releases built in Muse yet

Partial Builds

When we started building the code in the exercise above, we killed it because it would take too long. On a mu2egpvm node it will take an 45 min, and on mu2ebuild01 with its 16 cores, it will take 10min. (Note that these times are for a full build from scratch - any following incremental builds are usually much faster.) These long times are a common problem, so we have developed a way to build part of the code.

We need a fully built "base release" that already exists, such as the ones we just looked at. We then make a local directory with just a small piece of the code. We build this small piece, and then take all the rest of the compiled code from the "base release".

Warning In this method you must be aware of a big possible pitfall. If any header file that is compiled into your local small piece of code is different than the same header file compiled into the base release, then the code will have corrupt memory and is likely to fail horribly in unpredictable ways. It is best to use this partial build technique only when building a small part of the code base, and only changing cc files. It works great to do simple things like add a debugging print statement, or test a new module, or modify some fcl. If you have to modify header files, or do extensive code development, it is best to clone the whole Offline repo.

We will need to change the build directory organization, so we will have to start a new process.

mkdir -p /mu2e/app/users/$USER/tutorial_code/part
cd mu2e/app/users/$USER/tutorial_code/part
source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh
setup muse
muse list

Now organize the code in this area. First, take a look at the output of "muse list". Recall there were two section, "published release" which we used in the previous exercise, and the second list, the "CI builds". "CI" stand for continuous integration, which means every time the Offline repo changes, we build it and add it to this list. So you have a complete build of the most recent Offline code at your fingertips. You'll see the top line is the most recent, now cut and past that name to tell muse to use it as a backing build:

muse link master/be905d45

Of course the latest you see will be different from this example. The hex code is a "git hash" and labels one exact state of the Offline repo. Now we have indicated to Muse to use this build for Offline libraries.

Now we will introduce a small piece of Offline code. This piece of code is compiled and provided in the backing build we just linked, but if we also build it locally, Muse will prefer to use the local version, and take all the rest of Offline from the backing build.

To start the local subset of Offline code:

mgit init

You should see an "Offline" directory appear. Now lets add the HellowWorld subdirectory from Offline to this local area

cd Offline
mgit add HelloWorld
ls -l
cd ..

You should see that the HelloWorld subdirectory of Offline has been added locally. Now that the code is organized, tell Muse to finish setting up the paths:

 muse setup

and build

 muse build -j 4

Exploring The Partial Build

Here are some suggested exercises.

1. Look at the source code for the module HelloWorld/src/HelloWorld_module.cc . This file has a lot of structure that you can read about on the wiki page for Modules. The "payload" of the module is the two lines that printout the EventID for each event:

       cerr << "Hello, world with changes.  From analyze: "
            << event.id()
            << endl;
   

   The Mu2e wiki has a short writeup about EventIDs

2. Run the HelloWorld module:

     mu2e -c Offline/HelloWorld/test/hello.fcl 
   

   Identify the printout from the module:

   Hello, world.  From analyze: run: 1 subRun: 0 event: 1
   Hello, world.  From analyze: run: 1 subRun: 0 event: 2
   Hello, world.  From analyze: run: 1 subRun: 0 event: 3
   

   This will also print some warning messages, which will be fixed.

   The next tutorial has a lot of information about using fcl files so this tutorial will not discuss them much.

3. You can also run a larger number of events:

     mu2e -c HelloWorld/test/hello.fcl -n 20
   

4. Edit the file Offline/HelloWorld/src/HelloWorld_module.cc and change the printout. Then rebuild it:

    muse build
    mu2e -c Offline/HelloWorld/test/hello.fcl
   

   Observe that the printout made by the module has changed.

5. Make a new module by making a copy of a module and modifying the copy:
   1. Make a copy of HelloWorld/src/HelloWorld_module.cc

      cp Offline/HelloWorld/src/HelloWorld_module.cc Offline/HelloWorld/src/Bonjour_module.cc
      cp Offline/HelloWorld/test/hello.fcl Offline/HelloWorld/test/bonjour.fcl
      

   2. Edit the new .cc file and make a global change of HelloWorld to Bonjour. Change the printout again so that you can distinguish this module from the original.
   3. Edit the .fcl file to replace

       module_type : HelloWorld
      

      with

       module_type : Bonjour
      

   4. Build and run the new module

      muse build
      mu2e -c Offline/HelloWorld/test/bonjour.fcl
      

      Observe the changed printout. If you have questions about why you did not also change the module label, you can re-read the wiki page on Modules.

6. Make a new .fcl file that runs both modules in one job:

   cp Offline/HelloWorld/test/hello.fcl Offline/HelloWorld/test/both.fcl
   

   Edit both.fcl so it looks like:

     analyzers: {
       hello: {
         module_type : HelloWorld
       }
       hi : {
         module_type : Bonjour
       }
     }
   
     p1 : [ ]
     e1 : [hi, hello]
   

   Now run both modules in one job:

   mu2e -c HelloWorld/test/both.fcl
   

   Note that you do not need to rebuild the libraries because you did not change the source code.

7. Rebuild the code (muse build) even though there is nothing to do. You should see the following;

   scons: Reading SConscript files ...
   scons: done reading SConscript files.
   scons: Building targets ...
   scons: `.' is up to date.
   scons: done building targets.
   

8. In both.fcl, swap the order of "hi" and "hello" in the definition of "e1". Rerun. Notice that the order of the printout does not change! art reserves the right to run analyzer modules in an arbitrary order and we should never write code that depends on a particular order. art does respect user specified order for producer and filter modules.

You can also tell muse to make a tarball of your build area:

muse tarball

this is useful for when you want to run on the grid to generate or process large datasets using your the code in your build area.