Running Art Tutorial

Tutorial Session Goal

In this Tutorial you will learn how to run the Mu2e 'art' framework executable (mu2e), both interactively and on the grid.

Session Prerequisites and Advance Preparation

Perform the Tutorial on setting up the Mu2e Offline

Session Introduction

Art is a software framework for processing events with modular code with lots of run-time configurability. Art is controlled by scripts in a dedicated configuration language called fhicl (.fcl suffix). Art uses rootIO to store events.

This tutorial will cover how to build and run several different kinds of art jobs, and how to use the mu2e job tools to divide large projects into many separate jobs, and how to run those jobs in parallel on Fermigrid or the OSG (open science grid).

Exercises

Exercise 1: Running a simple module (Hello, Tutorial!) and basic FHiCL

In this exercise, we will run a simple module that will print a welcoming message.

First set up to run Offline
The executable we use to run Offline is "mu2e." Use the --help option to display all the command line options
FHiCL files (*.fcl) tell Offline what to do. We specify the fcl file we want to use every time we run Offline using the "-c" option. We will now run a simple job that prints a hello message using a premade fcl file.
We can now explore the hello.fcl file that configured this Offline job to see how it works.

In FHiCL, we make definitions using the syntax "variable : value". A group of definitions can be combined into a table by surrounding with braces {}.
Like in C++, we can refer to FHiCL code in other files by using "#include" statements
After defining the process name, you will see three main tables: source, services, physics
"source" configures the inputs to the job. If we are making new events from scratch, we use "EmptyEvent". If we are building on top of old files, we might use "RootInput." You can also see that this job is configured to run 3 events by default
"services" configures parts of art that are common to all modules, like the geometry, detector conditions, or ability to print out to files
"physics" is where we configure the modules that do all the work. There are "producer" modules that create data that is added to the event, and "analyzer" modules that read data and might make things like analysis TTrees. There are a couple different sections to the physics table. First we declare our producer and analyzer modules, then we define our "paths" (see below), and then we tell Art which paths we want to run.
To make the module run, we must tell art the list of modules and the order we want to run them in. We do this by defining a variable called a path to be this list of module names. Here there are two paths, p1 (which is empty), and e1. We then tell Art which paths to run using the definitions of "trigger_paths" and "end_paths". Producers (and filters) go in trigger paths, analyzers go in end paths.

You can see more detail about FHiCL at https://mu2ewiki.fnal.gov/wiki/FclIntro or check out the Art workbook and user guide chapter 9 (https://art.fnal.gov/wp-content/uploads/2016/03/art-workbook-v0_91.pdf)

Exercise 2: Module configuration with FHiCL

We will now see how to modify FHiCL to run different modules and even configure those modules at runtime

We have a new fcl file, hello2.fcl, try running that.
We can look at the source code for HelloWorld2 to see how we change Magic number.
Configure fcl to set Magic number to 5 by adding a line "magicNumber : 5" under module_type. Run the fcl again to check that it changed
You can also add this configuration to the end of the fcl file by using the full parameter location, i.e.
Finally, try running both this module and the original HelloWorld module by adding the module declaration from hello.fcl and adding it to your end_path. If you need help, check $TUTORIAL_BASE/solutions/hello2.fcl

Exercise 3: Using a more realistic Mu2e fcl to simulate an event

To fully simulate an event in Mu2e, we will need to run many more modules and services. Modules can become dependent on output from previous modules and may require certain services to be set up, so the final FHiCL for a functioning Mu2e Offline job ends up being somewhat complex. To help make things easier, we use a few FHiCL tricks.

Lets look at an example script to produce conversion electron events
At the top you will see a #include line. Lets look at the file it is including
You can see that this file includes several more files, and then starts with BEGIN_PROLOG. Prolog files are just a bunch of FHiCL definitions that then can be used later. You can see for example it defines a table called Primary.producers

Most directories in Offline have a prolog.fcl file that provide standard definitions for their modules and folders

In JobConfig/fcl/prolog.fcl (and in fcl/CeEndpoint.fcl) you will see several definitions using "@local" or "@table". This is how you reference a previously defined value (for example something defined in a prolog).
- @local references a standard definition (for example line 16 the definition of a module)
- @table references a table of several definitions but without the curly braces (for example line 18 adds several more module definition name:value pairs to the producers table)
- @sequence references a list of values separated by commas like for a path (for example line 62, each sequence adds the name of several modules to this path)
- For more details read https://mu2ewiki.fnal.gov/wiki/FclIntro
Back in fcl/CeEndpoint.fcl, you should see @table::Primary.producers, which we found in JobConfig/fcl/prolog.fcl. See if you can find out where the generate module in this fcl comes from and what it is running.
You can debug a complicated FHiCL with lots of #includes using the Offline option --debug-config. This will fully process the script, and print out to a file the results with all the @local etc. references made explicit. Lets try with our CeEndpoint.fcl
Finally, run fcl/CeEndpoint.fcl and generate 10 events

Some more general information about Mu2e simulation concepts can be found here https://mu2ewiki.fnal.gov/wiki/Simulation

Exercise 4: Exploring Offline outputs

The above exercise should produce two files, dig.owner.CeEndpoint.version.sequencer.art and nts.owner.CeEndpoint.version.sequencer.root (also located in $TUTORIAL_BASE/RunningArt/data). Both are actually root files, but they contain different information. The .root files produced by Offline are used for diagnostic histograms and TTrees, and analysis output like TrkAna that can be used in a normal root analysis. The .art files contain the actual c++ objects Offline uses to describe the event (both simulation information and reconstructed information), and so are in general meant to be processed by other Offline jobs.

Open both files in a root TBrowser to see their contents
We can use Offline modules to better understand the .art file contents.
In general, art dataproducts are much more difficult to work with in plain root scripts outside of the framework. The names are complicated, and the way objects reference other objects only works within the framework. Most of the time you will have some analysis module in Offline process the dataproducts .art file for you and produce a simpler TTree that you can run your root analysis on.

Exercise 5: Create your own primary production job

The JobConfig directory has the base scripts for all the production jobs, and so has examples of how to correctly configure most kinds of Offline jobs. If you need to do anything different, you can usually start with a JobConfig script and modify it slightly.

JobConfig/primary has scripts to generate primary only (no backgrounds) events without doing reconstruction
JobConfig/mixing has scripts to generate primary plus background events
JobConfig/reco has scripts to take output of the previous two and run reconstruction
These scripts are designed to be used with grid production scripts, and so don't include configuration of the random seed. To run the scripts as is, you will need to add two lines:

Lets try to make our own fcl script now. See if you can make and run a script to run primary only events with 100 MeV photons coming from the muon stops in the stopping target. Start by looking at $MU2E_BASE_RELEASE/EventGenerator/fcl/prolog.fcl, which has the default configuration for all of the event generators. There should be one called photonGun.

Tip: start by copying $MU2E_BASE_RELEASE/JobConfig/primary/CeEndpoint.fcl and replace the generator.
Don't forget to add the seed configuration!
NOTE: Are you getting an error about genId? The most recent version of Offline requires all generators to define this parameter. For photonGun, we want to set the parameter physics.genId to equal particleGun. See if you can figure out how!

Now see if you can turn on the StrawDigi diagnostic output. Looking at $MU2E_BASE_RELEASE/TrackerMC/src/StrawDigisFromStepPointMCs_module.cc, you will see a FHiCL parameter called diagLevel. Try increasing it in your script from 0 to 2.
Run your script and check the output .root file, you should see a new TDirectory with the StrawDigi diagnostics.

If you need help, check solutions/photongun.fcl

Exercise 6: Running event reconstruction

The mu2e scripts we have run so far have been digi production jobs, so they ended up producing mu2e::StrawDigis, mu2e::CrvDigis, etc. dataproducts. These represent the data as it would look coming off the detector. We need to use a different script to actually run the reconstruction algorithms. We will use the output of the digi production jobs (the .art file) as the input to these new jobs.

Take a look at $MU2E_BASE_RELEASE/JobConfig/reco/mcdigis-trig.fcl . You will see this script uses a RootInput source, which means it processes on top of an existing .art event. We can tell Offline which file to use three ways:

mu2e --source <path to .art file>
mu2e --source-list <path to text file with one .art file path per line>
add "source.fileNames : ["<path to .art file 1>","<path to .art file 2>",...] to the .fcl file

Since we are starting from a "primary only" art file (more on that later), we will run the mcdigis-primary.fcl script. Try running it now on your CeEndpoint digi output:
Run Print/fcl/dumpDataProducts.fcl on the new .art file you produced (mcs.owner.RecoMCTrig.version.sequencer.art), and compare to the data products in the original digi art file.

You will see that the mu2e::StrawDigis etc. dataproducts are still there, but now associated with the SelectRecoMCs module
Additional dataproducts related to the reconstruction like mu2e::TimeClusters, mu2e::HelixSeeds, mu2e::KalSeeds. The module names for the reconstruction follow the pattern something + <D/U> + <e/mu> + <M/P> for downstream/upstream electron/muon minus/plus. "KSF" is the seed fit and "KFF" is the final fit.

If you look at $MU2E_BASE_RELEASE/JobConfig/reco/prolog.fcl, you will see that all of the reconstruction algorithms are actually producer modules. Although reconstructing an event might seem like an analysis, since we saw that it is adding dataproducts to the .art file, in the Art framework it qualifies as a producer. This also means we won't want to do a root analysis on this file either, we will want to further condense it down to a useful .root file, we will see one example in the next exercise.

Exercise 7: Running TrkDiag to create TrkAna TTrees

We will now run some analysis modules on the output of the reconstruction job to get a TTree that we can look at in a TBrowser or analyze with a root script. To see more about this particular analysis tree see the upcoming tutorial (https://mu2ewiki.fnal.gov/wiki/TrkAna_Tutorial_Session).

We will use the script $MU2E_BASE_RELEASE/TrkDiag/fcl/TrkAnaReco.fcl. If you look at this file, you will see that unlike the other scripts we have used, this does not have an outputs table, so it will not be producing a .art file of dataproducts. Instead it will only produce analyzer output through the TFileService service. You can see the default output name configured in the fcl file. If you look in $MU2E_BASE_RELEASE/TrkDiag/fcl/prolog.fcl, you can see that this script will run the TrackAnalysisReco module.

When running a script that uses TFileService, you can configure the output filename from the command line using the -T option. Lets try running the TrackAnalysisReco module on our CeEndpoint reconstruction output, and lets write the output to trkana.root:

 mu2e -c $MU2E_BASE_RELEASE/TrkDiag/fcl/TrkAnaReco.fcl --source data/mcs.owner.RecoMCTrig.version.sequencer.art -T trkana.root

Open trkana.root and explore with the TBrowser. You will see more about what the parts of this TTree mean in the later tutorial.

Exercise 8: Using generate_fcl to prepare to run jobs on the grid

When you want to simulate a large number of events, you will need to split your work up over multiple Offline jobs, and you will often want to run them on the Fermilab grid. There is a series of scripts to help you set up your fcl files and to start and track your jobs. The first problem is creating the fcl scripts. If you want to generate 10000 conversion electron events for example but only want to run 500 in a single job, you will need 20 fcl files that do the same thing - but each will need to have a different random seed, and each will need to output to different file names! The script generate_fcl takes a base fcl and will build these 20 files for it.

The generate_fcl tool is in mu2etools so you will first need to setup that UPS product
In the $MU2E_BASE_RELEASE/JobConfig/examples directory, there are scripts to call generate_fcl for each of the fcl files in JobConfig/primary,JobConfig/mixing, and JobConfig/reco. Lets look at $MU2E_BASE_RELEASE/JobConfig/examples/generate_CeEndpoint.sh

You can see from the --include option that this script is using JobConfig/primary/CeEndpoint.fcl as the base fcl, and will be used to produce 2000000 events (200 jobs of 10000 events each)
The description, dsconf, and dsowner options configure the output filenames. If you are simulating a large production that will be widely used by others, it is important that the output filenames follow specific patterns, which is what generate_fcl tries to enforce. You can read more about the Mu2e filename patterns at https://mu2ewiki.fnal.gov/wiki/FileNames
generate_fcl will by default create directories named 000,001,002, etc. and put 1000 fcl files in each. In this script you will see that the 000 folder is automatically renamed to CeEndpoint
More detailed documentation for generate_fcl is located at https://mu2ewiki.fnal.gov/wiki/GenerateFcl

Run this script to produce 200 jobs

There should now be a CeEndpoint directory. If you ls inside it, you will see many .fcl files and .fcl.json files
The .json files are used during official productions to keep track of the provenance of every file produced, you can ignore them for now
If you look at one of the .fcl files, you will see it just #includes our base fcl, and the only change from file to file is a reconfiguration of the seed and output filenames.

Look at scripts/generate_reco-CeEndpoint-mix.sh (This is a slightly modified version of JobConfig/examples/generate_reco-CeEndpoint-mix.sh). Instead of njobs and events-per-job we now just provide a --inputs argument with the name of a file containing filenames of art files to read from, and a --merge-factor parameter. Normally reconstruction is fast enough that in one job you can process the output of many simulation jobs. The merge-factor configuration tells generate_fcl how many art files each reco job will use.
Run this script and explore the output

Since data/CeEndpoint-mix.txt had 60 entries and we have a merge-factor of 20, we will produce 3 output fcl files.

Exercise 9: Adding backgrounds with event mixing

So far when we've been simulating Mu2e events, we've only included the primary signal. But in the real experiment any signal event will sit on top of a lot of backgrounds from the beam etc. Simulating backgrounds from the beam takes a very long time, so they are done in separate jobs. The outputs of those dedicated background simulations can then be put on top of a signal only simulation to create something more realistic - this is called event "mixing".

First lets look at a production mixing script: open up JobConfig/mixing/CeEndpointMix.fcl and JobConfig/primary/CeEndpoint.fcl side by side and compare. You can see that by using prolog we've kept the scripts looking basically identical.
We want to randomize which background events are mixed on top of each signal event - we can do this using the generate_fcl script. If you ls data/*.txt you will see files for several different types of backgrounds. Each is a list of filenames of background only simulation art files. These files are not included in Offline, you will need to produce a copy of them making sure to use the most up to date background datasets before you run any mixing jobs. Open scripts/generate_CeEndpointMix.sh (this is just a lightly modified version of JobConfig/example/generate_CeEndpointMix.sh). It looks similar to the non-mixing version but now has several --aux-input values which point to these text files.
Build a set of mixing jobs:

Find out more about mixing here https://mu2ewiki.fnal.gov/wiki/Mixing

Exercise 10: Submitting grid jobs with mu2eprodsys

Here we will go over the actual process of running a large set of simulation jobs (like the fcl we produced in exercise 8) on the grid. A more detailed version of the whole grid production workflow can be found at https://mu2ewiki.fnal.gov/wiki/MCProdWorkflow , and details on submitting jobs can be found at https://mu2ewiki.fnal.gov/wiki/SubmitJobs

We need to make everything that our grid production needs will be accessible to the grid nodes:

The Offline code
The .fcl scripts we want to run
Any required input .art files

For something to be accessible to a grid node, we need to either send it directly to the node along with the job submission or put it on a shared disk resource. If you look at the background .art files we saw in exercise 9 for example (data/*.txt), you will see they are located on the mu2e machines in the /pnfs/mu2e/tape/ directory, which is accessible to the nodes. Additionally, the official Offline releases at /cvmfs/mu2e.opensciencegrid.org/Offline/ are accessible. More information on the various disk spaces can be found at https://mu2ewiki.fnal.gov/wiki/Dcache and https://mu2ewiki.fnal.gov/wiki/Cvmfs .

Lets try setting up a grid production (at least as much as the docker will allow).

First we will need to move our fcl files so they are accessible to the grid. We will need to set up the UPS products for the grid tools
Next, we will create a tarball of our fcl files produced with generate_fcl.
If we were really going to run this production on the fermilab grid we would put this tarball on a node accessible disk, for example:
The command we will run is mu2eprodsys. The options we need to provide it are:

--dsconf and --dsowner : should be same as in generate_fcl
--wfproject : sets name of directory where output will be placed
--setup : path to the setup.sh script for the version of Offline we want to run. This Offline must be on a grid accessible disk
--fcl : path to the fcl.tar.gz file we just created

Once you think you have the full command, try it out with the --dry-run option. You can compare against the script at solutions/mu2eprodsys.sh
If you want to run a non release version of Offline, we will need to provide mu2eprodsys with a tar containing the code to run. Using the example from https://mu2ewiki.fnal.gov/wiki/Mu2e_Offline_Tutorial, build a satellite release with any package you want. Then we will use gridexport https://mu2ewiki.fnal.gov/wiki/Gridexport
Run mu2eprodsys again but replace --setup with --code=/path/to/tar.gz that was just produced

Exercise 11: Submitting grid jobs with input art files

If you are running production jobs using previously generated data as input, you will often need to deal with getting data off the tape. You can find a lot more detail about this here: https://mu2ewiki.fnal.gov/wiki/Prestage and https://mu2ewiki.fnal.gov/wiki/Dcache .

This will mostly not be possible in the docker but we can at least go through some of the motions. Lets see how you would look at an already generated dataset and how you "prepare" a dataset to run reconstruction on.

Go to https://mu2ewiki.fnal.gov/wiki/MDC2018 and pick one of the digi datasets to prepare. (digi datasets start with "dig")
Lets figure out how many files are in this dataset. Run setup dhtools samweb count-files "dh.dataset=dig.name.of.your.digi.dataset.here.art"
Since there is a large amount of data being stored, sometimes it is written to tape drives to save space on the faster drives. Tape drives can be very slow to access, so we will need to "prestage" these files back to the disk drives before we actually use them. First we check if this dataset has been saved back to tape drives

Note this will fail because we are on a docker. Normally it would print somewhere between 0 and 100%

Lets pretend it all is actually on tape. To prestage, follow the directions in https://mu2ewiki.fnal.gov/wiki/Prestage , but generally samweb prestage-dataset --parallel=5 --defname=dig.name.of.your.digi.dataset.here.art
- again this will not work because we are on the docker. This could take a while depending on how many files are in your dataset
Now that our files are supposedly on disk, lets actually find one of them

This will give you the filenames but not their locations. You can then grab any of these file names and run

If you want to create a list of file locations to use with generate_fcl, you can do

Exercise 12: Running the event display

The event display is a module in Offline which visualizes events in an interactive display.

Run EventDisplay.fcl on an art file
Show MC tracks
Show reco tracks
Show hits

Tracker Hits: "KalSeed:KFFDmuM:" These are tracker hits which were used for the reconstruction.
Calo Hits: "CaloCrystalHit:SelectRecoMC:"
CRV Hits: CrvRecoPulse:SelectRecoMC:"

Zoom and orientation

miny: 2000 mm
maxy: 3000 mm
minz: 0 mm
maxz: 2000 mm

Next event
Animation

Reference Materials

https://mu2ewiki.fnal.gov/wiki/Code
https://mu2ewiki.fnal.gov/wiki/Workflows
art workbook
various DocDBs that reference production, satellite release, partial checkout, etc.