GitDirExample

Validation

This package is the center of offline validation in mu2e. It is used in nightly code validation and in various debugging efforts. It is also useful in test such as comparing the code on different platforms. The module and tools may be a convenient way for a user to get an overview of how a job is functioning. The comparison tools may be useful for a user to compare two histogram files. Eventually the module will probably be brought into standard production workflows. It is not decided if any online monitoring will be here.

This package contains components:

    Validation - a module that makes standard histograms of products in a file
    fcl to control Validation
    code to make histograms
    a set of classes that make comparing histograms (and profiles, etc) simple. These can be loaded directly into an interactive root session. These are in theinc and root subdirectories.
    an executable that uses the comparison tools to compare two files of histograms and make reports in various ways, including a web page 

It is useful to know how the histogram comparisons are done. The first histogram file is take as the standard. In plots it will be the gray histogram. The second file is taken as the file to be tested and appears as red dots.

Two criteria are used: a standard root KS test, and a fraction test. When comparing histograms, underflow and overflow bins are included by default, but you can switch this off. The fraction test is done by integrating through each histogram and at each step compute the difference in the two sums and save the largest difference found, then divide by the total entries in the standard histogram and subtract from one (so when they are similar, the result is near 1). When statistics get very small, the KS test fails but the fraction test still is useful. When statistics are very high, and there is any tiny systematic variation (like comparing two data runs) the KS test will fail. In this case the fraction test will tell you what you want to know. Very generally, if a comparison passes one of the two tests, it is "OK".

There are two comparison levels, a loose and tight, and also two modes. The two most common cases for comparison are either the files are supposed to be identical (like in nightly validation) or statistically independent (like comparing two simulation files which had different random seeds). If the files were supposed to be identical, the alarm levels for the tests are 0.999 (tight) and 0.99 (loose). If the files are independent, they are set at 0.01 (tight) and 0.001 (loose). The normalization of the two files can be scaled.

During build, this package makes:

    bin/valCompare - the comparison exe
    lib/libmu2e_Validation_Validation_module.so - the module
    lib/libmu2e_Validation_root.so - this can be loaded into a root session
    lib/libmu2e_Validation_root_dict_rdict.pcm
    lib/libmu2e_Validation_root_dict.rootmap
    lib/libmu2e_Validation.so 

Validation module
The module can be run by itself on a file:

mu2e -f Validation/fcl/val.fcl -s inputfile

This will produce validation.root. This will contain a Validation subdirectory and in there will be more directories, one for each art product the module could find and analyze. The directories are named by the product name. If there are many copies of products, such as StepPointMCs, then each copy will get its own directory and set of histograms. The module can also be run as part of a path:

services : {
# request standard geometry and conditions services
# request TFileService
}
# setup module
physics :{
  analyzers: {
    Validation : {
      module_type : Validation
      validation_level : 1
    }
  }
}
# put Validation in a path..

Using the module as part of your path will let you see all the products in the event, even ones that will get dropped on output. The validation level will control how in-depth the histogramming will go. So far, only level 1 is implemented. This is intended to be quick and robust, just histogramming a few variables from each product. When level 2 is implmented, it might histogram derived quantities or quantities derived from multiple products, or make cuts. The following products are analyzed:

CaloCluster
CaloHit
GenParticle
SimParticle
StepPointMC
StrawHit
TrackClusterMatch
TrackSummary

valCompare
This is in the path for the release. It operates on two file of histograms. It will only compare histograms that are in the same directories and with the same names, and ignore anything else.

valCompare -h
    lists all options
valCompare -s file1.root file2.root
    produces a summary of how many histograms compared
valCompare -r file1.root file2.root
    produces a line for each histogram and how it compared
valCompare -w dir/result.html file1.root file2.root
    produces the named web page and overlay plots for each histogram also in that dir

histogram comparison classes
These can be used in root:

root [0] .L lib/libmu2e_Validation_root.so
TValCompare c
c.SetFile1("file1.root")
c.SetFile2("file2.root")
c.Analyze()
c.Summary()
c.Browse()

They can also be used in code. The histograms are compared with TValHistH, TProfile with TValHistP and TEfficiency with TValHistE

#include "Validation/inc/root/TValHistH.h"
TH1D* h1 = new TH1D...
TH1D* h2 = new TH1D...
TValHistH comph()
comph.SetHist1(h1)
comph.SetHist2(h2)
comph.Analyze()
comph.Summary()
comph.Report()
comph.GetKsProb()
comph.GetFrProb()
comph.Draw()

Maintenance
This form of validation is based on products and new products will need to be added at times. It is probably easiest to start with an existing product. Each product has a code file.

cd Validation/inc
cat ValSimParticle.hh | sed 's/SimParticle/NewProduct/g' > ValNewProduct.hh
cd ../src
cat ValSimParticle.cc | sed 's/SimParticle/NewProduct/g' > ValNewProduct.cc

and edit in new histograms and variables. The new product will have to be included in Validation/src/Validation_module.cc