Validation

Validation tools

When you setup any modern Offline release, you have the validation tools in your path. There are two main tools. The first is a module to make a standard set of histograms. The second is a executable to compare two sets of histogram files.

Validation module

The module can be run by itself on a file:

mu2e -f Validation/fcl/val.fcl -s inputfile

This will produce validation.root (or whatever you specify with -T). This will contain a Validation directory and in there will be more directories, one for each art product the module could find and analyze. The directories are named by the product name. If there are many instances of products, such as StepPointMCs, then each copy will get its own directory and set of histograms. The module can also be run as part of a path:

services : {
# request standard geometry and conditions services
# request TFileService
}
# setup module
physics :{
  analyzers: {
    Validation : {
      module_type : Validation
      validation_level : 1
    }
  }
}
# put Validation in a path..

Using the module as part of your path will let you see all the products in the event, even ones that will get dropped on output. The validation level will control how in-depth the histogramming will go. So far, only level 1 is implemented. This is intended to be quick and robust, just histogramming a few variables from each product. When level 2 is implemented, it might histogram derived quantities or quantities derived from multiple products, or make cuts. The following products are analyzed:

• CaloCluster
• CaloCrystalHit
• CaloDigi
• CaloRecoDigi
• GenParticle
• SimParticle
• StepPointMC
• StrawHit
• TrackClusterMatch
• TrackSummary

valCompare

valCompare is an executable that uses the custom code to compare two files of histograms and make reports in various ways, including a web page.

The executable is run with arguments of two histogram filespecs. The first histogram file is take as the standard and in plots it will be the gray histogram. The second file is taken as the file to be tested and appears as red dots. It will only compare histograms that are in the same directories and with the same names, and ignore anything else, such as ntuples.

Two criteria are used: a standard root KS test, and a fraction test. When comparing histograms, underflow and overflow bins are included by default, but you can switch this off. The fraction test is done by integrating through each histogram and at each step compute the difference in the two sums and save the largest difference found, then divide by the total entries in the standard histogram and subtract from one (so when they are similar, the result is near 1). When statistics get very small, the KS test fails but the fraction test still is useful. When statistics are very high, and there is any tiny systematic variation (like comparing two data runs) the KS test will fail. In this case the fraction test will tell you what you want to know. Very generally, if a comparison passes one of the two tests, it is "OK".

There are two comparison levels, a loose and tight, and also two modes. The two most common cases for comparison are either the files are supposed to be identical (like in nightly validation) or statistically independent (like comparing two simulation files which had different random seeds). If the files were supposed to be identical, the alarm levels for the tests are <0.999 (tight) and <0.99 (loose). If the files are independent, they are set at <0.01 (tight) and <0.001 (loose). The normalization of the two files can be scaled.

valCompare options:

valCompare -h
    lists all options
valCompare -s file1.root file2.root
    produces a summary of the histograms compared
valCompare -r file1.root file2.root
    produces a line for each histogram and how it compared
valCompare -w dir/result.html file1.root file2.root
    produces the named web page and overlay plots for each histogram also in that dir

The core histogram comparison classes an can also be used in root:

root [0] .L lib/libmu2e_Validation_root.so
TValCompare c
c.SetFile1("file1.root")
c.SetFile2("file2.root")
c.Analyze()
c.Summary()
c.Browse()

They can also be used in code. The histograms are compared with TValHistH, TProfile with TValHistP and TEfficiency with TValHistE

#include "Validation/inc/root/TValHistH.h"
TH1D* h1 = new TH1D...
TH1D* h2 = new TH1D...
TValHistH comph()
comph.SetHist1(h1)
comph.SetHist2(h2)
comph.Analyze()
comph.Summary()
comph.Report()
comph.GetKsProb()
comph.GetFrProb()
comph.Draw()