GitDirExample
Jump to navigation
Jump to search
Validation This package is the center of offline validation in mu2e. It is used in nightly code validation and in various debugging efforts. It is also useful in test such as comparing the code on different platforms. The module and tools may be a convenient way for a user to get an overview of how a job is functioning. The comparison tools may be useful for a user to compare two histogram files. Eventually the module will probably be brought into standard production workflows. It is not decided if any online monitoring will be here. This package contains components: Validation - a module that makes standard histograms of products in a file fcl to control Validation code to make histograms a set of classes that make comparing histograms (and profiles, etc) simple. These can be loaded directly into an interactive root session. These are in theinc and root subdirectories. an executable that uses the comparison tools to compare two files of histograms and make reports in various ways, including a web page It is useful to know how the histogram comparisons are done. The first histogram file is take as the standard. In plots it will be the gray histogram. The second file is taken as the file to be tested and appears as red dots. Two criteria are used: a standard root KS test, and a fraction test. When comparing histograms, underflow and overflow bins are included by default, but you can switch this off. The fraction test is done by integrating through each histogram and at each step compute the difference in the two sums and save the largest difference found, then divide by the total entries in the standard histogram and subtract from one (so when they are similar, the result is near 1). When statistics get very small, the KS test fails but the fraction test still is useful. When statistics are very high, and there is any tiny systematic variation (like comparing two data runs) the KS test will fail. In this case the fraction test will tell you what you want to know. Very generally, if a comparison passes one of the two tests, it is "OK". There are two comparison levels, a loose and tight, and also two modes. The two most common cases for comparison are either the files are supposed to be identical (like in nightly validation) or statistically independent (like comparing two simulation files which had different random seeds). If the files were supposed to be identical, the alarm levels for the tests are 0.999 (tight) and 0.99 (loose). If the files are independent, they are set at 0.01 (tight) and 0.001 (loose). The normalization of the two files can be scaled. During build, this package makes: bin/valCompare - the comparison exe lib/libmu2e_Validation_Validation_module.so - the module lib/libmu2e_Validation_root.so - this can be loaded into a root session lib/libmu2e_Validation_root_dict_rdict.pcm lib/libmu2e_Validation_root_dict.rootmap lib/libmu2e_Validation.so Validation module The module can be run by itself on a file: mu2e -f Validation/fcl/val.fcl -s inputfile This will produce validation.root. This will contain a Validation subdirectory and in there will be more directories, one for each art product the module could find and analyze. The directories are named by the product name. If there are many copies of products, such as StepPointMCs, then each copy will get its own directory and set of histograms. The module can also be run as part of a path: services : { # request standard geometry and conditions services # request TFileService } # setup module physics :{ analyzers: { Validation : { module_type : Validation validation_level : 1 } } } # put Validation in a path.. Using the module as part of your path will let you see all the products in the event, even ones that will get dropped on output. The validation level will control how in-depth the histogramming will go. So far, only level 1 is implemented. This is intended to be quick and robust, just histogramming a few variables from each product. When level 2 is implmented, it might histogram derived quantities or quantities derived from multiple products, or make cuts. The following products are analyzed: CaloCluster CaloHit GenParticle SimParticle StepPointMC StrawHit TrackClusterMatch TrackSummary valCompare This is in the path for the release. It operates on two file of histograms. It will only compare histograms that are in the same directories and with the same names, and ignore anything else. valCompare -h lists all options valCompare -s file1.root file2.root produces a summary of how many histograms compared valCompare -r file1.root file2.root produces a line for each histogram and how it compared valCompare -w dir/result.html file1.root file2.root produces the named web page and overlay plots for each histogram also in that dir histogram comparison classes These can be used in root: root [0] .L lib/libmu2e_Validation_root.so TValCompare c c.SetFile1("file1.root") c.SetFile2("file2.root") c.Analyze() c.Summary() c.Browse() They can also be used in code. The histograms are compared with TValHistH, TProfile with TValHistP and TEfficiency with TValHistE #include "Validation/inc/root/TValHistH.h" TH1D* h1 = new TH1D... TH1D* h2 = new TH1D... TValHistH comph() comph.SetHist1(h1) comph.SetHist2(h2) comph.Analyze() comph.Summary() comph.Report() comph.GetKsProb() comph.GetFrProb() comph.Draw() Maintenance This form of validation is based on products and new products will need to be added at times. It is probably easiest to start with an existing product. Each product has a code file. cd Validation/inc cat ValSimParticle.hh | sed 's/SimParticle/NewProduct/g' > ValNewProduct.hh cd ../src cat ValSimParticle.cc | sed 's/SimParticle/NewProduct/g' > ValNewProduct.cc and edit in new histograms and variables. The new product will have to be included in Validation/src/Validation_module.cc