TrkQual: Difference between revisions

From Mu2eWiki
Jump to navigation Jump to search
(Creating TrkQual page)
 
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Under Construction =
= Under Construction =
This will be filled out soon!
This page is currently being written so please excuse any mistakes, errors or omissions.


= Training TrkQual =  
= Introduction =
For a worked example see [[TrkAna_Tutorial_Session#Exercise_3:_Retraining_TrkQual|this TrkAna tutorial exercise]]
TrkQual is an artificial neural network (ANN) that evaluates the reconstruction quality of tracks. It uses certain reconstructed parameters of the track (e.g. number of hits) to separate "good" tracks from "bad" tracks and is trained on a simulation sample.
 
There are two classes of users:
* '''analyzers''': people who use the result of TrkQual in their analyses; and,
* '''trainers''': people who train and calibrate the TrkQual ANN.
 
This wiki page tries to guide both groups to accomplish the tasks they need. If this wiki page is lacking (or out of date), then additional help can be found in the usual places (hypernews, Slack, DocDB etc.).
 
= For Everyone =
There may be a few different trainings of TrkQual available. Questions to ask:
* what simulation sample was it trained on?
* what input variables are used?
* what definition of good quality and bad quality was used?
* has the output been calibrated?
 
Calibration
 
= For Analyzers... =
== ...Using A TrkAna Tree ==
If you are using a [[TrkAna]] tree for your analysis, then there are two useful leaves that you can apply cuts to:
* '''dequal.TrkQual'''
* '''dequal.TrkQualCalib'''
which are the uncalibrated and calibrated output values respectively. Note that it is probably best to use the calibrated value, since the effect of this ''should'' stay the same even if the underlying training changes.
 
If you have configured the TrackAnalysisReco module correctly, then you may also have the branch:
* '''detrkqual'''
which contains all the input variables that were available to be trained on. Note that it is possible for some trainings to "comment-out" input variables and so it is best not to assume that ''all'' the input variables seen in this branch were used...
 
== ...Using An Art Module... ==
If you are using an art module for your analysis, then there are two data products you can use:
* '''RecoQual''', which contains the uncalibrated and calibrated output values; or,
* '''TrkQual''', which contains the values of the input variables and the uncalibrated output value.
 
Note that <code>RecoQual</code> is a general data product that can exist for other ANNs too (e.g. [[TrkPID]])
 
=== ...And RecoQual Collection ===
You can grab the <code>RecoQualCollection</code> like so:
<nowiki>auto recoQualCollsH = event.getValidHandle<RecoQualCollection>(_inputTag);
</nowiki>
 
where <code>_inputTag</code> is the <code>art::InputTag</code> for the module that produced the <code>RecoQualCollection</code> (e.g. <code>TrkQualDeM</code>).
 
You can then get the output value for each RecoQual object like this:
<nowiki>for(const auto& i_recoQual : *recoQualCollsH) {
  std::cout << "TrkQual Uncalibrated Output = " << i_trkQual._value << std::endl;
  std::cout << "TrkQual Calibrated Output = " << i_trkQual._calib << std::endl;
}
</nowiki>
 
=== ...And TrkQualCollection ===
You can grab the <code>TrkQualCollection</code> like so:
<nowiki>auto trkQualCollsH = event.getValidHandle<TrkQualCollection>(_inputTag);
</nowiki>
 
where <code>_inputTag</code> is the <code>art::InputTag</code> for the module that produced the <code>TrkQualCollection</code> (e.g. <code>TrkQualDeM</code>).
 
You can then get the output value for each TrkQual object like this:
<nowiki>for(const auto& i_trkQual : *trkQualCollsH) {
  std::cout << "TrkQual Output = " << i_trkQual.MVAOutput() << std::endl;
}
</nowiki>
Note that this is the ''uncalibrated'' output value.
 
= For Trainers =
TMVA has a nice GUI that you can use to check input variable distributions, output variable distributions, ROC curves etc. After you have performed a training, a ROOT file should be created (e.g. TrkQual.root), that you can open in the GUI like so:
> root -l
> TMVA::TMVAGui("TrkQual.root")
 
We also have a worked example in [[TrkAna_Tutorial_Session#Exercise_3:_Retraining_TrkQual|this TrkAna tutorial exercise]]
== ...Who Are Retraining TrkQual ==
Run the script AnalysisUtilities/scripts/RunTrainTrkQual.C. This contains file and tree names that you may want to change.
 
== ...Who Are Training a New TrkQual and... ==
=== ...Are Removing Variables ===
Can simply comment them out in AnalysisUtilities/scripts/TrainTrkQual.C (search for "AddVariable"). The Offline infrastructure handles this behaviour easily.
 
=== ...Are Adding Variables ===
If just want to test a new variable (or combination of variables) that already exist in TrkAna, you can add it to AnalysisUtilities/scripts/TrainTrkQual.C (search for "AddVariable")
 
If this variable becomes permanent, then it needs to:
* be added to the TrkQual data product (RecoDataProducts/inc/TrkQual.hh) (including enum and name);
* filled into the TrkQual object when it is created in TrkDiag/src/TrackQuality_module.cc (e.g. <code>trkqual[TrkQual::new_var] = x;</code>)
 
=== ...Are Changing Other Parameters ===
Plenty of parameters you might want to change in AnalysisUtilities/scripts/TrainTrkQual.C, including:
* signal and background definitions (search for "TCut signal" and "TCut bkg")
* background weight expression (search for "BackgroundWeightExpression")
 
== ...Who Are Calibrating ==
Run the script AnalysisUtilities/scripts/RunCalibTrkQual.C
 
This adds an XML block to the XML file. If this already exists the script won't run, so delete it from the XML file, if you need

Latest revision as of 18:40, 14 May 2020

Under Construction

This page is currently being written so please excuse any mistakes, errors or omissions.

Introduction

TrkQual is an artificial neural network (ANN) that evaluates the reconstruction quality of tracks. It uses certain reconstructed parameters of the track (e.g. number of hits) to separate "good" tracks from "bad" tracks and is trained on a simulation sample.

There are two classes of users:

  • analyzers: people who use the result of TrkQual in their analyses; and,
  • trainers: people who train and calibrate the TrkQual ANN.

This wiki page tries to guide both groups to accomplish the tasks they need. If this wiki page is lacking (or out of date), then additional help can be found in the usual places (hypernews, Slack, DocDB etc.).

For Everyone

There may be a few different trainings of TrkQual available. Questions to ask:

  • what simulation sample was it trained on?
  • what input variables are used?
  • what definition of good quality and bad quality was used?
  • has the output been calibrated?

Calibration

For Analyzers...

...Using A TrkAna Tree

If you are using a TrkAna tree for your analysis, then there are two useful leaves that you can apply cuts to:

  • dequal.TrkQual
  • dequal.TrkQualCalib

which are the uncalibrated and calibrated output values respectively. Note that it is probably best to use the calibrated value, since the effect of this should stay the same even if the underlying training changes.

If you have configured the TrackAnalysisReco module correctly, then you may also have the branch:

  • detrkqual

which contains all the input variables that were available to be trained on. Note that it is possible for some trainings to "comment-out" input variables and so it is best not to assume that all the input variables seen in this branch were used...

...Using An Art Module...

If you are using an art module for your analysis, then there are two data products you can use:

  • RecoQual, which contains the uncalibrated and calibrated output values; or,
  • TrkQual, which contains the values of the input variables and the uncalibrated output value.

Note that RecoQual is a general data product that can exist for other ANNs too (e.g. TrkPID)

...And RecoQual Collection

You can grab the RecoQualCollection like so:

auto recoQualCollsH = event.getValidHandle<RecoQualCollection>(_inputTag);
 

where _inputTag is the art::InputTag for the module that produced the RecoQualCollection (e.g. TrkQualDeM).

You can then get the output value for each RecoQual object like this:

for(const auto& i_recoQual : *recoQualCollsH) {
  std::cout << "TrkQual Uncalibrated Output = " << i_trkQual._value << std::endl;
  std::cout << "TrkQual Calibrated Output = " << i_trkQual._calib << std::endl;
}
 

...And TrkQualCollection

You can grab the TrkQualCollection like so:

auto trkQualCollsH = event.getValidHandle<TrkQualCollection>(_inputTag);
 

where _inputTag is the art::InputTag for the module that produced the TrkQualCollection (e.g. TrkQualDeM).

You can then get the output value for each TrkQual object like this:

for(const auto& i_trkQual : *trkQualCollsH) {
  std::cout << "TrkQual Output = " << i_trkQual.MVAOutput() << std::endl;
}
 

Note that this is the uncalibrated output value.

For Trainers

TMVA has a nice GUI that you can use to check input variable distributions, output variable distributions, ROC curves etc. After you have performed a training, a ROOT file should be created (e.g. TrkQual.root), that you can open in the GUI like so:

> root -l
> TMVA::TMVAGui("TrkQual.root")

We also have a worked example in this TrkAna tutorial exercise

...Who Are Retraining TrkQual

Run the script AnalysisUtilities/scripts/RunTrainTrkQual.C. This contains file and tree names that you may want to change.

...Who Are Training a New TrkQual and...

...Are Removing Variables

Can simply comment them out in AnalysisUtilities/scripts/TrainTrkQual.C (search for "AddVariable"). The Offline infrastructure handles this behaviour easily.

...Are Adding Variables

If just want to test a new variable (or combination of variables) that already exist in TrkAna, you can add it to AnalysisUtilities/scripts/TrainTrkQual.C (search for "AddVariable")

If this variable becomes permanent, then it needs to:

  • be added to the TrkQual data product (RecoDataProducts/inc/TrkQual.hh) (including enum and name);
  • filled into the TrkQual object when it is created in TrkDiag/src/TrackQuality_module.cc (e.g. trkqual[TrkQual::new_var] = x;)

...Are Changing Other Parameters

Plenty of parameters you might want to change in AnalysisUtilities/scripts/TrainTrkQual.C, including:

  • signal and background definitions (search for "TCut signal" and "TCut bkg")
  • background weight expression (search for "BackgroundWeightExpression")

...Who Are Calibrating

Run the script AnalysisUtilities/scripts/RunCalibTrkQual.C

This adds an XML block to the XML file. If this already exists the script won't run, so delete it from the XML file, if you need