Difference between revisions of "ConditionsData"

Revision as of 16:29, 5 August 2020

Introduction

In Mu2e it will be necessary to maintain a database of calibration constants, also known as conditions data. This will include information like the alignment of the detector, the gas gain in straws, the time-space relationship for the straws, the gain of SiPMs, the calibration curve for ADC's and so on.

The current system is set of two postgres databases, mu2e_conditions_dev, for development by experts and mu2e_conditions_prd for users. The database infrastructure is maintained by the database group of the computing division. A user can access data by configuring a service in their art job, which reads the database using an http protocol. The system is intended to support access from high-volume grid jobs. In addition, there is a mu2e bin called dbTool which can be used for dump database contents, or maintaining database contents.

The user configures an art executable by pointing to a particular set of calibration in the database. In code, the user can access database tables directly by the DbService, but most access will be through a high-level use-friendly layer called the Proditions service. Proditions allows deriving and caching quantities computed from many database tables and other Proditions quantities, while triggering quantity updates as the underlying tables change with the run number.

This page is the right place to start to understand the system, then you can also see:

ConditionsMaintenance for calibrators and calibration manager
ConditionsDbCode, ConditionsDbSchema and ProditionsCode for developers

Using the conditions database

selecting a calibration set

When accessing the conditions database, you select a purpose, such as "PRODUCTION" and version, such a "V1_1". These are entered in the DbService services stanza.

services : {
   DbService : {
      purpose :  PRODUCTION
      version :  v1_1
      dbName : "mu2e_conditions_prd"
      textFile : ["table.txt"]
      verbose : 1
   }
}

The version numbers may have up to three fields like v1_2_3:

1=major version number, this changes when there is a major change in the content of the calibration set, that a user should probably be aware of, such as going from unaligned data to aligned data. The modules and services that will run and their configuration are likely to change. Your physics results will probably change.
2=minor version number. This changes when the list of table types changes, or if there was a repair to the calibration set. The modules and services that will run and their configuration might change. Your physics results might change.
3=extension number. This changes when new runs are added to the calibration set - physics results do not change, but run which previously failed to run because they had no calibrations, will now run.

You always want to explicitly provide a purpose, but if you do not, the code will assume "PRODUCTION". If you have no interest in the version number, then you can leave it blank and the code will take the highest version available. If you specify only the major version number, then the highest available minor version number will be used. Your results might change between runs of your exe. If you provide the major and minor version numbers, then any run numbers that successfully ran before will always run the same way, but runs that previously failed because no calibrations were available might succeed at a later time due to an extension of the calibration set. This (specifying major and minor) is probably the right approach for most user work. Finally, if you specify the major, minor and extension number, then you will get the exact same result every time.

The database parameter should usually be mu2e_conditions_prd, and this is the default.

The table parameter allows you to add a table to the calibration set for testing, see below.

verbose set to 0 will give no output, set to 1 is intended to give interesting brief reports, and can be set up to 10.

You can see the available purposes with

dbTool print-purposes

and you can see what versions are available with

dbTool print-versions

overriding a table with a local file

The conditions code allows a user to provide a file (or several files) in the DbService fcl configuration, with the effect that the file content overrides or extends whatever is in the database. This is intended to make it easy to test new or updated tables. The table must be defined in the code in order to be included in the text file, but no database entries are needed before it can be used. The text file must be in a specific format.

access by dbTool

All table data can be dumped by the command line tool dbTool. For example, dumping a data table in canonical format:

 > dbTool print-table --name TstCalib1 --pretty
TABLE TstCalib1
#  cid 1
# channel   flag   dtoe   
        0,    12,  1.11
        1,    13,  2.11
        2,    11,  3.11
...

It can also tell you about what sets of calibrations are available

 > dbTool print-versions
VID       purpose     LID   major  minor          user              time                     comment
  1           TEST    1      1      0              rlc  2018-10-12 08:58:26.794692-05:00  initial version
  2          EMPTY    2      1      0              rlc  2018-10-12 08:58:26.798556-05:00  initial version
  3       TRK_TEST    3      1      0              rlc  2018-10-12 08:59:41.575732-05:00  first test of trk tables

this tool has built-in help.

dbTool is also the primary method to upload new data to the database.

Access in Modules

Once the DbService is configured with a calibration set, all the data contained that set is available to any module. Since the interval of validity is not known, the services will not return any values until the file is being read and run and events numbers are defined. The intent is that only a few use cases will access the database tables directly. Most uses will access data in user-friendly containers provided by the high-level ProditionsServices. Proditions allows the creation and caching of entities (collections of numbers, classes) derived from multiple database tables and other Proditions entities. An example is a Proditions entity holding the aligned geometry might be made from several tables of alignment values. Proditions will create and cache the aligned geometry, and know when to update it if any of the underlying tables changed, as you process new run numbers. Proditions hides the low-level dependencies from the user and optimizes the creation, updating and caching. Another example is a straw model entity made from both database tables and other Prodition entities representing straw calibration or straw electronics conditions.

Accessing ProditionsService Contents

The recommended access pattern is to make the handle a module class member. You must call "get" on the handle every event to make sure it is up to date. This method will return a handle to correct version of the entity (here "Tracker", the tracker geometry class).

// in the module cc file
#include "TrackerGeom/inc/Tracker.hh"
#include "ProditionsService/inc/ProditionsHandle.hh"

// in the class defintion

 ProditionsHandle<Tracker> _alignedTracker_h;

// at the top of produce(art::Event& event) or analyze(art::Event& event)
 Tracker const& tracker = _alignedTracker_h.get(event.id());
// use tracker here...

// pass tracker to utility routines:
   auto v = strawPostion(tracker);

Important Notes:

The recommended practice is hold ProditionsHandles as member data of your module class and to default construct them in the c'tor. Classes and functions that are not modules should never use a ProditionsHandle. (Proditions service can be accessed in other services, but the creation of the services must follow a set pattern, and the actual Proditions handles creation may need to be delayed.)
When a module member function needs a proditions entity it should get a const& to the entity from the handle at the start of the module member function ( eg produce, analyze, filter, beginSubRun ... ). Hold the const& to the entity as a function-local variable or as member data of a function local struct; never hold the const& to the entity in a way that persists to the next event since this will prevent the entity being updated when appropriate
When a non-module function needs an entity, that function should receive the entity as a const& argument from the calling module.
A side effect of 3) is that some classes/functions called from module member functions will receive the entity as an argument for the sole purpose of passing it on to lower level classes/functions. This is OK. If you have many entities that need to be passed this way the recommended practice is to collect them into a small struct and pass the struct by const reference; this is slightly more efficient than passing them individually and enhances readability. The small struct should be function-local data of the module member function.
Some Proditions quantities (such as alignment) will only be uploaded to the database at run or subrun boundaries, so it would seem to make sense to access the proditions entity in the beginRun or beginSubrun methods. While this is possible, we recommend against it because the entity update freqeuncy may be increased without the coder knowing, and, secondly, in highly-skimmed datasets, you will be updating the entity for subruns where there are no events. If you only update in the event, it is optimal in all cases. (If no update is needed, the call is very low cost, and by only updating in the event call, you never update unecessarily.)

Accessing DbService tables

Most uses will not employ this pattern where the database table are accessed directly. Most uses will employ the Proditions service explained above.

The code pattern is to create a handle to the table (TstCalib1 in this example) as a data member in the module

#include "DbService/inc/DbHandle.hh"
#include "DbTables/inc/TstCalib1.hh"
namespace mu2e {
  class DbServiceTest : public art::EDAnalyzer {
  private:
    mu2e::DbHandle<mu2e::TstCalib1> _testCalib1;
  };
};

in the event method, the user must update the handle with the run number:

  void DbServiceTest::analyze(const art::Event& event) {
    auto const& myTable = _testCalib1.get(event.id());
  };

once the user has the table filled with the correct content for this event, the access can be by several methods. At this point it is important to check with experts on how the table is intended to be used. There may be a channel or index column which has a certain meaning by convention. There may or may not be random access by maps. The number of rows may be guaranteed fixed or variable. Dead channels might have flag values, etc. Some examples of access are:

  int n = 0;
  for(auto const& r : myTable.rows()) {
     std::cout << "row " << n << " is channel "<<r.channel()<<" and has DtoE "<<r.dtoe()<<std::endl;
     n++;
  }

  int channel = 1;
  std::cout << "DtoE for channel "<<channel<<" is "<<myTable.row(channel).dtoe()<<std::endl;

  int index = 1;
  std::cout << "DtoE for row "<<index<<" is "<<myTable.rowAt(index).dtoe()<<std::endl;

Conventions

Intervals of validity

Intervals are inclusive, the end points stated are in the interval. You can't create an interval where the end is before the beginning, so all intervals contain at least one subrun.

String	Interpreted
EMPTY	0:0-0:0
MAX	0:0-999999:999999
ALL	0:0-999999:999999
1000	1000:0-1000:999999
1000-1000	1000:0-1000:999999
1000-MAX	1000:0-999999:999999
MIN-1000	0:0-1000:999999
MIN-MAX	0:0-999999:999999
1000-2000	1000:0-2000:999999
1000:10-2000	1000:10-2000:999999
1000:11-1001:23	1000:11-1001:23

text file format

The text file must have the following format:

TABLE <tableName> <IOV>
row1-col1, row1-col2, row1-col3
row2-col1, row2-col2, row2-col3
...

For example:

# my comment
TABLE TstCalib1 1001:2-1002
1,20,20.21
2,21,20.22
3,22,20.23

The table name is the same as the c++ class name. The IOV may be missing, in which case the table applies to all data. You can see the format of the IOV text here. The data is one line for each row in the table, typically a channel. The rows may need to be in a specific order, depending on how the table is coded and accessed. The columns are separated by commas, and in the order defined by the c++ representation of the table. Please see here for details on string columns. There may be several tables in one file.

strings

Arbitrary string input occurs at two places

when adding comments to a dbTool create action, such as creating a new purpose or version
when uploading a calibration table that has a string column

In general, there are three special characters to watch our for: double quotes, used for quoting ("), comma, used for column separation (,), and hash, used for comments (#).

When inputing a string to dbTool to include a comment, there are only two rules:

if more than one word, use double quotes
when using a double quote in a string, escape it

When writing a file that contain calibration data, the following rules apply

comments may be included by writing the hash (#) as the first character of a line. Comments are not allowed embedded in lines with data. The hash may be used in a string column.

# illegal comment - will crash on parse (part of number column)

# malformed comment - will appear in string column

OK, legal to use hash in string

commas must be quoted

will crash on parse

OK

embedded double quotes must escaped or doubled

OK

OK, multiple words OK (as long as no commas or quotes)

OK, single quotes OK

OK

OK, comma requires quotes

OK