Tracker Databases

Tracker Databases

There are a few tracker databases hosted at different sites. This wiki page currently focuses on the ones that interact with Fermilab-hosted mu2e_tracker_prd database and uses tools available in the TrackerHardwareDB git repository. If you have any questions or comments, contact Andy Edmonds.

Setting Up

To use the tools described on this wiki page, you need to clone the TrackerHardwareDB git repository. The following instructions are for running on the mu2egpvms but some tools (e.g. the panel QC viewer) could be run from any machine:

ssh -XY username@mu2egpvmXX.fnal.gov
cd /mu2e/app/users/username/
git clone git@github.com:Mu2e/TrackerHardwareDB

To get any code updates, you should periodically do

cd TrackerHardwareDB
git pull

FNAL Planes Database

The database for the plane construction at FNAL. The working version is not hosted on mu2e_tracker_prd but is copied to the imported.fnal_planes database table following the instructions below.

Importing

To copy to imported.fnal_planes:

cd TrackerHardwareDB/python
python3 create_imported_fnal_planesdb.py path/to/fnal/planesdb

This will create two .sql files. You will need the mu2e_tracker_admin database role to then do

cd ../sql
psql -h ifdb08 -p 5459 mu2e_tracker_prd < create_imported_fnal_planesdb.sql
psql -h ifdb08 -p 5459 mu2e_tracker_prd < insert_imported_fnal_planesdb.sql

The plane construction DB can now be viewed using the panel QC viewer below.

Some details:

• the first psql command creates a backup imported.fnal_planes_previous table before clearing the old one
• the second psql command insert all the new data
• both the backup and current tables are needed to extract data for the qc.panels and qc.planes tables

Extracting

The information in imported.fnal_planes needs to be extracted so that it can be inserted into the qc.panels and qc.planes tables.

cd TrackerHardwareDB/python
python3 extract_updates_from_imported_fnal_planesdb.py

This script will compare the previous and current versions of imported.fnal_planes and look for any changes in the contents of the following:

• 'Missing_straws', 'Missing_wires', 'High_currents', 'Blocked_straws', 'Sparks', 'Brooken_wires', 'Loose_omegas', 'Missing_anode_pins', 'Missing_cathode_pins', 'Missing_omegas'

During this process, the script may prompt you for input if it cannot parse an integer when it thinks it should.

Once the script is finished it will have created a run_create_update_qc_panels_table.sh file. This contains commands for the python script described in #Changing missing_straws, high_current_wires, etc.. You can then run:

. ./run_create_update_qc_panels_table.sh

to create the .sql files. These should be checked and then run manually.

Panel QC Database

The panel QC database that is stored in mu2e_tracker_prd

Viewing Database

The panel QC database viewer can be run on any machine connected to the internet with Node.js installed.

Below are instructions to run the viewer on the mu2egpvm machines but in principle it is runnable from anywhere.

Installation

To install the panel QC viewer:

cd TrackerHardwareDB/panel-qc-viewer/
npm install

Running

To run the panel QC viewer, you need to forward a port when you ssh to the mu2egpvm machine:

ssh -L 3000:localhost:3000 -XY username@mu2egpvmXX.fnal.gov
cd TrackerHardwareDB/panel-qc-viewer
npm start

Then you can open a browser and go to localhost:3000 on your local machine

To run on a different port you can type:

 PORT=XXXX npm start

Updating Database

There are a few python scripts that can be used to update the qc.panels table. To set up to run the python scripts you need to do the following:

cd TrackerHardwareDB/python
setup mu2e
muse setup Offline
setup pyana

Then each scripts follows a similar pattern:

1. Run a python scripts to create a .sql file
2. Check the .sql file for any obvious errors
3. Run commands contained in the .sql file. This puts the data in the database

To put data in the database, you need to have the mu2e_tracker_writer database role

Adding a new panel

To add a new panel to the qc.panels table:

cd TrackerHardwareDB/python
python3 create_insert_new_panels.py --panel_ids X Y Z
psql -h ifdb08 -p 5459 mu2e_tracker_prd < insert_new_panels.sql

This will create .sql files to insert new rows for panels X, Y, and Z. Note that X, Y, Z do not need to contain "mn" or "MN" (the script will remove these characters)

Adding a new plane

To add a new plane to the qc.planes table:

cd TrackerHardwareDB/python
python3 create_insert_new_planes.py --plane_ids X Y Z
psql -h ifdb08 -p 5459 mu2e_tracker_prd < insert_new_planes.sql

This will create .sql files to insert new rows for planes X, Y, and Z.

Changing missing_straws, high_current_wires, etc.

cd TrackerHardwareDB/python
python3 create_update_qc_panels_table.py --panel_id XX --{new/add/remove}_{column1} A B C --{new/add/remove}_{column2} D E F
psql -h ifdb08 -p 5459 mu2e_tracker_prd < update_qc_panels_table.sql

will create .sql files to update panel XX.

Some examples:

• This will replace the current contents of missing_straws column for panel 53 with straw numbers 1, 2, 3.

python3 create_update_qc_panels_table.py --panel_id 53 --new_missing_straws 1 2 3

• This will add the straw number 4 to the current list of high_current_wires for panel 76:

python3 create_update_qc_panels_table.py --panel_id 76 --add_high_current_wires 4

• This will remove the straw number 5 from the current list of missing_wires and add the straw numbers 88 and 89 to the current list of loose_omega_pieces for panel 121:

python3 create_update_qc_panels_table.py --panel_id 121 --remove_missing_wires 5 --add_loose_omega_pieces 88 89

Changing panels in a plane

cd TrackerHardwareDB/python
python3 create_update_qc_planes_table.py --plane_id XX --add_panels A B C --remove_panels D E F
psql -h ifdb08 -p 5459 mu2e_tracker_prd < update_qc_planes_table.sql

will create .sql files to update plane XX by adding panels A, B, and C, and removing panels D, E, and F.

Adding earboard test results

There are two things to do:

1. Add the generated pdf file to the repository

cd TrackerHardwareDB/panel-qc-viewer/public/images/earboard/
cp /path/to/images/plot_*.pdf .
git add *.pdf
git commit -m "Adding ear board plots"
git push

2. Update the result in the database with true or false

cd TrackerHardwareDB/python
python3 create_update_qc_panels_table.py --panel_id 121 --earboard true (or false)
psql -h ifdb08 -p 5459 mu2e_tracker_prd < update_qc_panels_table.sql

Adding Fe55 analyzed data

From the panel QC Fe55 analysis script, we get either csv files named like mn269_ch0_r3_delta.csv and mn269_ch1_r3_delta.csv (for cramp type 1) or a file named like mn269_r3_delta.csv (for cramp type 2). The data contained in either case can be put into the database like so:

For cramp type 1:

cd TrackerHardwareDB/python
python3 create_update_maxerf_risetime.py ../data/mn269_ch0_r3_delta.csv ../data/mn269_ch1_r3_delta.csv
psql -h ifdb08 -p 5459 mu2e_tracker_prd < update_maxerf_risetime.sql

For cramp type 2:

cd TrackerHardwareDB/python
python3 create_update_maxerf_risetime.py ../data/mn269_r3_delta.csv
psql -h ifdb08 -p 5459 mu2e_tracker_prd < update_maxerf_risetime.sql

The python script has some internel checks to make sure there are the correct number of channels and no duplicate channel numbers

Adding Fe55 plots

These are just stored in the repository so to add these do:

cd TrackerHardwareDB/panel-qc-viewer/public/images/hv_data/
cp /path/to/images/mn*.png .
git add *.png
git commit -m "Adding hv data plots"
git push

Then for other people to see the new images, they will need to pull in the updates.

Adding raw Fe55 data

We periodically backup the raw HV data csv files. The csv files will be stored alongside a README that contains comments in some directory on trackerpi3. These are too large to store in the database so instead we tarball them up and store them on dCache. Information about the files (both .csv and README files) is then stored in the database tables qc.panel_hv_data_files, which is then available through the viewer.

To steps to back up the data are:

cd TrackerHardwareDB/python
python3 create_insert_hv_data_files.py /path/to/directory1 /path/to/directory2
psql -h ifdb08 -p 5459 mu2e_tracker_prd < insert_hv_data_files.sql
scp *.tbz mu2e@trackerpi3.dhcp.fnal.gov:~/dropbox/upload

The python script does the following:

1. Walks through /path/to/directory and creates a list of files with the name "README" and a list of files with the suffix ".csv"
2. Reads each README file and creates a map of "csv filename" --> "README comment"
3. Loops through the csv files and creates a map of "tar filename" --> "list of csv files it will contain"
   1. files with the same panel number are put in the same tarball
   2. if there are two files with the name (i.e. in different directories), then the script will check if the files are identical. If the files are identical, only one will be tarballed, if they are different, then both will be tarballed
4. Loops through the tar filename map and creates the tarballs
   1. the tarball names have the form mcs.mu2e.PanelQC_HV.Fe55.mnNNN_YYYYMMDD_YYYYMMDD.tbz where NNN is the panel number and the two dates are the earliest and latest date covered by the contained data
5. Writes an sql file for uploading the file information to the database

The last step of copying the tarballs to trackerpi3 lets the RawDataMover (RDM) know to copy them to tape. Once the tarballs can be safely deleted, the RDM moves the files to the delete directory, where we need to manually delete them.

Retrieving raw Fe55 data

We do not store the original raw HV files in the database. Instead, the files are stored on dCache and their location can be retrieved from SAM as follows:

setup dhtools
sam locate-file bck.mu2e.PanelQC_HV.Fe55.XXXXXX.tbz

This will give you the location of the file. You will probably need to prestage the file. Once the file is on disk you can do:

tar -xvf bck.mu2e.PanelQC_HV.Fe55.XXXXXX.tbz

to unzip the tarball in your current directory.

The original README files can be viewed on the hv_readmes page of the viewer (can be accessed from a link on the HV data page). The README files are stored in the database in the qc.panel_hv_readme_files table