Computing Concepts: Difference between revisions
No edit summary |
No edit summary |
||
| Line 19: | Line 19: | ||
==Code== | ==Code== | ||
===Events=== | ===Events=== | ||
Very briefly, the experiment is driven by a short burst of millions of protons hitting the primary target every 1695 ns. This cycle is called a microbunch. After the burst interacts, muons migrate to, and come to rest in the stopping target. After this surge of particles dies down during the first part of the microbunch, the detector observes what happens to the stopped muons during the second part of | Very briefly, the experiment is driven by a short burst of millions of protons hitting the primary target every 1695 ns. This cycle is called a '''microbunch'''. After the burst interacts, outgoing muons migrate to, and come to rest in, the stopping target. After this surge of particles dies down during the first part of the microbunch, the detector observes what happens to the stopped muons during the second part of the microbunch. The data recorded during this ~900 ns is written out (if it passes the trigger) in a data structure called an event. Many events can be written in one file. Events have unique identifying numbers. Short periods of data-taking (minutes) are grouped into subruns with a unique ID number, and longer periods of stable running (a few hours) will be grouped into a run with unique ID number. | ||
===Sim and Reco=== | |||
Since we do not have data yet, we analyze '''simulation''' or '''sim''' events which are based on our expectation of what data will look like. We draw randomly from known physics processes and then trace the particles through the detector, and write out events. The interaction of particles in the detector materials is simulated with the [[geant]] software package. The simulated looks like the real data will, except it also contains the truth of what happened in the interactions. | |||
The output of simulation would typically be data events in the '''raw''' formats that we will see produced by the detector readout. These are typically ADC values indicating energy deposited, or TDC values indicating the time of a energy deposit. In the '''reconstruction''' or '''reco''' process, we run this raw data through modules that analyze the raw data and look for patterns that can be identified as evidence of particular particles. For example, hits in the tracker are reconstructed into individual particle paths, and energy in the calorimeter crystals is clustered into showers caused by individual electrons. | |||
Exactly how a reconstruction module does its work is called its '''algorithm'''. A lot of the work of physicists is invested in these algorithms, because they are fundamental to the quality of the experimental results. | |||
===Coding=== | ===Coding=== | ||
The mu2e simulation and reconstruction code is written in c++. We write modules which create data, or read data out of the event, process it, and write the results back into the event. The modules plug into a framework called '''art''', and this framework calls the modules to do the actual work, as the framework reads | The mu2e simulation and reconstruction code is written in c++. We write modules which create data, or read data out of the event, process it, and write the results back into the event. The modules plug into a framework called '''art''', and this framework calls the modules to do the actual work, as the framework reads an input file and writes an output file. The primary data format is determined by the framework, so it is called the '''art format''' and the file will have an extension .art. | ||
We use the '''git''' code management system to store and track our code. Currently, we have one main git repository which contains all our simulation and reconstruction code. You can check out this repository, or a piece of it, and build it locally. In this local area you can make changes to code and read, write and analyze small amounts of data. We build the code with a make system called '''scons'''. The code may be built optimized ('''prof''') or non-optimized and prepared for running a debugger ('''debug'''). | |||
At certain times, the code is tagged, built and published as a stable release. These releases are available on the /cvmfs disk area. cmfvs is a sophisticated distributed disk system with layers of servers and caches, but to us it just looks like a local disk, which can be mounted almost anywhere. We often run large projects off of these fixed releases. cmvfs is mounted on the interactive nodes, at remote institutions, on some desktops, and all the many farm nodes we use. | |||
===Executables=== | |||
Which modules are run and how they are configured is determined by a control file, written in '''fcl''' (pronounced fickle). This control file can change the random seeds for the simulation and the input and output file names, for example. A typical run might be to create a new simulation file. For various reasons, we often do our simulation in several stages, writing out a file between each run of the executable, or stage, and reading it in to start the next stage. A second type of job might be to run one of the simulation stages with a variation of the detector design, for example. Another typical run might be to take a simulation file as input and test various reconstruction algorithms, and write out reconstruction results. | |||
=== | ===Data products=== | ||
The data in an event in a file is organized into data products. Examples of data products include straw tracker hits, tracks, or clusters in the calorimeter. The fcl is often used to decide which data products to read, which one to make, and which ones to write out. There are data products which contain the information of what happened during the simulation, such as the main particle list, SimParticles. | |||
===Products=== | ===UPS Products=== | ||
Disambiguation of "products" - please note that we have both '''data products''' and '''UPS products''' which unfortunately are both referred to as "products" at times. Please be aware of the difference, which you can usually determine from the context. | |||
The art framework and fcl control language are provided as a '''product''' inside the '''UPS''' software release management system. There are several other important UPS products we use. This software is distributed as UPS products because many experiments at the lab use these utilities. You can control which UPS products are available to you ( which you can recognize as a setup command like "setup root v6_06_08") but most of this is organized as defaults inside of a setup script. | |||
The art framework and fcl control language are provided as a ''' | |||
===Histogramming=== | ===Histogramming=== | ||
Once you have an art file, how to actually make plots and histograms of the data? There are many ways to do this, so it is important to consult with the people you work with, and make sure you are working in a style that is consistent with their expertise and preferences, so you can work together effectively. | Once you have an art file, how to actually make plots and histograms of the data? There are many ways to do this, so it is important to consult with the people you work with, and make sure you are working in a style that is consistent with their expertise and preferences, so you can work together effectively. | ||
In any case, we always use the '''root''' | In any case, we always use the '''root''' UPS product for making and viewing histograms. | ||
There are two main ways to approach it. The first is to insert the histogram code into a module and write out a file which contains the histograms. The second method is to use a module to write out an '''ntuple''', also called a '''tree'''. This is a summary of the data in each event, so instead of writing out the whole track product, you might just write out the momentum and the number of hits in the nutple. The ntuple is very compact, so you can easily open this and make histogram interactively very quickly. | There are two main ways to approach it. The first is to insert the histogram code into a module and write out a file which contains the histograms. The second method is to use a module to write out an '''ntuple''', also called a '''tree'''. This is a summary of the data in each event, so instead of writing out the whole track data product, you might just write out the momentum and the number of hits in the nutple. The ntuple is very compact, so you can easily open this and make histogram interactively very quickly. | ||
==Workflows== | ==Workflows== | ||
===Designing larger jobs=== | ===Designing larger jobs=== | ||
After understanding data on a small level by running interactive jobs, you may want to run on larger datasets. If a job is working interactively, it is not too hard to take that workflow and adapt it for running on large datasets on the compute farms. First, you will need to understand the '''mu2egrid''' | After understanding data on a small level by running interactive jobs, you may want to run on larger datasets. If a job is working interactively, it is not too hard to take that workflow and adapt it for running on large datasets on the compute farms. First, you will need to understand the '''mu2egrid''' UPS product which is a set of scripts to help you submit jobs and organize the output. mu2egrid will call the '''jobsub''' UPS product to start your job on the farm. You data will be copied back using the '''ifdh''' UPS product, which is a wrapper to data transfer software. The output will go to '''dCache''', which is a high-capacity and high-throughput distributed disk system. We have 100's of terabytes of disk space here, divided into three types (a scratch area, a persistent disk area, and a tape-backed area). Once the data is written, there are procedures to check it and optionally concatenate the files and write them tape. We track our files in a database that is part of the '''SAM''' UPS product. You can see the files in dCache by looking under the '''/pnfs''' filesystem. Writing and reading files to dCache can have consequences, so please understand how to use dCache and also consult with an experienced user before running a job that uses this disk space. | ||
===Grid resources=== | ===Grid resources=== | ||
mu2e has access to a compute farm at Fermilab, called GPGrid. This is several thousand cores and | mu2e has access to a compute farm at Fermilab, called '''GPGrid'''. This is several thousand cores and mu2e is allocated a portion of the nodes, and we can use more if no one else needs it. Once you have used the interactive machines to build and test your code, you can submit a large job to the compute farms using the [["jobsub"]] system. You can get typically get 1000 nodes for a day before your priority goes down and you get fewer. If the farm is not crowded, which is not uncommon, you can get several times that. | ||
mu2e also has access to compute farms at other institutions through a collaboration call Open Science Grid (OSG). It is easy to modify your submit command to use these resources. We do not have a quota here, so we don't really know how many nodes we can get, but it is usually at least as much as we can get on GPGrid. This system is less reliable than GPgrid so we often see unusual failure modes | mu2e also has access to compute farms at other institutions through a collaboration call Open Science Grid (OSG). It is easy to modify your submit command to use these resources. We do not have a quota here, we can only access idle nodes, so we don't really know how many nodes we can get, but it is usually at least as much as we can get on GPGrid. This system is less reliable than GPgrid so we often see unusual failure modes or jobs restarting. | ||
==Your workflow== | ==Your workflow== | ||
What part of the offline system you | What part of the offline system you will need to be familiar with will depend on what tasks you will be doing. Let's identify three nominal roles. In all cases, you will need the accounts and authentication. | ||
# ntuple user. This is the simplest case. You probably will be given a ntuple, or a simple recipe to make an ntuple, then you will want to analyze the data. You will need to have a good understanding of c++ and root, but not much else. | # ntuple user. This is the simplest case. You probably will be given a ntuple, or a simple recipe to make an ntuple, then you will want to analyze the data. You will need to have a good understanding of c++ and root, but not much else. | ||
# art user. In this level you would be running art executables, so you will also need to understand fcl and products. if you will be doing processing of larger datasets, you will need to understand farms and workflows. | # art user. In this level you would be running art executables, so you will also need to understand fcl and data products. if you will be doing processing of larger datasets, you will need to understand farms and workflows. | ||
# developer. In this case, you will be writing modules, so you need to understand the art framework, products, c++ and standards in some detail. | # developer. In this case, you will be writing modules, so you need to understand the art framework, data products, c++ and standards in some detail. | ||
Revision as of 20:54, 2 February 2017
Introduction
This page is intended for physicists who are just starting to work in the mu2e computing environment. The following is a broad overview of the major components and how they work. There are many links into the rest of the mu2e documentation which is more terse and is intended as a reference once you get through the introductory material. We suggest reading through this page, then take a look at the "your workflow" notes to decide what to go back a dig further into.
Interactive logins
Collaborators can do interactive work in several places. Probably the best place to start is the interactive machines at Fermilab. These are set of virtual machines running SL6 (a variant of linux) . The disks that the user sees are located on specialized disk server hardware and the same disks are mounted by all the interactive machines. There are five machines named mu2egpvm01.fnal.gov through mu2egpvm05.fnal.gov. You will have an account and a home area here (the same area is on all machines) and some space for data. We prefer using the bash shell for all purposes. You can read more about the interactive machines here.
Collaborators can also compile and run mu2e code on their linux desktops or laptops. You can read more about that here. The code should be manageable on Mac's unix layer but we have not pursued this option officially. Finally, several of the collaboration institutions have set up working areas at their home institutions. For these options, someone would need to copy the code distribution and supporting software tools to the local machine, or mount the distributed code disk. You can read more about these options here, but best to discuss this with your group leaders first. Generally we recommend working on the lab interactive nodes unless the networking from your institution makes that prohibitive.
Authentication
You login to the virtual machines by using kerberos authentication. You will need a permanent ID called a kerberos "principal" which is looks like "xyz@FNAL.GOV", where xyz is your username. You will have a password associated with your principal. You will use this principal and password to log into linux desktops located at Fermilab or to ssh into the interactive machines from your home institution. You typically refresh your kerberos authentication every day.
The second identity you will need is the "services" principal, which looks like xyz@services.fnal.gov, or just xyz, and also has a password. You will need this identity to log into Fermilab email, the servicedesk web site and some other services based at the lab. You would typically only use this authentication as you log into the service.
Finally, you will need a CILogin certificate. This cert is the basis of authentication to the mu2e documents database, the computing farms, and other services. You will use this cert in two ways. The first way is to load it into your browser, which then gives you access to web pages and web services. The second is by using your kerberos authentication to access a copy of your certificate maintained in a remote database. You get this certificate once and then renew it only once a year.
Please follow the procedure on the ComputingAccounts page to setup your accounts.
Code
Events
Very briefly, the experiment is driven by a short burst of millions of protons hitting the primary target every 1695 ns. This cycle is called a microbunch. After the burst interacts, outgoing muons migrate to, and come to rest in, the stopping target. After this surge of particles dies down during the first part of the microbunch, the detector observes what happens to the stopped muons during the second part of the microbunch. The data recorded during this ~900 ns is written out (if it passes the trigger) in a data structure called an event. Many events can be written in one file. Events have unique identifying numbers. Short periods of data-taking (minutes) are grouped into subruns with a unique ID number, and longer periods of stable running (a few hours) will be grouped into a run with unique ID number.
Sim and Reco
Since we do not have data yet, we analyze simulation or sim events which are based on our expectation of what data will look like. We draw randomly from known physics processes and then trace the particles through the detector, and write out events. The interaction of particles in the detector materials is simulated with the geant software package. The simulated looks like the real data will, except it also contains the truth of what happened in the interactions.
The output of simulation would typically be data events in the raw formats that we will see produced by the detector readout. These are typically ADC values indicating energy deposited, or TDC values indicating the time of a energy deposit. In the reconstruction or reco process, we run this raw data through modules that analyze the raw data and look for patterns that can be identified as evidence of particular particles. For example, hits in the tracker are reconstructed into individual particle paths, and energy in the calorimeter crystals is clustered into showers caused by individual electrons. Exactly how a reconstruction module does its work is called its algorithm. A lot of the work of physicists is invested in these algorithms, because they are fundamental to the quality of the experimental results.
Coding
The mu2e simulation and reconstruction code is written in c++. We write modules which create data, or read data out of the event, process it, and write the results back into the event. The modules plug into a framework called art, and this framework calls the modules to do the actual work, as the framework reads an input file and writes an output file. The primary data format is determined by the framework, so it is called the art format and the file will have an extension .art.
We use the git code management system to store and track our code. Currently, we have one main git repository which contains all our simulation and reconstruction code. You can check out this repository, or a piece of it, and build it locally. In this local area you can make changes to code and read, write and analyze small amounts of data. We build the code with a make system called scons. The code may be built optimized (prof) or non-optimized and prepared for running a debugger (debug).
At certain times, the code is tagged, built and published as a stable release. These releases are available on the /cvmfs disk area. cmfvs is a sophisticated distributed disk system with layers of servers and caches, but to us it just looks like a local disk, which can be mounted almost anywhere. We often run large projects off of these fixed releases. cmvfs is mounted on the interactive nodes, at remote institutions, on some desktops, and all the many farm nodes we use.
Executables
Which modules are run and how they are configured is determined by a control file, written in fcl (pronounced fickle). This control file can change the random seeds for the simulation and the input and output file names, for example. A typical run might be to create a new simulation file. For various reasons, we often do our simulation in several stages, writing out a file between each run of the executable, or stage, and reading it in to start the next stage. A second type of job might be to run one of the simulation stages with a variation of the detector design, for example. Another typical run might be to take a simulation file as input and test various reconstruction algorithms, and write out reconstruction results.
Data products
The data in an event in a file is organized into data products. Examples of data products include straw tracker hits, tracks, or clusters in the calorimeter. The fcl is often used to decide which data products to read, which one to make, and which ones to write out. There are data products which contain the information of what happened during the simulation, such as the main particle list, SimParticles.
UPS Products
Disambiguation of "products" - please note that we have both data products and UPS products which unfortunately are both referred to as "products" at times. Please be aware of the difference, which you can usually determine from the context.
The art framework and fcl control language are provided as a product inside the UPS software release management system. There are several other important UPS products we use. This software is distributed as UPS products because many experiments at the lab use these utilities. You can control which UPS products are available to you ( which you can recognize as a setup command like "setup root v6_06_08") but most of this is organized as defaults inside of a setup script.
Histogramming
Once you have an art file, how to actually make plots and histograms of the data? There are many ways to do this, so it is important to consult with the people you work with, and make sure you are working in a style that is consistent with their expertise and preferences, so you can work together effectively.
In any case, we always use the root UPS product for making and viewing histograms. There are two main ways to approach it. The first is to insert the histogram code into a module and write out a file which contains the histograms. The second method is to use a module to write out an ntuple, also called a tree. This is a summary of the data in each event, so instead of writing out the whole track data product, you might just write out the momentum and the number of hits in the nutple. The ntuple is very compact, so you can easily open this and make histogram interactively very quickly.
Workflows
Designing larger jobs
After understanding data on a small level by running interactive jobs, you may want to run on larger datasets. If a job is working interactively, it is not too hard to take that workflow and adapt it for running on large datasets on the compute farms. First, you will need to understand the mu2egrid UPS product which is a set of scripts to help you submit jobs and organize the output. mu2egrid will call the jobsub UPS product to start your job on the farm. You data will be copied back using the ifdh UPS product, which is a wrapper to data transfer software. The output will go to dCache, which is a high-capacity and high-throughput distributed disk system. We have 100's of terabytes of disk space here, divided into three types (a scratch area, a persistent disk area, and a tape-backed area). Once the data is written, there are procedures to check it and optionally concatenate the files and write them tape. We track our files in a database that is part of the SAM UPS product. You can see the files in dCache by looking under the /pnfs filesystem. Writing and reading files to dCache can have consequences, so please understand how to use dCache and also consult with an experienced user before running a job that uses this disk space.
Grid resources
mu2e has access to a compute farm at Fermilab, called GPGrid. This is several thousand cores and mu2e is allocated a portion of the nodes, and we can use more if no one else needs it. Once you have used the interactive machines to build and test your code, you can submit a large job to the compute farms using the "jobsub" system. You can get typically get 1000 nodes for a day before your priority goes down and you get fewer. If the farm is not crowded, which is not uncommon, you can get several times that.
mu2e also has access to compute farms at other institutions through a collaboration call Open Science Grid (OSG). It is easy to modify your submit command to use these resources. We do not have a quota here, we can only access idle nodes, so we don't really know how many nodes we can get, but it is usually at least as much as we can get on GPGrid. This system is less reliable than GPgrid so we often see unusual failure modes or jobs restarting.
Your workflow
What part of the offline system you will need to be familiar with will depend on what tasks you will be doing. Let's identify three nominal roles. In all cases, you will need the accounts and authentication.
- ntuple user. This is the simplest case. You probably will be given a ntuple, or a simple recipe to make an ntuple, then you will want to analyze the data. You will need to have a good understanding of c++ and root, but not much else.
- art user. In this level you would be running art executables, so you will also need to understand fcl and data products. if you will be doing processing of larger datasets, you will need to understand farms and workflows.
- developer. In this case, you will be writing modules, so you need to understand the art framework, data products, c++ and standards in some detail.
Tutorials
- Testing the ROOT display
- Testing the Geant4 based event display
- Notes on dynamic libraries
- The First Step: the art workbook
- Running G4 within art: The first examples.
- Mu2e maintained FAQs: C++ FAQ, Unix/Linux FAQ, ROOT FAQ, Geant4 Notes