MCProdWorkflow
Introduction
This workflow is for production-style simulation jobs. It can be used for stage-1 jobs which start with a generator, or later simulation stages which start with the output files of previous stages. It can also be used for concatenation of a dataset. The output may stay on disk, or the output files can be concatenated and uploaded to tape and properly documented through the SAM database. Uploading is intended for cases where the output needs to be saved for more than a month or so, or that might be used by many collaborators, or needs to be carefully documented. This process may be run out of the mu2epro account for collaboration samples, or a personal account for personal samples.
You will need to plan the project in some detail before starting this production workflow.
- the physics
- the basic fcl to perform the job
- the output dataset names
- the job plan in terms of number and length of jobs, etc.
This page assumes that the user is familiar with the basic infrastructure and its references:
- Simulation, fcl
- file names, file tools, SAM
- grids, dCache, data transfer, enstore
- grid, job planning, monitoring
- prestaging, concatenation, mu2egrid
The basic steps, expanded below are
- prestage input files, if needed
- generate a set of fcl files, one for each job
- copy fcl files to dCache and optionally register the fcl dataset with SAM
- submit jobs
- check output and recover failed jobs
- optionally concatenate output files for each dataset
- generate a set of fcl files
- copy fcl files to dCache and optionally register the fcl dataset with SAM
- submit jobs
- check output and recover failed jobs
- optionally upload output files
- optionally tar and upload log files
The mu2egrid
and related packages provide Mu2e-specific code
required for submitting jobs and manipulating files. Most scripts support the
--help
option. Look for the
--dry-run
and --verbose
options to
show what will be done without performing the action.
Directories
It is useful to have a working area:
/mu2e/data/users/$USER/projects/my_project
and areas for the jobs main fcl
/mu2e/data/users/$USER/projects/project_name/job/fcl
and for each concatenation
/mu2e/data/users/$USER/projects/project_name/output1/fcl
For official collaboration work, the output will go to
/pnfs/mu2e/persistent/users/mu2epro/workflow/project_name/STATUS
for individual's work, the output will go to
/pnfs/mu2e/scratch/users/$USER/workflow/project_name/STATUS
where STATUS is
outstage
for output from grid jobsgood
for jobs that have been checked and passedfailed
for jobs that have been checked and passed
Prestage input dataset
Prestaging makes sure the input dataset has been copied off tape to disk, so it is ready to use. If there is no input dataset, or it is known to be on disk (in scratch dCache for example) skip this step.
We recommend starting the prestaging as soon as possible since it can take several days. Please follow the prestage instructions on the input dataset.
Generate fcl
In this step you take a fcl file which works interactively to generate a simulation sample, and scale it up for the grid. Please follow the instructions for generating fcl. There are examples of a job which starts from a generator (stage 1) or a later stage that take input art files from a previous stage, as well as a fcl file.
Submit Jobs
Please follow the instructions for submitting jobs.
Concatenate output datasets
Some discussion and overview is available in job planning and concatenation. If you decide to concatenate the output datasets, for each dataset:
This procedure is essentially the same as the main job: generate fcl, submit and recover jobs, and you will end up with the new datasets as output, and those will be uploaded in the next step.
Production datasets of the first stage of a simulation project often have the string g4s1 in the dataset name, and we have been replacing this with cs1 to create the concatenated dataset name.
Upload output files
Please see the background information on the upload page then follow the recipes in MC art files section.
Archive log files
Please see the instructions on the log files section of the upload page.