MCProdWorkflow: Difference between revisions

From Mu2eWiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
The <code>mu2egrid</code> package provides Mu2e-specific code
required for using jobsub.  All scripts in the package support the
<code>--help</code> option. If it is present on the command line, all
other options will be ignored. All scripts also
support <code>--dry-run</code> and <code>--verbose</code> options to
show what will be done without performing the action.


==Introduction==
This workflow is for production-style simulation jobs. It can be used for stage-1 jobs which start with a generator, or later simulation stages which start with the output files of previous stages.  It will result in the output files being concatenated and uploaded to tape and properly documented through the SAM database.  It is intended for cases where the output needs to be saved for more than a month or so, or that might be used by many collaborators, or needs to be carefully documented.  If your work doesn't need to be uploaded or is more personal or temporary, you can follow the [[MCScrWorkflow|scratch workflow]] which does not concatenate or upload.  Most commonly this procedure would be part of a collaboration simulation effort and would be run out of the [[Mu2epro|mu2epro account]], but is can be run in a personal account.
The user should prepare the physics, and relevant fcl in detail and perform basic test before starting this procedure.
This page assumes that the user is familiar with the basic infrastructure and its references:
* [[Fclintro|fcl]]
* [[Simulation]]
* [[Job planning|job planning]]
* [[Grids|grid]]
* [[FileNames|file names]]
* [[FileTools|file tools]]
* [[SAM]]
* [[Grids|grids]]
* [[Dcache|dCache]]
* [[Enstore|enstore]]
* [[Prestage|prestaging]]
* [[Concatenate|concatenation]]
* [https://cdcvs.fnal.gov/redmine/projects/mu2egrid/wiki mu2egrid]
*[[OfflineOps|monitoring]]
The basic steps, expanded below are
* if needed, prestage input files
* generate a set of fcl files
* register the fcl dataset with SAM, and copy fcl files to dCache
* submit jobs
* check output and recover failed jobs
* if needed, concatenate output files
* upload output files
* tar and upload log files


To be able to use the full functionality of the system, such as
The <code>mu2egrid</code> and [[FileTools|related]] packages provides Mu2e-specific code
recovering failed jobs and/or storing outputs on tape, more steps are required:
required for submitting jobs and manipulating files.  Most scripts support the
<code>--help</code> option.  Look for the  
<code>--dry-run</code> and <code>--verbose</code> options to
show what will be done without performing the action.


<ol>
==Prestage inputs==
  <li>[generate_fcl.shtml Generate a set of fcl files]  that completely define the jobs to be run.
  <li>[fcl_upload.shtml Register]  the fcl dataset with SAM, and make fcl files available on /pnfs.
  <li>Make sure that input data files, if any, are [prestage.shtml pre-staged to disk] .
  <li>[mu2eprodsys.shtml Submit mu2eprodsys grid jobs] .
  <li>[postprocess.shtml#check Check results] , identify and re-run failed jobs.
  <li>[postprocess.shtml#store Store]  outputs on tape:
</ol>

Revision as of 20:29, 4 April 2017

Introduction

This workflow is for production-style simulation jobs. It can be used for stage-1 jobs which start with a generator, or later simulation stages which start with the output files of previous stages. It will result in the output files being concatenated and uploaded to tape and properly documented through the SAM database. It is intended for cases where the output needs to be saved for more than a month or so, or that might be used by many collaborators, or needs to be carefully documented. If your work doesn't need to be uploaded or is more personal or temporary, you can follow the scratch workflow which does not concatenate or upload. Most commonly this procedure would be part of a collaboration simulation effort and would be run out of the mu2epro account, but is can be run in a personal account.

The user should prepare the physics, and relevant fcl in detail and perform basic test before starting this procedure.

This page assumes that the user is familiar with the basic infrastructure and its references:

The basic steps, expanded below are

  • if needed, prestage input files
  • generate a set of fcl files
  • register the fcl dataset with SAM, and copy fcl files to dCache
  • submit jobs
  • check output and recover failed jobs
  • if needed, concatenate output files
  • upload output files
  • tar and upload log files

The mu2egrid and related packages provides Mu2e-specific code required for submitting jobs and manipulating files. Most scripts support the --help option. Look for the --dry-run and --verbose options to show what will be done without performing the action.

Prestage inputs

Retrieved from "http://mu2ewiki.fnal.gov/w/index.php?title=MCProdWorkflow&oldid=1973"