Workflows: Difference between revisions
Jump to navigation
Jump to search
| Line 6: | Line 6: | ||
* [[MCScrWorkflow| scratch simulation workflow]] - how to run simulation to scratch areas for smaller jobs | * [[MCScrWorkflow| scratch simulation workflow]] - how to run simulation to scratch areas for smaller jobs | ||
* [[MARSWorkflow| MARS and G4Beamline workflow]] - these are not in the art framework | * [[MARSWorkflow| MARS and G4Beamline workflow]] - these are not in the art framework | ||
* [[JobPlan|Job | * [[JobPlan|Job planning]] - considerations of project CPU, disk, memory and file size | ||
* [[Prestage]] - make sure files have been retrieved off tape and are ready to use | * [[Prestage]] - make sure files have been retrieved off tape and are ready to use | ||
* [[GenerateFcl]] - generate fcl for simulation jobs | * [[GenerateFcl]] - generate fcl for simulation jobs | ||
* [[SubmitJobs]] - submit, monitor and recover jobs in the workflow | * [[SubmitJobs]] - submit, monitor and recover jobs in the workflow | ||
* [[Concatenate]] - merging root and art files | * [[Concatenate]] - merging root and art files | ||
* [[Upload]] - upload files | * [[Upload]] - upload files to tape | ||
* [https://cdcvs.fnal.gov/redmine/projects/mu2egrid/wiki mu2egrid] - reference for scripts in workflows | * [https://cdcvs.fnal.gov/redmine/projects/mu2egrid/wiki mu2egrid] - reference for scripts in workflows | ||
Revision as of 20:46, 11 April 2017
Workflow for jobs
- Grids - overview of the computing farms and job submission
- production simulation workflow - how to run simulation, concatenate and upload the files
- scratch simulation workflow - how to run simulation to scratch areas for smaller jobs
- MARS and G4Beamline workflow - these are not in the art framework
- Job planning - considerations of project CPU, disk, memory and file size
- Prestage - make sure files have been retrieved off tape and are ready to use
- GenerateFcl - generate fcl for simulation jobs
- SubmitJobs - submit, monitor and recover jobs in the workflow
- Concatenate - merging root and art files
- Upload - upload files to tape
- mu2egrid - reference for scripts in workflows
Data Handling
- dCache - the large aggregated data disk system
- enstore - the mass storage tape system
- Data transfer - how to move larger datasets, necessary for grid jobs
- File names - how to name files for upload or for production
- File families - the logical grouping of data on tapes
- SAM - file metadata and data handling management system
- SAM metadata - the metadata stored for each file
- SAM expert - some notes for the future, not of general interest
- File tools - tools for manipulating large file datasets and metadata
- FTS Upload - deprecated method to upload by the FTS area