Workflows: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
| Line 4: | Line 4: | ||
* [[Grids|Grids]] - overview of the computing farms and job submission | * [[Grids|Grids]] - overview of the computing farms and job submission | ||
* [[HPC]] - high performance computing resources | * [[HPC]] - high performance computing resources | ||
* [[Docker]] software containers for submitting to special farms | |||
* [[AnalysisWorkflow| analysis workflow]] - how to read existing datasets | * [[AnalysisWorkflow| analysis workflow]] - how to read existing datasets | ||
* [[MCProdWorkflow| production simulation workflow]] - how to run simulation, concatenate and upload the files | * [[MCProdWorkflow| production simulation workflow]] - how to run simulation, concatenate and upload the files | ||
Revision as of 17:06, 15 October 2019
Workflow for jobs
- Grids - overview of the computing farms and job submission
- HPC - high performance computing resources
- Docker software containers for submitting to special farms
- analysis workflow - how to read existing datasets
- production simulation workflow - how to run simulation, concatenate and upload the files
- MARS and G4Beamline workflow - these are not in the art framework
- Job planning - considerations of project CPU, disk, memory and file size
- Prestage - make sure files have been retrieved off tape and are ready to use
- GenerateFcl - generate fcl for simulation jobs
- gridexport - export a build of Offline or a satellite release for use on the grid
- SubmitJobs - submit, monitor and recover jobs in the workflow
- Concatenate - merging root and art files
- Upload - upload files to tape
- ErrorRecovery - how to recover from common errors
- Validation - check how code is performing
- Datasets - production datasets
Data Handling
- dCache - the large aggregated data disk system
- enstore - the mass storage tape system
- Data transfer - how to move larger datasets, necessary for grid jobs
- File names - how to name files for upload or for production
- File families - the logical grouping of data on tapes
- SAM - file metadata and data handling management system
- SAM metadata - the metadata stored for each file
- SAM expert - some notes for the future, not of general interest
- File tools - tools for manipulating large file datasets and metadata
- FTS Upload - deprecated method to upload by the FTS area