Workflows

From Mu2eWiki
Revision as of 16:29, 29 January 2018 by Kutschke (talk | contribs)
Jump to navigation Jump to search


Workflow for jobs

  • Grids - overview of the computing farms and job submission
  • HPC - high performance computing resources
    • Docker software containers for submitting to special farms
  • production simulation workflow - how to run simulation, concatenate and upload the files
  • MARS and G4Beamline workflow - these are not in the art framework
  • Job planning - considerations of project CPU, disk, memory and file size
  • Prestage - make sure files have been retrieved off tape and are ready to use
  • GenerateFcl - generate fcl for simulation jobs
  • gridexport - export a build of Offline or a satellite release for use on the grid
  • SubmitJobs - submit, monitor and recover jobs in the workflow
  • Concatenate - merging root and art files
  • Upload - upload files to tape
  • mu2egrid - reference for scripts in workflows
  • Validation - check how code is performing
  • Datasets - production datasets

Data Handling

  • dCache - the large aggregated data disk system
  • enstore - the mass storage tape system
  • Data transfer - how to move larger datasets, necessary for grid jobs
  • File names - how to name files for upload or for production
  • File families - the logical grouping of data on tapes
  • SAM - file metadata and data handling management system
    • SAM metadata - the metadata stored for each file
    • SAM expert - some notes for the future, not of general interest
  • File tools - tools for manipulating large file datasets and metadata
  • FTS Upload - deprecated method to upload by the FTS area

Operations