Workflows

Workflow for jobs

• Grids - overview of the computing farms and job submission
• POMS - grid campaign organization, submission and recovery tool
• HPC - high performance computing resources
• Docker software containers for submitting to special farms
• analysis workflow - how to read existing datasets
• production simulation workflow - how to run simulation, concatenate and upload the files
• MARS and G4Beamline workflow - these are not in the art framework
• Job planning - considerations of project CPU, disk, memory and file size
• Prestage - make sure files have been retrieved off tape and are ready to use
• SimulationFCL - an overview how MDC2020 fcl works
• GenerateFcl - generate fcl for simulation jobs
• SubmitJobs - submit, monitor and recover jobs in the workflow
• JustInTimeFcl - a technique for simplifying fcl generation
• Concatenate - merging root and art files
• Upload - upload files to tape
• ErrorRecovery - how to recover from common errors
• Validation - check how code is performing, using MC
• DQM - Data Quality Monitoring
• Datasets - production datasets
• RunNumbers - run number ranges and uses
• ProductionLogic - production job logic

Data Handling

• dCache - the large aggregated data disk system
• enstore - the mass storage tape system
• File Transfer Service (FTS) - service to create SAM records and copy files to tape
• Raw Data Mover (RDM) - Mu2e service to copy teststand data files to tape
• Data transfer - how to move larger datasets, necessary for grid jobs
• File names - how to name files for upload or for production
• File families - the logical grouping of data on tapes
• SAM - file metadata and data handling management system
  • SAM metadata - the metadata stored for each file
  • SAM expert - some notes for the future, not of general interest
• File tools - tools for manipulating large file datasets and metadata
• FTS Upload - deprecated method to upload by the FTS area
• Rucio - next-gen data handling

Operations

• operations links
• Mu2epro account