Rucio: Difference between revisions

From Mu2eWiki
Jump to navigation Jump to search
Line 1: Line 1:


==Introduction==
==Introduction==
Rucio is a CERN software system for storing file metadata and organizing the delivery of that data to users.  Its primary features are scalability, flexibility, adaptive file replication, and built-in monitoring.  It can use various backends for databases, various platforms for its servers and daemons, various transfer and storage method plug-ins, and a command line and python interface for users.   
Rucio is a CERN software system for storing file metadata and organizing the delivery of that data to users.  Its primary features are scalability, flexibility, adaptive file replication, and built-in monitoring.  It can use various backends for databases, various platforms for its servers and daemons, various transfer and storage method plug-ins, and a command line and python interface for users.   


Line 7: Line 8:
* Rucio - a database of file locations, and servers which can move and track data, responding to user rules
* Rucio - a database of file locations, and servers which can move and track data, responding to user rules
* Data Dispatcher - a modern replacement for SAM project file delivery
* Data Dispatcher - a modern replacement for SAM project file delivery
 
* mdh - Mu2e data-handling commands added to supplement the above systems
At Fermilab the status of these systems is:
* Metacat - in use by ProtoDUNE
* Rucio - in use by ProtoDUNE
* Data Dispatcher - will be ready on time to support the next ProtoDUNE run at CERN, April 2023.
 
All 3 components must be available before POMS can be ported to move from SAM to these components.  As of September 2021, the expecation was that this would take "a few months" following the availability of Data Dispatcher.


==Implementation==
==Implementation==

Revision as of 20:58, 15 June 2023

Introduction

Rucio is a CERN software system for storing file metadata and organizing the delivery of that data to users. Its primary features are scalability, flexibility, adaptive file replication, and built-in monitoring. It can use various backends for databases, various platforms for its servers and daemons, various transfer and storage method plug-ins, and a command line and python interface for users.

The new system would consist of these parts

  • Metacat - a new database of file metadata
  • Rucio - a database of file locations, and servers which can move and track data, responding to user rules
  • Data Dispatcher - a modern replacement for SAM project file delivery
  • mdh - Mu2e data-handling commands added to supplement the above systems

Implementation

export METACAT_SERVER_URL=http://dbweb5.fnal.gov:9094/mu2e_meta_prod/app
export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/mu2e
export DATA_DISPATCHER_URL=https://metacat.fnal.gov:9443/mu2e_dd_prod/data
export DATA_DISPATCHER_AUTH_URL=https://metacat.fnal.gov:8143/auth/mu2e

References

Retrieved from "http://mu2ewiki.fnal.gov/w/index.php?title=Rucio&oldid=22124"