Rucio: Difference between revisions
Jump to navigation
Jump to search
| Line 17: | Line 17: | ||
* files must be named by the [[FileNames|naming convention]] (by policy) | * files must be named by the [[FileNames|naming convention]] (by policy) | ||
* all files are readable by all users | * all files are readable by all users | ||
==Quick start== | |||
===setup=== | |||
setup mu2e | |||
setup mdh | |||
will setup all related data-handling tools | |||
===Authentication=== | |||
metacat auth login -m token $USER | |||
* if you get a token ''file not found'', please run getToken, or see [[token docs]] | |||
* if you get ''Authentication failed'', you might not have an account | |||
==Implementation== | ==Implementation== | ||
Revision as of 21:26, 15 June 2023
Introduction
Rucio is a CERN software system for storing file metadata and organizing the delivery of that data to users. Its primary features are scalability, flexibility, adaptive file replication, and built-in monitoring. It can use various backends for databases, various platforms for its servers and daemons, various transfer and storage method plug-ins, and a command line and python interface for users.
The new system would consist of these parts
- Metacat - a database of file metadata (docs)
- Rucio - a database of file locations, and servers which can move and track data, responding to user rules (docs)
- Data Dispatcher - a modern replacement for SAM project file delivery (docs)
- mdh - Mu2e data-handling commands added to supplement the above systems (see
mdh -h)
A few overarching concepts to keep in mind
- these system only recognition authentication with tokens
- metacat requires you to be authenticated to write to the database (create files, datasets)
- all files belong to a namespace
- if you create a namespace, it must start with your username (by policy)
- files must be named by the naming convention (by policy)
- all files are readable by all users
Quick start
setup
setup mu2e setup mdh
will setup all related data-handling tools
Authentication
metacat auth login -m token $USER
- if you get a token file not found, please run getToken, or see token docs
- if you get Authentication failed, you might not have an account
Implementation
export METACAT_SERVER_URL=http://dbweb5.fnal.gov:9094/mu2e_meta_prod/app export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/mu2e export DATA_DISPATCHER_URL=https://metacat.fnal.gov:9443/mu2e_dd_prod/data export DATA_DISPATCHER_AUTH_URL=https://metacat.fnal.gov:8143/auth/mu2e