Difference between revisions of "Rucio"
Jump to navigation
Jump to search
| Line 49: | Line 49: | ||
export DATA_DISPATCHER_URL=https://metacat.fnal.gov:9443/mu2e_dd_prod/data | export DATA_DISPATCHER_URL=https://metacat.fnal.gov:9443/mu2e_dd_prod/data | ||
export DATA_DISPATCHER_AUTH_URL=https://metacat.fnal.gov:8143/auth/mu2e | export DATA_DISPATCHER_AUTH_URL=https://metacat.fnal.gov:8143/auth/mu2e | ||
| + | |||
| + | /pnfs/mu2e/tape | ||
| + | /pnfs/mu2e/persistent/datasets | ||
| + | /pnfs/mu2e/scratch/datasets (expect to have a greedy cleanup of two weeks) | ||
</pre> | </pre> | ||
Revision as of 16:46, 16 June 2023
Introduction
Rucio is a CERN software system for storing file metadata and organizing the delivery of that data to users. Its primary features are scalability, flexibility, adaptive file replication, and built-in monitoring. It can use various backends for databases, various platforms for its servers and daemons, various transfer and storage method plug-ins, and a command line and python interface for users.
The new system would consist of these parts
- Metacat - a database of file metadata (docs)
- Rucio - a database of file locations, and servers which can move and track data, responding to user rules (docs)
- Data Dispatcher - a modern replacement for SAM project file delivery (docs)
- mdh - Mu2e data-handling commands added to supplement the above systems (see
mdh -h)
A few overarching concepts to keep in mind
- these system only recognition authentication with tokens
- metacat requires you to be authenticated to write to the database (create files, datasets)
- all files belong to a namespace
- if you create a namespace, it must start with your username (by policy)
- files must be named by the naming convention (by policy)
- all files are readable by all users
- metacat has roles. Users can be members of a role, and then the role can create and own objects
Quick start
setup
setup mu2e setup mdh
will setup all related data-handling tools
Authentication
Authenticate yourself
metacat auth login -m token $USER
- if you get a token file not found, please run getToken, or see token docs
- if you get Authentication failed, you might not have an account
Your authentication lasts as long as your token valid period
To check your authtication
metacat auth list
There is no logout
Listing
list namespaces
metacat namespace list
list your namespaces
metacat namespace list -u $USER
Implementation
export METACAT_SERVER_URL=http://dbweb5.fnal.gov:9094/mu2e_meta_prod/app export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/mu2e export DATA_DISPATCHER_URL=https://metacat.fnal.gov:9443/mu2e_dd_prod/data export DATA_DISPATCHER_AUTH_URL=https://metacat.fnal.gov:8143/auth/mu2e /pnfs/mu2e/tape /pnfs/mu2e/persistent/datasets /pnfs/mu2e/scratch/datasets (expect to have a greedy cleanup of two weeks)