Rucio: Difference between revisions

From Mu2eWiki
Jump to navigation Jump to search
No edit summary
Line 26: Line 26:
===Authentication===
===Authentication===
  metacat auth login -m token $USER
  metacat auth login -m token $USER
* if you get a token ''file not found'', please run getToken, or see [[token docs]]
* if you get a token ''file not found'', please run getToken, or see [[Authentication#Tokens|token docs]]
* if you get ''Authentication failed'', you might not have an account
* if you get ''Authentication failed'', you might not have an account



Revision as of 21:27, 15 June 2023

Introduction

Rucio is a CERN software system for storing file metadata and organizing the delivery of that data to users. Its primary features are scalability, flexibility, adaptive file replication, and built-in monitoring. It can use various backends for databases, various platforms for its servers and daemons, various transfer and storage method plug-ins, and a command line and python interface for users.

The new system would consist of these parts

  • Metacat - a database of file metadata (docs)
  • Rucio - a database of file locations, and servers which can move and track data, responding to user rules (docs)
  • Data Dispatcher - a modern replacement for SAM project file delivery (docs)
  • mdh - Mu2e data-handling commands added to supplement the above systems (see mdh -h)

A few overarching concepts to keep in mind

  • these system only recognition authentication with tokens
  • metacat requires you to be authenticated to write to the database (create files, datasets)
  • all files belong to a namespace
    • if you create a namespace, it must start with your username (by policy)
  • files must be named by the naming convention (by policy)
  • all files are readable by all users

Quick start

setup

setup mu2e
setup mdh

will setup all related data-handling tools

Authentication

metacat auth login -m token $USER
  • if you get a token file not found, please run getToken, or see token docs
  • if you get Authentication failed, you might not have an account

Implementation

export METACAT_SERVER_URL=http://dbweb5.fnal.gov:9094/mu2e_meta_prod/app
export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/mu2e
export DATA_DISPATCHER_URL=https://metacat.fnal.gov:9443/mu2e_dd_prod/data
export DATA_DISPATCHER_AUTH_URL=https://metacat.fnal.gov:8143/auth/mu2e

References