Upload: Difference between revisions

From Mu2eWiki
Jump to navigation Jump to search
(Created page with "== Introduction == Keeping all Intensity Frontier data on disks is not practical, so large datasets must be written to tape. At the same time, the data must always be avail...")
 
No edit summary
Line 1: Line 1:
==  Introduction ==
==  Introduction ==
Keeping all Intensity Frontier data on disks is not practical,
mu2e has several forms of [[Disk|disk space]] and large aggregated disks are available in the [[Dcache|dCache]] system. 
so large datasets must be written to tape.  At the same time, the data
But we also write a large part of the files we produce to tape, which is less expensive, and can hold much more data.
must always be available and delivered efficiently. 
We usually write data to tape for one or more of the following reasons
The solution is coordinating several subsystems:
* to make room for new activity
* to keep it safe
* to make a permanent record


<ul>
The tape system is called [[Enstore|enstore]] and consists of several tape libraries and many tape drives with good connections to dCache.  We write to tape by copying files into tape-backed dCache and they are copied automatically to tape. The files will, on the scale of weeks if they are unused, be deleted off disk so they are only on tape.  We can get them copied from tape to disk again by [[Presatge|prestaging]] them.
<li><b>dCache</b>: a set of disk servers, a database of files
on the servers, and services to deliver those files with high throughput
<ul>
<li>[scratchDcache.shtml <b>scratch dCache</b>] : a dCache where least used files are purged as space is needed.
<li><b>tape-backed dCache</b>: a dCache where all files are on tape and are
cycled in and out of the dCache as needed
</ul>
<li><b>pnfs</b>: an nfs server behind the /pnfs/mu2e parition
which looks like a file system to users, but
is actually a interface to the dCache file database.
<li><b>Enstore</b>: the Fermilab system of tape and tape drive management
<li><b>SAM</b>: Serial Access to Metadata, a database of file metadata
and a system for managing large-scale file delivery
<li><b>FTS</b>:File Transfer Service, a process which manages the intake
of files into the tape-backed dCache and SAM.
<li><b>jsonMaker</b>: a piece of mu2e code which helps create
and check metadata when creating a SAM record of a file
<li><b>SFA</b>: Small File Aggregation, enstore can tar up small
files into a single large file before it goes to tape,
to increase tape efficiency.
</ul>


'''All data written to tape must follow certain conventions.'''  Please familiarize yourself with the links in this list.
* all files are named by [[FileNames|mu2e conventions]]
* all files will have a [[SAM|SAM record]] with [[SamMetadata|SAM metadata]], including file location
* all files are uploaded using [[FileTools|standard tools]], see especially [[FileTools#jsonMaker|jsonMaker]].


The basic procedure is for the user to run the jsonMaker
The following are the common workflow cases.
on a data file to make the  
 
==Upload steps==
===Generate the metadata===
[http://json.org json file] ,  
[http://json.org json file] ,  
then copy both the data file and the json
===Generate the metadata===
into an FTS area in </a "scratchDcache.shtml">scratch dCache</a>
===Generate the metadata===
called a dropbox.
===Generate the metadata===
The json file is
essentially a set of metadata fields with the
corresponding values.  The
[http://mu2esamgpvm02.fnal.gov:8787/fts/status FTS] 
will see the file with its json
file, and copy the file to a permanent location in
the tape-backed dCache and use the json to create a metadata record in SAM.
The tape-backed dCache will migrate the file to tape quickly
and the SAM record will be updated with the tape location.
Users will use [sam.shtml SAM]  to read the files
in tape-backed dCache.
 
 
Since there is some overhead in uploading, storing and retrieving
each file, the ideal file size is as large as reasonable.
This size limit should be determined by how long an executable
will typically take to read the file.  This will vary according to
exe settings and other factors, so a conservative estimate
should be used.    A file should be sized so that the longest
jobs reading it should take about 4 to 8 hours to run, which
generally provides efficient large-scale job processing.
A grid job that reads a few files in 4 hours is nearly as efficient,
so you can err on the small size.
You definately want to avoid a single
job section requiring only part of a large file.
Generally,
file sizes should not go over 20 GB in any case because they get
less convenient in several ways.
Files can be concatenated
to make them larger, or split to make them smaller.
Note - we have agreed that a subrun will only appear in one file.
Until we get more experience with data handling, and
see how important these effects are, we will
often upload files in the size we make them or find them.


Once files have been moved into the FTS directories,  
==MC production workflow, art files==
please do not try to move or delete them since this will
In the standard MC [[MCProdWorkflow|workflow]], there are three times you might upload files:
confuse the FTS and require a hand cleanup.  Once files
* after generating the fcl, uploading the fcl files is part of [[GenerateFcl|that procedure]]
are on tape, there is an expert procedure to delete them,
* after producing art files (including concatenation if needed), this is described in this section
and files of the same name can then be uploaded to replace
* upload log files are archive, which is handled [[here]]
the bad files.


<!********************************************************>
After the jobs have completed, the output datasets will be below a directory like the following, where you will be working:
== Recipe==
  cd /pnfs/mu2e/persistent/users/mu2epro/workflow/project_name/good
Below this directory, there are directories for each cluster, and below that directories for each job.
Each output art file named "a.b.c.d.e.f" should have a associated json file called "a.b.c.d.e.f.json" produced as part of the grid job and containing the SAM record metadata.


If you are about to run some new Monte Carlo in the official framework,
There are two steps.  First, declare the files to the SAM database
then the upload will be built into the scripts and documented
<pre>
with the mu2egrid
  mu2eClusterFileList --dsname <dataset> --json <cluster_number>  | mu2eFileDeclare
[ Monte Carlo submission] process.
</pre>
<font color=red>this is under development,
where <code>dataset</code> is the dataset name of the files to find and upload and
please ask Andrei for the status</font>
the <code>cluster_directory</code> is one of the cluster subdirectories.


The second step is to move the files to the final location in tape-backed dCache:
<pre>
mu2eClusterFileList --dsname <dataset> <cluster_directory>  | mu2eFileUpload --tape
</pre>


Existing files on local disks can be uploaded using the following steps.
The third step is to tell SAM where the files are in the tape system, to add their "location" to the SAM record.
The best approach would be to read quickly through the rest of this
<pre>
page for concepts then focus on the
mu2eDatasetLocation --add=tape <dataset>
[uploadExample.shtml upload examples]  page.
</pre>
Since it takes about a day, or sometimes more, for a file to migrate to tape and establish its tape location, after being copied to tape-backed Cache, it makes sense to wait a day before running this command


<ul>
This command should be as many times as needed in order to get the "Nothing to do" message, which means all the files in the dataset now have their location recorded:
<li>choose values for the [#metadata SAM Metadata] , including
<pre>
      the appropriate [#ff file family]
> mu2eDatasetLocation --add=tape sim.mu2e.cd3-pions-cs1.v563.art
<li>record the above items in a json file fragment
  No virtual files in dataset sim.mu2e.cd3-pions-cs1.v563.art. Nothing to do on Mon Nov 21 18:11:29 2016.
that will apply to all the files in your dataset
  SAMWeb times: query metadata = 0.00 s, update location = 0.00 s
<li>[#name rename]  your files by the upload convention
  Summary1: out of 0 virtual dataset files 0 were not found on tape.
(This can also be done by jsonMaker in the next step.)
  Summary2: successfully verified 0 files, added locations for 0 files.
<li>setup an offline release and run the [#jsonMaker jsonMaker] 
  Summary3: found 0 corrupted files and 0 files without tape labels.
to write the json file, which will include the fragment from the previous step
</pre>
<li>use "ifdh cp" to copy the data file and the full json file to the FTS area
/pnfs/mu2e/scratch/fts (This step can also can be done by jsonMaker.)
<li>use [sam.shtml SAM]  to access the file or its metadata
</ul>


</ol>
==MC workflow, Log Files==
The following is some detail you should be aware of in general, but
We usually upload the ntuple (TFileService) files along with the log files.
a detailed knowledge is not required.

Revision as of 22:02, 11 April 2017

Introduction

mu2e has several forms of disk space and large aggregated disks are available in the dCache system. But we also write a large part of the files we produce to tape, which is less expensive, and can hold much more data. We usually write data to tape for one or more of the following reasons

  • to make room for new activity
  • to keep it safe
  • to make a permanent record

The tape system is called enstore and consists of several tape libraries and many tape drives with good connections to dCache. We write to tape by copying files into tape-backed dCache and they are copied automatically to tape. The files will, on the scale of weeks if they are unused, be deleted off disk so they are only on tape. We can get them copied from tape to disk again by prestaging them.

All data written to tape must follow certain conventions. Please familiarize yourself with the links in this list.

The following are the common workflow cases.

Upload steps

Generate the metadata

json file ,

Generate the metadata

Generate the metadata

Generate the metadata

MC production workflow, art files

In the standard MC workflow, there are three times you might upload files:

  • after generating the fcl, uploading the fcl files is part of that procedure
  • after producing art files (including concatenation if needed), this is described in this section
  • upload log files are archive, which is handled here

After the jobs have completed, the output datasets will be below a directory like the following, where you will be working: 

cd /pnfs/mu2e/persistent/users/mu2epro/workflow/project_name/good

Below this directory, there are directories for each cluster, and below that directories for each job. Each output art file named "a.b.c.d.e.f" should have a associated json file called "a.b.c.d.e.f.json" produced as part of the grid job and containing the SAM record metadata.

There are two steps. First, declare the files to the SAM database 

 mu2eClusterFileList --dsname <dataset> --json <cluster_number>  | mu2eFileDeclare

where dataset is the dataset name of the files to find and upload and the cluster_directory is one of the cluster subdirectories.

The second step is to move the files to the final location in tape-backed dCache: 

mu2eClusterFileList --dsname <dataset> <cluster_directory>  | mu2eFileUpload --tape

The third step is to tell SAM where the files are in the tape system, to add their "location" to the SAM record. 

mu2eDatasetLocation --add=tape <dataset>

Since it takes about a day, or sometimes more, for a file to migrate to tape and establish its tape location, after being copied to tape-backed Cache, it makes sense to wait a day before running this command

This command should be as many times as needed in order to get the "Nothing to do" message, which means all the files in the dataset now have their location recorded: 

> mu2eDatasetLocation --add=tape sim.mu2e.cd3-pions-cs1.v563.art
  No virtual files in dataset sim.mu2e.cd3-pions-cs1.v563.art. Nothing to do on Mon Nov 21 18:11:29 2016.
  SAMWeb times: query metadata = 0.00 s, update location = 0.00 s
  Summary1: out of 0 virtual dataset files 0 were not found on tape.
  Summary2: successfully verified 0 files, added locations for 0 files.
  Summary3: found 0 corrupted files and 0 files without tape labels.

MC workflow, Log Files

We usually upload the ntuple (TFileService) files along with the log files.

Retrieved from "http://mu2ewiki.fnal.gov/w/index.php?title=Upload&oldid=2309"