Enstore: Difference between revisions

From Mu2eWiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 2: Line 2:
The Scientific Computing Division maintains a system of data tapes called enstore ([http://cd-docdb.fnal.gov/cgi-bin/ShowDocument?docid=5420 manual], [https://cdcvs.fnal.gov/redmine/projects/enstore project]), which allows us to "write data to tape".  The tapes are held in libraries in Feynman Cmputing Center and the Grid Computing Center building.  When files are needed, a robot arm retrieves them and inserts them in a tape drive.
The Scientific Computing Division maintains a system of data tapes called enstore ([http://cd-docdb.fnal.gov/cgi-bin/ShowDocument?docid=5420 manual], [https://cdcvs.fnal.gov/redmine/projects/enstore project]), which allows us to "write data to tape".  The tapes are held in libraries in Feynman Cmputing Center and the Grid Computing Center building.  When files are needed, a robot arm retrieves them and inserts them in a tape drive.


There are currently two types 9/2018 of tapes and we are transitioning between them.
There are currently (9/2018) two types of tapes and we are transitioning from the first to the second.


'''T10K''' [https://en.wikipedia.org/wiki/StorageTek_tape_formats StorageTek] T10000KC and T10000KD which hold 5TB and 8TB each, respectively.  We currently (2017) share about 20 tape drives with all of the Intensity Frontier.  On 9/28/18 our media for new files was switched to [https://en.wikipedia.org/wiki/Linear_Tape-Open LTO8], in library CD-LTO8F1.
'''T10K''' [https://en.wikipedia.org/wiki/StorageTek_tape_formats StorageTek] T10000KC and T10000KD which hold 5TB and 8TB each, respectively.  A tape drive can read at 250MB/s so one tape can be read in about 5h.  We currently (2017) share about 20 tape drives with all of the Intensity Frontier.   


'''LTO''' The LTO8 drives have a quoted maximum speed between 300MB/s and 350MB/s depending on media type (as compared to the quoted maximum of 250MB/s for the T10KC drives currently used). The LTO8 tapes have a capacity of 12TB but as you know, there is a global unavailability of those tapes, so we're using "M8" (LTO7 formatted with LTO8) tapes which have a capacity of 9 TB.
'''LTO''' [https://en.wikipedia.org/wiki/Linear_Tape-Open Linear Tape Open] The LTO8 drives have a quoted maximum speed between 300 MB/s and 350 MB/s depending on media type . The LTO8 tapes have a capacity of 12TB but there is a global unavailability of those tapes, so we're using "M8" (LTO7 formatted with LTO8) tapes which have a capacity of 9 TB.  There is a total of 56 drives for the intensity frontier, but a large fraction are used for converting T10K data to LTO8.




We have a total of 36 drives in production now with an additional 20 to be added within the next few weeks. By the end of this calendar year, all experiments (with the exception of CMS) will be putting all new data in this library using those drives and a significant fraction of the drives will be utilized for migrating existing data from T2 media to LTO.
Mu2e tapes are divided into several [[FileFamilies| file families]] depending on the type of data: production or user data, raw or reco, beam data or sim.  The files assigned to a file family will go to one set of tapes for that file family.  It can be useful to segregate data like this so it can be treated specially. For example, raw data might be stored with two copies in different buildings, while this is unnecessary for sim data.


 
The maximum read rates of tapes are 100's of MB/s, but in reality, tapes are rarely read all the way though or efficiently - we typically access single files at a time. Typical access times are:
 
* 1m to find and mount the tape
 
mu2e tapes are divided into several [[FileFamilies| file families]] depending on the type of data: production or user data, raw or reco, beam data or sim.  The files assigned to a file family will go to one set of tapes for that file family.  It can be useful to segregate data like this so it can be treated specially.  For example, raw data might be stored with two copies in different buildings, while this is unnecessary for sim data.
 
A tape drive can read at 250MB/s so one tape can be read in about 5h.  In reality, tapes are rarely read all the way though - we typically access single files at a time. Typical access times are:
* 1m to find an mount the tape
* 1m to seek to the file
* 1m to seek to the file
* 10s to read the file
* 10s to read the file
* 1m to dismount and replace the tape
* 1m to dismount and replace the tape
If requests for multiple files from a single tape are queued, then those requests will be grouped and ordered to improve the drive efficiency and reduce wear.
Once the file is off the tape, it has to be copied to tape-backed [[Dcache|dCache]].  If the file is over 300MB, it was written to tape as lone file and the copy to dCache is immediate.  If it is smaller, then it was rolled in a tarball with other small files in a system called Small File Aggregation (SFA).  In this case, it has to be extracted fro the tarball before being written to dCache and this can add up to 15s latency.
Once the file is off the tape, it has to be copied to tape-backed [[Dcache|dCache]].  If the file is over 300MB, it was written to tape as lone file and the copy to dCache is immediate.  If it is smaller, then it was rolled in a tarball with other small files in a system called Small File Aggregation (SFA).  In this case, it has to be extracted fro the tarball before being written to dCache and this can add up to 15s latency.



Revision as of 17:24, 28 September 2018

The Scientific Computing Division maintains a system of data tapes called enstore (manual, project), which allows us to "write data to tape". The tapes are held in libraries in Feynman Cmputing Center and the Grid Computing Center building. When files are needed, a robot arm retrieves them and inserts them in a tape drive.

There are currently (9/2018) two types of tapes and we are transitioning from the first to the second.

T10K StorageTek T10000KC and T10000KD which hold 5TB and 8TB each, respectively. A tape drive can read at 250MB/s so one tape can be read in about 5h. We currently (2017) share about 20 tape drives with all of the Intensity Frontier.

LTO Linear Tape Open The LTO8 drives have a quoted maximum speed between 300 MB/s and 350 MB/s depending on media type . The LTO8 tapes have a capacity of 12TB but there is a global unavailability of those tapes, so we're using "M8" (LTO7 formatted with LTO8) tapes which have a capacity of 9 TB. There is a total of 56 drives for the intensity frontier, but a large fraction are used for converting T10K data to LTO8.


Mu2e tapes are divided into several file families depending on the type of data: production or user data, raw or reco, beam data or sim. The files assigned to a file family will go to one set of tapes for that file family. It can be useful to segregate data like this so it can be treated specially. For example, raw data might be stored with two copies in different buildings, while this is unnecessary for sim data.

The maximum read rates of tapes are 100's of MB/s, but in reality, tapes are rarely read all the way though or efficiently - we typically access single files at a time. Typical access times are:

  • 1m to find and mount the tape
  • 1m to seek to the file
  • 10s to read the file
  • 1m to dismount and replace the tape

If requests for multiple files from a single tape are queued, then those requests will be grouped and ordered to improve the drive efficiency and reduce wear.

Once the file is off the tape, it has to be copied to tape-backed dCache. If the file is over 300MB, it was written to tape as lone file and the copy to dCache is immediate. If it is smaller, then it was rolled in a tarball with other small files in a system called Small File Aggregation (SFA). In this case, it has to be extracted fro the tarball before being written to dCache and this can add up to 15s latency.

We can mix large and small files in our data. The largest file that can be written is something like 1TB, but as a practical matter, all mu2e files should be less than about 20GB, and when there is an option, they should be 2-5 GB.

We write to tape through the tape-backed dCache. We only write files that are properly named, organized and documented by the mu2egrid scripts.

To read files from tape, we would usually access them by the mu2egrid scripts or the /pnfs mount of dCache. See also SAM to help with the file names and locations. Large numbers of files, such as those required by grid jobs, require prestaging to make sure they are off tape and on disk before reading them.

A list of complete file listings is available, updated each day (use wget, too big for a browser).