View source for An example of how to archive files to tape

The procedure is for saving "random" assortments of files as tape
archives.  One would have to retrieve and expand a tar file later to
access its content; the individual files will not be accessible from
their tape storage in a transparent way.    This corresponds to the
"Random files" option [[Upload|here]].
The current page walks through steps in an example use case.


== Decide how to divide data ==

Decide how to divide data to be archived into chunks.  A good size target is 2GB per chunk.  For example
   

   $ du -ks /disk2/data/tmnguyen/socketudp/data*2020  | awk '{s+=$1; print s"\t"$0}'
   433728  433728  /disk2/data/tmnguyen/socketudp/data_Feb02_2020
   1058092 624364  /disk2/data/tmnguyen/socketudp/data_Feb03_2020
   1271012 212920  /disk2/data/tmnguyen/socketudp/data_Feb04_2020
   1476712 205700  /disk2/data/tmnguyen/socketudp/data_Feb06_2020
   2498948 1022236 /disk2/data/tmnguyen/socketudp/data_Feb07_2020
   2498956 8       /disk2/data/tmnguyen/socketudp/data_Feb09_2020
   3022512 523556  /disk2/data/tmnguyen/socketudp/data_Feb10_2020
   3022712 200     /disk2/data/tmnguyen/socketudp/data_Feb11_2020
   3042356 19644   /disk2/data/tmnguyen/socketudp/data_Feb12_2020
   3770908 728552  /disk2/data/tmnguyen/socketudp/data_Feb17_2020
   4705628 934720  /disk2/data/tmnguyen/socketudp/data_Feb19_2020
   4795148 89520   /disk2/data/tmnguyen/socketudp/data_Feb20_2020
   5092448 297300  /disk2/data/tmnguyen/socketudp/data_Feb21_2020
   5751636 659188  /disk2/data/tmnguyen/socketudp/data_Feb22_2020
   5755548 3912    /disk2/data/tmnguyen/socketudp/data_Feb25_2020
   5762860 7312    /disk2/data/tmnguyen/socketudp/data_Feb26_2020
   6084616 321756  /disk2/data/tmnguyen/socketudp/data_Jan03_2020
   6094680 10064   /disk2/data/tmnguyen/socketudp/data_Jan22_2020
   6129020 34340   /disk2/data/tmnguyen/socketudp/data_Jan23_2020
   6236776 107756  /disk2/data/tmnguyen/socketudp/data_Jan24_2020
   6241600 4824    /disk2/data/tmnguyen/socketudp/data_Jan29_2020
   6628736 387136  /disk2/data/tmnguyen/socketudp/data_Jan30_2020
   7372852 744116  /disk2/data/tmnguyen/socketudp/data_Jan31_2020


so the total size of all of the "data*2020" directories is about 7.3 GB.
Therefore they can be archived as a single tar file, or split into a few files.

== Prepare tar files ==

The naming of files for tape upload is important.
File names must follow the Mu2e convention explained [[FileNames|here]].
   
File names will look like

   data_tier.owner.description.configuration.sequencer.file_format

A critical point is that every file name must  be unique,
the "sequencer" field is to make this possible when uploading
a "dataset" of similar files.

Continuing with the example,

  $ cd  /disk2/data/tmnguyen/socketudp
  $ tar jcvf /disk2/bck.tmnguyen.socketudp.v0.2020part1.tbz data*Jan*2020
  $ tar jcvf /disk2/bck.tmnguyen.socketudp.v0.2020part2.tbz data*Feb0*2020
  $ tar jcvf /disk2/bck.tmnguyen.socketudp.v0.2020part3.tbz data*Feb1*2020

== Copy the files to a mu2egpvm machine ==

   ssh mu2egpvm01.fnal.gov
   $ mkdir /mu2e/data/users/tmnguyen
   $ cd  /mu2e/data/users/tmnguyen
   $ scp -p mu2etest.fnal.gov:/disk2/bck.\*.tbz .

(and remove them from mu2etest)

== Tape upload ==

This step is best done using VNC or a terminal server like "screen"
or "tmux", because the command will probably take more than a day to complete.
A terminal server will prevent the command from being killed if ssh
session disconnects for any reason.
   
   ssh mu2egpvm01.fnal.gov
   $ screen
   $ mu2einit
   $ setup mu2efiletools
   $ kx509
   $ cd /mu2e/data/users/tmnguyen
   $ mu2eFileMoveToTape bck.*.tbz

Another way to ensure that the command will complete is to run it under nohup. A complication here is the buffering of the nohup.out file.

== Check ==

Check the next day whether the process is complete.  
You can start a new ssh shell and re-connect to the
existing "screen" session:

   ssh mu2egpvm01.fnal.gov
   $ screen -d -r

If everything worked fine you'll see the source files given to the
mu2eFileMoveToTape command deleted.  If they are still there and
you are SURE that the original mu2eFileMoveToTape process is not
running any more (on any mu2egpvm node!), you can re-run the same
mu2eFileMoveToTape command again, until the upload succeeds.