FileTools
<!********************************************************>
jsonMaker
The jsonMaker is a python script which lives in the dhtools product and should be available at the command line after "setup dhtools." Please see the [uploadExample.shtml upload examples] page for details.
All files to be uploaded should be processed by the jsonMaker,
which writes the final json file to be included with the
data file in the FTS input directory. Even if all
the final json could be written by hand, the jsonMaker
checks certain required fields are present and other rules,
checks consistency, and writes in a known correct format.
Simply run the maker with all the data files and json fragment(s)
as input. The help of the code is below. The most
useful practical reference is the
[uploadExample.shtml upload examples] page.
jsonMaker [OPTIONS] ... [FILES] ... Create json files which hold metadata information about the file to be uploaded. The file list can contain data, and other types, of files (foo.bar) to be uploaded. If foo.bar.json is in the list, its contents will be added to the json for foo.bar. If a generic json file is supplied, it's contents will be added to all output json files. Output is a json file for each input file, suitable to presenting to the upload FTS server together with the data file. If the input file is an art file, jsonMaker must run a module over the file in order to extract run and event information, so a mu2e offline release that contains the module must be setup. -h print help -v LEVEL verbose level, 0 to 10, default=1 -x perform write/copy of files. Default is to evaluate the upload parameters, but not not write or move anything. -c copy the data file to the upload area after processing Will move the json file too, unless overidden by an explicit -d. -m mv the data file to the upload area after processing. Useful if the data file is already in /pnfs/mu2e/scratch where the FTS is. Will move the json file too, unless overidden by an explicit -d. -e just rename the data file where it is -s FILE FILE contains a list of input files to operate on. -p METHOD How to match a input json file to a data file METHOD="none" for no json input file for each data file (default) METHOD="file" pair an input json file with a data file based on the fact that if the file is foo, the json is foo.json. METHOD="dir" pair a json file and a data file based on the fact that they are in the same directory, whatever their names are. -j FILE a json file fragment to add to the json for all files, typically used to supply MC parameters. -i PAR=VALUE a json file entry to add to the json for all files, like -i mc.primary_particle=neutron -i mc.primary_particle="neutron" -i mc.simulation_stage=2 Can be repeated. Will supersede values given in -j -a FILE a text file with parent file sam names - usually would only be used if there was one data file to be processed. -t TAG text to prepend to the sequencer field of the output filename. This can be useful for non-art datasets which have different components uploaded at different times with different jsonMaker commands, but intended to be in the same dataset, such as a series of backup tarballs from different stages of processing. -d DIR directory to write the json files in. Default is ".". If DIR="same" then write the json in the same directory as the the data file. If DIR="fts" then write it to the FTS directory. If -m or -c is set, then -d "fts" is implied unless overidden by an explicit -d. -f FILE_FAMILY the file_family for these files - required -r NAME this will trigger renaming the data files by the pattern in NAME example: -r mcs.batman.beam-2014.fcl-100..art The blank sequencer ".." will be replaced by a sequence number like ".0001." or first run and subrun for art files. -l DIR write a file of the data file name and json file name followed by the fts directory where they should go, suitable for driving a "ifdh cp -f" command to move all files in one lock. This file will be named for the dataset plus "command" plus a time string. -g the command file will be written (implies -l) and then when all files are evaluated and json files written, execute the command file with "ifdh cp -f commandfile". Useful to use one lock file to execute all ifdh commands. Nullifies -c and -m. Requires python 2.7 or greater for subprocess.check_output and 2.6 or greater for json module. version 2.0