ErrorRecovery

Introduction

Some errors occur regularly, such as when authorization expires, dCache is struggling, or a procedure is repeated when it can't be repeated. Some common situations are recorded here with advice on how to handle them. Errors that can easily be googled, such as syntax errors, will not appear here.

art and fcl

art exit codes

art exit codes

cling::AutoloadingVisitor

At beginning of run time you see a spew of errors like this.

Error in cling::AutoloadingVisitor::InsertIntoAutoloadingState:
   Missing FileEntry for MCDataProducts/inc/ExtMonFNALPatRecTruthAssns.hh
   requested to autoload type mu2e::ExtMonFNALTrkFit

These are caused by cling system as implemented by root. Eventually this system will pre-compile header files (and maybe dictionaries?) so that compiling source files is very fast. In root 6, this not fully implemented and cling needs to find the header file for reasons that are not clear to us. It is searching the path given by ROOT_INCLUDE_PATH which should be set adequately by the setups. The error message is harmless, we know of no consequence except the spew.

Grid Workflows

mu2eFileDeclare: Metadata is invalid

 3610  OK:       /pnfs/mu2e/scratchError: got server response 400 Bad Request.
Metadata is invalid:
Parent file cnf.rhbob.pbarTracksFromAscii.pbarTracksFromAscii.001000_00006374.fcl not found

This error can occur when you are trying to declare a file to SAM. The declaration include submitting a metadata file to the SAM. This file contains all the information you want to store in the SAM database about the file. One the useful bits is the "parents" of the file. These are the fcl files and the input files used to create this file. If a parent file is "not found", that means that parent file was not declared in the SAM database. In the normal workflow, you would have declared the fcl files and the input file before you even started the job which produced the file at hand. Here are two recovery procedures:

• go back to the area and the log file for when you declared the file in the error message. In this case it was the fcl file which drove the creation of the file being declared. It might be possible to see what went wrong there, and fix it by re-declaring the fcl file, for example.
• if you do not need every file, and you do not see this error too often, you can move or delete the directory containing the result from this job, and restart mu2eFileDeclare.

SSL negotiation

Error creating dataset definition for ...
500 SSL negotiation failed: .

Your certificate is not of the right form

dCache hangs

A simple access to dCache (accessing filespecs like /pnfs/mu2e) can sometimes hang for a long time. This is difficult to deal with because there are legitimate reasons dCache could respond slowly. First, please read dCache page for background information.

dCache could be operating normally yet respond slowly because

• your request was excessive, such as running find or a ls -l on a large number (>few hundred) files. If there are 1000's of files queried, this could take minutes, and much longer for larger numbers of files. Use file tools and plain ls where possible.
• you, or other users, or even other experiments could be overloading dCache. This is difficult to determine, see operations page for some monitors. dCache has several choke points and not all are easily monitored.
• the files you are accessing are on tape and you have to wait for them to come off tape. The solution is to prestage files

It is difficult to tell if dCache is overloaded, but if it is not, your problem could be caused by any of several failure modes inside dCache, and these failures are relatively common. Here are some guidelines for when to put in a ticket

• if a simple ls on a directory which does not contain many files or subdirectories hangs for more than 2 min.
• if file access in your MC workflow seems normal then suddenly hangs for more than one 1h
• if accessing random files not recently accessed, and known to be prestaged, when access hangs more than 8h.
• is prestaging does not progress after 8h

Sometimes a hang is occurring only on one node, due to a problem with its nfs server. In this case, you can put in a ticket and then work on another node.

Generally, dCache has a lot of moving parts and is fragle in some ways. There is no real cost to putting a ticket and the dCache maintainers are responsive, so when in doubt, put in a ticket. You will always learn something about dCache.

mkdir failed mu2eFileUpload

The first time you invoke mu2eFileUpload fo r a new dataset, you get a permission denied error in mkidr

mkdir /pnfs/mu2e/tape/usr-sim/sim/rlc: 
Permission denied at /cvmfs/mu2e.opensciencegrid.org/artexternals/mu2efiletools/v3_7/bin/mu2eFileUpload line 71

This is because you are writing to the dCache tape or persistent areas. By default no one can write to those areas and as needed, permission is given to write to specific directories. Please write to mu2eDataAdmin.