SearchPaths

Introduction

Mu2e Offline supports the concept of a search path for most files that configure either the behaviour of art itself or that of the Mu2e Offline code. This means that you can specify a partial path to such a file and the code will search for that file in an ordered list of places. The code will traverse the list of places and look for the file in each place. As soon as it successfully finds the file it will open that file and continue.

What happens if the file is present at more than one place on the list? The list is ordered and the code declares success on the first match; it never looks to see if there is more than one match.

If the code cannot find a match, it throws.

A search path is specified in an environment variable as colon separated list of directories. This is in the same spirit as the well known environment variables PATH, LD_LIBRARY_PATH and PRODUCTS.

The search algorithm treats absolute paths specially. The code can be configured so that, for absolute paths, it ignores the search path and simply looks to see if there is a file at the absolute path. If there is no such file, it throws. Alternatively it can be configured to disallow absolute paths and to only look for files relative to the search path; in this case it throws as soon as it sees an absolute path. This last feature is useful when the program is being used to in production runs for which it is important to maintain a strict audit trail. Mu2e Offline is normally configured to disallow absolute paths; this is to make sure that we do not have a lot of work to do when we start production.

The search algorithm can also be configured to treat paths with a leading "." with the same rules as apply to absolute paths.

Both tools have the same policy, that the file must be found exactly by adding the requested relative path onto each element of the search path. Suppose that we wisht to specify a file:

/A/B/C/D/E/F.txt

And the setup scripts have defined:

export MU2E_FILE_PATH=/A/B/I/:/A/G/I:/A/B/C

If we search for the file "F.txt" it will not be resolved because the system only looks for the following files: "A/B/I/F.txt", "A/G/I/F.txt", "A/B/C/F.txt". To find the file of interest with the path above, one must ask for "D/E/F.txt", which will find a match on the last element. Alternatively, if the search path were

export MU2E_FILE_PATH=/A/B/I/:/A/G/I:/A/B/C/D/E

then "F.txt" would match with the last element.

Three important files are not covered by this policy: event-data input files, event-data output files and the root file managed by the TFileService. These files are managed by other facilties which allow only two options: an absolute path or a path that is relative to the current working directory at the time of execution.

FHICL_FILE_PATH

This environment variable is the search path used to find the .fcl file named by the -c command line argument. It is also used to find any .fcl files that are included by the top level .fcl file, and recursively for all included .fcl files. As of Offline v1_0_0 this is defined as:

export FHICL_FILE_PATH=.:fcl;

This definition is done in Offline/setup.sh . This will soon be changed to something like:

export FHICL_FILE_PATH=$MU2E_BASE_RELEASE:$MU2E_BASE_RELEASE/fcl

And, when one sets up a satellite release it will be redfined as,

export FHICL_FILE_PATH=$MU2E_SATELLITE_RELEASE:$MU2E_SATELLITE_RELEASE/fcl:$FHICL_FILE_PATH

FHiCL is configured to allow an absolute path for the .fcl file specified on the command line but, for paths to included files, FHiCL only allows paths relative to FHICL_FILE_PATH.

There are two FHiCL prolog files that are included in many .fcl files:

#include "minimalMessageService.fcl"
#include "standardProducers.fcl"

These are found in Offline/fcl. We currently recommend that these includes be written as shown here, without the leading fcl.

As we get experience with FHiCL we will consider adjustments to these policies. The policies will be as open as possible during our development phase, with the constraint that we understand how to ensure a strict audit trail when the time comes for large scale production.

MU2E_SEARCH_PATH

All of the non-fcl run-time configuration and one special data stream search for their files using the environment variable MU2E_SEARCH_PATH. This distinction is a historical artifact: the formats of these other configuration files, and the tools to read them, were defined before FHiCL was created.

The class that supports this functionality is Mu2eUtilities/inc/ConfigFileLookupPolicy.hh; under the covers it uses technology from cetlib. Throughout the Mu2e Offline documentation, we will refer to this functionality as "the file lookup policy". This feature is used in:

1. Any file that is parsed by SimpleConfig. This includes:
   • The geometry file, read by the GeometryService.
   • The event generator run-time configuration file, read by the EventGenerator module.
   • The conditions data run-time configuration file, read by the ConditionsService.
   • Any files included into the above three files using #include.
2. The magnetic field maps, read by the BFieldManagerMaker.
3. The particle data table files, read by the ParticleDataTable class.
4. The G4 macro file optionally read by G4_plugin.cc
5. The beam arrival time distribution read by FoilParticleGenerator. ( This should be moved into the conditions service ).
6. Input particles in G4beamline format, read using EventGenerator/inc/FromG4BLFile.hh

In the following, it is presumed that the reader is familiar with the ideas of base releases and satellite releases. The environment variable MU2E_SEARCH_PATH is built at two places in the Mu2e setup scripts. In Offline/setup.sh it is set to

export MU2E_SEARCH_PATH=$MU2E_BASE_RELEASE:$MU2E_DATA_PATH;

The idea is that the code will search for files first in the base release and then in $MU2E_DATA_PATH. All run-time configuration files should be found under $MU2E_BASE_RELEASE. Large files that do not really part of a particular code release, such as magnetic field maps should be found under $MU2E_DATA_PATH.

In Offline/bin/addlocal.sh the environment variable is set to

export MU2E_SEARCH_PATH=$MU2E_TEST_RELEASE:$MU2E_BASE_RELEASE:$MU2E_DATA_PATH;

The idea here is that the code should first look for run-time configuration files in the test release and, if it is not found, to look again in the base release. One can also deploy alternate version of the large data files using at test release.

We are considering replacing the files parsed with SimpleConfig with FHiCL format files because the grouping of parameters will be enforced by the language and not just by convention.