CodeDistribution

Introduction

There are two main ways to work on mu2e code interactively.

• login to the interactive nodes at the lab, where everything is setup and all disks are mounted. This gives you the maximum environment, all disk areas, all data samples, and easy collaboration with everyone on mu2e.
• access the code at your home institution or on your desktop or laptop. While this may be necessary due to network speed and reliability, it will not have the above advantages. There are two ways to move the code locally:
  • mount the cvmfs distributed code disk - recommended
  • copy the code and supporting UPS packages locally

In either case the machine you want to work on must provide all generic (not Mu2e or art) system libraries that Offline depends on. The art team maintans a checkPrerequisites shell script, which should be run to verify that all the libraries are available. If some are missing, the script will print out instructions to install missing RPM packages. The script can be downloaded from this page: https://scisoft.fnal.gov/scisoft/bundles/tools/

If an RPM package is not found you probably need to enable extra repositories, see LinuxFAQ.

Why stop and go no further on this page

We strongly recommend that you use the cvmfs distributed code system for accessing tagged code releases and the supporting UPS products. Not only it is fully prepared in one place, it is always maintained by the experts and automatically up-to-date with new releases and the latest versions of the products. Any distribution you copy locally will be out of date in a matter days unless you rigorously maintain it. cvmfs does require root access once, but since it is the LHC standard, many institutions have already installed it and in any case, it is usually quite easy.

Copying the code distribution

Are you sure you can't use cvmfs? Please consult with your group leader or mu2e mentor to make sure this is the best solution. If you are sure, then proceed.

The instructions below were written in 2017 and are very out of date. They are also written for code distrubtion using UPS, which is being phased out in favor of Spack.

These instructions explain what you must install in order to run the Mu2e introductory examples. If you wish to do other work, you may also need to download the appropriate event-data input files and appropriate run-time configuration files.

Should you need/wish to build the tool chain from source, you can follow the instructions for building the tool chain from source.

The difference between the last two options is where you put the version number of the requested version of art. In the script for v1_12_02 and earlier a new download script is created for each new release and the version number is embedded into the screipt name. In the script for v1_12_04 and script does not change from version to version and the version number is supplied as an argument to the script.

Version v1_12_03 was an integration release will not be distributed.

Components of Offline

The tool chain needed to run Mu2e Offline at a remote site includes the following parts:

• A UPS repository containing art, the UPS packages on which it depends, geant4, scons and a few other packages. You may also wish to read the Mu2e documentation on UPS.
• The site specific setup script; you need to copy this file from Fermilab and make a few edits to describe your local environment. At Fermilab, this is the script that is invoked when you type the command mu2einit.
• Supporting data data files, including
  • The magnetic field map files.
  • The example stopped muon input file; this is needed to run the introductory examples but may not been needed for other purposes.
• You will download the actual mu2e code via a git url, and built it against the above packages

The above files may be put into a directory tree rooted anywhere in your local or network mounted file systems; the Mu2e system does not presume any absolute path names. When necessary, information about absolute pathnames is distributed using environment variables.

The above files may be owned by any user, provided they are readable, and, if appropriate, executable, by Mu2e users at your site. However we do suggest that you create a separate account for installing these files; this way you cannot accidentally delete them when logged in to your own account; if you decide to use a separate account, there are no restrictions on the name of the account but it should be in the same group as regular Mu2e users and the files should be group readable and, if appropriate, group executable. In summary, using a separate account to own the tool chain is not necessary but it is a recommended safety practice.

Supported Operating Systems

Here is a list of the supported platforms from the art documentation. In general, the list includes several versions of Scientific Linux and Max OS. Builds are 64-bit.

Here are few platform notes collected over the years, which may have become obsolete...

• For the MAC OSX versions, the letter "d" in the mnemonic names "d12" and "d13" refers to "Darwin", which is Apple's name for OSX. The d12 version is known to work when run on a few specific earlier releases of Mac OS, specifically, Snow Leopard (OSX 10.8) and Lion (OSX 11).

• Starting with OSX 10.11, El Kapitan, Apple made some security changes referred to as SIP; this also affects OSX 10.12, Sierra. A lot of our software will not work on machines that have SIP enabled; more information is available on the Mu2e page about SIP on MAC OS El Kapitan and above.

• Officially the art team does not support Scientific Linux CERN (SLC) or plain Scientific Linux (SL) but several users have reported that installing the SLF binaries on SLC or SL machines "just works". This is not surprising since the differences among these OS versions involve only authentication. Some users have reported success using SLF binaries on other Redhat based linux versions, such as Fedora.

• The art team supports Ubuntu14.04 and will soon support Ubuntu16; they plan to discontinue support for Ubuntu14.04 soon after Ubuntu16 support is available; there will likely be one release of art with support for both versions of Unbuntu.

• If you wish to use the Mu2e Offline on a platform not discussed above, you will need to build the tool chain from source.

• If you have a large computing resource that runs an OS not listed above, you can ask the art team to provide a binary distribution for that platform. For this to happen you will need to give them access to an appropriate machine on which they can build the software. The art team is only likely to agree to this if you are bringing a signficant computing resource to the table.

Collecting Information

Before you run the installation tools you need to decide where to install the software and you need to learn some information about the platform on which you are doing the installation.

1. Decide where to install the tool chain. In the following we will presume that you have described this choice by defining the environment variable MU2E:

   export MU2E=/absolute/path/to/your/local/mu2e/top_level/directory

2. Decide the memonic name of the platform that you wish to install: one of slf5, slf6, d13 or d14.
3. Decide on which art version you wish to install, for example

   export ART_VERSION=v1_17_07

4. Consult Offline/setup.sh to learn which version of art you need to download; you can also consult the Mu2e software team.
5. Learn the correct the UPS options for the packages in the tool chain, for example:

   export ART_OPTIONS=s30-e9

6. See the table below to learn which ART_OPTIONS are required/allowed for which ART_VERSION.
7. Learn the correct version of the BTrk tracking code to use. You can learn this by looking at Offline/setup.sh. At this writing the correct version is:

   export BTRK_VERSION=v1_01_01

The art team maintains a page that describes the art options. The two options shown in the above example are e9 and s30; each of the numbered e options chooses a compiler version plus compiler flags. The numbered s options are used by products that depend on an art version (for example ifdh_art), not by art itself; the s option is a mnemonic for which version of art it depends on.

The table below gives the correct values of ART_VERSION, ART_OPTIONS and BTRK_VERSION for all versions of art that have been used by Mu2e, starting from art v1_12_04.

ART_VERSION	ART_OPTIONS	BTRK_VERSION
v1_12_04	s5-e6
v1_13_01	s7-e7
v1_15_00	s13-e7	v1_00_08
v1_17_02	s20-e9	v1_01_00
v1_17_07	s30-e9	v1_01_01
v2_04_00a	s42-e10	v1_01_03
v2_06_02a	s47-e10	v1_01_08
v2_07_03	s50-e14	v1_02_02

Mu2e has not used every version of art; normally we jump to the most recent version when it has a feature we need or when we have a lull in other computing work.

Cheat Sheet for art v1_17_07

To pull products for a different version of art, use the following as a model but consult the table above for the appropriate values of version numbers and qualifiers.

The notes provided by SCD are at the following url. They may help to understand what is happening at each stage of this cheat sheet.

Make a top Level directory to hold everything, assign it to MU2E

cd ${MU2E}

Make subdirectories to hold the UPS repository and the magnetic field maps

mkdir artexternals
mkdir DataFiles

The following instructions presume that you used exactly these subdirectory names and that you defined the environment variables described in Collecting Information. Download the script that will do the installation of the UPS repository

mkdir tmp
cd tmp
curl -O http://scisoft.fnal.gov/scisoft/bundles/tools/pullProducts
chmod u+x pullProducts

If curl is not available on your machine, wget may be:

wget http://scisoft.fnal.gov/scisoft/bundles/tools/pullProducts

Run the script to install the UPS products:

./pullProducts ${MU2E}/artexternals slf6 mu-${ART_VERSION} ${ART_OPTIONS} prof

If you also want the debug builds of the UPS products, run pullProducts a second time:

./pullProducts ${MU2E}/artexternals slf6 mu-${ART_VERSION} ${ART_OPTIONS} debug

The pull Products script is smart enough that it will only copy UPS products that are not already present; for example the Geant4 data files

Copy the BTrk ups product. This is not yet managed by pull products.

scp your_kerberos_principle@mu2egpvm01.fnal.gov:/grid/fermiapp/products/mu2e/Downloads/BTrk_${BTRK_VERSION}.tar.gz .
cd ../artexternals
tar xzf ../tmp/BTrk_${BTRK_VERSION}.tar.gz

Clean up

cd ..
/bin/rm -rf tmp

Copy the site specific setup script and edit it:

scp your_kerberos_principal@mu2egpvm01.fnal.gov:/grid/fermiapp/products/mu2e/setupmu2e-art.sh .

Be careful, there is a second argument to scp, a dot, asking scp to copy the requested file into the current directory. Make the following edits to this file:

Redefine MU2E_DATA_PATH with the absolute path to the DataFiles directory created above. setupmu2e-art.sh ensures that git is available to users. If your system does not automatically provide git in a users environment you may wish to modify setupmu2e-art.sh so that git is available to all users that source this script.

If you want to run the art workbook at your site, set the 3 workbook-related environment variables. to values appropriate for your site. The variables are:

export ART_WORKBOOK_OUTPUT_BASE=/mu2e/data/users
export ART_WORKBOOK_WORKING_BASE=/mu2e/app/users
export ART_WORKBOOK_QUAL="s3:e5"

Copy over the GA05 magnetic field maps. This requires about 4.2GB of disk space for the B fields and 2.4GB for the CRV lookup tables.

mkdir -p DataFiles/BFieldMaps
scp -r your_kerberos_principle@mu2egpvm01.fnal.gov:/cvmfs/mu2e.opensciencegrid.org/DataFiles/BFieldMaps/GA05 DataFiles/BFieldMaps
mkdir -p DataFiles/CRVConditions
scp -r your_kerberos_principle@mu2egpvm01.fnal.gov:/cvmfs/mu2e.opensciencegrid.org/DataFiles/CRVConditions/v2_0 DataFiles/CRVConditions

Copy the text files that contain the positions and times of stopped muons; these are used by the old style event generators and will soon be replaced by different files.

scp -r mu2egpvm01.fnal.gov:/cvmfs/mu2e.opensciencegrid.org/DataFiles/mergedMuonStops DataFiles

You will also need to edit your fcl files to tell the event generators where to find the stopped muon files. When you log in to the machine to work on mu2e, the analog of mu2einit is

 source ${MU2E}/setupmu2e-art.sh