UPS
The End of UPS/UPD
This page describes a software tool named UPS that has been used by Mu2e since it's inception. Fermilab will end support for UPS with the retirement of the Scientific Linux 7 (SL7) operating system on June 30, 2024 and it's replacement with Alma Linux 9 (AL9). On AL9 systems most of functions of UPS will be replaced by a system named Spack.
This page describes UPS as it worked in SL7. It may remain relevant after June 30, 2024 for two cases. First, Mu2e may sometimes run legacy software in SL7 containers that run on AL9 hosts; in that case we will use legacy software distributed by UPS. Second, UPS will continue to work in a limited way after June 30. It is Mu2e's intention to have 100% of our software moved from UPS to Spack by that time but, depending on the available workforce, some packages may remain UPS based for a short time after June 30, 2024.
What is UPS/UPD
UPS/UPD is a tool, developed and supported by Fermilab, for management of external software products in a unix environment. From the Mu2e point of view, "external" means everything that does not come from our GitHub repository; this includes art, Geant4, ROOT, CLHEP, boost and so on. It also includes some stand-alone products such as G4beamline. Most Mu2e code is NOT deployed as a UPS product. The acronyms stand for Unix Product Distribution and Unix Product Support. This page gives an overview relevant for Mu2e; for additional information consult the UPS Manual: PDF versionl or wiki version; there is also a getting started guide linked from UPS/UPD wiki home page.
UPS/UPD is installed on all Fermilab managed computers that are used by Mu2e. When you follow the instructions to install the external producuts on a remote computer ( your laptop or a computer at your home institution), UPS/UPD is included in the package.
When you wish to use an external product, such as geant4, you issue a command like:
setup -B geant4 v4_9_6_p02 -q+e4:+prof
The next few paragraphs will talk about each piece of this command.
In the example above, the command setup not the standard unix setup command (man 2 setup); it is a UPS/UPD command that is defined for you when you initialize UPS/UPD. As you keep reading this document you will learn that UPS/UPD is initialized for you when you initialize the Mu2e working environment. Few Mu2e users will need to access the unix setup command; if you do need to, and if it has been shadowed by UPS/UPD setup, you can access the unix setup command by giving its full path, usually /usr/bin/setup. The UPS/UPD setup command predates the unix setup command and it is rare that one would want to use both in a single shell; so we have never changed the name.
In the above, "v4_9_6_p02" is the version of Geant4 that is requested; at any given time, Mu2e might be completing some projects with an older version of Geant4 while starting new projects with a newer version. Typically the Mu2e UPS/UPD repository will contain several versions of Geant4.
After you issue a setup command, new environment variables will be defined and, in many cases, new directories will be added to your path and to your LD_LIBRARY_PATH. For the example, UPS/UPD defines the environment variable GEANT4_DIR, which points to the root directory of the version of the geant4 product that you requested. In general, the root directory for a product named XXX is defined by the environment variable XXX_DIR. For some, but not all, products, the source code is available as XXXSOURCE_DIR. For many products the environment variables XXX_INC, XXX_LIB and XXX_BIN are defined as the location to look for header files, libraries and executables. Some external products have native environment variables, such as G4INCLUDE and ROOTSYS; UPS/UPD setup also defines these environment variables.
The argument "-q+e4:+prof" is called a qualifier. The UPS/UPD system allows us to make serveral different builds of one version of a product; these builds might differ in the compiler version, optimziation level or in the required version of some other product, such as CLHEP. UPS/UPD uses the notion of qualifiers to keep the bookkeeping straight for multiple builds of one version of a product. In the above example, the qualifier "e4" is a shorthand defined by the art team for a compiler version, a set of compiler switches and a version of CLHEP; when Geant4 is built with the "e4" qualifier, it can be linked against a version of art that was also built with the "e4" qualifier. The qualifier "prof" requests that we find a version of Geant4 that was built with maximum optimization and with some, but not all, debugger symbols retained; the symbols retained are enough to profile the code and to get a traceback from a core dump; it is not enough to do full line by line debugging. The name "prof" is motivated by the notion that the build retains enough debugger symbols for profiling.
You will sometimes see the qualifier string written with the plus signs and sometimes without. UPS/UPD has the notion of a "not-exact-but-good-enough" qualifier match. Using the plus sign on each qualifier disables this feature and tells UPS that you require an exact match. For products used by Mu2e Offline, Mu2e always requires an exact match.
The -B option will be discussed in the section on mismatch detection; the short story is that it tells UPS to issue an error if we accidentally try to setup two differnt versions of the same product.
A more complete version of the above command might look like:
setup -B geant4 v4_9_6_p02 -f wLinux64bit+2.6-2.5 -q+e4:+prof
Here the "-f Linux64bit+2.6-2.5" requests a specfic "flavor" of the product. Flavor is used to distinguish different operating systems, for example SLF5 vs SLF6; in the above example, the flavor specifies SLF5. When UPS/UPD was first developed the lab was supporting heterogeneous unix instalaltions that included Sun OS, IBM AIX, SGI and DEC Alphas. Most of the time the correct flavor can be determined automatically by the setup command and it is not necessary to specify it. The reason why the string "Linux64bit+2.6-2.5" means SLF5 has a long history behind it; for now just learn the translation.
The manager of a UPS/UPD repository can configure it so that some products have a default version and default qualifiers. In this case, one could simply type:
setup XXX
In UPS/UPD-speak the default version is called the current version.
Mu2e does not allow the use of current version for products that may influence physics results and requires that we always ask for exactly the version and qualifiers that we want. The reason for this is to problem that a default qualifier might change in a poorly documented fashion, making it difficult to reproduce results. Moreover, the current version may differ from one site to the next so that code run at Fermilab gives a different result than code run at, say, LBL! Mu2e records the required versions of each product a Muse envset (see Muse). When you give the command "muse setup", it will ensure that the correct versions are chosen.
For products such as mu2egrid, jobsub and ifdhc that are used as a part of grid workflows, Mu2e does allow the use of the current version.
Finally, UPS/UPD products can be declared to depend on specific qualifiers of specific versions of other products. A particular version of art, for example, depends on particular versions of ROOT, CLHEP, boost, and several other products. When you issue the command
setup -B art v1_08_09 -q+e4:+mu2e:+prof
it will, behind the scenes, setup the appropriate versions of the products on which it depends.
It is possible to setup product A that requires a particular version of product C, and then to setup product B that requires a different version of product C. UPS/UPD has an optional feature that will automatically detect such conflicts; this will be discussed in the section on mismatch detection.
The names of qualifiers are not 100% systematic. For example, during migration to a new compiler it is common to see two builds of a product version, one with no qualifier ( meaning the old compiler ) and one with a qualifier that is a compiler version (meaning the new compiler).
When a product has qualifiers and/or flavors, the path to the binary files ( bin/ and lib/ subdirectories), contains a mangled name that encodes the flavor+qualifier. For example, if you setup Offline v4_0_6 on mu2egpvm02 ( SLF5 64 bit ), you will see
> echo $G4LIB /grid/fermiapp/products/mu2e/artexternals/geant4/v4_9_6_p02/Linux64bit+2.6-2.5-e4-prof/lib64
The string "Linux64bit+2.6-2.5" identifies the flavor; it says that this release was built for a particular version of SLF5 on 64 bit hardware; for historical reasons operating system is encoded using the version number of the kernel. The string "-e4-prof" identifies the qualifiers.
If you look at the G4 product root directory you will see that a six different builds are available:
> ls $GEANT4_DIR Linux64bit+2.6-2.12-e4-debug Linux64bit+2.6-2.12-e4-prof Linux64bit+2.6-2.5-e2-debug Linux64bit+2.6-2.5-e2-prof Linux64bit+2.6-2.5-e4-debug Linux64bit+2.6-2.5-e4-prof source ups
The directories whose names contain "Linux64bit+2.6-2.12" contain binaries that were built on an SLF6 machine. The directories whose names contain "e2" were built for use with an older version of art. In each case both debug and profile builds are provided.
If your build system needs access to the mangled flavor+qualifier string, it is usually available in one of two ways, either via an environment variable named XXX_FQ, or via an environment variable named XXXX_FQ_DIR. For example,
> printenv GEANT4_FQ_DIR /grid/fermiapp/products/mu2e/artexternals/geant4/v4_9_6_p02/Linux64bit+2.6-2.5-e4-prof > printenv ROOT_FQ Linux64bit+2.6-2.5-mu2e-e4-prof
I don't know why some packages are done one way and some the other.
The bottom line is that UPS/UPD exists to
- Distribute many versions of many products.
- For each version of each product, distribute many distinct builds, that might differ by OS for which they were built, the optimization level with which they were built or the version+qualifier of the other products on which they depend.
For each UPS/UPD product only one flavour+qualifier of one version of the product may be setup at a time.
A Common Error Message
From time to time you will issue a UPS command and get the following error message:
You are attempting to run "setup" which requires administrative privileges, but more information is needed in order to do so. Password for root:
This means that the UPS setup command is not known to your shell and your shell is instead running the unix system command /usr/bin/setup, which is used to coqntrol hardware.
In most cases this means one of two things:
- You do not have the correct login scripts; these are described in the wiki page on Shells.
- Your login scripts were not executed
If you do not have the correct login scripts, fix the problem, log out and log in again.
If you do have the correct login scripts, then source them in your current shell; watch for error messages. If there are error messages either resolve them yourself or report them to the Mu2e software team. If you have modified your login scripts, and if you introduced an error before the code that initializes ups, then the script will exit before it has a chance to initialize ups.
Unsetup
A product that has been setup can removed from your enviroment by the UPS/UPD command unsetup. For example,
unsetup geant4
This will delete the environment variables that were defined by the setup command and will remove any elements that were inserted into your path or your LD_LIBRARY_PATH.
Historically unsetup has been been very fragile; not all product maintainers have properly maintained the unsetup actions for their product. For that reason it is not widely used.
Now suppose that you do the following:
setup A v0 .... do some work setup A v1 .... do some more work
that is, you setup v0 of a product A and do some work using it. Then you setup v1 of the same product. Behind the scenes, the second setup first does an unsetup of version v0; then it does the setup of v1. If product A has a properly maintained unsetup, this will work fine. On the other hand, if product A does not have a properly maintained unsetup, this can lead to a broken environment that sometimes finds version v0 and sometimes finds version v1.
For this reason, we recommend that, in your login scripts, you only setup products that will never be unsetup during your login session.
We further recommend that you never setup more than one of Mu2e Offline, G4Beamline or MARS at the same time; it is possible to do this correctly but it is very, very fragile. A corollary of this is that we recommend that you do NOT setup any of these three things in your login scripts.
Relocatable UPS/UPD
The traditional use of UPS/UPD was to install it as root in system wide disk space; a common variant is to create an account named products and have that account own the the UPS/UPD disk space. A second tradition was to embed hard coded paths into the installation.
The very earliest releases of UPS/UPD required hard coded paths in many places. Fairly soon UPS/UPD was redesigned to allow one to only use relative paths, thereby making a UPS/UPD directory tree relocatable. However it was tedious to do so and few did so.
In the very early days of Mu2e, Ron Rechenmacher from FNAL CD made a changes to UPS/UPD that make it much easier to use UPS/UPD in a relocatable way and Mu2e is doing so; we can make a tar file for the full UPS/UPD area, copy it to a remote site and untar it. This installation is usable as is, with no changes. We refer to this as "Relocatable UPS/UPD".
Mu2e Use of Relocatable UPS/UPD
In the instructions to start a Mu2e example when logged into a Mu2e computer at Fermilab, the first step is:
mu2einit
This is simply a shorthand for:
source /grid/fermiapp/products/mu2e/setupmu2e-art.sh
This script is called the Mu2e site specific setup script and is discussed in the next section.
Mu2e Site Specific Setup Script
This section needs to be updated
At Fermilab, and on any other site that mounts the Mu2e cvmfs area, the site specific setup script is:
source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh
If you have installed the recommended login scripts (see Shells#setup_scripts), you can also run this by typing the much shorter command:
mu2einit
If you are running at a site that does not mount the Mu2e cvmfs area, speak with the site administrator to learn where the correct setup script is. Once this is done, then all Mu2e Offline code should work exactly the same at all sites. If this is not true then let the Mu2e software team know so that we can fix it. A few other definitions that are common to all sites and that do not have a more natural home are also found in this script.
This script does five jobs:
- It initializes the UPS/UPD system.
- It defines the environment variable MU2E_DATA_PATH.
- It defines the environment variables CVSROOT and CVS_RSH (only a few small Mu2e subgroups still use cvs).
- It defines the environment variable GROUP.
- It does a UPS setup of git
- It does a UPS setup of muse
Of these, only item 2 should need maintenance at remote sites.
If this script is installed in the correct place, at the top of the UPS/UPD installation, it will automatically find the UPS/UPD installation and will not need site specfic editing.
The second step defines where on the remote site to find the information that, at Fermilab, is found at:
/cvmfs/mu2e.opensciencegrid.org/DataFiles/
This directory tree holds things like magnetic field maps, the Muse envsets and other auxiliary files.
The third step is needed so that cvs commands will work correctly. The fourth step is used by the software that submits grid jobs; we expect that, in the future, we will support job submission from remote sites; in the mean time it does not hurt to define this one extra environment variable.
The last two steps setup UPS products that are needed to begin work on Mu2e.
Chaining Repositories
When you initialize UPS/UPD, it defines the environment variable PRODUCTS. This environment variable contains a colon separated list of directory names. UPS treats this list as a search path for products; that is, when you give the command
setup A vn -qQUAL
UPS will in the first directory on the list, presume it to be a UPS repository and look in it for a product named A with the specified version and qualifiers and with the flavor that matches the machine; if UPS finds such a product, it will set it up; if it does not find such a product, UPS will try the next directory in the list; and so on. The first match wins. Once PRODUCTS has been defined, you may add the directory names of other UPS repositories to the list ( at the beginning, middle or end). On each invocation of the setup command, UPS will use the current value of PRODUCTS as the search path. It is OK if PRODUCTS contains a mix of old-style and relocatable style repositories.
Using Both Old and New UPS/UPD on FNAL Machines
This section needs to be updated
When you log in to one of the mu2egpvm* machines, it will execute a lab supplied ~/.profile script. This script contains the fragment:
if [ -f "/afs/fnal.gov/ups/etc/setups.sh" ]
then
. "/afs/fnal.gov/ups/etc/setups.sh"
fi
This initializes UPS and sets the PRODUCTS environment variable to point at /grid/fermiapp/products/common/db. This is an old-style ups repository that contains some products that are shared among all of the intensity frontier experiments, such as jobsub. Most of these products contain some assumptions about the Fermilab site and, in general, will not work at remote sites. When you execute the command
muu2einit
it looks for a product named mu2e in /grid/fermiapp/products/common/db. This is a trival product that does nothing more than
source /grid/fermiapp/products/mu2e/setupmu2e-art.sh
On the interactive machines there is both an old style UPS/UPD installation, supported by the lab, and a relocatable style UPS/UPD installation containing Mu2e specific packages and supported by Lynn Garren. Offically released Mu2e code uses only the relocatable UPS/UPD installation.
From time to time, however, a development effort needs to use a product that is found in the old style UPS/UPD but not yet in the Mu2e UPS/UPD. In this case, the path of least resistance is to use the required package from the old style installation. This is complicated by the fact that old style UPS/UPD on the interactive machines are maintained by different people and contain different packages. We can always ask for a new package, or a new version of an existing package, to be installed in any UPS/UPD installation.
I believe, but am not 100% certain, that the old style UPS/UPD is not available on the grid worker nodes.
On mu2egpvm* the old style UPS/UPD can be enabled by
source /afs/fnal.gov/ups/etc/setups.sh
or
source /afs/fnal.gov/ups/etc/setups.csh
On any machine it is possible use both the old and new style installations at the same time. I believe that it is recommended to setup the old style version first and the new style second; this ensures that the new system takes precedence. Also, the old style system obeys slightly different conventions for which environment variables are defined by each setup.
The default fermilab login scripts always setup the old style UPS/UPD. We may wish to change this some day.
You can see which UPS/UPD databases have been setup by
> echo $PRODUCTS
This will give a colon separated list of all of the UPS/UPD root directories that they system will look in to find products.
Listing Available Products and Versions
To list all available versions of all products:
> ups list -aK+
To list all available versions of, for example, geant4
> ups list -aK+ geant4
If you are logged in to one of the Mu2e machines at fermilab, the above commands will list all of the products found either in the Mu2e relocatable UPS/UPD or in the system wide standard FNALU UPS/UPD. To restrict the commands to look only at the Mu2e relocatable UPS/UPD,
> ups list -aK+ geant4 -z $MU2EPRODUCTS
( Aside: the environment variable MU2EPRODUCTS was defined on Oct 31, 2011. The variable will not be defined in your shell unless you have logged in since then. ) This same syntax will work on your own laptop if you have deployed the mu2e UPS/UPD products on your laptop.
Mismatch Detection
Suppose that there are three products A, B and C, where product A requires v2 of product C, while product B requires v1 of product C. Consider the sequence of commands
setup A vA setup B vB
where vA and vB are the required versions of products A and B. Behind the scenes the following happens:
setup C v2 // Caused by setup A unsetup C // Caused by setup B setup C v1
This is done silently. If product A happens to work correctly with v1 of product C, then all is well. If not, there will be some sort of error that may be very difficult to detect and understand.
You can tell UPS to detect this situation and issue an error when it detects it.
setup -B A vA setup -B B vB
In this case UPS/UPD will detect the mismatched during the setup of product B and will issue an error message.
UPD install
Widely-used relocatable products are usually stored in the upd ftp server. A few infrastructure (as opposed to art-maintained) and older products are only stored here. If so, you can install it directly:
setup upd ups list -a mkdir myproducts cd myproducts rsync -aur /cvmfs/mu2e.opensciencegrid.org/artexternals/.up?files . mkdir prd # this seems to prevent some errors in some packages, may be a better way... unsetup python upd (for help) upd list -aK+ upd list -aK+ cigetcert upd install -v -j cigetcert v1_16_1 -z $PWD upd install -v -j cpn v1.7 -z . upd install -G "-c -q stken" encp v3_11c -q dcache -z $PWD
Creating a Local UPS Products Repository
At times it might be useful to create a local version of a UPS product that is usually published and served from cvmfs. This might be a new product for development, or testing a new version of a product with the intent to later publish it, or overriding an existing published version of a product with local changes.
- Create a directory to hold your ups products. The name is not important but the two traditional choices are products and artexternals. Then add some configuration information needed by ups. You only need to do this step once. $ mkdir products
- Install new ups products into this directory following the instructions for those packages. Your local products directory should contain the directories which are simply the product names and inside those directories, there should be version-numbered directories.
- On each login, setup your environment and add your new products directory to the PRODUCTS path. The local path should appear before the cvmfs products paths in order to override the cvmfs product versions. This step should be done after "mu2einit," which defines the standard PRODUCTS path, and before "muse setup," which uses the path to setup UPS products. $ source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups
$ cd products
$ cp -R /cvmfs/mu2e.opensciencegrid.org/artexternals/.upsfiles/ .
$ mu2einit
$ export PRODUCTS=/path/to/your/new/products:${PRODUCTS}
$ muse setup ..
Example local build
- setup your environment $ source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups
- get the product tar file for the source code (it doesn’t matter where you put the tar file, but I like to keep them in the top level of my products directory) $ cd path/my_PRODUCTS
- untar the source code into your products directory $ tar xf geant4-4.10.4-source.tar.bz2 -C prod/external
- make changes to the build script as needed $ nano prod/external/geant4/v4_10_4/build_geant4.sh
- execute the build script with the first argument being the absolute path to the products directory, the second being the qualifiers, and the third being the build type. you can always excite the script without any arguments in order to get some guidance about the arguments $ ./prod/external/geant4/v4_10_4/build_geant4.sh /path/my_PRODUCTS/prod/external e15:cl23 prof
$ mu2einit
$ curl -O https://scisoft.fnal.gov/scisoft/packages/geant4/v4_10_4/geant4-4.10.4-source.tar.bz2 (see Scisoft)
UPS Expansion
As we expand to more platforms and operating systems, we find that the kernel number is no long enough to define exactly what the flavor should be. UPS 6.0, will provide for more distinctions - kernel, glibc, and distribution (SL, CentOS, Ubuntu). Here is the plan. Mu2e is using this system now, as of 1/2018.
UPS Override
The Fermilab linux machines are primarily Scientific Linux 7 and our code and supporting UPS products are built for this OS. This build is derived from a redhat linux release, but there are other linux builds that are also derived from redhat, such as CentOS, AlmaLinux and Fedora. It is likely that these other derived builds can use our SL7 code build without modification. One important factor is a compatible GLIBC version. When trying our code builds on one of these systems, it is necessary to tell UPS to use the SL7 build, and not try to find a build specific to the actual OS. This is done with a UPS flavor override, before any setups:
export UPS_OVERRIDE="-H Linux64bit+3.10-2.17-sl7"
This has been successful on
- CentOS Linux release 7.9.2009 (Core)
- some versions of Fedora