User:Kutschke/Mu2eEnvironments

Introduction

This note provides the big picture for the transition of Mu2e Environment management and build management that started in the spring of 2024 and is ongoing. The target audience is people who are new to Mu2e software (or have been away from it for a while) but who are experienced with the concepts of environment and build management.

UPS

Until recently Mu2e used a Fermilab developed system called UPS for management of external software products such as g++, art, root, geant4 etc. When we build Mu2e Offline we use about 60 external software products. Most of the UPS products were curated by CSAID, built and packaged for distribution as tar.bz2 files. Here curated means that CSAID ensured that

1. the set of products chosen was internally consistent (ie ensure that version A of product 1 is known to work with version B of product 2; if products 1 and 2 both depend on product 3, choose a version of product 3 that is known to work with both.)
2. all products were compiled and linked with the same compiler and similar enough compiler options
3. the build options for each product were selected appropriately. For example xrootd can be built with support for authentication by proxies, tokens, both or neither. And root can be built with and without xrootd support.

Spack

CSAID has decided to discontinue support for UPS and to migrate to a new system for environment management named Spack. It also supports the build and distribution of a curated set of software packages (UPS product means the same as spack package). The reasons for making this change are that

1. UPS is 30+ years old and the people who have experience supporting it are near retirement.
2. Rather than invest new resources in an old technology we should choose an industry standard solution.
3. Spack is the industry standard for US supercomputer centers. By adopting it for our regular use we hope to streamline our onboarding into the supercomputer centers.
4. Curation of UPS products takes a lot of CSAID resources and spack promises to reduce the amount of effort required to do this. (Personal comment: this has not yet proven to be the case; at this stage in the learning curve it is quite the opposite. Perhaps we will get there).

Spack is a big powerful tool and users can make many choices about how to use it. So it has a steep, long learning curve. CSAID, Mu2e and other Fermilab based experiments are still working through the learning curve. You can expect that policies and procedures for how to use Spack will evolve over the next year or so.

Transition to Spack

CSAID decided that, for the AL9 operating system, CSAID supplied software will only be distributed via Spack. On June 30, 2024, the SL7 operating system reached end-of-support-life. All Mu2e interactive machines now run AL9. We can still work in SL7 docker containers on our AL9 machines; see Legacy Access to SL7. So we still have all of our UPS distributions of CSAID software. We expect that new CSAID supplied software will be supplied only for AL9 machines and only via Spack.

There are some UPS products that "script only" and these continue to work on AL9 machines just as they worked on SL7 machines. Mu2e uses some such products in our AL9 environment. They will be transition to spack as we better understand how to use spack. In UPS-speak, script only products are called "NULL flavored".

Because we are still using UPS in this limited say, you will still see "setup abc". Be a little suspicious of these instructions; please check with Mu2e computing management to see if that instruction is still current.

Offline vs Online

UPS Era Usage

The Mu2e online and offline communities have made different choices about environment management and build systems.

The offline community wrote a thin scripting layer named Muse that manages the environment by setting up a curated set of UPS products and defining some additional environment variables and bash functions. When you do a build using the muse build command it delegates the build to the scons build system, which can build multiple repositories concurrently. Each source directory contains a file named SConscript that contains instructions for scons.

The online community chose to use Fermilab supplied build systems named MRB (Multi Repository Build) to manage their environment and oversee coordinated builds of many repositories. The underlying build system is cmake. They also use UPS to provide external software such as the compiler, art, root, clhep etc.

MRB is the product that CSAID uses to build all of their UPS supplied software products. The online group chose to use MRB because it is the official CSAID supported environment and build management system. The offline group already had a mature environment when the online group made their choice and a cost benefit analysis reached the decision that we would stay with muse/scons. The driver was that MRB required a higher level of expertise than did Muse/scons, so Muse/scons was better suited to novice users.

Most of the repositories managed by the online group are independent of the offline software so these worlds co-existed independently for a long time. There are two points of contact.

1. The repository artdaq_core_mu2e contains the raw data formats. It is used by both online and offline. It is owned and maintained by the online community.
2. The trigger runs art modules that are found in repositories managed by the offline community and are developed in the offline world.

When it is necessary to update artdaq_core_mu2e for use in the offline world, an expert from the offline community builds it and installs it in UPS using MRB. Once it is in UPS, it's just another UPS product like art, root, geant4 and so on.

For many years the offline community provided a tool to package a build of the Offline repository as a UPS product. When the online community needed a new version of Offline, they built it using the native offline tools and installed it in UPS. Again, it's just another UPS package.

This worked well throughout the UPS era.

The online and offline worlds maintain separate collections of UPS products. The reason is that each world needs to be able to operate when the other is down.

Spack Era

In the spack era, CSAID developed a spack based system for environment and build management. They have completely retired UPS. The Mu2e online world has adopted this system. The build system component is cmake based. To be a bit more specific, there is component of the build system named cetmodules that was adapted from UPS awareness to spack awareness. Modern versions of cetmodules work with both UPS and spack but at some time support of UPS will be withdrawn. Online end users need to learn a new sequence of commands to use the new spack based environment.

The offline community adapted the backend of Muse so that it has a UPS aware backend and a spack aware backend. The default is that it chooses the UPS aware backend when it is run on an SL7 machine and the spack aware backend when it is run on an AL9 machine. From an end user viewpoint, nothing changed.

As with UPS, the online and offline worlds maintain separate collections of spack packages. The reason is the same.

When a new version of artdaq_core_mu2e is needed in the offline environment, an offline expert builds it natively and installs it as spack package in the offline spack area.

To get access to modules from the Mu2e Offline repo, the online team populated the Mu2e Offline repository with CMakeLists.txt files. These will build Offline in their spack based environment.

The instructions to build Offline using cmake and spack can be found at: Building_Offline_with_cmake#Using_Spack. These instructions may change at any time.

Offline Use

For Offline software, Mu2e created a software product named Muse to manage the Mu2e software environment. It's a thin layer that collects commonly used sequences of commands into a single muse command. Muse defines the concept of an envset, short for "environmental set", which is a script that creates a complete Mu2e environment by choosing a curated set of UPS products and defining some additional environment variables. Envsets are versioned with names like p001, p002 ... p057, which is the current one as of this writing. Any version of the Mu2e software is guaranteed to work with at least one envset and will often work with nearby versions.

Muse delegates building code to scons, which Mu2e has used from it's early days.

When it was time to upgrade to a new version of art, we

1. unwound a curated set of tar files onto /cvmfs/mu2e.opensciencegrid.org/artexternals/
2. built and tested a few Mu2e software repositories using the new products and installed them into /cvmfs/mu2e.opensciencegrid.org/artexternals/ (BTrk, KinKal, artdaq_core_mu2e).
3. Created a new Muse envset to chose the new set of products
4. Tested building and running Offline and some other Mu2e repositories against the products in the new envset. Update our code as necessary.
5. At an appropriate time update the default envset to the new one.

Users can choose to continue to use an old envset or migrate to the new default. All builds are labelled with the envset used for the build.