User:Kutschke/Mu2eEnvironments

Introduction

This note provides the big picture for the transition of Mu2e Environment management and build management that started in the spring of 2024 and is ongoing. The target audience is people who are new to Mu2e software (or have been away from it for a while) but who are experienced with the concepts of environment and build management.

UPS

Until recently Mu2e used a Fermilab developed system called UPS for management of external software products such as g++, art, root, geant4 etc. When we build Mu2e Offline we use about 60 external software products. Most of the UPS products were curated by CSAID, built and packaged for distribution as tar.bz2 files. Here curated means that CSAID ensured that

1. the set of products chosen was internally consistent (ie ensure that version A of product 1 is known to work with version B of product 2; if products 1 and 2 both depend on product 3, choose a version of product 3 that is known to work with both.)
2. all products were compiled and linked with the same compiler and similar enough compiler options
3. the build options for each product were selected appropriately. For example xrootd can be built with support for authentication by proxies, tokens, both or neither. And root can be built with and without xrootd support.

Spack

CSAID has decided to discontinue support for UPS and to migrate to a new system for environment management named Spack. It also supports the build and distribution of a curated set of software packages (UPS product means the same as spack package). The reasons for making this change are that

1. UPS is 30+ years old and the people who have experience supporting it are near retirement.
2. Rather than invest new resources in an old technology we should choose an industry standard solution.
3. Spack is the industry standard for US supercomputer centers. By adopting it for our regular use we hope to streamline our onboarding into the supercomputer centers.
4. Curation of UPS products takes a lot of CSAID resources and spack promises to reduce the amount of effort required to do this. (Personal comment: this has not yet proven to be the case; at this stage in the learning curve it is quite the opposite. Perhaps we will get there).

Spack is a big powerful tool and users can make many choices about how to use it. So it has a steep, long learning curve. CSAID, Mu2e and other Fermilab based experiments are still working through the learning curve. You can expect that policies and procedures for how to use Spack will evolve over the next year or so.

Transition to Spack

CSAID decided that, for the AL9 operating system, CSAID supplied software will only be distributed via Spack. On June 30, 2024, the SL7 operating system reached end-of-support-life. All Mu2e interactive machines now run AL9. We can still work in SL7 docker containers on our AL9 machines; see Legacy Access to SL7. So we still have all of our UPS distributions of CSAID software. We expect that new CSAID supplied software will be supplied only for AL9 machines and only via Spack.

There are some UPS products that "script only" and these continue to work on AL9 machines just as they worked on SL7 machines. Mu2e uses some such products in our AL9 environment. They will be transition to spack as we better understand how to use spack. In UPS-speak, script only products are called "NULL flavored".

Because we are still using UPS in this limited say, you will still see "setup abc". Be a little suspicious of these instructions; please check with Mu2e computing management to see if that instruction is still current.

Offline Use

For Offline software, Mu2e created a software product named Muse to manage the Mu2e software environment. It's a thin layer that collects commonly used sequences of commands into a single muse command. Muse defines the concept of an envset, short for "environmental set", which is a script that creates a complete Mu2e environment by choosing a curated set of UPS products and defining some additional environment variables. Envsets are versioned with names like p001, p002 ... p057, which is the current one as of this writing. Any version of the Mu2e software is guaranteed to work with at least one envset and will often work with nearby versions.

Muse delegates building code to scons, which Mu2e has used from it's early days.

When it was time to upgrade to a new version of art, we

1. unwound a curated set of tar files onto /cvmfs/mu2e.opensciencegrid.org/artexternals/
2. built and tested a few Mu2e software repositories using the new products and installed them into /cvmfs/mu2e.opensciencegrid.org/artexternals/ (BTrk, KinKal, artdaq_core_mu2e).
3. Created a new Muse envset to chose the new set of products
4. Tested building and running Offline and some other Mu2e repositories against the products in the new envset. Update our code as necessary.
5. At an appropriate time update the default envset to the new one.

Users can choose to continue to use an old envset or migrate to the new default. All builds are labelled with the envset used for the build.

The Mu

CSAID has decided to discontinue support for UPS and to migrate to a new system for environment management named Spack. The short term plan for spack is that CSAID will build and distribute