SconsUPS

This page is about creating the UPS product tarball for new versions of scons.

Necessary Permissions

To clone the git repo that contains the build scripts for building scons, you need to make sure you have permission on redmine as a developer on the scons project. To get these permissions, email the Scisoft team using the team email scisoft-team@fnal.gov.

After you have the necessary redmine permissions, you should just need kinit.

If you can see the repository that contains the build scripts, you should have all of the permissions that you need. The repository for the build scripts is located at https://cdcvs.fnal.gov/redmine/projects/build-framework/repository/scons-ssi-build

Setting Up a UPS Product Working Area

Build scripts:

These commands will pull the necessary scripts to build scons and a table of dependencies for the scons product. I created a new directory in /mu2e/app/users/ to hold the scripts. Before we can execute the scripts to build scons, we also need to setup UPS.

git clone ssh://p-build-framework@cdcvs.fnal.gov/cvs/projects/build-framework-scons-ssi-build
setup mu2e

After cloning this area, you will see three scripts and a directory inside of the build-framework-scons-ssi-build repo:

bootstrap.sh: grabs product from the web and makes the source code tarball and helper product files
build_scons.sh: builds the actual UPS product tarball
autobuild.sh: for building in distributions only
ups directory: table file which lists dependencies on other packages; for scons, there are only python dependencies, so in the table file is a list of specific python versions which can be built against

Output directory for products:

The area to which output files will be directed needs to be an area that is writeable for UPS products, so we need to copy the .upsfiles directory from /cvmfs/.

mkdir /mu2e/data/users/<your_username>/<output_dir_name>
cp -pr /cvmfs/mu2e.opensciencegrid.org/artexternals/.upsfiles /mu2e/data/users/<your_username>/<output_dir_name>

Running the Scripts and Making Tarballs

After the build script and output file areas are set up, there are two main build scripts to run to link all of the necessary dependencies and produce the tarball of the UPS product. These are bootstrap.sh and build_scons.sh.

To build a specific version of scons, there are a few flags in bootstrap.sh and build_scons.sh that need modified. Typically, the only flags that need modified to build a new version are the flags origpkgver and pkgver. Search for these strings in both bootstrap.sh and build_scons.sh and change them in both scripts before running the next commands. Here's what the two flags mean:

origpkgver: package version like vN_N_N
pkgver: only used for appending a letter if a patch is released; add the letter on the end if needed and remove if not needed

In build_scons.sh, the other place that you may want to keep an eye on are the python qualifiers that are allowed to build against. If the version that you need is not there, it will also need to be added to the table file in the ups directory.

To create the necessary tarballs execute these two commands:

./bootstrap.sh /mu2e/data/users/<your_username>/<output_dir_name>

From this script, you'll see two directories and two tarballs - a directory and tarball for scons and a directory and tarball for the helper product ssibuildshims. This scons tarball will be the source code tarball and will contain the word source. The ssibuildshims tarball contains links for this helper product.

The way to then actually build the scons UPS product tarball is to call the build_scons.sh script, which requires some flags to configure. After specifying the output directory, the next flag is to select the python version of the product, here we are using p392. The next flag, 'tar', says to actually write out the tarball. Without this flag, no tarball will be produced. In the command below, we then direct the output into a log file.

./build_scons.sh /mu2e/data/users/<your_username>/<output_dir_name> p392 tar > scons_vNNN.log

From this script, you'll see the UPS product tarball for scons which will be named something like scons-N.N.N-noarch-p392.tar.bz2. This will be what you use with /cvmfs/.

Adding a New Python Dependency

If you want to build against a version of Python that is not already listed in the ups/scons.table file, you simply need to add the desired dependency to the table file in a block like the one below for p392. The relevant Python files will be sourced from cvmfs.

Flavor=ANY
Qualifiers=p392

  Action = ExtraSetup
    setupRequired( python v3_9_2 )

When upgrading art, one should check and make sure that the necessary Python version is supported by scons and add the correct dependency if it is missing.

Next Steps

Once you have the tarball from the build_scons.sh script, this tarball can be copied to /cvmfs/mu2e.opensciencegrid.org/artexternals and untarred to make the new versions of scons available using the standard procedures.

Before using the new version anywhere in the head of Mu2e repos, be sure to test the new version in a clean working area. The easiest way to do this is to make a clean muse working area and create a new uNNN muse envset which chooses the new version. In the working area, clone all of the repos listed in the file /cvmfs/mu2e.opensciencegrid.org/DataFiles/Muse/linkOrder. Try to build and watch for any warnings or errors.

If necessary, push the changes that you have made to the build scripts back to the redmine build-framework-scons-ssi-build repo. If you need to do this, simply commit and push.

Testing a New Scons

After building all of the repos in /cvmfs/mu2e.opensciencegrid.org/DataFiles/Muse/linkOrder, there are a few more checks that should be done to test a new version of scons.

Run muse build -j 20 RELEASE

Run two executables to check things like the trigger fcl build. Here, just 10 events are substantial. If these tests are successful, then everything should be working correctly.
- mu2e -c -n 10 Production/Validation/ceSimReco.fcl
- mu2e -c -n 10 Production/Validation/reco.fcl -s /pnfs/mu2e/persistent/users/mu2epro/valjob/reco_210611/dig.brownd.CeEndpointMix.MDC2020d.001203_00000000.art