Scons

Online documentation

• SCons web site
• SCons online documenation(version 4.4.0)
  • To discover the version of scons we are currently using, do the following after "muse setup": echo $SETUP_SCONS
• SconsUPS - how to build a new version of scons, packaged for UPS

FAQ

1. What scons command do you issue to do a cleanup?
   Use the -c option. For example:

   scons -c -j 4

2. What is a .os file?
   It's just an ordinary .o file. Scons uses the .os suffix a bookkeeping device. Files ending in .o are put into static libraries (.a) while files ending in .os are put into shared libraries (.so). While Mu2e only builds shared libraries, it is possible to build both static and shared libraries in one build. In the general case object files destined for shared libraries need to be built with different compiler switches than those destined for static libraries. Therefore a build system needs some bookkeeping device to identify which object files are built for which time of library; the .os vs .o convention is the one that scons has chosen.
3. How can I make scons run faster?
   
   • Use multiple threads
   • Do a partial rebuild
   • Use cached dependency information

   The last two options can be dangerous - use them with care.

4. How can I build debug?
   Log in a new window and navigate to the Offline directory. Set the build to debug:

   ./buildopts --build=debug

   re-setup UPS packages to chose the debug version of packages

   source setup.sh

   wipe out any previous builds, which may have been prof

   scons -c -j 4

   and build debug

   scons -j 4

   You can reverse this choice with "--build=prof". The choice is "sticky" in that if you log in again and run setup.sh, you will get prof or debug, whichever was the most recent choice.

Introduction

SCons is the tool that Mu2e is using to compile and link our software; in short, it is a replacement for the widely used gmake tool. The role formerly played by Makefiles is now filled by one file named SConstruct at the top of the directory tree, and by many files named SConscript, one per directory to be built.

One particularly nice feature is that SCons, itself, keeps track of dependencies on included header files whereas make and its derivatives required a separate tool for this. From the SCons web site:

SCons is an Open Source software construction tool that is a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.

We have also considered using cmake instead of scons. This is discussed below.

Multiple threads

Scons can run in parallel threads, which makes it run much faster on multi-core machines. To ask scons to use mulitple threads

scons -j n

where n is number; at the times during the build when it is possible for several things to be done in parallel, scons will use up to n threads. Our experience is that, on a lightly used machine, n should be about about 1.5 to 2 times the number of available cores. Why more than 1 times the number of coress? This keeps the cores busy while some of the jobs are waiting for disk IO to complete. Typical desktop machines these days have 2 cores; mu2egpvm* machines have 4 cores and mu2ebuild01 has 16 cores. In the Mu2e instructions it suggests using -j 4 on mu2egpvm*; this smaller number was chosen because you will rarely be the only user on the machine. In particular, if we run a tutorial and 10 people all decide to build at the same time, the machine will slow to a crawl. mu2ebuild01 will run the build much faster is in tended only for building code, so we recommend using the full power, -j 24.

Here are the current (v6_3_1) times for a full build

mu2egpvm05	scons -j 4	38m
mu2ebuild01	scons -j 24	10m

Partial rebuilds

If you have a complete build and edit only a few files, it is possible to rebuild only a restricted portion of the code base. This is a very dangerous operation because you are personally taking responsibility for understanding the dependency relations among the entire code base. If you get it wrong, your build will be inconsistent and many run-time errors are possible, including the corruption of existing data files.

If you change any header files you should do a full rebuild:

scons -j 4

There is really only one safe case for doing partial rebuilds. Suppose that you already have a full, clean build and are repeatedly editing several files in MyPackage/src. For example,

 MyPackage/src/My_module.cc
 MyPackage/src/File1.cc
 MyPackage/src/File2.cc

Note that there are no header files in this list. You can build just these files by issuing the command:

  scons -j 4 lib/libMyPackage.so lib/libMy_module.so

This will only consider the following files for rebuilding

• the non-module and non-service .cc files in MyPackage/src
• the module MyPackage/src/My_module.cc

In particular it will NOT rebuild the following:

• Any files outside of MyPackage/src
• Any other module files within MyPackage/src
• Any service files withing MyPackage/src

Why is this faster? When you run

scons

scons must traverse the entire code base to determine which files need to be recompiled and which libraries need to be rebuilt. When you run

 scons target_name

scons only needs to check the dependency tree of target_name.

You can also do a partial clean up,

  scons -c -j 4 lib/libMyPackage.so lib/libMy_module.so

Cached Dependencies

This section describes a feature that is very useful when used correctly but very dangerous when used incorrectly. Read to the end of this section to learn about when the dangers arise.

When you run scons, the first thing that it does is to scan the code base to build a dependency tree. This dependency tree encodes the information: suppose I change a particular file, what other files need to be recompiled or relinked? When scons is finished, it saves the dependency tree in the .sconsign.dblite file (found in the same directory as the SConstruct file).

The default behaviour of scons is that it will not trust the cached dependency tree and will always recompute it. It does this every time that your run scons. If you know that your changes did not change any of the dependency information, then you can tell scons to trust it's cached dependency tree. For example,

scons -j 4 --implicit-deps-unchanged

You can also combine this with partial builds. Suppose that you have edited only Analyses/src/ReadBack_module.cc, you can rebuild it with:

scons -j 4 --implicit-deps-unchanged lib/libmu2e_Analyses_ReadBack_module.so

You can read more about this at: scons documentation. You can use the Prev and Next buttons on that page to read more about managing dependencies in scons.

Here are the results of some real-life tests for builds in which everything was up to date so scons did not have to build anything, just check for work to do. The table below gives results for running on mu2egpvm04, using Offline v6_3_1:

mu2egpvm04

scons -j 4	105s
scons -j 4 --implicit-deps-unchanged	45s

mu2ebuild01

scons -j 24	53s
scons -j 24 --implicit-deps-unchanged	26s
scons -j 4 --implicit-deps-unchanged	26s
scons -j 1 lib/libmu2e_Analyses_ReadBack_module.so	12s
scons -j 1 lib/libmu2e_Analyses_ReadBack_module.so --implicit-deps-unchanged	8s
scons -j 24 lib/libmu2e_Analyses_ReadBack_module.so	12s
scons -j 24 lib/libmu2e_Analyses_ReadBack_module.so --implicit-deps-unchanged	8s

The bottom line is that --implicit-deps-unchanged offers a speedup between about a factor of 1.5 and 2.5. But the times are still slow enough to be noticeable.

What is the dangerous situation? If you have made edits that modify the dependency tree, you use must not use the --implicit-deps-unchanged option. If you do, you may get an inconsistent build and your code may not execute with subtle, hard to detect errors. As a rough guideline, you must not use this option if you have done any of the following:

• Edited any header file.
• Changed any link list.
• Changed versions of external products
• Changed compiler options

The bottom line is that if you are unsure whether or not it is safe to use this option, then do not use it.

Overview of Mu2e Usage of SCons

The scons executable is one of many third party products used by the Mu2e software. These are often referred to as "external products" or just "products" and they are managed by the Fermilab UPS/UPD system. The scons binaries are located under the directory $SCONS_DIR; this environment variable is established when you source setup.sh from the top level of Offline.

After you check out Mu2e software, in the top level directory you will find a file named SConstruct. This defines rules that are common to building all parts of the Offline software.

In each Mu2e PackageName/src directory you will find a file named SConscript. This provides additional information that is needed to build the files in that src/ directory; usually this additional information is only the list of link-time library dependencies, which differ from library to library. Most of the SConscript files are identical or differ only by their link lists; one notable exception is Mu2eG4/src/SConscript which contains the rules needed to find the G4 header files, the G4 compiler -D options and the G4 libraries.

In the following, pay careful attention to whether the sentence is describing the SConstruct file ( the unique file at the top level ) or the many SConscript files.

The SConstruct file defines the information that is needed globally, such as the include paths (-I), the link library paths (-L), compiler switches, linker switches, and the commands needed to build dictionaries. Most of the SConscript files only specify the first order link dependencies of the libraries to be built. A few SConscript files also specify compiler flags, link flags, additional -L and additional -I flags that are needed only for that particular directory.

Some of you may recognize that the SConstruct and SConscript files are python scripts (python is a widely used scripting language). Most Mu2e users will not need to learn python to use SCons; we can think of the SConstruct and SConscript files as just configuration files with a small grammar that we need to learn.

What's in the SConstruct file

The SConstruct file depends on a couple of helpers which hide the details and make this top-level file more readable. These files are under scripts/build/python in the release.

# Functions that do small tasks and build lists
from sconstruct_helper import *
# handles how the input files are collected and the output files are named
from mu2e_helper import *

After adding a few features, the first main operation is to create an scons object called an Environment.

env = Environment( CPPPATH = cppPath(mu2eOpts),   # $ART_INC ...
                   LIBPATH = libPath(mu2eOpts),   # /lib, $ART_LIB ...
                   ENV = exportedOSEnvironment(), # LD_LIBRARY_PATH, ROOTSYS, ...
                   FORTRAN = 'gfortran',
                   BABARLIBS = BaBarLibs()
                 )

The command env=Environment(...) creates a new python variable named env that contains environment information; think of this as a list of environment variables as is available in a standard unix shell. Initially the variable env is local to SConstruct. The above code fragment adds three enviroment variables to env, CPPPATH, LIBPATH and ENV. Note the env and ENV are distinct. CPPPATH and LIBPATH have special meaning to SCons and both have the meaning that you might expect from previous experience with make: CPPPATH is a list of directories that the C-Preprocessor (CPP) will use to resolve #include directives LIBPATH is a list of directories in which the linker will look to find libraries. The variable ENV contains copies of some information from the environment in the shell that called SCons; it does not have special meaning to SCons but will be used by the SConscript files. We use the helper functions (cppPath, etc) to provide the list of includes and libraries which are extracted from the environment for a fixed list of external packages.

The philosophy behind ENV is to discourage writers of SConscript files from directly accessing the external environment but instead to get information about the calling environment via ENV. In this way, SConstruct can augment or override information found in the calling environment.

SConscript files may add elements to CPPPATH and LIBPATH and the syntax for this will be discussed later. Actually the SConscript files can delete or totally redefine CPPPATH or LIBPATH but I cannot imagine a situation in which that would be useful.

Before describing the CPPPATH in detail, you need to know one more thing: scons issues all of its commands from the directory from which you invoked scons. When scons compiles, for example, the file EventGenerator/src/ConversionGun.cc, it will issue the command:

> g++ -o EventGenerator/src/ConversionGun.os {flags} {-Ixxx -Iyyy ... } EventGenerator/src/ConversionGun.cc

What is a .os file? SCons anticipates that some people will want to build both static (.a) libraries and shared (.so) libraries from the same source files. Object files destined for the two libaries usualy require different compiler options ( such as -fpic for shared libaries ); to keep the bookkeeping straight, if an object file is destined for a .a library, it is named .o; on the other hand, if an object file is destined for a .so library, scons names it .os.

Now look in a little more detail at CPPPATH. The environment variable CPPPATH is a list of directory names; For each name in the list, scons will add a -I element to the compile command. The directories are taken from product environmental variables for a fixed set of packages that the release depends on.

We encourage the use of the satellite releases to build parts of Offline while allowing the bulk of Offline to default to a fixed published release. In this case CPPPATH will include the satellite release directories before the base release. LIBPATH is handled similarly.

The next main block of SConstruct is:

genreflex = Builder(action="genreflex ${SOURCES[0]} -s ${SOURCES[1]} ...
...
env.Append(BUILDERS = {'DictionarySource' : genreflex})

These lines tell scons how to generate root dictionary and map files for the data products; this operation uses the root genreflex program. The next few lines are:

# this sets the build flags, like -std=c++14 -Wall -O3, etc
SetOption('warn', 'no-fortran-cxx-mix')
env.MergeFlags( mergeFlags(mu2eOpts) )

These tell scons to add various flags to the compile and link commands; scons is smart enough to figure out which commands are part of the compile command, which are part of the link command and which are part of both.

The part

env.Append( ROOTLIBS = rootLibs() )
...

adds some variables to the enviroment, so that SConscript files can refer to them like:

rootlibs = env['ROOTLIBS']

Then "export" the Environment, so that all SConscript files can "import" it.

Export('env')

The last fragment in SConstruct searches the directory heirarchy to find SConscript files. This is stored in the local variable ss.

ss = sconscriptList(mu2eOpts)
...
env.SConscript(ss)

The line env.SConscript(ss) tells SCcons to execute each of these SConscript scripts.

What About cmake?

We have also considered changing to the build system cmake. The principal argument in its favor is that this is the build tool used by the Fermilab people who maintain art and its underlying tool chain, which gives us a natural support base. We initially chose scons because these same people had been using scons, which gave us a natural support base.

At present our evaluation is that scons is more forgiving to less sophisticated users, at a small enough cost to more advanced users. When no new files have been added, both systems do a build with a single command, make or scons. When a new file has been added, however, cmake requires that your first rerun cmake before running make. On the other hand, with scons, you only need to issue the scons command.

One weakness of scons is that the scons command must be issued from the top level directory, which requires that scons scan the full work tree to discover that only a subset need be done. With cmake, on the other hand, the cmake and make commands may be issued from any directory and they will rebuild only the subset of the project that descends from that directory. While the choice of scons makes work slower for advanced users, it makes it much, much safer for novice users who might not be aware when it is unsafe to issue the cmake command from a lower level directory.

We believe that we can teach scons to do out-of-source builds but we have not yet tried.