Scons

Online documentation

• SCons web site
• SCons online documenation(version 2.3.4)

FAQ

1. What scons command do you issue to do a cleanup?
   Use the -c option. For example:

   scons -c -j 4

2. What is a .os file?
   It's just an ordinary .o file. Scons uses the .os suffix a bookkeeping device. Files ending in .o are put into static libraries (.a) while files ending in .os are put into shared libraries (.so). While Mu2e only builds shared libraries, it is possible to build both static and shared libraries in one build. In the general case object files destined for shared libraries need to be built with different compiler switches than those destined for static libraries. Therefore a build system needs some bookkeeping device to identify which object files are built for which time of library; the .os vs .o convention is the one that scons has chosen.
3. How can I make scons run faster?
   
   • Use multiple threads
   • Do a partial rebuild
   • Use cached dependency information

   The last two options can be dangerous - use them with care.

4. How can I build debug?
   Log in a new window and navigate to the Offline directory. Set the build to debug:

   ./buildopts --build=debug

   re-setup UPS packages to chose the debug version of packages

   source setup.sh

   wipe out any previous builds, which may have been prof

   scons -c -j 4

   and build debug

   scons -j 4

   You can reverse this choice with "--build=prof". The choice is "sticky" in that if you log in again and run setup.sh, you will get prof or debug, whichever was the most recent choice.

Introduction

SCons is the tool that Mu2e is using to compile and link our software; in short, it is a replacement for the widely used gmake tool. The role formerly played by Makefiles is now filled by one file named SConstruct at the top of the directory tree, and by many files named SConscript, one per directory to be built.

One particularly nice feature is that SCons, itself, keeps track of dependencies on included header files whereas make and its derivatives required a separate tool for this. From the SCons web site:

SCons is an Open Source software construction tool that is a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.

We have also considered using cmake instead of scons. This is discussed below.

Multiple threads

Scons can run in parallel threads, which makes it run much faster on multi-core machines. To ask scons to use mulitple threads

scons -j n

where n is number; at the times during the build when it is possible for several things to be done in parallel, scons will use up to n threads. Our experience is that, on a lightly used machine, n should be about about 1.5 to 2 times the number of available cores. Why more than 1 times the number of coress? This keeps the cores busy while some of the jobs are waiting for disk IO to complete. Typical desktop machines these days have 2 cores; mu2egpvm* machines have 4 cores and detsim has 32 cores. In the Mu2e instructions it suggests using -j 4 on mu2egpvm* or detsim; this smaller number was chosen because you will rarely be the only user on the machine. In particular, if we run a tutorial and 10 people all decide to bulid at the same time, the machine will slow to a crawl. Even two people doing scons -j 16 will make detsim unusable by anyone else.

Partial rebuilds

If you have a complete build and edit only a few files, it is possible to rebuild only a restricted portion of the code base. This is a very dangerous operation because you are personally taking responsibility for understanding the dependency relations among the entire code base. If you get it wrong, your build will be inconsistent and many run-time errors are possible, including the corruption of existing data files.

If you change any header files you should do a full rebuild:

scons -j 4

There is really only one safe case for doing partial rebuilds. Suppose that you already have a full, clean build and are repeatedly editing several files in MyPackage/src. For example,

 MyPackage/src/My_module.cc
 MyPackage/src/File1.cc
 MyPackage/src/File2.cc

Note that there are no header files in this list. You can build just these files by issuing the command:

  scons -j 4 lib/libMyPackage.so lib/libMy_module.so

This will only consider the following files for rebuilding

• the non-module and non-service .cc files in MyPackage/src
• the module MyPackage/src/My_module.cc

In particular it will NOT rebuild the following:

• Any files outside of MyPackage/src
• Any other module files within MyPackage/src
• Any service files withing MyPackage/src

Why is this faster? When you run

scons

scons must traverse the entire code base to determine which files need to be recompiled and which libraries need to be rebuilt. When you run

 scons target_name

scons only needs to check the dependency tree of target_name.

You can also do a partial clean up,

  scons -c -j 4 lib/libMyPackage.so lib/libMy_module.so

Cached Dependencies

This section describes a feature that is very useful when used correctly but very dangerous when used incorrectly. Read to the end of this section to learn about when the dangers arise.

When you run scons, the first thing that it does is to scan the code base to build a dependency tree. This dependency tree encodes the information: suppose I change a particular file, what other files need to be recompiled or relinked? When scons is finished, it saves the dependency tree in the .sconsign.dblite file (found in the same directory as the SConstruct file).

The default behaviour of scons is that it will not trust the cached dependency tree and will always recompute it. It does this every time that your run scons. If you know that your changes did not change any of the dependency information, then you can tell scons to trust it's cached dependency tree. For example,

scons -j 4 --implicit-deps-unchanged

You can also combine this with partial builds. Suppose that you have edited only Analyses/src/ReadBack_module.cc, you can rebuild it with:

scons -j 8 --implicit-deps-unchanged lib/libmu2e_Analyses_ReadBack_module.so

You can read more about this at: scons documentaion. You can use the Prev and Next buttons on that page to read more about managing dependencies in scons.

Here are the results of some real-life tests for builds in which everything was up to date so scons did not have to build anything, just check for work to do. The table below gives results for running on mu2egpvm04, using code very close to Offline v5_4_6 and for running on detsim using Offline v5_4_6:

mu2egpvm04

scons -j 8	105s
scons -j 8 --implicit-deps-unchanged	45s

detsim

scons -j 20	84s
scons -j 20 --implicit-deps-unchanged	44s
scons -j 4 --implicit-deps-unchanged	41s
scons -j 1 lib/libmu2e_Analyses_ReadBack_module.so	19s
scons -j 1 lib/libmu2e_Analyses_ReadBack_module.so --implicit-deps-unchanged	12s
scons -j 20 lib/libmu2e_Analyses_ReadBack_module.so	19s
scons -j 20 lib/libmu2e_Analyses_ReadBack_module.so --implicit-deps-unchanged	12s

The bottom line is that --implicit-deps-unchanged offers a speedup between about a factor of 1.5 and 2.5. But the times are still slow enough to be noticeable.

What is the dangerous situation? If you have made edits that modify the dependency tree, you use must not use the --implicit-deps-unchanged option. If you do, you may get an inconsistent build and your code may not execute with subtle, hard to detect errors. As a rough guideline, you must not use this option if you have done any of the following:

• Edited any header file.
• Changed any link list.
• Changed versions of external products
• Changed compiler options

The bottom line is that if you are unsure whether or not it is safe to use this option, then do not use it.

Overview of Mu2e Usage of SCons

The scons executable is one of many third party products used by the Mu2e software. These are often referred to as "external products" or just "products" and they are managed by the Fermilab UPS/UPD system. The scons binaries are located under the directory $SCONS_DIR; this environment variable is established when you source setup.sh from the top level of Offline.

After you check out Mu2e software, in the top level directory you will find a file named SConstruct. This defines rules that are common to building all parts of the Offline software.

In each Mu2e PackageName/src directory you will find a file named SConscript. This provides additional information that is needed to build the files in that src/ directory; usually this additional information is only the list of link-time library dependencies, which differ from library to library. Most of the SConscript files are identical or differ only by their link lists; one notable exception is Mu2eG4/src/SConscript which contains the rules needed to find the G4 header files, the G4 compiler -D options and the G4 libraries.

In the following, pay careful attention to whether the sentence is describing the SConstruct file ( the unique file at the top level ) or the many SConscript files.

The SConstruct file defines the information that is needed globally, such as the include paths (-I), the link library paths (-L), compiler switches, linker switches, and the commands needed to build dictionaries. Most of the SConscript files only specify the first order link dependencies of the libraries to be built. A few SConscript files also specify compiler flags, link flags, additional -L and additional -I flags that are needed only for that particular directory.

Some of you may recognize that the SConstruct and SConscript files are python scripts (python is a widely used scripting language). Most Mu2e users will not need to learn python to use SCons; we can think of the SConstruct and SConscript files as just configuration files with a small grammar that we need to learn.

What's in the SConstruct file

The section below includes the first few lines of the SConstruct file:

import os
import sys

# Check that the release-specific setup has been run.
if not os.environ.has_key('MU2E_BASE_RELEASE'):
    sys.exit('You must setup a Mu2e base release before running scons.\nExiting.')

# Extract information from the shell environment.
art_inc       = os.environ['ART_INC']
art_lib       = os.environ['ART_LIB']
base          = os.environ['MU2E_BASE_RELEASE']
boost_lib     = os.environ['BOOST_LIB']
boost_inc     = os.environ['BOOST_INC']
clhep_inc     = os.environ['CLHEP_INC']
clhep_lib     = os.environ['CLHEP_LIB_DIR']

The first two lines tell scons to import information about the local operating system and operating environment; this includes all of the environment variables defined at the time of the call to scons. The next line checks that one of the required environment variables is present; if you are using the standard mu2e setup scripts this is a sufficient check that you have run the correct scripts. The remaining lines copy some environment variables into variables that are local to SConstruct; these lines have been truncated.

The section that has an if branch for test releases will be discussed later.

The next major section of SConstruct has the following structure. Some of the long lists have been truncated to make it easier to see the gross structure:

env = Environment( CPPPATH=[ cpppath_frag,
                             base,
                             base+'/BaBar/include',
                             art_inc,
                               ... deleted lines
			   ],
		   LIBPATH=[ libpath_frag,
                             base+'/lib',
                             art_lib,
                               ... deleted lines
                           ],
		   ENV={ 'LD_LIBRARY_PATH': os.environ['LD_LIBRARY_PATH'],
                         'GCC_FQ_DIR' : os.environ['GCC_FQ_DIR'], # For GCCXML
                               ... deleted lines
		       }
                 )

       ... lots of stuff deleted

Export('env')

The command env=Environment(...) creates a new python variable named env that contains environment information; think of this as a list of environment variables as is available in a standard unix shell. Initially the variable env is local to SConstruct. The command Export('env') makes env visible to all of the SConscript files. The above code fragment adds three enviroment variables to env, CPPPATH, LIBPATH and ENV. Note the env and ENV are distinct. CPPPATH and LIBPATH have special meaning to SCons and both have the meaning that you might expect from previous experience with make: CPPPATH is a list of directories that the C-Preprocessor (CPP) will use to resolve #include directives LIBPATH is a list of directories in which the linker will look to find libraries. The variable ENV contains copies of some information from the environment in the shell that called SCons; it does not have special meaning to SCons but will be used by the SConscript files.

The philosophy behind ENV is to discourage writers of SConscript files from directly accessing the external environment but instead to get information about the calling environment via ENV. In this way, SConstruct can augment or override information found in the calling environment.

SConscript files may add elements to CPPPATH and LIBPATH and the syntax for this will be discussed later. Actually the SConscript files can delete or totally redefine CPPPATH or LIBPATH but I cannot imagine a situation in which that would be useful.

Before describing the CPPPATH in detail, you need to know one more thing: scons issues all of its commands from the directory from which you invoked scons. When scons compiles, for example, the file EventGenerator/src/ConversionGun.cc, it will issue the command:

> g++ -o EventGenerator/src/ConversionGun.os {flags} {-Ixxx -Iyyy ... } EventGenerator/src/ConversionGun.cc

What is a .os file? SCons anticipates that some people will want to build both static (.a) libraries and shared (.so) libraries from the same source files. Object files destined for the two libaries usualy require different compiler options ( such as -fpic for shared libaries ); to keep the bookkeeping straight, if an object file is destined for a .a library, it is named .o; on the other hand, if an object file is destined for a .so library, scons names it .os.

Now look in a little more detail at CPPPATH:

      CPPPATH=[ cpppath_frag,
                base,
                base+'/BaBar/include',
                art_inc,
                mesfac_inc,
                fhicl_inc,
                cetlib_inc,
                cpp0x_inc,
                boost_inc,
                clhep_inc,

The environment variable CPPPATH is a list of directory names; For each name in the list, scons will add a -I element to the compile command. Most of the lines defining CPPPATH are local variables within the SConstruct python script and must be defined. The exception is cpppath_frag which is itself a list of directory names; if that list happens to be empty cpppath_frag adds nothing to CPPPATH; if it non-empty it adds all of its elements to CPPPATH.

The meaning of cpppath_frag is the following. If one checks out and builds a complete release of Offline, then the environment variable MU2E_BASE_RELEASE is defined but MU2E_TEST_RELEASE is empty. If on the other hand, one is working in a test release, then MU2E_TEST_RELEASE points to the current working area (the test release) while MU2E_BASE_RELEASE points to a complete build of Offline (the base release) against which the test release builds. When compiling code from a test release, the command line must first have the required -I elements from the test release, followed by the -I elements from the base release, followed by the remaining -I elements. If one is compiling the code from a base release, there is, by definition, no test release so the first -I elements must be those of the base release. They gymnastics with cpppath_frag ensures that the -I elements for the test release are present if and only if they are required.

Hopefully the above explanation is enough to understand the LIBPATH variable as well.

One few final note about this section: in python it is perfectly OK to have an apparently superfluous comma at the end of a list; it can be there or not but the usual style is to put it there.

The next main block of SConstruct is:

genreflex_flags =  .... lots deleted
aa= ...  lots deleted

genreflex = Builder(action=aa)
env.Append(BUILDERS = {'DictionarySource' : genreflex})

These lines tell scons how to generate root dictionary and map files for the data products; this operation uses the root genreflex program. The next few lines are:

SetOption('warn', 'no-fortran-cxx-mix')
env.MergeFlags('-g')
env.MergeFlags('-O2')
env.MergeFlags('-rdynamic')
env.MergeFlags('-Wall')

These tell scons to add various flags to the compile and link commands; scons is smart enough to figure out which commands are part of the compile command, which are part of the link command and which are part of both.

The next fragment extracts the version of the g++ compiler and adds it to env. And we have already discussed the Export command.

ff = os.popen('g++ --version'); ll = ff.readline(); ff.close()
gcc_version = ll[10:13]
env.gcc_ver=gcc_version.replace('.','')

Export('env')

The last fragment in SConstruct searches the directory heirarchy to find SConscript files. This is stored in the local variable ss. The line env.SConscript(ss) tells SCcons to execute each of these SConscript scripts.

ss=[]
for root,dirs,files in os.walk('.'):
    for file in files:
	if file == 'SConscript': ss.append('%s/%s'%(root[2:],file))
	pass
    pass
env.SConscript( ss )

What About cmake?

We have also considered changing to the build system cmake. The principal argument in its favor is that this is the build tool used by the Fermilab people who maintain art and its underlying tool chain, which gives us a natural support base. We initially chose scons because these same people had been using scons, which gave us a natural support base.

At present our evaluation is that scons is more forgiving to less sophisticated users, at a small enough cost to more advanced users. When no new files have been added, both systems do a build with a single command, make or scons. When a new file has been added, however, cmake requires that your first rerun cmake before running make. On the other hand, with scons, you only need to issue the scons command.

One weaknesses of scons is that the scons command must be issued from the top level directory, which requires that scons scan the full work tree to discover than only a subset need be done. With cmake, on the other hand, the cmake and make commands may be issued from any directory and they will rebuild only the subset of the project that decends from that directory. While the choice of scons makes work slower for advanced users, it makes it much, much safer for novice users who might not be aware when it is unsafe to issue the cmake command from a lower level directory.

We believe that we can teach scons to do out-of-source builds but we have not yet tried.

Computing Computing/Code