CodeDebugging
Introduction
The two main tools we use to debug code are the standard gnu debugger gdb and the memory debugger valgrind. Some people use gprof or openspeedshop as a profiler.
gcc manual useful for understanding flags.
art tools
art has some debugging tools built in art tools, including time and memory checking systems.
gdb
gdb is the gnu standard text-only debugger. We set up a compatible version when you set up an Offline version.
Here are a few simple commands to get started, there is an online manual.
If your command is
mu2e -s somefile.art -c Print/fcl/print.fcl
Evoke gdb with:
gdb --args mu2e -s somefile.art -c Print/fcl/print.fcl
Start execution:
(gdb) run
You can also restart execution by typing "run" again. If you run once, the libraries won't all be loaded, but when you re-run they will be. Show the call stack:
(gdb) where
Select second frame in stack:
(gdb) frame 2
Typically, the exe will have been built on another machine, so gdb can't find the source code. You can tell it about source directories like:
(gdb) dir /cvmfs/mu2e.opensciencegrid.org/Offline/v7_0_4/SLF6/prof/Offline/Print/src
These commands can be put in a .gdbinit file. To remap paths of libraries built elsewhere, use:
(gdb) set substitute-path from to
Step one line, stepping over function calls:
(gdb) n
Step one line, stepping into function calls:
(gdb) s
Set break by function (tab completion available if libraries are loaded)
(gdb) break 'mu2e::CaloHitPrinter::Print(art::Event const& event, std::ostream& os)'
Set break by line number
(gdb) break 'CaloHitPrinter.cc:102'
Continue after a break:
(gdb) cont
Run to the end of a stack call:
(gdb) finish
Print local variable "x":
(gdb) p x
Print stl vector "myVector"
(gdb) print *(myVector._M_impl._M_start)@myVector.size()
list code line n
(gdb) list n
Catch art throws
(gdb) catch throw
It is possible to to do very much more such as setting break on a memory location write, attach to a running process, examine threads, call functions, set values, break after certain conditions, etc.
ddt
valgrind
valgrind is a memory debugger which largely works by inserting itself into heap memory access. It can detect:
- use of unintialized variables
- accessing memory freed or never allocated
- double deletion
- memory leaks
It is installed on all interactive machines, or you can UPS setup a particular version.
Here is a typical command
setup valgrind v3_21_0
valgrind --leak-check=yes --error-limit=no -v \
--demangle=yes --show-reachable=yes --track-origins=yes --num-callers=20 \
mu2e -c my.fcl
The additional memory checking causes the exe to run much. much slower.
You may find that packages like root libraries may have so many (probably not consequential) errors detected that it drowns out the useful messages. "Conditional jump or move depends on uninitialised value(s)" is a common ROOT error. valgrind allows you to suppress errors that are not important to you. We have an example suppressions file which can be added with
valgrind --leak-check=yes --error-limit=no -v \
--suppressions=$MU2E_BASE_RELEASE/scripts/valgrind/all_including_geant_callbacks.supp \
--demangle=yes --show-reachable=yes --track-origins=yes --num-callers=20 \
mu2e -c my.fcl
This file is aggressive in removing benign errors, and it may also remove some useful errors. For example, the easiest way to suppress the thousands of errors from geant, is to suppress errors that include the geant paths in the call stack. Unfortunately, the way geant is setup, we register our simulation code with geant, and geant calls our code at appropriate times. To valgrind, this makes our code look like a part of geant and the naive suppression method may also suppress errors from our simulation code. There is no easy way around this, but it may be possible with sufficient effort.
One known problem is a large stack item which leads to this warning, with a way to suppress
==30413== Warning: client switching stacks? SP change: 0x1ffeff2ba8 --> 0x1ffed07d30 ==30413== to suppress, use: --max-stackframe=3059320 or greater
vector bounds check
Overrunning the bounds of a vector is a common problem which might show up as seg faults, wrong results, or unstable results.
The standard method for checking bounds is to set _GLIBCXX_DEBUG. We would do this
Muse/python/sconstruct_helper.py
method mergeFlags(mu2eOpts)
where it would be added as
-D_GLIBCXX_DEBUG
to the list of flags. In our code, this doesn't work because the method replaces
std::vector
with
std::__debug::vector
this works, except at the interface to a third-party package, such as root or fhiclcpp. If those interfaces include passing a vector, the signatures will no longer match, because those packages are not compiled with this gcc flag defined.
Here is another method which worked 9/2021.
1. create a Muse work dir containing Offline, Production, Muse, and MuseConfig
2. run codetools/python/vectorCheck.py
vectorCheck.py Offline
You can also run it on subsets of directories
vectorCheck.py Offline/Validation
The effect will be to change each instance of
#include <vector>
to
#include "vectorCheck.h"
3. create your new vector include file
cp /cvmfs/mu2e.opensciencegrid.org/artexternals/gcc/v9_3_0/Linux64bit+3.10-2.17/include/c++/9.3.0/ ./vectorCheck.h
inside vectorCheck.h, find the line
#include <bits/stl_vector.h>
comment it out, and edit in the file content itself
/cvmfs/mu2e.opensciencegrid.org/artexternals/gcc/v9_3_0/Linux64bit+3.10-2.17/include/c++/9.3.0/bits/stl_vector.h
to be sure you have the right files, you can remove -o and add -E to dump the include sources.
Now put in the range check. Find the const and non-const operator[] and edit in the range change. (Do not call at() since that calls [].)
reference
operator[](size_type __n) _GLIBCXX_NOEXCEPT
{
__glibcxx_requires_subscript(__n);
_M_range_check(__n);
return *(this->_M_impl._M_start + __n);
}
4. Remove the error flag from the build. (This could be made a switch.) Edit Muse/python//sconstruct_helper.py and remove the -DWerror flag from the list in mergeFlags method. This was necessary because a header in fchilcpp was giving a "misleading indentation" warning. The code is in boost, far from our mods and the warning seems to be incorrect, so we are assuming it is a spurious warning.
4. Create a fake Muse setup
export MUSE_DIR=$PWD/Muse export PATH=$MUSE_DIR/bin:$PATH export MUSE_ENVSET_DIR=$PWD/MuseConfig/envset alias muse="source muse"
and execute it.
5. Build the area and run in gdb. If there is a range error, gdb will stop and you can issue where to see the code location.
fsanitize bounds check
gcc provides a switch to insert address checking code many places in a build. It seems to be a little fragile and not very well maintained, but it has found vector bounds errors and an local c-style array overrun. You can read more about sanitize in the gcc manual and AddressSanitizer docs
This method is different than the above section on compiler defines. The above method replaces std::vector with std::__debug::vector, but
the sanitize method seems to work more like valgrind, in putting hooks in system calls, like malloc.
To turn on sanitize, you have to build with certain switches set:
muse build ... --mu2eSanitize
and set the following environmental, which prevents the exe from exiting at the first warning:
export ASAN_OPTIONS="verify_asan_link_order=0:alloc_dealloc_mismatch=0:detect_leaks=0"
and then run the exe. It will run several times slower than a normal exe.
Sanitize will only print one line of traceback. If that line is in an stl method, for example, you will want a full traceback. One way is to set a break in gdb
break '__ubsan::ScopedReport::ScopedReport(__ubsan::ReportOptions, __ubsan::Location, __ubsan::ErrorType)'
Historical notes
To activate this method add the following lines to SConstruct in Muse/python
env.PrependUnique(CCFLAGS=['-fsanitize=address']) env.PrependUnique(LINKFLAGS=['-fsanitize=address'])
after MergeFlags and build.
A mu2e exe command will probably result in the following error:
==19064==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
which I think is due to the fact that the "mu2e" exe is inside art and is therefore not compiled with this switch. In a local toy example, I don't get this error. The error can be avoided:
LD_PRELOAD=/cvmfs/mu2e.opensciencegrid.org/artexternals/gcc/v9_3_0/Linux64bit+3.10-2.17/lib64/libasan.so.5 \ mu2e -n 1000 -c Production/Validation/ceSimReco.fcl &>check4.log
this can also be solved with
ASAN_OPTIONS=verify_asan_link_order=0
if you need to set multiple ASAN options, serparate with colons:
ASAN_OPTIONS=strict_string_checks=1:detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1
I also hit this error
==29458==ERROR: AddressSanitizer: alloc-dealloc-mismatch (malloc vs operator delete) on 0x611005473a00
inside a root routine. The root code was not compiled with this flag, but this was in a header file, so it might be compiled locally. Luckily, it comes with this advice:
==29458==HINT: if you don't care about these errors you may set ASAN_OPTIONS=alloc_dealloc_mismatch=0
so adding
export ASAN_OPTIONS="alloc_dealloc_mismatch=0"
caused it not stop at this point any more.
This system is supposed to have flags which prevent the code from stopping on errors,
-fsanitize-recover=all or -fsanitize-recover=address
but neither worked for me, even in a toy example.
art floating point service
This section discusses the management of floating point exceptions. The for those who need background information, Wikipedia has a good article about floating point arithmetic: [1]; of most relevance is the section that discusses floating point exceptions: [2]
art provides an interface that let's us customize how the processsor responds to floating point exceptions. The art wiki includes documentation about the art FloatingPointControl service https://cdcvs.fnal.gov/redmine/projects/art/wiki/FloatingPointControl ]. You can check if the documentation is current using art's command line help:
mu2e --print-description FloatingPointControl
More complete information is available in the gnu documentation.
The recommended configuration is:
services.FloatingPointControl: {
enableDivByZeroEx : true
enableInvalidEx : true
enableOverFlowEx : true
enableUnderFlowEx : false # See note below
setPrecisionDouble : false # see note below
reportSettings : true
}
- When one of these parameters is set to true, if the named condition occurs, the floating point until will trap and raise the signal SIGFPE. This will immediately stop execution of the program and, if enabled, dump core. If you are running inside of gdb you can ask gdb to show you the line of code that produced the error. If the variable is false, the code will set the result to a value described in the gnu documentation and continue execution.
- enableUnderFlowEx. Setting it to false is usually safe. Setting it true is likely to produce many false positives but it can be used to identify poorly written code in which a different order of operations would give a more precise result.
- setPrecisonDouble.
- Inside the FPU, the registers are 80 bits long but normal CPU registers are the size of a double, 64 bits.
- setPrecisionDouble : true tells the FPU to round results to the precision of a double after each operation; the alternative is to retain all 80 bits for use in the next operation. This results in a loss of precision that is usually unimportant. One good use case for setting it true is when validating a new compiler or validating an architecture with AVX vs one without. In both cases it improves the chances of getting exact bit-for-bit identical results.
- From Marc Paterno: "The IEEE-754 double has an effective 53 bit mantissa (including an implied leading ‘1’ bit, for all except subnormals). The Intel FPU has 80-bit extended precision that uses a 64 bit mantissa (no implied leading ‘1’. The SSE and AVX units do not have the extended precision. setPrecisionDouble: true in effect makes the FPU more like the SSE and AVX units. I am not sure what effect it has on speed, but it does have an effect on accuracy. In particular, it can help avoid some catastrophic cancellations (although careful coding can also help avoid them)."
When you are developing your code we recommend that you enable floating point exceptions as described above. One issue with this system is that it only permits configuration of FPEs on a whole job basis; if there are buggy modules ahead of yours in the art job, the work around is to run those modules in a separate job, without enabling FPEs, and write an output file. Then do your development, with FPEs enabled, using that file as your input.
stack corruption
Interesting blog post
gprof
openspeedshop
This profiling package needs to be installed from documents.
osspcsamp "mu2e -s somefile.art -c Print/fcl/print.fcl" >& log
Arm forger profile
arm forge's map program:
setup forge_tools map --profile --start --nompi $(type -p mu2e) -c full.fcl <some file>.root
which will generate a map output file. If you then run
map <generated file>.map