KinKal: Difference between revisions
(4 intermediate revisions by the same user not shown) | |||
Line 2: | Line 2: | ||
==Local KinKal development with Muse== | ==Local KinKal development with Muse== | ||
===Introduction=== | |||
== | <font color=red size=+1>These instructions only work on SL7/UPS, not on AL9/Spack.</font> | ||
This section describes how to do simultaneous development of KinKal and Offline. Because KinKal uses a different build system than does Offline, you need to take a few additional steps when you start. Once you have taken these steps "muse build" will first build KinKal and install it into a local UPS area; it will then build Offline using the local UPS area | This section describes how to do simultaneous development of KinKal and Offline. Because KinKal uses a different build system than does Offline, you need to take a few additional steps when you start. Once you have taken these steps "muse build" will first build KinKal, but only if necessary, and install it into a local UPS area; it will then build Offline using KinKal from the the local UPS area. | ||
This writeup assumes that you have a basic knowledge of [[UPS]] and of [[Muse]]; in particular you should understand what a Muse envset is and how to use it. Normally you will choose the envset recommended by Offline and this example is written assuming that's what you want to do. If you are new to this work and you need to use a different envset, your mentor will tell you which one to use. Section [[#Using An Alternate envset]] describes how the workflow must be changed in order to use a different envset. | This writeup assumes that you have a basic knowledge of [[UPS]] and of [[Muse]]; in particular you should understand what a Muse envset is and how to use it. Normally you will choose the envset recommended by Offline and this example is written assuming that's what you want to do. If you are new to this work and you need to use a different envset, your mentor will tell you which one to use. Section [[#Using An Alternate envset]] describes how the workflow must be changed in order to use a different envset. | ||
Line 38: | Line 38: | ||
<li> Setup the Mu2e development environment | <li> Setup the Mu2e development environment | ||
<pre> | <pre> | ||
mu2einit | |||
setup codetools | setup codetools | ||
</pre> </li> | </pre> </li> | ||
<li> Choose a UPS version number for your local build of KinKal. The number will have no meaning outside of your muse working directory so you can use any number that is not already in use. Choosing version | <li> Choose a UPS version number for your local build of KinKal. The number will have no meaning outside of your muse working directory so you can use any number that is not already in use. Choosing version 99.9.0 is likely to be safe for a long time to come. The version number must have 3 dot separated fields.</li> | ||
<li> Build KinKal; create a local ups directory and install the build of KinKal into that UPS area. | <li> Build KinKal; create a local ups directory and install the build of KinKal into that UPS area. | ||
<pre> | <pre> | ||
kk2ups -v " | kk2ups -v "99.9.0" -b -i -m "" -j 8 | ||
</pre> | </pre> | ||
On | On mu2ebuild02 you may increase 8 to 32. This command will build both the profile and debug versions of KinKal. The option -m with an empty string tells Kkk2ups to choose the version of root that matches the default for building Offline. See section [[#Using An Alternate envset]] for when you should provide a non-empty string. | ||
</li> | </li> | ||
<li> If everything worked correctly, the output of this command will end with | <li> If everything worked correctly, the output of this command will end with | ||
Line 62: | Line 62: | ||
This will produce the output: | This will produce the output: | ||
<pre> | <pre> | ||
v99_09_00 v99_09_00.version | |||
</pre> | </pre> | ||
Also do | Also do | ||
<pre> | <pre> | ||
ls artexternals/KinKal/ | ls artexternals/KinKal/v99_09_00 | ||
</pre> | </pre> | ||
This will produce output similar to: | This will produce output similar to: | ||
Line 80: | Line 80: | ||
cp /cvmfs/mu2e.opensciencegrid.org/DataFiles/Muse/pxyz unnn | cp /cvmfs/mu2e.opensciencegrid.org/DataFiles/Muse/pxyz unnn | ||
</pre> | </pre> | ||
where unnn is the name of your copy envset, for example u046 | where unnn is the name of your copy envset, for example u046. The name unnn has no meaning outside of your muse working directory. | ||
</li> | </li> | ||
<li> Edit the envset unnn to make two changes | <li> Edit the envset unnn to make two changes | ||
<ol> | <ol> | ||
<li> Find the line that sets up KinKal and replace the version number with | <li> Find the line that sets up KinKal and replace the version number with v99_09_00 | ||
<pre> | <pre> | ||
setup KinKal | setup KinKal v99_09_00 -q +${MUSE_BUILD}:+${MUSE_COMPILER_E}:+${MUSE_PYTHON}' | ||
</pre> | </pre> | ||
</li> | </li> | ||
<li> Add the following line after the last setup command: | <li> Add the following line after the last setup command: | ||
<pre> export MUSE_PREBUILD="kk2ups -v \" | <pre> export MUSE_PREBUILD="kk2ups -v \"99.9.0\" -b -i -j 24" </pre> | ||
This line is not the same as the command given in step 6; it omits the -m argument. When muse executes this command a version of root will already be setup in your | This line is not the same as the command given in step 6; it omits the -m argument. When muse executes this command a version of root will already be setup in your environmen and kk2ups will use that version of root. | ||
</li> | </li> | ||
</ol> | </ol> | ||
Line 99: | Line 99: | ||
<li> Build Offline and any other repos that you cloned in step 3; if anything has changed in KinKal since the last build, this command will also rebuild KinKal and reinstall it in the local artexternals. | <li> Build Offline and any other repos that you cloned in step 3; if anything has changed in KinKal since the last build, this command will also rebuild KinKal and reinstall it in the local artexternals. | ||
<pre>muse build -j 8 </pre> | <pre>muse build -j 8 </pre> | ||
On | On mu2ebuild02 you may increase 8 to 32.</li> | ||
<li> Now you can edit files in any of KinKal, Offline and any other locally cloned repos. When you want to rebuild you can do so with a single command: | <li> Now you can edit files in any of KinKal, Offline and any other locally cloned repos. When you want to rebuild you can do so with a single command: | ||
<pre>muse build -j 8 </pre> | <pre>muse build -j 8 </pre> | ||
On | On mu2ebuild02 you may increase 8 to 32.</li> | ||
</ol> | </ol> | ||
Line 111: | Line 111: | ||
<li> (-b) build the existing checkout in the local clone of KinKal | <li> (-b) build the existing checkout in the local clone of KinKal | ||
<li> (-i) if necessary, create a local directory named artexternals, configure it as a UPS repositor | <li> (-i) if necessary, create a local directory named artexternals, configure it as a UPS repositor | ||
<li> (-i) copy the build of KinKal into artexternals/KinKal/ | <li> (-i) copy the build of KinKal into artexternals/KinKal/v99_09_00 and configure it as a UPS product. | ||
</ol> | </ol> | ||
</li> | </li> | ||
Line 117: | Line 117: | ||
<li> The full syntax of the kk2ups option -m is: -m "args", where args is will be given as arguments to muse setup. For example to request envset pabc | <li> The full syntax of the kk2ups option -m is: -m "args", where args is will be given as arguments to muse setup. For example to request envset pabc | ||
<pre> | <pre> | ||
kk2ups -v " | kk2ups -v "99.9.0" -b -i -m "-q pabc" -j 8 | ||
</pre> | </pre> | ||
Internally kk2ups looks up the version of root that is chosen by "muse setup $args" and uses that version of root to build KinKal. In step 6, args is an empty string, which tells kk2ups to use the default envset. Because you have a clone of Offline in your Muse working area, the default envset is that specified by Offline/.muse. | Internally kk2ups looks up the version of root that is chosen by "muse setup $args" and uses that version of root to build KinKal. In step 6, args is an empty string, which tells kk2ups to use the default envset. Because you have a clone of Offline in your Muse working area, the default envset is that specified by Offline/.muse. | ||
Line 134: | Line 134: | ||
<li> In step 6, supply the alternate envset as the value of the -m argument to kk2ups | <li> In step 6, supply the alternate envset as the value of the -m argument to kk2ups | ||
<pre> | <pre> | ||
kk2ups -v " | kk2ups -v "99.9.0" -b -i -m "-q pabc" -j 8 | ||
</pre></li> | </pre></li> | ||
<li> In step 10.1, make a copy of the alternate envset | <li> In step 10.1, make a copy of the alternate envset | ||
Line 155: | Line 155: | ||
<pre> | <pre> | ||
cd 'workdir' | cd 'workdir' | ||
mu2einit | |||
setup codetools | setup codetools | ||
muse setup -q unnn | muse setup -q unnn | ||
</pre> | </pre> | ||
where unnn is the envset you created during the initial build. | where unnn is the envset you created during the initial build. You can then edit files and build as needed. | ||
Line 169: | Line 169: | ||
<pre> | <pre> | ||
cd 'myworkdir' | cd 'myworkdir' | ||
mu2einit | |||
setup codetools | setup codetools | ||
muse setup -q unnn:debug | muse setup -q unnn:debug | ||
muse build -j 8 | muse build -j 8 | ||
</pre> | </pre> | ||
On | On mu2ebuild02 you may change the 8 to 32. | ||
This will make a debug build of KinKal and install in it the local artexternals. Then it will build Offline and any other repos you cloned in step 3. | This will make a debug build of KinKal and install in it the local artexternals. Then it will do a debug build of Offline and any other repos you cloned in step 3. If you have not modified KinKal since you first executed step 6, the build of KinKal will be fast because there is no work to do. | ||
===What if you update Offline?=== | ===What if you update Offline?=== | ||
If you update the version of Offline in your working area | If you update the version of Offline in your working area the first step is to compare the old and new versions of Offline | ||
to see if the envset specified in Offline/.muse has changed. If it has not changed, there are no additional steps and you can proceed as described in | |||
[[#Returning to the area]]. | |||
# | |||
If it has changed, do the folllowing in a clean shell. In this example the new envset is named pdef. | |||
<ol> | |||
<li> Setup the environment: | |||
<pre> | <pre> | ||
cd 'myworkdir' | cd 'myworkdir' | ||
mu2einit | |||
setup codetools | |||
</pre> | </pre> | ||
<li> Choose a new version number for the UPS version of KinKal, say 9.9.1. and repeat step 6</li> | |||
< | |||
<pre> | <pre> | ||
kk2ups -v "99.9.1" -b -i -m "" -j 8 | |||
</pre> | </pre> | ||
There is no need to specify the new envset because kk2ups will pick it up from Offline/.muse. | |||
</li> | |||
<li> Choose a new name for your modified envset, say u047. | |||
<li> Repeat steps 9 and 10 using the new envset name, pdef, the new KinKal version 9.9.1 and the new modifed envset name u047 | |||
<li> In step 11 use u047 instead of u046. | |||
</ol> | |||
When this work is completed you will see both versions of KinKal in artexternals and both builds of Offline in your build subdirectory. If do not expect to return to the old builds it is safe to delete them. | |||
</ | |||
< | |||
< | |||
< | |||
</ | |||
Latest revision as of 13:27, 26 July 2024
Local KinKal development with Muse
Introduction
These instructions only work on SL7/UPS, not on AL9/Spack.
This section describes how to do simultaneous development of KinKal and Offline. Because KinKal uses a different build system than does Offline, you need to take a few additional steps when you start. Once you have taken these steps "muse build" will first build KinKal, but only if necessary, and install it into a local UPS area; it will then build Offline using KinKal from the the local UPS area.
This writeup assumes that you have a basic knowledge of UPS and of Muse; in particular you should understand what a Muse envset is and how to use it. Normally you will choose the envset recommended by Offline and this example is written assuming that's what you want to do. If you are new to this work and you need to use a different envset, your mentor will tell you which one to use. Section #Using An Alternate envset describes how the workflow must be changed in order to use a different envset.
Initial build
- In a clean shell, create a Muse workDir and cd there
mkdir 'myworkdir' cd 'myworkdir'
- clone KinKal into the Muse workdir as your would normally. Add the Mu2e KinKal repo as a remote and checkout the main branch of the Mu2e KinKal remote.
git clone git@github.com:<your GitHub username>/KinKal cd KinKal git remote add -f mu2e https://github.com/KFTrack/KinKal git checkout -b work_branch mu2e/main cd ..
- Clone your fork of Offline into the muse workdir as you would normally. Add the Mu2e Offline repo as a remote and checkout the main branch of the Mu2e Offline remote. If you need other muse-aware repos (Production, TrkAna, ...) for your workflow, you can add them too
git clone git@github.com:<your GitHub username>/Offline cd Offline git remote add -f mu2e https://github.com/Mu2e/Offline git checkout -b work_branch mu2e/master cd .. # Optional: repeat the above pattern for any of Production, TrkAna ...
- Setup the Mu2e development environment
mu2einit setup codetools
- Choose a UPS version number for your local build of KinKal. The number will have no meaning outside of your muse working directory so you can use any number that is not already in use. Choosing version 99.9.0 is likely to be safe for a long time to come. The version number must have 3 dot separated fields.
- Build KinKal; create a local ups directory and install the build of KinKal into that UPS area.
kk2ups -v "99.9.0" -b -i -m "" -j 8
On mu2ebuild02 you may increase 8 to 32. This command will build both the profile and debug versions of KinKal. The option -m with an empty string tells Kkk2ups to choose the version of root that matches the default for building Offline. See section #Using An Alternate envset for when you should provide a non-empty string.
- If everything worked correctly, the output of this command will end with
Begin install. Begin install of prof version. Begin install of debug version. KinKal_to_UPS/build completed with status 0
- Look at the files created by kk2ups:
ls artexternals/KinKal/
This will produce the output:
v99_09_00 v99_09_00.version
Also do
ls artexternals/KinKal/v99_09_00
This will produce output similar to:
include slf7.x86_64.debug.e20.p3915 slf7.x86_64.e20.p3915.prof source ups
As Mu2e Offline evolves to newer versions of g++ and python, the qualifiers e20 and p3915 will change.
- Look at Offline/.muse to find the name of the default envset for building Offline. In these instructions we will call that pxyz. Make a local muse subdirectory and make a copy of pxyz inside of it.
mkdir muse cd muse cp /cvmfs/mu2e.opensciencegrid.org/DataFiles/Muse/pxyz unnn
where unnn is the name of your copy envset, for example u046. The name unnn has no meaning outside of your muse working directory.
- Edit the envset unnn to make two changes
- Find the line that sets up KinKal and replace the version number with v99_09_00
setup KinKal v99_09_00 -q +${MUSE_BUILD}:+${MUSE_COMPILER_E}:+${MUSE_PYTHON}'
- Add the following line after the last setup command:
export MUSE_PREBUILD="kk2ups -v \"99.9.0\" -b -i -j 24"
This line is not the same as the command given in step 6; it omits the -m argument. When muse executes this command a version of root will already be setup in your environmen and kk2ups will use that version of root.
- Find the line that sets up KinKal and replace the version number with v99_09_00
- cd back to the muse working directory and setup muse using the envset you just created
cd .. muse setup -q unnn
- Build Offline and any other repos that you cloned in step 3; if anything has changed in KinKal since the last build, this command will also rebuild KinKal and reinstall it in the local artexternals.
muse build -j 8
On mu2ebuild02 you may increase 8 to 32. - Now you can edit files in any of KinKal, Offline and any other locally cloned repos. When you want to rebuild you can do so with a single command:
muse build -j 8
On mu2ebuild02 you may increase 8 to 32.
About kk2ups
- The command kk2ups is short for "KinKal to UPS" and is found in codetools. The command you issued in step 6 will
- (-b) build the existing checkout in the local clone of KinKal
- (-i) if necessary, create a local directory named artexternals, configure it as a UPS repositor
- (-i) copy the build of KinKal into artexternals/KinKal/v99_09_00 and configure it as a UPS product.
- The full syntax of the kk2ups option -m is: -m "args", where args is will be given as arguments to muse setup. For example to request envset pabc
kk2ups -v "99.9.0" -b -i -m "-q pabc" -j 8
Internally kk2ups looks up the version of root that is chosen by "muse setup $args" and uses that version of root to build KinKal. In step 6, args is an empty string, which tells kk2ups to use the default envset. Because you have a clone of Offline in your Muse working area, the default envset is that specified by Offline/.muse.
- kk2ups has an online help feature:
kk2ups -?
- To learn more about kk2ups you can read https://github.com/Mu2e/codetools/blob/main/KinKal_to_UPS/README.md.
Using An Alternate envset
If you need to use an alternative envset, you need to modify two steps in the above procedure. In this example, the alternate envset is named pabc:
- In step 6, supply the alternate envset as the value of the -m argument to kk2ups
kk2ups -v "99.9.0" -b -i -m "-q pabc" -j 8
- In step 10.1, make a copy of the alternate envset
mkdir muse cd muse cp /cvmfs/mu2e.opensciencegrid.org/DataFiles/Muse/pabc unnn
Be sure to use the same envset in both places.
Rebuilding KinKal
In earlier versions of this workflow there were explicit steps required to rebuild KinKal when needed. These steps are now done automatically/
Returning to the area
If you want to restart work in a new shell, follow these steps:
cd 'workdir' mu2einit setup codetools muse setup -q unnn
where unnn is the envset you created during the initial build. You can then edit files and build as needed.
Adding debug build
Due to the way that UPS works, it is not possible to work on both prof and debug builds in the same shell. You need to do each in a separate shell.
To add a debug build to an existing prof build, start a clean shell and:
cd 'myworkdir' mu2einit setup codetools muse setup -q unnn:debug muse build -j 8
On mu2ebuild02 you may change the 8 to 32.
This will make a debug build of KinKal and install in it the local artexternals. Then it will do a debug build of Offline and any other repos you cloned in step 3. If you have not modified KinKal since you first executed step 6, the build of KinKal will be fast because there is no work to do.
What if you update Offline?
If you update the version of Offline in your working area the first step is to compare the old and new versions of Offline to see if the envset specified in Offline/.muse has changed. If it has not changed, there are no additional steps and you can proceed as described in #Returning to the area.
If it has changed, do the folllowing in a clean shell. In this example the new envset is named pdef.
- Setup the environment:
cd 'myworkdir' mu2einit setup codetools
- Choose a new version number for the UPS version of KinKal, say 9.9.1. and repeat step 6
- Choose a new name for your modified envset, say u047.
- Repeat steps 9 and 10 using the new envset name, pdef, the new KinKal version 9.9.1 and the new modifed envset name u047
- In step 11 use u047 instead of u046.
kk2ups -v "99.9.1" -b -i -m "" -j 8
There is no need to specify the new envset because kk2ups will pick it up from Offline/.muse.
When this work is completed you will see both versions of KinKal in artexternals and both builds of Offline in your build subdirectory. If do not expect to return to the old builds it is safe to delete them.