SpackMaintenance

Simple Installs

Installing a simple isolated package from binaries with no building or linking, or the package is all scripts. An example here might be something like bzip2, curl or ack. To see everything in buildcaches

spack buildcache list -al

to install, you can open a cvmfs tranasaction on the cvmfs node and

# to get spack setup
mu2einit
spack install --cache-only mypackage@myversion

and publish. the cache-only switch says do no try to build during the install. Generallywe do not want to build on the cvmfs node since we don't control the environment there and it is not installed like our other nodes.

A simple install with a build has to be done outside the cvmfs machine, on our build nodes. It is still simple because it does not involve linking to other packages, like root, art or offline. An example here is millepede or igprof.

mu2einit
smack subspack mydev
source mydev/setup-env.sh
spack install mypackage@myversion ...
# put it in our buildcache
# create sync.sh and cvmfs.sh scripts
smack synccache
# you must have setup a gpg key for the cache
export MYKEY=...
# this has commands to copy your local builds into a local buildcache (./bc)
./synch.sh
# push your local buildcache to to the mirror site
smack pushcache

then on the cvmfs node, start a transaction then copy over and run the cvmfs.sh script which has the install commands

mu2einit
./cvmfs.sh

Debugging

Bad caches

When don't know how this happens, but it is possible to upload to our mirror a build with an incorrect path. Builds in caches are supposed to have padding so they can be installed anywhere. If they captured a non padded path, you might get a "CannotGrowString" error during installation

==> Error: Failed to install perl due to CannotGrowString: Cannot replace b'/cvmfs/mu2e.opensciencegrid.org/packages/perl/5.38.0/linux-almalinux9-x86_64_v2-gcc-11.4.1-5gvcjfeptnu6tfyg7dbfjjb76mk5moaz' with b'/cvmfs/mu2e.opensciencegrid.org/packages2/perl/5.38.0/linux-almalinux9-x86_64_v2-gcc-11.4.1-5gvcjfeptnu6tfyg7dbfjjb76mk5moaz' because the new prefix is longer. To fix this, compile with more padding (config:install_tree:padded_length), or install to a shorter prefix.

Solution is to remove these builds from the cache and re-index it

ssh products@fifeutilgpvm03.fnal.gov
cd rlc_tmp
mv /spack_cache/mu2e/build_cache/linux-almalinux9-x86_64_v2/gcc-11.4.1/perl-5.38.0 .
mv /spack_cache/mu2e/build_cache/linux-almalinux9-x86_64_v2-gcc-11.4.1-perl-5.38.0-5gvcjfeptnu6tfyg7dbfjjb76mk5moaz.spec.json.sig .
( . /home/products/spack/share/spack/setup-env.sh spack ; spack --debug buildcache update-index -k /spack_cache/mu2e )

you may need to figure out how to replace the build with a correct one.

NoVerifyException

This means that the packages you are trying to install are gpg signed, but but you haven't talk spack to trust the signatures.

==> Error: Failed to install bzip2 due to NoVerifyException: Spack found new style signed binary packages, but was unable to verify any of them.  Please obtain and trust the correct public key.  If these are public spack binaries, please see the spack docs for locations where keys can be found.

This command will go through the mirrors and find the public key signatures, and copy them in the local install

spack buildcache keys --install --trust