Difference between revisions of "Spack"

Latest revision as of 04:01, 26 April 2024

Introduction

Spack is a software package setup and build system. It can replace UPS, MRB (Multi-Repo Build) and Muse "package managers". Spack was developed for the supercomputer environment and is common use there today. The driving force that inspired spack is the need to install many packages while coordinating their dependencies.

The computing division will provide their software (art, ifdhc) within the spack framework. They have requested that we adopt spack as far as possible, so that the lab can converge to one package manager which will simplify maintenance and support. It will also prepare us to be more efficient in our use of supercomputers in the future. The lab has decided to end support for UPS and only provide software in the spack framework starting with he adoption of the AlmaLinux operating system, which will fully adopted by the hard deadline of 6/30/2024.

spack is designed to manage your code from github repo, to build, to installation in the final location. However, for a few reasons, we are adopting spack in two phases. For historical reasons, they are phase 2 and 3. In phase 2, we build our Offline repos locally, driven by the muse scripts, and link to the art suite libraries which are delivered in spack format. This has the primary advantage that all muse commands and functionality stay the same.

Phase 3, which is full spack adoption, we expect will come in two forms. The first is "bare spack" for experts who are comfortable working with spack directly, and a "wrapped' version with Muse-like script which will hide most details for beginners and casual code users. A basic build in bare spack is being used in the online setting.

Phase 2 Usage

The initial setup, which works for sl7 (with UPS) and al9 (with spack) is

source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh

Since we can't use UPS, we can't "setup mu2e". After this, muse will be in your path and will work normally. You can "muse setup" and "muse build" as usual. You can also make tarballs, and submit to the grid, or access musings, like "muse setup SimJob".

To access the data-handling tools, both SAM and the new (metacat) tools, you can

muse setup ops

which can be run with or without setting up a Muse Offline build.

We are preparing

muse setup ana

to provide access to a python analysis tool suite, like pyana.

Notes

show what spack knows about architectures

spack arch --known-targets

full listing of dependencies

spack find --long --show-flags --deps --variants ifdhc

diff between two hashes

spack diff art-root-io/jrcjyn4 art-root-io/h43e5rd

How to group sets of setups into one. You can make a spack environment, like

spack env create uboone_analysis_current_sl7_x86_64

then

spack activate uboone_analysis_current_sl7_x86_64

and "spack install" packages into it; then if you

spack env activate uboone_analysis_current_sl7_x86_64

those will be the packages you see, and they're all spack loaded at once. Marc stills need to add support for that to the cvmfs scripts though...

geant names

geant data packages have different names in spack. From Julia:

g4able is G4ABLA
g4emlow is G4EMLOW
g4neutron is G4NDL
g4nucleonsxs is G4SAIDDATA
g4nuclide is G4ENSDFSTATE 
g4photon is PhotonEvaporation
g4pii is G4PII
g4radiative is RadioactiveDecay
g4tendl is G4TENDL
g4particlexs is G4PARTICLEXS
g4incl is G4INCL

G4NEUTRONXS appears to be obsolete.

References

lab intro talk
spack official docs
public packages
lab spack wiki and tutorial
githubs
- FNALssi (spack, spack tools)
- fermitools (metacat, jobsub, POMS)
- art framework
- artdaq
recipe repos
- mu2e-spack (ots,mdh)
- fnal_art (art,canvas,ifdhc)
- artdaq
- scd_recipes (metacat)
- builtin lab spack
cetmodules
cmake
gfal2

@@ Line 1: / Line 1: @@
 ==Introduction==
-Spack is a multi-package setup and build system.  It replaces [[UPS]] and MRB (Multi-Repo Build) and related packages. The computing division will work to build and provide their software (art, ifdhc) within spack. Mu2e will probably not use the spack build features, but will have to use spack to access pre-built software, instead of ups setup command.
+Spack is a software package setup and build system.  It can replace [[UPS]], MRB (Multi-Repo Build) and [[Muse]] "package managers". Spack was developed for the supercomputer environment and is common use there today.  The driving force that inspired spack is the need to install many packages while coordinating their dependencies.
-Scripts are provided to allow a spack setup of a UPS product and ''vice versa''.  The user doesn't see the difference.
+The computing division will provide their software (art, ifdhc) within the spack framework. They have requested that we adopt spack as far as possible, so that the lab can converge to one package manager which will simplify maintenance and support.  It will also prepare us to be more efficient in our use of supercomputers in the future.  The lab has decided to end support for UPS and only provide software in the spack framework starting with he adoption of the AlmaLinux operating system, which will fully adopted by the hard deadline of 6/30/2024.
-[https://docs.google.com/presentation/d/16CxC40Hs0V0m7pwTQaSoUdo-gqy5UqojNZY824AiCBY/edit#slide=id.g23f597c85bd_0_0 draft intro talk]
+spack is designed to manage your code from github repo, to build, to installation in the final location. However, for a few reasons, we are adopting spack in two phases.  For historical reasons, they are phase 2 and 3.  In phase 2, we build our Offline repos locally, driven by the muse scripts, and link to the art suite libraries which are delivered in spack format.  This has the primary advantage that all muse commands and functionality stay the same.
-==Usage==
+Phase 3, which is full spack adoption, we expect will come in two forms.  The first is "bare spack" for experts who are comfortable working with spack directly, and a "wrapped' version with Muse-like script which will hide most details for beginners and casual code users. A basic build in bare spack is being used in the [[Building_Offline_with_cmake| online setting]].
- setup mu2e
- source /cvmfs/fermilab.opensciencegrid.org/packages/common/spack/rollout/NULL/share/spack/setup-env.sh
-defines
- SPACK_ROOT=/cvmfs/fermilab.opensciencegrid.org/packages/common/spack/rollout/NULL
-  MODULEPATH=$SPACK_ROOT/share/spack/modules/linux-scientific7-x86_64
- DK_NODE=$SPACK_ROOT/share/spack/dotkit/linux-scientific7-x86_64
- PATH= ... /cvmfs/fermilab.opensciencegrid.org/packages/common/spack/rollout/NULL/bin
- BASH_FUNC_spack()=... load ... unload ...
-and some useful commands
+==Phase 2 Usage==
+The initial setup, which works for sl7 (with UPS) and al9 (with spack) is
+ source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh
+Since we can't use UPS, we can't "setup mu2e".  After this, muse will be in your path and will work normally.  You can "muse setup" and "muse build" [[Muse| as usual]].  You can also make tarballs, and submit to the grid, or access musings, like "muse setup SimJob".
- spack -h
+To access the data-handling tools, both SAM and the new (metacat) tools, you can
- spack find -h
+  muse setup ops
- spack arch
+which can be run with or without setting up a Muse Offline build.
-  spack find
- spack find --variants art_root_io arch=linux-scientific7-x86_64_v2 %gcc@13.1.0
- spack find -ldv
-You can see the common lab packages are available in spack: ifdhc, jobsub_client, sam_web_client.
+We are preparing
+  muse setup ana
-  spack load ifdhc
+to provide access to a python analysis tool suite, like pyana.
- # load (like ups setup) using a hash for the version and qualifiers
- spack load ifdhc/242t7pk
 ==Notes==
@@ Line 69: / Line 59: @@
 ==References==
-*[https://fifewiki.fnal.gov/wiki/Spack New Fermilab spack wiki]
+*[https://docs.google.com/presentation/d/16CxC40Hs0V0m7pwTQaSoUdo-gqy5UqojNZY824AiCBY/edit#slide=id.g23f597c85bd_0_0 lab intro talk]
-** [https://fifewiki.fnal.gov/wiki/Using_fermi-spack-tools_for_test_releases Using fermi-spack-tools for test releases]
+*[https://spack.readthedocs.io/en/latest/index.html spack official docs]
-*[https://cdcvs.fnal.gov/redmine/projects/spack-infrastructure/wiki/Wiki Old Fermilab spack wiki]
+*[https://packages.spack.io/ public packages]
-*[https://spack.readthedocs.io/en/latest/ Spack docs]
+*[https://fifewiki.fnal.gov/wiki/Spack lab spack wiki and tutorial]
-*[https://github.com/FNALssi fnal ssi github] [https://github.com/FNALssi/fnal_art/tree/develop/packages recipes] [https://github.com/FNALssi/spack/tree/fnal-develop/var/spack/repos/builtin/packages builtin recipes]
+* githubs
-*[https://github.com/spack/spack github]
+**[https://github.com/FNALssi FNALssi] (spack, spack tools)
-*[https://github.com/FNALssi/fermi-spack-tools spack-tools github]
+**[https://github.com/fermitools fermitools] (metacat, jobsub, POMS)
+**[https://github.com/art-framework-suite/art.git art framework]
+**[https://github.com/art-daq/ artdaq]
+* recipe repos
+** [https://github.com/Mu2e/mu2e-spack mu2e-spack] (ots,mdh)
+** [https://github.com/FNALssi/fnal_art fnal_art] (art,canvas,ifdhc)
+** [https://github.com/art-daq/artdaq-spack artdaq]
+** [https://github.com/marcmengel/scd_recipes/tree/master/packages scd_recipes] (metacat)
+** builtin [https://github.com/FNALssi/spack/tree/fnal-develop/var/spack/repos/builtin lab] [https://github.com/spack/spack/tree/develop/var/spack/repos/builtin spack]
 *[https://fnalssi.github.io/cetmodules/ cetmodules]
+*[https://cmake.org/cmake/help/latest/ cmake]
+*[https://grid-deployment.web.cern.ch/grid-deployment/dms/dmc/docs/gfal2-python-1.9.0/html/index.html gfal2]
 [[Category:Computing]]
 [[Category:Code]]
 [[Category:CodeManagement]]