CodeDevelopment: Difference between revisions

From Mu2eWiki
Jump to navigation Jump to search
No edit summary
 
(3 intermediate revisions by the same user not shown)
Line 14: Line 14:


# '''user'''  - setup pre-built code and run utilities and standard mu2e fcl.  Example: compare filter rates by modifying fcl, make a gdml file, run tutorials.  Needs simple and fool-proof procedures.
# '''user'''  - setup pre-built code and run utilities and standard mu2e fcl.  Example: compare filter rates by modifying fcl, make a gdml file, run tutorials.  Needs simple and fool-proof procedures.
# '''analyzer''' - checkout, build user code against pre-built offline, commit changes. Example: user makes ntuples from  personal code or Stntuple makes an ntuple.  Needs user code well-incorporated with offline - possibly using the same build system, possibly coordinated directory structures and commands. Fool-proof user-to-offline dependencies.
# '''analyzer''' - checkout, build user code against pre-built offline, commit changes. Example: user makes ntuples from  personal code or Stntuple.  Needs user code well-incorporated with offline - possibly using the same build system, possibly coordinated directory structures and commands. Fool-proof user-to-offline dependencies.
# '''minor''' - make localized modifications to offline head, test and commit them, as in specific product maintenance.  Examples: fix a bug in an algorithm, add an accessor.  Needs quick and reliable partial build of arbitrary pieces. Needs to catch dependency mistakes. Needs to switch quickly between prof and debug.
# '''analyzerPlus''' - checkout, build user code against pre-built offline, commit changes.  Also checkout offline code consistent with prebuilt release and make some small modificstion.  Do not commit back to offline. Example: user makes ntuples and needs to tweak seom algorithm code to get the right variable out in the ntuple.  Needs quick and reliable partial build of arbitrary pieces. Needs to catch dependency mistakes. Needs to switch quickly between prof and debug.
# '''minor''' - make localized modifications to offline head or branch, test and commit them, as in specific product maintenance.  Examples: fix a bug in an algorithm, add an accessor.  Needs quick and reliable partial build of arbitrary pieces. Needs to catch dependency mistakes. Needs to switch quickly between prof and debug.
# '''major''' - make larger and more widespread changes, test and commit them.  Examples: modifying major include files, reco sequences, products.  Probably needs full build of offline.  Needs quick checkout and build.
# '''major''' - make larger and more widespread changes, test and commit them.  Examples: modifying major include files, reco sequences, products.  Probably needs full build of offline.  Needs quick checkout and build.
# '''expert''' - infrequent complex development - move code between packages, build prototype products with offline, build code releases, build trigger code.
# '''expert''' - infrequent complex development - move code between packages, build prototype products with offline, build code releases, build trigger code packages.


In my opinion, the system should serve the first 3 primarily, under the assumption that the others are done less often and are done by people who are less likely to make mistakes.
In my opinion, the system should serve the first three or four primarily, under the assumption that the others are done less often and are done by people who are less likely to make mistakes.


==Some Issues==
==Some Issues==
Line 53: Line 54:
* split out Analysis package?
* split out Analysis package?
* consider separating sim and reco more sharply
* consider separating sim and reco more sharply
==Summary of packages==
Many of the longest compile times are in libraries where little or none of the code is needed by 95% of users.
<pre>
                            files  modules lines  compile time (s)
                  Alignment    14      0    331    14
                  Analyses    162    98  28297    868
                BFieldGeom    18      0    770    10
                BFieldTest      5      2    444    22
                  BTrkData      6      0    428    13
                BTrkHelper    11      0    169    21
              BeamlineGeom    15      0      7      5
                CRVAnalysis      5      0    142    16
                CRVResponse    134    11  6991    123
          CRYEventGenerator      6      1    543    26
                  CalPatRec    108    15  21857    219
                CaloCluster    19      3  1212    46
                    CaloMC    15      4  1909    63
                  CaloReco    13      2  1715    52
            CalorimeterGeom    17      0    708      7
            ConditionsBase      1      0      0      0
          ConditionsService    64      0    379    20
                ConfigTools    10      0  1958    11
        CosmicRayShieldGeom    25      0    485      9
        CutAndCountAnalysis      7      2    495    50
                        DAQ    16      6  2261    64
            DAQDataProducts      6      0    128      8
              DataProducts    28      0    320    10
      DetectorSolenoidGeom      4      0      5      4
              EventDisplay    39      1  3432    54
            EventGenerator    91    25  10075    385
                EventMixing    36      6  1561    68
      ExternalShieldingGeom      7      0    51      9
      ExtinctionMonitorFNAL    168    29  8634    310
                    Filters    33    31  6769    293
                  G4Helper      7      0    169    18
          GeneralUtilities    48      0  1069    22
            GeomPrimitives    19      0    242      6
            GeometryService    86      1  11907    160
    GlobalConstantsService    15      0    639    25
                HelloWorld      8      3    197    26
                  HitMakers    21      9  4054    102
                  JobConfig    113      0      0      0
                KalmanTests    17      0      0      0
                    MBSGeom      2      0      0      0
            MCDataProducts    99      0  1192    41
MECOStyleProtonAbsorberGeom      4      0    56      4
                  Mu2eBTrk      8      0    267    17
                    Mu2eG4    810      2  39016    536
              Mu2eHallGeom      5      0    58      8
            Mu2eInterfaces      7      0    33      5
                  Mu2eReco      4      0      0      0
              Mu2eUtilities    92      0  3806    122
                ParticleID    19      4  3867    70
                      Print    75      3  4011    155
    ProductionSolenoidGeom    11      0    98    11
      ProductionTargetGeom      5      0    14      8
                  Projects    71      1    387      0
        ProtonBeamDumpGeom      5      0    47      7
          RecoDataProducts    126      1  2247    44
              ReleaseNotes    188      0      0      0
              RootAnalyzer      5      0      0      0
                SConstruct      1      0      0      0
                    STMGeom      7      0      0      0
                    Sandbox    83    29  2516    222
                SeedService    20      1    450    17
              ServicesGeom      5      0    35      7
                    Sources      7      0    783    32
        StoppingTargetGeom      6      0    58      4
              TTrackerGeom    14      0    175      7
                  TestTools      3      0    84      3
          TrackCaloMatching    33      6  3250    77
          TrackerConditions      9      0    704    27
                TrackerGeom    12      0    367      7
                  TrackerMC    20      1  1621    32
                    Trigger      2      1    81    15
                TrkAdapter      4      0    31    16
              TrkChargeReco    19      0    541    18
                    TrkDiag    93    10  5288    130
                    TrkExt    21      1  3143    49
                TrkFilters    15      5  1012    61
                  TrkPatRec    19      4  2009    71
                    TrkReco    33      0  4407    85
            UtilityModules    11      2    132    22
                Validation    57      1  2923    83
</pre>
[[Category:Computing]]
[[Category:Code]]

Latest revision as of 18:51, 27 November 2018

Introduction

A page to write some principles and questions for code development.

  • 13265 Andrei's recent talk
  • 7591 Rob's offline work list
  • 6240 Ray's older talk
  • 6077 Rob's comments
  • 6053 Andrei's older talk
  • 5741 Andrei's older talk

Use Cases

Each level incorporates the levels above.

  1. user - setup pre-built code and run utilities and standard mu2e fcl. Example: compare filter rates by modifying fcl, make a gdml file, run tutorials. Needs simple and fool-proof procedures.
  2. analyzer - checkout, build user code against pre-built offline, commit changes. Example: user makes ntuples from personal code or Stntuple. Needs user code well-incorporated with offline - possibly using the same build system, possibly coordinated directory structures and commands. Fool-proof user-to-offline dependencies.
  3. analyzerPlus - checkout, build user code against pre-built offline, commit changes. Also checkout offline code consistent with prebuilt release and make some small modificstion. Do not commit back to offline. Example: user makes ntuples and needs to tweak seom algorithm code to get the right variable out in the ntuple. Needs quick and reliable partial build of arbitrary pieces. Needs to catch dependency mistakes. Needs to switch quickly between prof and debug.
  4. minor - make localized modifications to offline head or branch, test and commit them, as in specific product maintenance. Examples: fix a bug in an algorithm, add an accessor. Needs quick and reliable partial build of arbitrary pieces. Needs to catch dependency mistakes. Needs to switch quickly between prof and debug.
  5. major - make larger and more widespread changes, test and commit them. Examples: modifying major include files, reco sequences, products. Probably needs full build of offline. Needs quick checkout and build.
  6. expert - infrequent complex development - move code between packages, build prototype products with offline, build code releases, build trigger code packages.

In my opinion, the system should serve the first three or four primarily, under the assumption that the others are done less often and are done by people who are less likely to make mistakes.

Some Issues

  • is it amenable to a tutorial?
  • how will it run on OSG without /mu2e/app?
  • will there be a different build time vs run time environment?
  • is initial setup different from re-setup?
  • what windows does the user have open? How does one know what a window is good for?
  • what are likely points of confusion?
  • is the directory structure organized and intuitive? Is offline separate from user code?
  • is it fast and easy for an analyzer or junior to start a new project?
  • what does a user type most often? Can it be shorter and simpler?
  • can all dependencies be checked automatically?
  • what does switching between prof and debug look like?
  • how does an analyzer switch between pre-built releases?
  • what build commands are supported for user code, how much are they forced to be coordinated with offline?
  • are there clear lines of code responsibility?
  • who coordinates pieces and tags and validates the whole (and how)?
  • does it support code reviews and pull requests?
  • does it support out-of-source builds?
  • what parts of the dependencies are tagged and which are outside the tag?
  • what do paths look like? do they include "mu2e"?
  • what does validation do - continuous and nightly?
  • what is the maintenance effort required?
  • does it follow KISS?
  • is it staged? does it fit into a long-term plan?
  • is there a transition plan? How disruptive is it?
  • does a major change like introducing the calibration database cause unnecessary disruption?
  • is git history preserved?
  • have we considered the best ways to speed up compilation?
  • split out fcl partially or fully?
  • split out Analysis package?
  • consider separating sim and reco more sharply


Summary of packages

Many of the longest compile times are in libraries where little or none of the code is needed by 95% of users. 

                             files   modules lines  compile time (s)
                  Alignment     14      0    331     14
                   Analyses    162     98  28297    868
                 BFieldGeom     18      0    770     10
                 BFieldTest      5      2    444     22
                   BTrkData      6      0    428     13
                 BTrkHelper     11      0    169     21
               BeamlineGeom     15      0      7      5
                CRVAnalysis      5      0    142     16
                CRVResponse    134     11   6991    123
          CRYEventGenerator      6      1    543     26
                  CalPatRec    108     15  21857    219
                CaloCluster     19      3   1212     46
                     CaloMC     15      4   1909     63
                   CaloReco     13      2   1715     52
            CalorimeterGeom     17      0    708      7
             ConditionsBase      1      0      0      0
          ConditionsService     64      0    379     20
                ConfigTools     10      0   1958     11
        CosmicRayShieldGeom     25      0    485      9
        CutAndCountAnalysis      7      2    495     50
                        DAQ     16      6   2261     64
            DAQDataProducts      6      0    128      8
               DataProducts     28      0    320     10
       DetectorSolenoidGeom      4      0      5      4
               EventDisplay     39      1   3432     54
             EventGenerator     91     25  10075    385
                EventMixing     36      6   1561     68
      ExternalShieldingGeom      7      0     51      9
      ExtinctionMonitorFNAL    168     29   8634    310
                    Filters     33     31   6769    293
                   G4Helper      7      0    169     18
           GeneralUtilities     48      0   1069     22
             GeomPrimitives     19      0    242      6
            GeometryService     86      1  11907    160
     GlobalConstantsService     15      0    639     25
                 HelloWorld      8      3    197     26
                  HitMakers     21      9   4054    102
                  JobConfig    113      0      0      0
                KalmanTests     17      0      0      0
                    MBSGeom      2      0      0      0
             MCDataProducts     99      0   1192     41
MECOStyleProtonAbsorberGeom      4      0     56      4
                   Mu2eBTrk      8      0    267     17
                     Mu2eG4    810      2  39016    536
               Mu2eHallGeom      5      0     58      8
             Mu2eInterfaces      7      0     33      5
                   Mu2eReco      4      0      0      0
              Mu2eUtilities     92      0   3806    122
                 ParticleID     19      4   3867     70
                      Print     75      3   4011    155
     ProductionSolenoidGeom     11      0     98     11
       ProductionTargetGeom      5      0     14      8
                   Projects     71      1    387      0
         ProtonBeamDumpGeom      5      0     47      7
           RecoDataProducts    126      1   2247     44
               ReleaseNotes    188      0      0      0
               RootAnalyzer      5      0      0      0
                 SConstruct      1      0      0      0
                    STMGeom      7      0      0      0
                    Sandbox     83     29   2516    222
                SeedService     20      1    450     17
               ServicesGeom      5      0     35      7
                    Sources      7      0    783     32
         StoppingTargetGeom      6      0     58      4
               TTrackerGeom     14      0    175      7
                  TestTools      3      0     84      3
          TrackCaloMatching     33      6   3250     77
          TrackerConditions      9      0    704     27
                TrackerGeom     12      0    367      7
                  TrackerMC     20      1   1621     32
                    Trigger      2      1     81     15
                 TrkAdapter      4      0     31     16
              TrkChargeReco     19      0    541     18
                    TrkDiag     93     10   5288    130
                     TrkExt     21      1   3143     49
                 TrkFilters     15      5   1012     61
                  TrkPatRec     19      4   2009     71
                    TrkReco     33      0   4407     85
             UtilityModules     11      2    132     22
                 Validation     57      1   2923     83
Retrieved from "http://mu2ewiki.fnal.gov/w/index.php?title=CodeDevelopment&oldid=10654"