Git: Difference between revisions

From Mu2eWiki
Jump to navigation Jump to search
(Replaced content with " This page is obsolete, please see GitHubWorkflow")
Tag: Replaced
 
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:


==Introduction==
This page is obsolete, please see [[GitHubWorkflow]]
The primary source code management system for the Mu2e Offline software is a [http://git-scm.com/ git] repository. Git is a popular open-source software version control system. Currently there are two primary repositories being maintained in parallel, one hosted on the Fermilab [https://cdcvs.fnal.gov/redmine Redmine] site, and one hosted on the commercial git hosting site [https://www.github.com/mu2e/Offline github]. Redmine is a software project management system which contains, for each project, a log, a wiki (which may contain useful info), and possibly gantt charts and bug tracking along with the git respositories. GitHub, in addition to hosting git repositories, allows the creation and documentation of issues, code comments, and pull requests.
 
Based on the git repository's url, you can check out code, and, if you are a code developer, check in modifications. Git also has capabilities for tagging code, tracking history and supports splitting off and merging development branches, among many other features.
 
Depending on your experience or needs, please also see the basic [[CodeRecipe|code recipe]] or the official [[GitHubWorkflow | GitHub workflow]].  You can continue with the [[#quickStart|quick start]] on this page or the expanded git [[GitIntro|introduction page]].
See [[#References|the references]] for more details on Redmine and git. There is a [https://cdcvs.fnal.gov/redmine/projects/fermi-redmine/wiki/Ssh_notes page for troubleshooting] redmine git ssh connections.
 
Some older parts of Mu2e software is managed using [https://cdcvs.fnal.gov/redmine/projects/mu2eofflinesoftwaremu2eoffline/wiki/Revision_Control_Systems other source code management systems].  Some code, both critical and specialized, is in other, smaller repositories.  See [[#Repositories|Repositories]] below for more details on repositories.
 
<div id="quickStart">
 
==Quick Start (DEPRECATED)==
 
This section gives a quick guide to using git with the Redmine workflow. For the GitHub version, see [[GitHubWorkflow]].
 
Please also see the basic [[CodeRecipe|code recipe]] for an introduction to setting up the environment, checking out code and building.
 
If you are going to commit anything, you must understand and follow the [[#gitCommits|mu2e commit instructions]]
 
If this is the first time you are using git in the mu2e environment, please follow the [[GitIntro#configuringGit|few steps]] listed in order to set up your preferences in ~.gitconfig.
 
A more detailed introduction to all mu2e repositories is [https://cdcvs.fnal.gov/redmine/projects/mu2eofflinesoftwaremu2eoffline/wiki/Revision_Control_Systems here] and our introduction to git is [[GitIntro|here]].
 
* checkout the main repository
** readonly: <pre>git clone http://cdcvs.fnal.gov/projects/mu2eofflinesoftwaremu2eoffline/Offline.git</pre>
** with kerberos authentication for committing code <pre>git clone ssh://p-mu2eofflinesoftwaremu2eoffline@cdcvs.fnal.gov/cvs/projects/mu2eofflinesoftwaremu2eoffline/Offline.git</pre>
* checkout the head to a local working branch, here named "work" (recommended)
<pre>git checkout -b work</pre>
* see the history of a file
<pre>git log fileSpec</pre>
* list tags
<pre>git tag -l</pre>
* checkout a tag or branch or commit hash into the working area.  Use this to switch between branches, it will NOT overwrite your work.
<pre>git checkout tagOrBranchNameorHash </pre>
* update the local repository to the origin repository (Redmine)
<pre>git fetch</pre>
* update the local repository to changes staged in the working area (does not affect the origin repository)
<pre>git commit -m"'Your descriptive message'"</pre>
* update the origin repository (Redmine) to changes in your local repository
<pre>git push</pre>
* update the local repository to the origin repository (REDMINE)
<pre>git fetch</pre>
* update the working area to the local repository
<pre>git merge</pre>
* update the local repository and update the working area both together
<pre>git pull</pre>
* merge a branch (including master) into your branch. NB: do NOT merge your branch into master unless authorized
<pre>git checkout otherbranch
git pull
git checkout mybranch
git pull
git checkout mybranch
git merge otherbranch</pre>
* see what the status is
<pre>git status</pre>
* command line help
<pre>git status help</pre>
* git browser (to see a graphical history of commits)
<pre>gitk --all</pre>
 
 
Please see the references below or many web resources for details on git concepts, commands and features.
If you are going to commit anything, you must understand and follow the [[#gitCommits|commit workflows]].
 
<div id="gitCommits">
 
==Workflows for committing code==
 
As the group of mu2e code developers grows at the same time that the need for a stable code base increases, individual code contributions may start to clash with other contributions, so we are using a "request to pull" procedure.  In this procedure, the developer does code development on a git branch and commits that branch to the repository frequently.  This branch does not affect other users, so it can be in any state the developer finds useful - it might not even compile. Once the developer believes the work is complete, they make a request to the Offline group heads to merge or "pull" the branch into the master branch.  If the change is very simple or low-risk, the heads may ask the developer to go ahead and do that merge.  If the change is large or complex, they may request additional documentation or testing before the pull.  The branch should be prepared so it includes the head of the master branch, so it can be merged with "fast forward". 
 
The procedure of developing on a branch, then requesting to merge is explained in detail on the [[GitMajorWorkflow|git  workflow page]]. 
 
If a change is very small, or planned in advance, or urgent, the Offline group heads may allow the commit to follow the [[GitMinorWorkflow|minor workflow]] which ends in a commit directly to the head of master.
 
Once the switch to GitHub is made, we will begin following the [[GitHubWorkflow|GitHub workflow]].
 
==GitHub Pull Request Procedures and FNALbuild==
 
When you open a Pull Request (PR) on the Mu2e/Offline GitHub repository, you will receive a greeting message from @FNALbuild, which is Fermilab's build bot account. @FNALbuild acts as a glue between the GitHub collaborative environment and Jenkins, and is used by the repository maintainers to trigger tests that may be required to pass before your changes are merged. The tests (or continuous integration, CI, actions) currently are a supplement to the existing review process.
 
'''N.B. buildmaster.fnal.gov links (Jenkins server) can only be accessed via Fermilab onsite VPN.'''
 
Some CI actions are triggered automatically by the creation of the PR. They can also be triggered by posting a command in a comment on a Pull Request; see below for recognized commands.  fnalbuild-users GitHub Team indicates which GitHub users and Teams in the Mu2e organisation are able to trigger CI actions; some branches have additional users authorized to trigger actions; in addition the requestor of the Pull Request may also trigger actions.
 
'''@FNALbuild run build test'''
 
In regular use. Merges the PR branch into the current HEAD of the base branch and builds the code, then runs a series of small jobs (10 events) to check for runtime errors.
 
'''@FNALbuild run code checks'''
 
Currently not in use. Checks the files that were changed by a PR for trailing whitespace and hard tabs - fails if modifications are needed. FNALbuild will provide a patch to fix these problems, if any.
 
This is not an integration test - it only tests the PR branch as-is.
 
 
'''@FNALbuild run validation'''
 
Not in regular use. Pulls the built code from a build test and runs a validation job (ceSimReco) over 5000 events, and produces validation plots.
Builds, or pulls a cached build of master (not containing any PR changes) and runs the same validation job, producing plots.
Runs valCompare between the two sets of plots, and produces a comparison which is published on the PR in a comment.
 
The Jenkins jobs that run the tests are located [https://buildmaster.fnal.gov/buildmaster/view/mu2e/job/GitHubPRTests/ here]
 
 
'''Where the code is stored'''
 
The scripts that are run to test a PR are kept on GitHub in [https://github.com/Mu2e/codetools/tree/master/bin/github/jenkins_tests Mu2e/codetools].
 
The FNALbuild handler scripts are kept [https://github.com/Mu2e/CI here].
 
The Mu2e version of this bot was based on and re-written using the original process_pr.py script from FNALbuild/cms-bot:
 
* '''process_pr.py''': Where everything happens. given a PR, process all comments and figure out which tests to trigger based on those comments.
* '''comment_gh_pr.py''': this is called from the Jenkins jobs to post a comment on the PR when the result of a test has been determined, or when the status of a test has changed e.g. it is running.
* '''test_suites.py''': some configuration variables to set up the regular expressions that are used to search for commands that trigger tests.
* '''watchers.yaml''': Users may add themselves to this to 'watch' a specific folder in Offline for changes, such that they are notified when someone wishes to change it. Regex is supported.
* '''auth_teams.yaml''': Configuration file of which GitHub teams are able to launch CI actions on a base branch.
 
==Tips and Tricks==
 
===Branch prompt===
 
Many people work in a style where they are switching between branches frequently, and everyone does this switch at times.  You can put the branch name in your prompt so you are unlikely to get confused about what branch you are on.
 
In .bash_profile:
<pre>
parse_git_branch() {
    git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* (.*)/(1)/'
}
 
export PS1="\h \w $(parse_git_branch) "
</pre>
 
===Finding deleted files===
 
As a policy we have decided to not keep code that does not compile.  Some code may be useful as examples or to revive an abandoned effort, so we should be able to recover them.  Here are three methods:
<ol>
<li>grep the release notes </li>
grep mystring ReleaseNotes/*/*
<li>if you know the file name, look at its history</li>
git log myfile
<li>search git history.  The following git command will list every commit that contained the
deletion of a a file and it will list the names of all files deleted in that commit:</li>
git log --diff-filter=D --summary
and can be pipe do to grep
</ol>
Once you find the file name, you can check it out based on the commit that deleted it.
git checkout <deleting_commit>^ -- <file_path>
 
If you are deleting non-trivial files, please note the full file paths in the releases notes.
 
===What branch is Tracking What Other Branch?===
 
<pre>
git branch -avv
</pre>
will show a list of all branches in your working area, both local and remote braches.  The additional information provided by -vv is:
 
# The SHA of the commit at the head of the branch
# The comment text associcated with that commit
# If the branch is tracking another, there is a notation of the form [branch name] between items 1 and 2.
 
In the example below the local branch master is tracking Mu2e/master.
<pre>
* work                            9e5e9f0 Merge pull request #152 from goodenou/MT_fcl_fix
  master                          9e5e9f0 [Mu2e/master] Merge pull request #152 from goodenou/MT_fcl_fix
  remotes/Mu2e/MDC2018            7da9814 Merge pull request #74 from gianipez/trkTrig1
  remotes/Mu2e/master              9e5e9f0 Merge pull request #152 from goodenou/MT_fcl_fix
  remotes/origin/HEAD              -> origin/master
  remotes/origin/MDC2018          7da9814 Merge pull request #74 from gianipez/trkTrig1
  remotes/origin/bfield_xyzvec_1  8207ed0 Add accessor that uses XYZVec for input argument and return value.
  remotes/origin/branch_with_error 7b70a3e Deliberate error in order to see how CMS-BOT behaves with an error.
  remotes/origin/master            26bb554 Merge pull request #1 from Mu2e/master
</pre>
 
===What Remotes are Available in my Working area===
 
<pre>
git remote -v
</pre>
Combined with git branch -avv this allows you to determine which remote branches are attached to which remote repositories.
 
===Long commit histories===
 
[https://stackoverflow.com/questions/16306012/github-pull-request-showing-commits-that-are-already-in-target-branch avoid] posting long commit histories
 
==Repositories==
 
One important piece of the mu2e offline code is the [[BTrk]] UPS product, which contains our Kalman track fit code, and is stored in the [https://github.com/KFTrack/BTrk BTrk github repo]. 
 
All of the rest of the mu2e Offline code and tools are in Redmine repositories.
You can browse [https://cdcvs.fnal.gov/redmine/projects all Redmine projects] and
[https://cdcvs.fnal.gov/redmine/projects/mu2e mu2e Redmine].  The mu2e Redmine has subprojects (and sub-subprojects) and each has a repository associated with them.  For example in the [https://cdcvs.fnal.gov/redmine/projects/standalone Standalone] subproject, there is a sub-subproject '''[https://cdcvs.fnal.gov/redmine/projects/mu2eccfc Mu2eCCFC]'''. In addition, the main mu2e Offline repository has sub-repositories which can be seen on the right sidebar menu on the repository [https://cdcvs.fnal.gov/redmine/projects/mu2eofflinesoftwaremu2eoffline/repository main page].  An example is the '''dhtools''' sub-repo.
 
Here is a (possibly incomplete) list of the repositories.
 
{|style="width: 100%"
|-
|style="width:20%"|'''content'''
|style="width:30%"|'''user'''
|style="width:35%"|'''name'''
|style="width:15%"|'''alias'''
|-
|-
| || p-mu2e_artdaq || mu2e_artdaq ||
|-
|  raw data format || p-mu2e_artdaq || mu2e_artdaq-core ||
|-
| [https://cdcvs.fnal.gov/redmine/projects/mu2eccfc F-C limit setting] || p-mu2eccfc || mu2eccfc-mu2eccfc || mu2eccfc.git
|-
| CI validation || p-mu2e_ci || mu2e_ci ||
|-
|  || p-mu2e-daq-firmware || mu2e-daq-firmware ||
|-
|  || p-mu2e-daq-firmware || mu2e-daq-firmware-dtc ||
|-
|  || p-mu2e-dcs || mu2e-dcs ||
|-
|  || p-mu2e_detector_construction_db || mu2e_detector_construction_db ||
|-
|  || p-mu2eextmontestdaq || mu2eextmontestdaq ||
|-
| [[MCProdWorkflow|grid tools]]  || p-mu2egrid || mu2egrid ||
|-
| Offline || p-mu2eofflinesoftwaremu2eoffline || mu2eofflinesoftwaremu2eoffline || Offline.git
|-
| [[Jenkins]]  || p-mu2eofflinesoftwaremu2eoffline || mu2eofflinesoftwaremu2eoffline-codetools || codetools.git
|-
| [[DataTransfer|data handling]] || p-mu2eofflinesoftwaremu2eoffline || mu2eofflinesoftwaremu2eoffline-dhtools || dhtools.git
|-
|  || p-mu2eofflinesoftwaremu2eoffline || mu2eofflinesoftwaremu2eoffline-geomdisplay || geomdisplay.git
|-
| satellite releases  || p-mu2eofflinesoftwaremu2eoffline || mu2eofflinesoftwaremu2eoffline-satellite ||
|-
| gridexport product  || p-mu2eofflinesoftwaremu2eoffline || mu2eofflinesoftwaremu2eoffline-gridexport || gridexport.git
|-
|  || p-mu2eofflinesoftwaremu2eoffline || mu2eofflinesoftwaremu2eoffline-test ||
|-
| ''obsolete''  || p-mu2eofflinesoftwaremu2eoffline || mu2eofflinesoftwaremu2eoffline-validation || validation.git
|-
| Stntuple framework || p-mu2eofflinesoftwarestntuple || mu2eofflinesoftwarestntuple || Stntuple.git
|-
|  || p-mu2eofflinesoftwareuser || mu2eofflinesoftwareuser-betasource || betasource.git
|-
|  || p-mu2eofflinesoftwareuser || mu2eofflinesoftwareuser-example01 || example01.git
|-
|  || p-mu2eofflinesoftwareuser || mu2eofflinesoftwareuser-streamline ||
|-
|  || p-mu2epbarmodels || mu2epbarmodels ||
|-
|  || p-mu2epbaridanalysis || mu2epbaridanalysis ||
|-
|  || p-mu2e-raw-data || mu2e-raw-data ||
|-
|  || p-mu2e-simulations || mu2e-simulations ||
|-
|  || p-mu2estat || mu2estat ||
|-
| || p-mu2e-tools || mu2e-tools-bintools || mu2ebintools
|-
|[[FileTools]]  || p-mu2e-tools || mu2e-tools-filetools || mu2efiletools
|-
| [[FileTools]] || p-mu2e-tools || mu2e-tools-mu2efilename || mu2efilename
|-
| [[GenerateFcl|generate_fcl]]  || p-mu2e-tools || mu2e-tools-tools || mu2etools
|}
 
Creating a Redmine subproject is done by the "add subproject" button on the [https://cdcvs.fnal.gov/redmine/projects/mu2e?jump=repository Redmine project web page].  This is probably more appropriate for a project not closely associated with the Offline.
Creating a sub-repository associated to the main offline repo is explained in the
[https://cdcvs.fnal.gov/redmine/projects/mu2eofflinesoftwaremu2eoffline/wiki/Making_a_git_repository_in_redmine mu2e instructions] and
[https://cdcvs.fnal.gov/redmine/projects/fermi-redmine/wiki/CreatingRepositories SCD instructions]. This is probably more appropriate for a project closely associated with the Offline.
Write privileges to each repository can be controlled by the <code>.k5login</code> contents of each group username associated with the repo - note that this file should be edited by the Redmine project web page (settings->members), not directly.  The login to the group account is controlled by the <code>.admin</code> file.
 
A git repository is accessed through a url and the url can be read-only or read/write, which allows commits back to the repo.
Taking values from the table of Redmine repos above, a repo without an alias will have read-only and read/write urls:
<nowiki>http://cdcvs.fnal.gov/projects/</nowiki>'''name'''
ssh://'''user'''@cdcvs.fnal.gov/cvs/projects/'''name'''
and if it has an alias:
<nowiki>http://cdcvs.fnal.gov/projects/</nowiki>'''name'''/'''alias'''
ssh://'''user'''@cdcvs.fnal.gov/cvs/projects/'''name'''/'''alias'''
 
A good view of all repositories can be found by logging is as one of the user names where you have write permissions:
ssh '''user'''@cdcvs.fnal.gov
The projects can be seen
cd /cvs/projects
ls -l | grep mu2e
there is one directory per repository and the username can be seen as the owner of the repository directory. If the repository has an alias, it will appear as a subdirectory or a circular link under
/cvs/projects/'''name'''
The <code>.k5login</code> file for a repository is
/home/'''name'''/.k5login
 
==Redmine git hooks==
 
To add or modify a hook, log into the Mu2e Offline redmine project
  ssh p-mu2eofflinesoftwaremu2eoffline@ccdcvsvm.fnal.gov
The repo is hosted at:
/cvs/projects/mu2eofflinesoftwaremu2eoffline/Offline.git
Look in the hooks/ subdirectory.  I think what you want is a post-receive hook.  There is already a post receive hook file - this is the code that sends email to registered subscribers on every push.
 
==Looking at the Merge History==
 
So long as we only modify master by merging in pull requests, the following procedure will allow us to find the state master at an arbitrary time in the past.  The procedure is robust against people merging master into their working branches.
 
This information is taken from:
[https://stackoverflow.com/questions/21623699/is-it-possible-to-get-a-list-of-merges-into-a-branch-from-the-github-website-or] and I made a few small modifications in the details of the printout.
 
The following git command will list all merge commits of merges into master and print some information about each.
<pre>
git log --merges --first-parent master --pretty=format:"%h %<(10,trunc)%ae %C(yellow)%<(15)%aI%Creset %C(red bold)%<(15)%D%Creset %s"
</pre>
 
The result on the morning of Jan 29, 2020 looks like:
<pre>
f6bdbaebc kutschke.. 2020-01-28T14:33:45-06:00 HEAD -> master, tag: v08_02_01, Mu2e/master Merge pull request #125 from goodenou/art3_MT
651cf9425 kutschke.. 2020-01-28T14:31:30-06:00 tag: v08_02_00  Merge pull request #124 from resnegfk/g4105
14eeda454 kutschke.. 2020-01-25T17:27:40-06:00 tag: v08_01_00  Merge pull request #123 from ryuwd/refactor-useprodTracker
ae02fa010 kutschke.. 2020-01-24T13:32:14-06:00                Merge pull request #121 from resnegfk/g4stepper
2c1fd533c Dave_Bro.. 2020-01-23T17:25:06-08:00                Merge pull request #122 from bonventre/reflections
040a6fd0d kutschke.. 2020-01-22T20:31:50-06:00                Merge pull request #120 from brownd1978/schema1
</pre>
The color information was removed by the shell when I captured the output.  If you run the command yourself you will see the coloring.
 
 
 
 
==References==
*[https://cdcvs.fnal.gov/redmine/projects/fermi-redmine/wiki/Ssh_notes redmine git troubleshooting]
*[https://cdcvs.fnal.gov/redmine/projects/mu2eofflinesoftwaremu2eoffline/wiki/Revision_Control_Systems mu2e git and CVS]
*[[GitMinorWorkflow|mu2e minor commit workflow]]
*[[GitMajorWorkflow|mu2e major commit workflow]]
*[https://cdcvs.fnal.gov/redmine/projects/mu2eofflinesoftwaremu2eoffline/wiki/Git_Workflow_for_Mu2e old redmine commit instructions]
*[http://git-scm.com/ git home]  [http://git-scm.com/doc  Documentation] [https://git-scm.com/docs reference]
*[https://cdcvs.fnal.gov/redmine/projects/mu2eofflinesoftwaremu2eoffline/repository redmine repository browser]
*[https://cdcvs.fnal.gov/redmine Fermilab redmine]
*[http://mu2e-docdb.fnal.gov:8080/cgi-bin/ShowDocument?docid=4527Andrei's talk] at the Software and Simulation Meeting, Sept 17, 2014.
*[https://cdcvs.fnal.gov/redmine/projects/cet-is-public/wiki/Marc's_suggested_git_workflow Marc P's suggested git workflow] for simple git projects
*[https://cdcvs.fnal.gov/redmine/projects/fermi-redmine/wiki/GitWorkshop Fermilab git workshop] and related materials
*[https://cdcvs.fnal.gov/redmine/projects/mu2eofflinesoftwaremu2eoffline/wiki managing and creating] new redmine repositories
 
[[Category:Computing]]
[[Category:Code]]
[[Category:CodeManagement]]

Latest revision as of 20:11, 13 May 2021

This page is obsolete, please see GitHubWorkflow

Retrieved from "http://mu2ewiki.fnal.gov/w/index.php?title=Git&oldid=18339"