GitMinorWorkflow

Introduction

This page describes a recommended git workflow for use with the Mu2e Offline code for minor revisions. It is for small changes, such as changing fcl parameters, fixing bugs, adding minor features, or starting a new module. These modifications are committed directly to the head and should leave the head in a state that compiles and runs. You should work on your local work branch, but after the commit, there should be no branch visible in the main repo.

For more comlex revision use the git major workflow.

This page tells the same story 4 times:

• An Overview comprising a set of 10 bullet points
• A picture version that show the state of the repository after each sep
• A Cheat Sheet - just the git commands without commentary
• A Detailed Version

This page presumes that you are familiar with the Mu2e Introduction to git.

If you are already familiar with the background material, you can jump directly to the Cheat Sheet.

Overview

The recommended workflow has the following steps:

• Start from a clean local master branch that is up to date with the local tracking master branch.
• Synchronize the master branch in your local repository with that in the redmine repository.
• Create a local working branch starting from the local master branch. This new branch is temporary and will not appear in the redmine repository.
• Do your work on the local working branch and make all commits to that branch.
• Check that there are no uncommitted changes on your work branch.
• Synchronize your local master branch with the redmine repository.
• Use git rebase so that your working branch appears as if you had branched it off of the new head of the local master branch. This is the place at which you may need to resolve collisions with changes committed by others.
• Use git merge to move your working branch onto the head of the local master branch.
• Use git push to synchronize both the redmine repository and your local master tracking branch with your master branch.
• Delete your working branch.

The result is that your working branch never appears in the redmine repository; its contents are merged onto to the head of the master branch before that branch is pushed to the redmine repository.

The careful reader will notice that there is a race condition between that last git fetch/pull and the git push; this is addressed in the main body of the discussion.

The above description gives the example of a single working branch but you may many such branches; for example, you might have a separate project underway on each branch. When it is time to rebase and merge one of these branches you can leave the other branches unchanged and continue to work on them at a later time.

Picture Version

This section restates the information above but adds detail and uses figures to help tell the story. The numbered steps correspond to the same numbers in the previous section.

1) Start from a clean local master branch that is up to date with the local tracking master branch.

Check that your master branch is clean

git status

which should produce the output:

# On branch master
nothing to commit, working directory clean

If it does not, consult the page Cleaning your master branch.

2) Synchronize the master branch in your local repository with that in the redmine repository.

There are two ways to perform Step 2. The one step method is to issue the command

git pull

There is also a two step method.

git fetch
git merge --ff-only

There is a discussion of both methods at fetch vs pull.

The following error message may be generated by fetch or pull:

X11 forwarding request failed on channel 0

You can safely ignore it.

For this workflow example we imagine the starting point to be a redmine repository in which there is a master branch that ends in two commits named m1 and m2 (m1 and m2 are mnemonic names for SHA-1 hash codes). In all of these figures, commits are denoted by outline boxes with a white interior and a mnemonic name inside; the arrow pointing from m2 to m1 denotes that m1 is the parent of m2. A git branch is just a lightweight pointer to a commit. In Figure 1 there is one branch, named master, which points to the last commit, m2. In this figure branches are denoted by solid boxes with the branch name inside.

Figure 1.

While the Mu2e redmine repository does contain other branches, none are involved in this workflow example and they are not shown.

After issuing the above command(s), the local repository has been synchronized with the redmine repository. This is illustrated in Figure 2. These figures use the convention that figures representing the redmine repository are shown in blue while those representing the local repository are shown in red.

Figure 2.

The local repository has both a branch named master and a tracking branch named origin/master. Both are shown in Figure 1 and both point to the commit named m2. In most of the upcoming figures, the origin/master branch points to the same commit as does the master branch. When this is the case, only the master branch is shown. When this is not the case, both master and origin/master will be drawn on the figure.

3) Create a local working branch starting from the local master branch.

git checkout -b work

Figure 3 shows the state of the local master branch after Step 3; a new solid red box has been added to represent the newly created working branch; in this example the name of the branch is simply work.

Figure 3.

The redmine repository master branch and the local master tracking branch are unchanged by Step 3.

4) Do your work on the local working branch and make all commits to that branch.

Now suppose that you make 4 commits, named w1 through w4. Figure 4 shows the state of the local repository after those commits: there are four new commits; the local master branch still points at the commit m2; the work branch now points at commit w4.

Figure 4

5) Check that there are no uncommitted changes on your work branch.

git status

If there are uncommitted changes, commit them now. If the output does not say that your checked out branch is work, consult an expert.

6) Synchronize your local master branch with the redmine repository.

git checkout master
git pull

An alternative to the last step is to use git fetch and git merge; see fetch vs pull.

For purposes of this example, suppose that, after you synchronized with redmine repository in Step 2, someone pushed commits m3 and m4 to the master branch in redmine repository. The state of the redmine repository is illustrated in Figure 5:

Figure 5.

Two new commits have been added and the master branch now points at the commit named m4. This figure is drawn in blue to remind you that it represents the stat of the redmine repository, not the local repository.

In step 6, you synchronize the branches master and origin/master in your local repository with the redmine repository. After this step, your local repository will look like Figure 6:

Figure 6.

The commits m3 and m4 have been added and the master tag has been moved to point at m4; the origin/master tag has also been moved to point at m4.

In this picture it becomes clear why we use the name "branch" for the master and work branches.

7) This step is the key to the workflow and its action is to transform Figure 6 into Figure 7. This step in called rebasing.

A full discussion of the commands you need to issue at this time don't fit in this section; for the full discussion see the discussion of Step 7 in the last section on this page. The sequence of commands begins with:

git checkout work
git rebase master

Figure 7 shows the local repository after rebasing:

Figure 7.

Note that the commits in the work branch have new names: w1 became w1-prime, and so on. In words, Step 7 tells git:

• Look up the changes need to transform commit m2 into commit w1
• Apply these changes starting at m4
• Call the result a new commit named w1-prime.
• Repeat for the changes needed to transform w1 to w2, w2 to w3 and w3 to w4.

Very often this "just works" but there are times when it does not. An example of when it will "just work" is when the set of files modified by commits m3 and m4 has no overlap with the set of files modified by w1 through w4. In this case it is said that there are no conflicts.

An example of when rebasing will not work is when both commits m3 and w1 changed the same line of one file and that the two commits made different changes. In this case git does not know what to do and will ask for your help - this case will be described in more detail below. This situation is called and unresolveable conflict.

There are also resolvable conflicts. An example might be when commit m3 deletes one line in a file and commit w1 deletes a different line in the same file. In this case git will produce w1-prime in which both lines are removed.

The git jargon is that rebasing rewrites history to produce the w1-prime through w4-prime that you would have made yourself had you started to work at m4.

If there are problems during rebasing, you can revert to the state prior to the rebase by issuing the command:

git rebase --abort

After a successful rebase the original commits w1, w2, w3 and w4 are still present in your local repository but they are no longer part of any branch. It is possible to recover them but how to do that is out of the scope of this discussion.

8) Use git merge to move your working branch onto the head of the local master branch.

git checkout master
git merge --ff-only work

After this step your local repository will look like Figure 8:

Figure 8.

Note that the origin/master branch still points at m4 but that the master branch now points to w4-prime; this is the only figure in the workflow in which master and origin/master point at different commits. If, at this point, you give the git status command, it will tell you that master is 4 commits ahead of origin/master.

9) Use git push to synchronize both the redmine repository and your local master tracking branch with your master branch.

git push

After this step, the local repository looks like Figure 9:

Figure 9.

In this figure, master and origin/master once again point at the same place. Figure 10 shows the state of the redmine repository after the push; it does not have the work branch but it is otherwise identical to the local repository.

Figure 10.

10) Delete your working branch.

git branch -d work

Figure 11 shows the state of the local repository after step 10, deleting the work branch.

Figure 11.

To summarize this story: there was an intermediate state (Figure 6) when there was a manifest branch in the commit history. But, by the right use of git rebase and git merge, this branch was removed before the push to the redmine repository. The net result is that observers of the redmine repository will see a linear commit history.

Cheat Sheet

This section states the workflow with minimal comments; the sections below will describe it in detail. The workflow contains some redundant git status and git checkout commands; they are there to remind you to make sure that you have not forgotten a commit or that you have not wandered off to an incorrect branch. The numbered steps follow the same numbering scheme as the sections above.

Make sure that the local master branch is clean and is up to date with the local master tracking branch.

   git status
   git checkout master
   git status

Synchronize the local master branch with the redmine repository

   git pull

or

   git fetch
   git merge --ff-only origin/master

For a discussion about the differences between these two options see fetch vs pull.

The following error message may be generated by fetch or pull:

X11 forwarding request failed on channel 0

You can safely ignore it.

Create a working branch (the branch will be temporary and the name is not important):

   git checkout -b work

Do your work. Edit files; add/delete/rename files. Commit all of your work

   git add newfile1
   git commit -m  "comment" newfile1 oldfile1 oldfile2
    ... and so on until your work is done ...

Make sure you have no uncommitted changes

   git status

Go back to your master branch and sync with the redmine repository

   git checkout master
   git pull

or

   git checkout master
   git fetch
   git merge --ff-only origin/master

For a discussion about the differences between these two options see fetch vs pull.

Rebase. This step is not necessary if git pull did not download any commits.

   git checkout work
   git rebase master         
    ... Git issues an message saying that there is a merge conflict
    ... You need to hand edit the indicated file to resolve the conflict.
   git add hand-edited-file
   git rebase --continue
    ... Git issues an message saying that there is a merge conflict
    ... You need to hand edit the indicated file to resolve the conflict.

If there are no merge conflicts the git rebase command will complete the first time it is issued.

Merge your rebased working branch into the master branch

   git checkout master
   git merge --ff-only work

Push your commits to the redmine repository

   git push

Delete the working branch

   git branch -d work

There is a race condition between the last git pull and the git push; it will be discussed below.

Detailed Version

Step 1) Start from a clean local master branch that is up to date with the local tracking branch.

Check the status of your working environment:

git status

The output must look like the following:

# On branch master
nothing to commit, working directory clean

If the output is anything else, you have work to do before you can proceed. The steps you need to do are described on the wiki page Cleaning your master branch.

Step 2) Synchronize Local Master with the Redmine Repository

This step is not required but it is strongly recommended. If you do it, it will minimize the work that you need to do when you get to the rebase step. To do this, issue the git command:

git pull

or

git fetch
git merge --ff-only origin/master

For a discussion about the differences between these two options see fetch vs pull.

Step 3) Create a working branch

You will ultimately merge your working branch into the master branch and the existence of the working branch will be lost to history. Therefore the name of the branch is not important and we will call it work. If the intention is that the branch will become part of the git history, then you should give it a better name. To create and checkout the working branch issue the following git command:

git checkout -b work

In several places the Mu2e git instructions caution you against checking out a branch when you have uncommitted changes in your working tree - this may cause the loss of those changes. There is one prominent exception to this recommendation. If you have uncommitted changes on your master branch it is safe to create a new branch; in this case the uncommitted changes will remain in your working tree and, if you commit them, they will be committed to the new branch.

Now use the git branch command to look at the local branches:

git branch

which will make the output:

  master
* work

You can see that there are now two local branches; the position of the asterisk tells you that the working tree is an image of the branch named work.

Step 4) Do your work and commit it

Do the edits that you plan to make. Create new files as needed. Then follow the instructions in Wiki for adding files, deleting files, moving files and committing all of these changes.

For this example I removed the CVS meta-data comments from the file SConscript. To be specific I removed the following 4 lines:

# $Id: SConstruct,v 1.54 2014/08/02 05:23:26 gandr Exp $
# $Author: gandr $
# $Date: 2014/08/02 05:23:26 $
#

Now, check the status:

git status

which produces the output

# On branch work
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#    modified:   SConstruct
#
no changes added to commit (use "git add" and/or "git commit -a")

This tells you that you have modified the file SConstruct but have not yet committed it.

You can see all of the differences between the working tree and its parent branch by using the command:

git diff

If you have modified many files, you can check the changes in just the file SConstruct by giving the command:

git diff SConstruct

The next step is to commit the change:

git commit -m "Remove CVS comments which are no longer needed." SConstruct

which produced the output

[work bac5d3c] Remove CVS comments which are no longer needed.
 1 file changed, 1 insertion(+), 5 deletions(-)

Step 5) Check that there are no uncommitted changes Double check that there are no uncommitted changes:

git status
# On branch work
nothing to commit, working directory clean

This says that the working tree is clean

Another way to check for a clean working tree is to use the command

git diff

which will produce no output because the working tree is clean.

You can also compare the working tree to the local master branch

git diff master

Or you can compare it to the local clone of the redmine master branch

git diff origin/master

You do not need to type the "remotes" part of the branch name but you can if you want.

Since the local master and the local clone of the remote master are the same, both of the above diff commands produce:

diff --git a/SConstruct b/SConstruct
index 2f520ce..9ff2699 100644
--- a/SConstruct
+++ b/SConstruct
@@ -2,10 +2,6 @@
 #
 # Build a Mu2e base release or test release.
 #
-# $Id: SConstruct,v 1.54 2014/08/02 05:23:26 gandr Exp $
-# $Author: gandr $
-# $Date: 2014/08/02 05:23:26 $
-#
 # Original author Rob Kutschke.
 #
 import os, re, string

In this example I only changed one file. You may change many files, add many files, delete many files or rename many files. Just be sure to commit all changes. You may make commit each file individually, commit everything with one big commit or do something in between. Normally it makes sense to commit groups of files that together make one logical change.

The most important thing is to use git status to check that your working tree is clean before proceeding to the next step.

Step 6) Sync the local master branch with redmine

First, checkout the local master branch. This changes your working tree to be an image of the local master branch and discards any uncommitted changes to tracked files.

git checkout master

You can double check that you are on the right branch using git branch, which produces the ouptut

* master
  work

The main part of this step is to use git pull to:

• Contact the redmine repository and update the local tracking branch origins/master.
• Merge the changes from origins/master into the local master

Because we have made not changes to the local master branch, the merge is guaranteed to work without conflicts. Issue the command:

git pull

If there have not been any changes committed to the redmine repository since you did your previous git pull, this will produce the output:

Already up-to-date.

If there have been changes to the redmine repository, the git pull command will produce output like:

remote: Counting objects: 568, done. remote: Compressing objects: 100% (142/142), done. remote: Total 405 (delta 277), reused 376 (delta 259) Receiving objects: 100% (405/405), 89.99 KiB | 68.00 KiB/s, done. Resolving deltas: 100% (277/277), completed with 94 local objects. From ssh://cdcvs.fnal.gov/cvs/projects/mu2eofflinesoftwaremu2eoffline/Offline

  236c81fd3..7aa1da4ff  master          -> origin/master
* [new branch]          art_v2_09_03    -> origin/art_v2_09_03

Updating 236c81fd3..7aa1da4ff Fast-forward

CalPatRec/fcl/prolog.fcl                  |  14 ++-
CalPatRec/inc/CalHelixFinderAlg.hh        |   5 +-
CalPatRec/inc/CalHelixFinderData.hh       |   5 ++
CalPatRec/inc/CalHelixFinder_types.hh     |   8 +-

In either case the working tree should be the checked out master branch. Check it with git status, which should show:

# On branch master
nothing to commit, working directory clean

Step 7) Rebase the working branch

This step is the meat of the entire workflow. It is the place at which conflicts might arise; if they do arise they need to be resolved before moving on.

If the git pull in Step 6) told you that your local repository was already up to date, then this step will do nothing. You can choose to execute it, just to be safe; or you can skip it.

Switch back to your working branch and issue the rebase command.

git checkout work
git rebase master

The action of the rebase command is described in the figures earlier on this page. If this command completes successfully, there were no unresolvable conflicts and this step is done. You can proceed to Step 8).

If there is an unresolvable conflict, git rebase will stop and issue a diagnostic telling the name of a file that contains a conflict. You should edit the the file to fix the conflict. When that is done, give the git command

git add relative-path-to-file-that-you-fixed
git rebase --continue

If there is another unresolvable conflict fix that by hand and repeat the above commands. And so on. Eventually the rebase will complete and you can move on to Step 8.

Step 8) Merge the working branch into the master branch

git checkout master
git merge --ff-only work

The ff in "--ff-only" tells git to do a "fast forward" merge. A fast forward merge will only work when there have been no changes to the local master branch since the last rebase of the working branch ( or since the creation of the branch if you have not rebased ). Because the previous step was to rebase, the -ff-only is formally redundant but it is a good idea to include the --ff-only option as a safety check that the rebase actually succeeded.

The output of this step should look like:

Updating aa98377..42447ae
Fast-forward
 SConstruct | 4 ----
 1 file changed, 4 deletions(-)

Step 9) Push the results to redmine repository and clean up

Issue the command:

git push

This will push the changes from the local master branch to the redmine repository and also update the local tracking branch. The push command will produce output like:

git push
Counting objects: 8, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 583 bytes, done.
Total 6 (delta 4), reused 0 (delta 0)
To ssh://p-mu2eofflinesoftwaremu2eoffline@cdcvs.fnal.gov/cvs/projects/mu2eofflinesoftwaremu2eoffline/Offline.git
   aa98377..42447ae  HEAD -> master

In rare cases someone may push to the redmine repository after your last pull but before you push.

Fixme: describe procedure to deal with this.

Step 10) Delete the working branch

git branch -d work

If you made an error in the above and there remain un-merged commits on this branch, then the delete will fail. You will need to figure out what the error is and how to recover from it.

If you want to force the delete of a branch that fails to delete with the -d option, try the -D option:

git branch -D work

The -D option tells git to delete the branch even if the safety checks fail.

In this workflow, the delete is done after the push succeeds. This ensures that the branch is still accessible in the event that there we need to recover from a failed push.

Additional Notes

This section has notes that need to find a home in the main body of the text.

Scenario 1

• I have a local uncommitted change in file6.txt
• I do a git fetch and discover that file6.txt has already been changed in the redmine repository.
• explore resolving the conflict using stash

git status
# On branch master
# Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
#   (use "git pull" to update your local branch)
#
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#    modified:   file6.txt
#
no changes added to commit (use "git add" and/or "git commit -a")

git diff origin/master
diff --git a/file6.txt b/file6.txt
index 978a23e..f318806 100644
--- a/file6.txt
+++ b/file6.txt
@@ -1,3 +1,3 @@
 This is file 6.  Added in the second repo.
 Pushed before committing file 5 in the first repo.
-Added this line in the second repo and pushed.
+Added this line in first repo.  Then git fetch.

git merge --ff-only origin/master
Updating 754536d..3c940d3
error: Your local changes to the following files would be overwritten by merge:
    file6.txt
Please, commit your changes or stash them before you can merge.
Aborting

git rebase origin/master
Cannot rebase: You have unstaged changes.
Please commit or stash them.

git merge origin/master
Updating 754536d..3c940d3
error: Your local changes to the following files would be overwritten by merge:
    file6.txt
Please, commit your changes or stash them before you can merge.
Aborting

git stash
Saved working directory and index state WIP on master: 754536d hand corrections to merge in file 7
HEAD is now at 754536d hand corrections to merge in file 7

git status
# On branch master
# Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
#   (use "git pull" to update your local branch)
#
nothing to commit, working directory clean

git merge --ff-only origin/master
Updating 754536d..3c940d3
Fast-forward
 file6.txt | 1 +
 1 file changed, 1 insertion(+)

git status
# On branch master
nothing to commit, working directory clean

git stash pop
Auto-merging file6.txt
CONFLICT (content): Merge conflict in file6.txt

 git status
# On branch master
# Unmerged paths:
#   (use "git reset HEAD <file>..." to unstage)
#   (use "git add <file>..." to mark resolution)
#
#    both modified:      file6.txt
#
no changes added to commit (use "git add" and/or "git commit -a")

git add file6.txt

 git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#    modified:   file6.txt
#

git commit -m "Hand merge file6" file6.txt

git status
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#   (use "git push" to publish your local commits)
#
nothing to commit, working directory clean

gitk shows a linear history

git stash list
stash@{0}: WIP on master: 754536d hand corrections to merge in file 7

git stash drop
Dropped refs/stash@{0} (4b3c222fd438f2f0c886ddebbce760143d75fbd2)

git stash list  

( no output )

Scenario 2

• I have a local committed change in file7.txt
• The redmine repo has a conflicting change to file7.txt
• I do a git pull

This will trigger a merge that will put conflict markers in file7.txt. To recover

edit file7.txt
git add file7.txt
git commit -m "hand corrections to merge in file 7" -a
git push
Explore resolving the conflict using rebase

This makes a loop in the redmine repository.