GitHubWorkflow

Introduction

This page describes a recommended git workflow for use with the Mu2e Offline code in GitHub.

The Mu2e computing group manages the Mu2e organization within GitHub. To have full access to Mu2e software, Mu2e collaborators should create a GitHub account, and request to join the Mu2e organization.

1. To execute this workflow make sure you have defined the mu2e environment by executing "setup mu2e" in your shell
2. Make sure you have a GitHub account, and that you are added to the Mu2e GitHub organization (https://github.com/orgs/Mu2e/people)
3. In order to authenticate with github, you will need to set up your ssh keys on the machine you plan to clone / push from, following the instructions here
4. Before we do any developing, we will create our own fork of the official Mu2e Offline repo using the GitHub web interface (instructions here). This fork will be your personal sandbox on GitHub; you can do anything you want to it and it will have no effect on anyone else!

The main Mu2e repository is 'Offline', which contains the algorithms, data structures, art modules and configuration used in simulation production, online filtering, offline reconstruction, and other tasks. The main development branch within Offline is 'master', which should be used for all code development. Offline also contains legacy branches associated with particular data sets, which are maintained for a limited time after the master branch has been developed past those data, as described in the table below.

Mue2/Offline branches

branch name	branch purpose	support end date
master	development	end of Mu2e
MDC2018	MDC2018 dataset support	end of 2020

To work on a branch other than master:

> git checkout 'BRANCH' 

where 'BRANCH' is the name of the branch

Authentication

If you are following the instructions for accessing Offline as a user, not a developer, you will never need to authenticate yourself to github. If you will do work as a developer than there will be some steps for which you will need to authenticate yourself to github. The page Authentication#Authenticating to github has instructions on two ways to do so

Migrating an existing redmine clone directory to GitHub

You can repoint an existing Offline clone from redmine to GitHub as a way of making an easy transition. To follow these instructions, you must already have a GitHub account that's registered in the Mu2e Organization. The following assumes your redmine clone is in /mu2e/app/Me/MyRedmineClone/Offline, that your GitHub username is MyGitHubName, and that you are working on a branch called MyDevelopmentBranch.

1. Login to GitHub
2. Fork Mu2e/Offline in GitHub using the 'Fork' button on the top right of the Mu2e Offline repo page
3. Add GitHub as a remote to your clone

   > cd /mu2e/app/Me/MyRedmineClone/Offline

   > git remote rename origin Redmine

   > git remote add mu2e https://github.com/Mu2e/Offline

   > git remote add origin https://github.com/MyGitHubName/Offline

   These commands are very fast as no data is actually transferred. You can see your remotes with:

   > git remote -v

   Your clone is now connected to the main Mu2e github repository and your own fork in addition to the Redmine repository. Additionally, we have changed your fork to be the default remote repository (origin)

4. Push your working branch to your GitHub fork:

   > git push -u origin MyDevelopmentBranch

After this, you can continue developing in this directory according to the general GitHub workflow described below.

Downloading Offline as a user and NOT a developer

Option 1: you want the default primary version of the code (most people):

1. clone the repo:
git clone https://github.com/mu2e/Offline
2. Done!

Option 2: A particular collaborator has a version or branch you want to use:

1. Find their github user name
2. Determine what the name of the branch they are working on is (Note: this can be master!)
3. Clone their fork:
git clone https://github.com/<user name>/Offline cd Offline git checkout origin/<branch name>
4. Done!

Option 3: You want to use pgit to avoid a long compilation time (EXPERIMENTAL)

1. Create new directory to put your Offline repo in and move to that directory
mkdir Offline cd Offline
2. As in Option 2, determine fork and branch name you wish to use
3. Create a partial checkout clone
pgit2 setup https://github.com/<user name (or mu2e)>/Offline <branch name>
4. You can now use as normal:
source setup.sh scons -j 4
5. You might need to source the following before 'source setup.sh':
source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh

Developer Workflow

To start, make sure you understand what a remote is, the difference between a branch and a fork, and what a pull request is.

1. Create our own fork of the official Mu2e Offline repo using the GitHub web interface (instructions here). This fork will be your personal sandbox on GitHub; you can do anything you want to it and it will have no effect on anyone else! Note that you only need to do this once; you can reuse this fork for all your development projects.
2. Create a local clone of your Offline fork. This is another step that you only need to do once and you can use this local clone for multiple development projects in you want to. When you push code to another repository, the default will be to push to your fork own instead of the official mu2e version.

setup mu2e
git init Offline
cd Offline
git remote add -f origin git@github.com:<your github username>/Offline
git remote add mu2e https://github.com/mu2e/Offline
git branch -d -r origin/master
git fetch mu2e master

3. The above steps will create a git repository with the following properties:
   • There are two remotes: your fork and mu2e/Offline
   • All of the branches from your fork, except master, will be visible as remote branches
   • The master branch from mu2e/Offline will be visible as a remote branch.
   • There will be no local branches and no checked out files; so you cannot yet build or run anything.
   • You can inspect the status of remotes and branches using git remote -v and git branch -avv.
4. When you want to start a new development project, the first step is to make a new local branch on which to do the work. If two development efforts are not intrinsically coupled to each other, each should be done on its own branch. This branch is local to your fork, so has no impact on the main Offline. The branch name is used for sharing your development work with other people while it is still in progress, and for making your pull request, but has no intrinsic meaning.
git fetch mu2e master git checkout --no-track -b <development branch name> mu2e/master
5. After you have done the previous step, your directory will be populated with files; you can start to edit, build and run. The two commands in the previous step ensure that your new development work starts from the up-to-date head of the master branch in the official mu2e/Offline. Note that the master branch from your own github fork is never a part of this work; in general it will always be very out of date. This is why step 2 is a little complicated: among other things it ensures that the master branch from your fork is not visible in your clone.
6. Do your work and commit it locally. This makes a local 'backup' of your work.
git commit -m "brief comment describing the changes you are committing"
7. When your development is complete, or when you wish to back it up remotely, or share your work with others, push your branch back to your GitHub fork. Note this requires you to have setup your github authentication.
git push -u origin <development branch name>
8. In the previous step the -u option tells git that your local branch should track the branch in your fork. If you push the branch again, the -u option is not needed but it won't hurt if it is present. You can use git branch -avv to see that your development branch is now tracking the version of itself in your github fork.
9. When your development is complete and tested and ready to be re-integrated with the rest of Mu2e/Offline, go to the GitHub web site and request that your branch be pulled back into Mu2e/Offline. Your pull request will start the code review process (see Code Review), which may take anywhere from a few hours to a few days. Note that GitHub will not allow merging a pull request that conflicts with other development merged since your clone.
   1. In a web browser, open https://github.com/<your user name>/Offline
   2. push the 'new request pull' button associated with your development branch
   3. More info is available at instructions
10. If your pull request shows conflicts with Mu2e/Offline, you must reconcile those before you can request a pull. This involves first bringing in the latest Mu2e/Offline master branch, and then merging it with your branch. Please do NOT perform these steps if there are no conflicts, as the merging process complicates the git history.
git fetch mu2e master git checkout <development branch name> git merge mu2e/master
11. if the 'git merge master' step has unresolved conflicts, you will need to edit the code to resolve them. There is no formula for this step; you will have to look at the 2 versions of the conflicting code blocks and decide how to best merge both functionality. If you have questions about the intent of the previously-merged conflicting code, work together with the author of those changes to figure that out. You can figure out who last changed a line in a file using the 'git blame' command.
git blame mu2e/master <name of file that has conflicts>
12. Once all conflicts from the merge are resolved, commit the merge and push it back to your fork. After this, GitHub will allow you to request a pull.
git add <files that were part of resolving conflicts> git remove <any files that need to be removed to resolve conflicts> git commit -m "Resolve conflicts message" git push origin <development branch name>
13. If changes are requested during the code review process, make those on the same development branch as your pull request. When the changes are complete, commit them, and push your changes back to your fork. GitHub will automatically add the new commits to the pull request, so the reviewers and the GitHub record will show them.
<edit code as requested by reviewer> git commit -m "Address review comment X" git push origin <development branch name>
14. When the code reviewers are satisfied, one of the software coordinators will merge the pull into Mu2e/Offline. Once your pull request is merged your changes (commits) will be part of Mu2e/Offline master, and your development branch can be deleted. If you are uncertain if your branch has been merged or not, select the branch, and push the 'compare' button. If this comes back stating there is 'nothing to compare to', it means all your changes were already merged. If it shows differences, those have NOT been merged, so do NOT delete your branch. To delete your branch in GitHub, just push the trash can icon. You can also delete the branch in your shell, as
git branch -d <my branch name> git push origin --delete <my branch name> (this deletes the branch from your github fork as well)
15. To reuse your working directory for a new development, first refresh to the current head of master, then create a new branch as described above. Do NOT reuse branches for new development, as updating those to the head of Mu2e/Offline master will confuse the git history.
git fetch mu2e master git checkout -b <new development branch name>
16. We encourage you to commit your work frequently and push to your github fork frequently; this is the best way to backup your work. You do NOT need to wait until you are ready for a pull request to push to your fork.

Adapting your Existing Clones to the New Workflow

Many of you have existing clones of your own github fork that were created using an older recommended github workflow. This section discusses how to bring your clone into compliance with the new recommended workflow. We cannot give an exact script to follow since everyone's local clone is likely to be different; so the following is general guidance:

1. If you have unpushed commits on your local master, see the instructions in the hints section below.
2. If you have uncommitted changes in your checked out area, commit them to a local development branch that you intend to keep.
   1. If needed: git stash; create a new development branch starting from the current head; git stash pop.
3. If the old workflow you will have have a local branch named master and a branch named remotes/origin/master. These come from the master branch of your github fork; if you still have these branches, delete them.
   1. You cannot delete the branch that is currently checked out; therefore, to delete your local master you need first to checkout a different branch.
4. You must have a branch named remotes/mu2e/master that comes from the Offline repository in the Mu2e github instance; actually the middle "mu2e" in the branch name is just an arbitrary identifier than you can choose but it would be weird to chose anything except mu2e or Mu2e.
5. If you have existing development branches that started from something other than mu2e/master, just keep them and proceed as described in the recommended workflow; if you are lucky they will merge cleanly. If not, follow the merge/rebase procedure.
6. Start all new development branches with:

 git fetch mu2e master
 git checkout -b <new development branch name>

Here are a few hints and reminders of things to watch for:

1. Use git remote -v and git branch -avv to learn what remotes you have and what branches are tracking what other branches.
2. If your local master branch has commits on it that you need to keep, then start a new development branch from the head of master. Just develop on it normally and push it to your fork when you are ready and make a pull request when you are ready for that. If you are lucky it will merge cleanly; if not follow the merge/rebase procedure. Once you have created the new development branch you can delete the local master branch.

Collaborating on a feature

Sometimes you may want to collaborate on a feature branch with other developers. In this case since the main Offline repository no longer has all the development branches we need to do a couple extra steps

1. First make sure you actually need to work on the same branch. Are you actually working on the same feature? Can the problem be split into smaller features that can be developed asynchronously? Just because features are related doesn't mean they need to be developed on the same branch
2. Determine if a large number of people will be developing on the same branch for a significant amount of time. In this case it should become an official branch in the mu2e/Offline, like MDC2018
3. Decide which user's fork will be the primary repo for this feature branch, and which branch on that fork you are going to use. If a new branch is needed, the owner of that fork start the new branch as follows. First make sure that you have done steps 1 and 2 in #Developer_Workflow. Then do the following:
git fetch mu2e master git checkout --no-track -b <branch name> mu2e/master git push -u origin <branch name>
4. There are then a couple options for moving forward: either add all other developers as collaborators on the primary fork, or use pull requests to the primary fork
5. To add developers as collaborators:
1. The owner of the primary fork opens https://github.com/<their user name>/Offline
2. click settings on the right, then collaborators
3. In the collaborators box, type the github user name of each other developer and hit "Add collaborator"
4. The other collaborators can then either create a read/write access clone of the primary fork, or add it as a remote to an existing offline repo
git clone git@github.com:<primary user name>/Offline or git remote add primaryfork git@github.com:<primary user name>/Offline
5. The other collaborators can now push directly to the primary fork as if it was their own:
git push primaryfork <branch name>
6. To use pull requests:
1. The owner of the primary fork can just push to it as normal following the normal developer workflow
2. Other developers clone their own fork, but add the primary fork as a remote
git remote add primaryfork git@githbub.com:<primary user name>/Offline
3. Other developers can pull in and merge changes from collaborators by fetching/pulling/merging from this remote
git fetch primaryfork git merge primaryfork/<branch name>
4. Other developers push to their own fork
git push origin <branch namee>
5. Like in the normal developer workflow, they open a pull request. But then in the compare window before creating the request, change the "base repository" from mu2e/Offline to <primary user name>/Offline (see here)
6. the owner of the primary fork will need to accept and merge it in
7. everything else goes like the normal workflow

Code Review

An important part of the GitHub workflow is reviewing new code before putting it back into the repository. Reviews are intended to minimize the risk that the requested changes break anything, check that the content of the changes are sensible, and enforce Mu2e coding standards and policies. Some reviews are automated, such as testing that the code builds and can run a few events of some standard apps. Automated code formatting checks will also be deployed soon.

Offline repo managers are responsible for assigning reviewers to each Pull Request (PR), as well as a manager in charge of each particular PR. The PR author may also assign or suggest reviewers. All assigned reviewers must approve the PR before the assigned manager will merge it in.

PRs can cover multiple subject areas. Reviewers should concentrate on reviewing code in areas in which they have personal expertise and/or subject knowledge. Reviewers are not expected to learn about areas outside their experience, as other reviewers will cover those. If you feel you were incorrectly assigned to a review, contact the repo manager assigned to the PR to request clarification or to be removed as reviewer. Reviewers should attempt to complete their reviews within a few days. Large PRs may take longer to review, and PR authors should plan accordingly. If an assigned reviewer is unavailable, they or the PR author should contact the assigned repo manager to request a substitution. The Offline repo managers should be alerted if a review becomes stuck for any reason.

Reviewers should look at the content of the PR commits for code correctness, good design and efficient implementation. Reviewers don’t need to build or run the code, that’s for the automated tests. The github commit differences referenced in the PR are the easiest way to see and review the changes. Review feedback should be inserted as comments at the relevant lines in the github diff where the reviewer has a concern. After reviewing all files and commits, reviewers should complete their review using the github interface. If you feel changes are required submit your review with that box checked. If you simply have questions submit your review checking the 'neutral' box; this neither approves the review or requires changes, it just requires a response on the part of the author. PR requesters should respond to all review comments or questions in the PR thread and/or by making a new commit inside the PR. Once all the reviewer's concerns and questions have been addressed, the reviewer should re-submit their review checking the 'approved' box. The repo manager assigned to the PR should merge the PR after all reviewers have approved it.

I have a branch on Redmine that's not on GitHub! How do I use it now?

The official Mu2e GitHub repo will usually have only the master branch (+ maybe some tagged release branches). Currently Redmine has >100 branches. If you have a Redmine branch that is not merged in to master, you will move this branch to your fork to work on it instead.

To start, you should read the development workflow and understand remotes and forks, and have a github account and a fork of the Mu2e Offline repo.

1. Create a local clone of your Offline fork. This will be identical to cloning the official mu2e repo (if your fork is up to date), except that the default remote (called "origin") where you will push to by default will point to your fork instead of the official mu2e version.
git clone git@github.com:<your user name>/Offline
2. Add Redmine as a remote to your local clone:
cd Offline git remote add redmine http://cdcvs.fnal.gov/projects/mu2eofflinesoftwaremu2eoffline/Offline.git
3. Fetch a list of the commits and branches from redmine, and check out whatever branch you need
git fetch redmine git checkout -b <branch name> redmine/<branch name>
4. Push this branch to your fork
git push origin <branch name>
5. You should now be able to see your branch on github.com/<your user name>/Offline. You can now continue working on that branch and pushing to your fork, and when it is ready you can submit a pull request to the main Mu2e repo using the developer workflow