Skip to contentSkip to conversation

Workflow question

DISCUSSION Russ Brown 6 participants 17 messages Jun 15, 2016

lore
Share & tools
lore

Russ Brown

author
#0 lore raw

Hi,

I've been trying to think of a git workflow that we could use to replace our current svn/svk setup without simply using git in exactly the same way that we use svn/svk.

Basically, we develop, maintain and enhance a website. On the central repo is trunk which represents live, and any number of project branches. Developers don't use local branches: they check out the project branches they're working on and commit to those. Developers merge from trunk to project branch from time to time to keep them current, and when a project rolls out the branch is merged to trunk.

In addition to the obvious advantages that git would give us (such as properly tracking that code author as opposed to the person who did the merge), I'm wanting to gain the following benefits:

* The repository is very large (multiple gigabytes) and mirroring using
svk obviously takes a lot of time and space, so I'm keen to bring that down, most likely by the developer not needing to mirror branches he doesn't care about, or by being able to throw away branches he's done with.
* The repository is full of revisions that fail review (or break
things) and are fixed by subsequent revisions. We'd much rather be able to have the developer fix his revisions before they get committed 'upstream' (whatever that ends up meaning).

I asked earlier about the email-based model that git itself uses, and while it appears to work very well for a widely-dispersed open-source project, I think it will be too cumbersome and slow for a fast-paced internal development team who make a number of live releases every day.

So, I've been thinking and have come up with this, which I'd appreciate comments about:

1. On a server we stick a git repository which contains the master
branch, which represents what trunk did (i.e. the live platform). This branch contains the full history for the live platform.
2. On the same server we clone that repo to create a second repository
which is the developer area. In this we track master from the live repo, and also create project branches.
3. Developers clone this developer repo, but I'd like them to be able
to decide which branches they actually want to clone from that repository rather than simply cloning them all. Is this possible?
4. Developers create a local branch of the project they
are working on and commit to that.
5. Once they think they're done, they publish their branch to the
development repo and request for comments.
6. If all is not well, the developer creates a new local branch and
moves good revisions from his previous one to the new one, modifying things as he goes, and republishes his new branch.
7. If all is well, their works gets merged or rebased onto the main
project branch, and once that's ready it gets pushed to the master and rolled to live. The developer's individual branches get deleted from the dev repo since they're no longer required.
8. From time to time the master branch gets merged to the project
branches. Developer's local branches can be rebased against the project branch as they please.

Firstly, is all of this possible, and if so would it be considered a good way of going about it?

Any comments appreciated.

Signature · 1 line
Russ
#1 lore raw
Quoted 26 lines · Russ Brown

Hi,

I've been trying to think of a git workflow that we could use to replace our current svn/svk setup without simply using git in exactly the same way that we use svn/svk.

Basically, we develop, maintain and enhance a website. On the central repo is trunk which represents live, and any number of project branches. Developers don't use local branches: they check out the project branches they're working on and commit to those. Developers merge from trunk to project branch from time to time to keep them current, and when a project rolls out the branch is merged to trunk.

In addition to the obvious advantages that git would give us (such as properly tracking that code author as opposed to the person who did the merge), I'm wanting to gain the following benefits:

* The repository is very large (multiple gigabytes) and mirroring using
svk obviously takes a lot of time and space, so I'm keen to bring that down, most likely by the developer not needing to mirror branches he doesn't care about, or by being able to throw away branches he's done with.
* The repository is full of revisions that fail review (or break
things) and are fixed by subsequent revisions. We'd much rather be able to have the developer fix his revisions before they get committed 'upstream' (whatever that ends up meaning).

Quoted 5 lines

I asked earlier about the email-based model that git itself uses, and while it appears to work very well for a widely-dispersed open-source project, I think it will be too cumbersome and slow for a fast-paced internal development team who make a number of live releases every day.

We came to the same conclusion at our workplace. Email works great, but it's faster and better to just walk over to your colleague and ask what he/she thinks about something.

Quoted 6 lines

So, I've been thinking and have come up with this, which I'd appreciate comments about:

1. On a server we stick a git repository which contains the master
branch, which represents what trunk did (i.e. the live platform). This branch contains the full history for the live platform.

A must-have for any more-than-two-developers setup, so so far so good ;-)

Quoted 3 lines

2. On the same server we clone that repo to create a second repository
which is the developer area. In this we track master from the live repo, and also create project branches.

This isn't necessary. Branches in git are very nearly zero-cost, so having them in the same repo as the master branch won't hurt a bit. You can add an update-hook on the mothership repo to restrict access to the master branch if you like, but creating two separate repos will likely give more headache than it's worth.

Quoted 3 lines

3. Developers clone this developer repo, but I'd like them to be able
to decide which branches they actually want to clone from that repository rather than simply cloning them all. Is this possible?

Yes, although I'd actually recommend you to clone the full repo anyway. Since the various branches are likely to share quite a lot of history the added overhead of a few extra branches will most likely be negligible. git makes even very large codebases appear small and unobtrusive. The linux kernel history since 2.6.12 contains 554853 objects and compresses down to 178MiB.

I think KDE is the largest repo imported to git so far. I've forgotten the exact numbers, but everyone was very impressed, and quite surprised, at the vast difference between SVN and git storage requirements.

Quoted 4 lines

4. Developers create a local branch of the project they
are working on and commit to that.
5. Once they think they're done, they publish their branch to the
development repo and request for comments.

Using topic-branches is a much better strategy, usually, since that allows each feature to be evaluated and improved on on its own, rather than having to merge *all* of a particular developers changes just to get desirable feature X. Note that cherry-pick provides ways of doing that anyways, but in a much less elegant way, and your integrator/ release engineer will likely tear his hair out on a daily basis without topic branches.

Quoted 7 lines

6. If all is not well, the developer creates a new local branch and
moves good revisions from his previous one to the new one, modifying things as he goes, and republishes his new branch.
7. If all is well, their works gets merged or rebased onto the main
project branch, and once that's ready it gets pushed to the master and rolled to live. The developer's individual branches get deleted from the dev repo since they're no longer required.

Topic branches would work the same, basically, except they can be pushed up for review a lot faster.

If all the pushing gets cumbersome, it also makes it easy to send the patches out as emails for discussion. It's usually easier to let git handle the actual code transmissions, but discussing patches in emails works quite well if it's intended for a wider audience.

Quoted 4 lines

8. From time to time the master branch gets merged to the project
branches. Developer's local branches can be rebased against the project branch as they please.

criss-cross merging can turn kinda nasty though, as you may have a hard time finding *the* common point when you run into that rogue merge with conflict markers everywhere (it happens for everyone sooner or later).

I'd suggest you rebase the developer/topic branches onto master with regular intervals instead.

Signature · 3 lines
Andreas Ericsson                   [email protected]
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231
#2 lore raw
Quoted 3 lines · Andreas Ericsson

We came to the same conclusion at our workplace. Email works great, but it's faster and better to just walk over to your colleague and ask what he/she thinks about something.

One of the projects I am working on does things this way, but I have to admit that I miss the email code-review process. There are often small fixups (stylistic, minor nits, "I would have done it this way...", etc) that are worth pointing out at the time, but are more painful to go back and correct much later.

And documenting those discussions can really help other developers besides the author and reviewer.

-Peff

#3 lore raw
Quoted 17 lines · Jeff King
Quoted 5 lines · Andreas Ericsson

We came to the same conclusion at our workplace. Email works great, but
it's faster and better to just walk over to your colleague and ask what
he/she thinks about something.

One of the projects I am working on does things this way, but I have to
admit that I miss the email code-review process. There are often small fixups (stylistic, minor nits, "I would have done it this way...", etc)
that are worth pointing out at the time, but are more painful to go back
and correct much later.

Quoted 2 lines

And documenting those discussions can really help other developers besides the author and reviewer.

Google has a pretty interesting internal code review system:

<http://video.google.com/videoplay?docid=-8502904076440714866>

Cheers,
Wincent

#5 lore raw
Quoted 9 lines · Jeff King
Quoted 3 lines · Wincent Colaiuta

Google has a pretty interesting internal code review system:

<http://video.google.com/videoplay?docid=-8502904076440714866>

Interesting pointer, thanks. Though I did get a little concerned when I
saw that it is based on Perforce. ;)

Yes (poor Google employees!), and not only that, I'm not aware of any public access to the code. But still, there are some interesting ideas about the review process in there.

W

author
#6 lore raw
In reply to Andreas Ericsson
Quoted 34 lines · Andreas Ericsson
Quoted 27 lines · Russ Brown

Hi,

I've been trying to think of a git workflow that we could use to replace our current svn/svk setup without simply using git in exactly the same way that we use svn/svk.

Basically, we develop, maintain and enhance a website. On the central repo is trunk which represents live, and any number of project branches. Developers don't use local branches: they check out the project branches they're working on and commit to those. Developers merge from trunk to project branch from time to time to keep them current, and when a project rolls out the branch is merged to trunk.

In addition to the obvious advantages that git would give us (such as properly tracking that code author as opposed to the person who did the merge), I'm wanting to gain the following benefits:

* The repository is very large (multiple gigabytes) and mirroring using
svk obviously takes a lot of time and space, so I'm keen to bring that down, most likely by the developer not needing to mirror branches he doesn't care about, or by being able to throw away branches he's done with.
* The repository is full of revisions that fail review (or break
things) and are fixed by subsequent revisions. We'd much rather be able to have the developer fix his revisions before they get committed 'upstream' (whatever that ends up meaning).

Quoted 5 lines

I asked earlier about the email-based model that git itself uses, and while it appears to work very well for a widely-dispersed open-source project, I think it will be too cumbersome and slow for a fast-paced internal development team who make a number of live releases every day.

Quoted 4 lines

We came to the same conclusion at our workplace. Email works great, but it's faster and better to just walk over to your colleague and ask what he/she thinks about something.

Very true, with the minor exception that at my place there are developers working at different sites, so the walk-over method will only work for specific subsets of the team. :)

Quoted 13 lines
Quoted 6 lines

So, I've been thinking and have come up with this, which I'd appreciate comments about:

1. On a server we stick a git repository which contains the master
branch, which represents what trunk did (i.e. the live platform). This branch contains the full history for the live platform.

A must-have for any more-than-two-developers setup, so so far so good ;-)

Quoted 3 lines

2. On the same server we clone that repo to create a second repository
which is the developer area. In this we track master from the live repo, and also create project branches.

Quoted 6 lines

This isn't necessary. Branches in git are very nearly zero-cost, so having them in the same repo as the master branch won't hurt a bit. You can add an update-hook on the mothership repo to restrict access to the master branch if you like, but creating two separate repos will likely give more headache than it's worth.

Ah, right. I'm just trying to remember why it was I came up with that idea in the first place, but I'm struggling a bit. :)

Quoted 11 lines
Quoted 3 lines

3. Developers clone this developer repo, but I'd like them to be able
to decide which branches they actually want to clone from that repository rather than simply cloning them all. Is this possible?

Yes, although I'd actually recommend you to clone the full repo anyway. Since the various branches are likely to share quite a lot of history the added overhead of a few extra branches will most likely be negligible. git makes even very large codebases appear small and unobtrusive. The linux kernel history since 2.6.12 contains 554853 objects and compresses down to 178MiB.

Makes sense. Thing is, with git-svn (which I've been using for a while now) it's possible to 'bring in' an upstream branch at will by adding it to your config file, and on next git-svn fetch revisions that affect that branch are automatically fetched. I figured it would just be a little more efficient, though I appreciate it involves more 'fiddling' by the client users.

Here's a question: is the creation and deletion of a branch also version controlled as it is in Subversion? In other words, if I create a branch, develop on it and delete it without merging it anywhere, will the revisions hang around and get pulled down by future people cloning the repository, or do they get thrown away?

Quoted 10 lines

I think KDE is the largest repo imported to git so far. I've forgotten the exact numbers, but everyone was very impressed, and quite surprised, at the vast difference between SVN and git storage requirements.

Quoted 4 lines

4. Developers create a local branch of the project they
are working on and commit to that.
5. Once they think they're done, they publish their branch to the
development repo and request for comments.

Quoted 8 lines

Using topic-branches is a much better strategy, usually, since that allows each feature to be evaluated and improved on on its own, rather than having to merge *all* of a particular developers changes just to get desirable feature X. Note that cherry-pick provides ways of doing that anyways, but in a much less elegant way, and your integrator/ release engineer will likely tear his hair out on a daily basis without topic branches.

I've seen the term 'topic-branch' used here quite a bit but it's unfamiliar to me. It is basically synonymous with what we call a 'project branch'? i.e. Management decide that feature X is required so we create a branch to develop it on which all developers on the project commit to.

Note that in step 4 above I mean the developer takes a local branch of the topic branch. For example, we start projectX/main, and create branch projectX on the shared repo. Developer 'jeff' works on the project and so creates local branch projectX/jeff and begins work. In step 5 they push this local branch to the shared repo so everyone can see it (alternative to the 'walk-over' method or emailing). Note that all changes jeff making in projectX/jeff are specific to the project branch, so he can rebase against other changes that get committed to that project branch.

If colleagues don't like his changes he can create projectXjeff2 and try again.

Jeff can also have other local branches to keep separate changes he is making on the other projects he is involved in.

That's how I'd thought of it happening...

Quoted 8 lines
Quoted 7 lines

6. If all is not well, the developer creates a new local branch and
moves good revisions from his previous one to the new one, modifying things as he goes, and republishes his new branch.
7. If all is well, their works gets merged or rebased onto the main
project branch, and once that's ready it gets pushed to the master and rolled to live. The developer's individual branches get deleted from the dev repo since they're no longer required.

Quoted 9 lines

Topic branches would work the same, basically, except they can be pushed up for review a lot faster.

If all the pushing gets cumbersome, it also makes it easy to send the patches
out as emails for discussion. It's usually easier to let git handle the actual code transmissions, but discussing patches in emails works quite well if it's intended for a wider audience.

Yes, I am going to experiment with this a little too to see just how much work it would involve for the developers (if it's too much they won't do it) :)

Quoted 5 lines
Quoted 4 lines

8. From time to time the master branch gets merged to the project
branches. Developer's local branches can be rebased against the project branch as they please.

Quoted 9 lines

criss-cross merging can turn kinda nasty though, as you may have a hard time
finding *the* common point when you run into that rogue merge with conflict markers everywhere (it happens for everyone sooner or later).

I'd suggest you rebase the developer/topic branches onto master with regular
intervals instead.

Having been using git-svn for a while I really like the clean history result that rebase gives, however my understanding was that you should never rebase any published branch as it could screw up clones of that branch. In fact, this is what has me the most confused: how to rebase a project branch that is on a shared repository against master when everyone will have it cloned? Or is this something that I clearly don't understand properly?

(Thanks for your answers BTW Andreas)

Signature · 1 line
Russ
#7 lore raw
Quoted 9 lines · Russ Brown
Quoted 6 lines

This isn't necessary. Branches in git are very nearly zero-cost, so having them in the same repo as the master branch won't hurt a bit. You can add an update-hook on the mothership repo to restrict access to the master branch if you like, but creating two separate repos will likely give more headache than it's worth.

Ah, right. I'm just trying to remember why it was I came up with that idea in the first place, but I'm struggling a bit. :)

It's not entirely true that branches are zero-cost. They cost nothing to make, but fetching a branch that has commits that you don't want is going to cost something. And since git-clone doesn't have good support for "fetch only this subset of the branches" you end up getting them all.

That being said, you will be amazed how _little_ that cost is unless one of the branches doesn't delta well (e.g., if one branch introduces 100M of unrelated images, then getting that branch is going to cost). But it's almost certainly worth seeing what your workload is like.

Quoted 5 lines

Here's a question: is the creation and deletion of a branch also version controlled as it is in Subversion? In other words, if I create a branch, develop on it and delete it without merging it anywhere, will the revisions hang around and get pulled down by future people cloning the repository, or do they get thrown away?

Branch creation isn't version controlled, but your frame of reference is obviously svn branches, not git branches. A git repo is just a collection of refs, and each ref is a pointer to some commit (which points to a tree and to further commits). Branches are just refs in a given namespace (where the namespace refs/heads/ indicates "it's ok to make further commits on top of these").

So if you create a branch in your private repo, it is a totally private thing; the central repo has no notion of it. When you push that branch to the central repo, you are creating a new ref at the central location that points to some commits you made. People who clone it will clone that ref and your commits. If you delete the branch locally, again it has no effect on the master repo. If you delete the branch from the central repo, then nobody will fetch those commits anymore (since nothing will be pointing to them).

Quoted 5 lines

I've seen the term 'topic-branch' used here quite a bit but it's unfamiliar to me. It is basically synonymous with what we call a 'project branch'? i.e. Management decide that feature X is required so we create a branch to develop it on which all developers on the project commit to.

Yes, that's exactly right. You have a "branch" for working on a "topic".

Quoted 4 lines

Note that in step 4 above I mean the developer takes a local branch of the topic branch. For example, we start projectX/main, and create branch projectX on the shared repo. Developer 'jeff' works on the project and so creates local branch projectX/jeff and begins work. In step 5 they

You are actually talking about per-developer, per-topic branches. Which is fine, too. It sounds like these projects are big (many participants over an extended period), so you may want to do things hierarchically:

1. project X starts, so you make a branch projectX and assign Bob to be the integrator (or if you prefer, nobody is the integrator).
2. Jeff starts to work on project X, feature Y. He makes a branch projectX/featureY, based on projectX. He may publish this to the shared repo if he wants to communicate his progress.
3. When featureY is "ready", he tells Bob to pull it into the main projectX branch (or if no integrator, he does it himself).

Any developer can see the progress of any "topic" by looking at its branch. But only things which have advanced to the main "projectX" branch are used as the basis for other topics. If you want, you can replace featureY with "jeff" to indicate "this is the work Jeff is doing on projectX", but then you will run into issues when Jeff is working on two different topics of projectY.

Quoted 7 lines

Having been using git-svn for a while I really like the clean history result that rebase gives, however my understanding was that you should never rebase any published branch as it could screw up clones of that branch. In fact, this is what has me the most confused: how to rebase a project branch that is on a shared repository against master when everyone will have it cloned? Or is this something that I clearly don't understand properly?

You generally don't want to rebase that branch. If there are other branches based on some work, then it will become more difficult to merge those branches into the rebased work. IOW, rebase works well only at the "outermost" level of development. So if branch "featureY" is branched from "projectX" which is branched from "main", then it is reasonable to rebase featureY against projectX. But rebasing projectX against master will make integrating "featureY" more difficult. So you should either:
- just do regular merges of projectX into main (and hopefully, if projectX is an aggregate of features, it won't have _that_ many merges, so the history should still be quite readable)
- wait until all such "featureY" branches are merged into projectX, announce a freeze of branching from projectX, and then rebase it forward.

-Peff

author
#8 lore raw
Quoted 17 lines · Jeff King
Quoted 8 lines · Russ Brown
Quoted 6 lines

This isn't necessary. Branches in git are very nearly zero-cost, so having them in the same repo as the master branch won't hurt a bit. You can add an update-hook on the mothership repo to restrict access to the master branch if you like, but creating two separate repos will likely give more headache than it's worth.

Ah, right. I'm just trying to remember why it was I came up with that idea in the first place, but I'm struggling a bit. :)

It's not entirely true that branches are zero-cost. They cost nothing to make, but fetching a branch that has commits that you don't want is going to cost something. And since git-clone doesn't have good support for "fetch only this subset of the branches" you end up getting them all.

Quoted 5 lines

That being said, you will be amazed how _little_ that cost is unless one of the branches doesn't delta well (e.g., if one branch introduces 100M of unrelated images, then getting that branch is going to cost). But it's almost certainly worth seeing what your workload is like.

That's pretty much what I thought. Given that we'll probably only port across a few (active) branches if we do decide to convert, I figure we might as well start off by cloning them all and see how it goes.

Quoted 6 lines
Quoted 5 lines

Here's a question: is the creation and deletion of a branch also version controlled as it is in Subversion? In other words, if I create a branch, develop on it and delete it without merging it anywhere, will the revisions hang around and get pulled down by future people cloning the repository, or do they get thrown away?

Quoted 7 lines

Branch creation isn't version controlled, but your frame of reference is obviously svn branches, not git branches. A git repo is just a collection of refs, and each ref is a pointer to some commit (which points to a tree and to further commits). Branches are just refs in a given namespace (where the namespace refs/heads/ indicates "it's ok to make further commits on top of these").

I keep reading things similar to this and bit by bit I'm starting to get it. :) I suppose this is one case in which it's definitely a disadvantage to have a good understanding of svn before coming to git...

<yoda>You must unlearn what you have learned</yoda>

Quoted 9 lines

So if you create a branch in your private repo, it is a totally private thing; the central repo has no notion of it. When you push that branch to the central repo, you are creating a new ref at the central location that points to some commits you made. People who clone it will clone that ref and your commits. If you delete the branch locally, again it has no effect on the master repo. If you delete the branch from the central repo, then nobody will fetch those commits anymore (since nothing will be pointing to them).

If you delete a branch that has commits on it that aren't referenced by any other branches, will those commits be removed by something like git pack or git gc?

Quoted 25 lines
Quoted 5 lines

I've seen the term 'topic-branch' used here quite a bit but it's unfamiliar to me. It is basically synonymous with what we call a 'project branch'? i.e. Management decide that feature X is required so we create a branch to develop it on which all developers on the project commit to.

Yes, that's exactly right. You have a "branch" for working on a "topic".

Quoted 4 lines

Note that in step 4 above I mean the developer takes a local branch of the topic branch. For example, we start projectX/main, and create branch projectX on the shared repo. Developer 'jeff' works on the project and so creates local branch projectX/jeff and begins work. In step 5 they

You are actually talking about per-developer, per-topic branches. Which is fine, too. It sounds like these projects are big (many participants over an extended period), so you may want to do things hierarchically:

1. project X starts, so you make a branch projectX and assign Bob to be the integrator (or if you prefer, nobody is the integrator).
2. Jeff starts to work on project X, feature Y. He makes a branch projectX/featureY, based on projectX. He may publish this to the shared repo if he wants to communicate his progress.
3. When featureY is "ready", he tells Bob to pull it into the main projectX branch (or if no integrator, he does it himself).

Quoted 7 lines

Any developer can see the progress of any "topic" by looking at its branch. But only things which have advanced to the main "projectX" branch are used as the basis for other topics. If you want, you can replace featureY with "jeff" to indicate "this is the work Jeff is doing on projectX", but then you will run into issues when Jeff is working on two different topics of projectY.

Yes, that's kinda like what I was thinking. The distinction between a developer working on a named branch or a feature branch I suppose is one to think about.

I suppose what has me the most confused is how a developer works with a remote branch: I've come to understand that a developer should never check out and work on a remote branch, and always create a local one and work on that. If he does that using the above hierarchy, there then becomes main->projectX->featureY->jeff_local_branch_of_featureY. Or is is possible for a developer to work directory on a remote branch?

Quoted 22 lines
Quoted 7 lines

Having been using git-svn for a while I really like the clean history result that rebase gives, however my understanding was that you should never rebase any published branch as it could screw up clones of that branch. In fact, this is what has me the most confused: how to rebase a project branch that is on a shared repository against master when everyone will have it cloned? Or is this something that I clearly don't understand properly?

You generally don't want to rebase that branch. If there are other branches based on some work, then it will become more difficult to merge those branches into the rebased work. IOW, rebase works well only at the "outermost" level of development. So if branch "featureY" is branched from "projectX" which is branched from "main", then it is reasonable to rebase featureY against projectX. But rebasing projectX against master will make integrating "featureY" more difficult. So you should either:
- just do regular merges of projectX into main (and hopefully, if projectX is an aggregate of features, it won't have _that_ many merges, so the history should still be quite readable)
- wait until all such "featureY" branches are merged into projectX, announce a freeze of branching from projectX, and then rebase it forward.

Quoted 1 line

-Peff

Ah,I see... The light is beginning to come on somewhat here, though it's dimmed somewhat by the remote/local branch confusion I mention above, which tends to imply that rebase is only really useful in local branches since it is always the outer-most branch (assuming that my understanding on that is correct, which it may well not be).

I actually quite like the idea of the freezing before re-basing in the sub-branches. However, to answer the question of which merge strategy would work best for us I think I need to actually set this up and have a play with it to see how it all pans out using the various options available.

Thanks for your help!

Signature · 1 line
Russ
#9 lore raw
Quoted 5 lines · Russ Brown

I keep reading things similar to this and bit by bit I'm starting to get it. :) I suppose this is one case in which it's definitely a disadvantage to have a good understanding of svn before coming to git...

<yoda>You must unlearn what you have learned</yoda>

I prefer to think of it like a war movie, where I keep having nightmarish flashbacks to CVS.

But yes, from the outside git _looks_ a lot like other SCMs you may have used, and so it's tempting to keep their mental models. But that can easily end up being confusing, as you have found. Personally, I think it pays to learn a little about what's going on under the hood, and then all of the commands Just Make Sense.

There are several explanations floating around; this is a pretty concise one:

http://eagain.net/articles/git-for-computer-scientists/

Quoted 3 lines

If you delete a branch that has commits on it that aren't referenced by any other branches, will those commits be removed by something like git pack or git gc?

The 'git-prune' command will do this, though it is not run as part of git-gc unless you specify --prune.

Quoted 6 lines

I suppose what has me the most confused is how a developer works with a remote branch: I've come to understand that a developer should never check out and work on a remote branch, and always create a local one and work on that. If he does that using the above hierarchy, there then becomes main->projectX->featureY->jeff_local_branch_of_featureY. Or is is possible for a developer to work directory on a remote branch?

As Junio noted, you can't, because they're remote. What you have in your local repository is a remote _tracking branch_, which is a local ref that tracks your idea of what the remote's branches are. And git will feel free to overwrite the contents of that tracking branch whenever you do a fetch or pull. So if you make commits on it, they are subject to being overwritten (and we note this property of the refs by putting them in the refs/remotes hierarchy, rather than refs/heads).

So in the case of our developer Jeff, his local repository will have a "projectX/featureY" branch that he works on. And he will also have a remote tracking branch "origin/projectX/featureY" which indicates where his local repo thinks the remote repo points. And the remote repo will have a "projectX/featureY" branch, of course.

Quoted 5 lines

Ah,I see... The light is beginning to come on somewhat here, though it's dimmed somewhat by the remote/local branch confusion I mention above, which tends to imply that rebase is only really useful in local branches since it is always the outer-most branch (assuming that my understanding on that is correct, which it may well not be).

Yes, although the important distinction is not so much "this is a local branch" but rather "this is a _published_ branch" which implies that other people are looking at (and possibly basing work on) it.

Quoted 4 lines

I actually quite like the idea of the freezing before re-basing in the sub-branches. However, to answer the question of which merge strategy would work best for us I think I need to actually set this up and have a play with it to see how it all pans out using the various options available.

Yes, it is easy to get into academic discussions of setups, but in practice you need to find a workflow that is smooth for your team.

On one web-based project I work on, we have a setup like this (which is very centralized):

- a central repo resides on a development server with two branches, "master" and "production"
- each developer clones the repo with a working tree
- some developers develop directly on 'master' if they have small changes, or only work on one thing at a time; other developers use topic branches to work on simultaneous changes
- any developer can push to master; it is expected that your code is in a reasonable state, since it will now be consumed by other developers. Anything that has made it into master is considered published and should not be rebased. It is up to developers whether they want to rebase their work before publishing or to simply merge.
- some developers communicate directly with each other: "hey, check out branch 'foo' in my repo" or "what do you think of this patch?"
- The live site has a repo cloned from the central repo, pointing at "production".
- there is a test site with a repo cloned from the master. Occasionally the master branch is pulled and tested. If it passes, it is pushed to the "production" branch. In addition, small immediate fixes can go onto "production", tested, and then pushed to the central repo's "production"

So this is not necessarily using the distributed nature of that much, but it allows those developers who aren't very comfortable with SCMs to stick to a "pull, hack, push" workflow. And those who want to can do more interesting things if it helps them.

-Peff

#10 lore raw
Quoted 7 lines · Jeff King

Personally, I think it pays to learn a little about what's going on under the hood, and then all of the commands Just Make Sense.

There are several explanations floating around; this is a pretty concise one:

http://eagain.net/articles/git-for-computer-scientists/

I agree. Once you understand that history in git is just a DAG of commits, and that "branches" are just named pointers into this DAG to help the user remember where to attach new commits, everything starts to Just Make Sense.

(And FWIW, that article was quite good as I recall.)

Signature · 2 lines
Karl Hasselström, [email protected]
      www.treskal.com/kalle
author
#11 lore raw
In reply to Jeff King
Quoted 17 lines · Jeff King
Quoted 5 lines · Russ Brown

I keep reading things similar to this and bit by bit I'm starting to get it. :) I suppose this is one case in which it's definitely a disadvantage to have a good understanding of svn before coming to git...

<yoda>You must unlearn what you have learned</yoda>

I prefer to think of it like a war movie, where I keep having nightmarish flashbacks to CVS.

But yes, from the outside git _looks_ a lot like other SCMs you may have used, and so it's tempting to keep their mental models. But that can easily end up being confusing, as you have found. Personally, I think it pays to learn a little about what's going on under the hood, and then all of the commands Just Make Sense.

Quoted 5 lines

There are several explanations floating around; this is a pretty concise one:

http://eagain.net/articles/git-for-computer-scientists/

Yes, this is very helpful indeed: thank you for that. /me bookmarks. I hadn't actually realised that rebase creates new commits and replaces your old ones: I'd thought they just got 'moved' (dunno how I thought it worked though!)

Quoted 7 lines
Quoted 3 lines

If you delete a branch that has commits on it that aren't referenced by any other branches, will those commits be removed by something like git pack or git gc?

The 'git-prune' command will do this, though it is not run as part of git-gc unless you specify --prune.

Another thing makes sense now (helped by the link above also).

Quoted 15 lines
Quoted 6 lines

I suppose what has me the most confused is how a developer works with a remote branch: I've come to understand that a developer should never check out and work on a remote branch, and always create a local one and work on that. If he does that using the above hierarchy, there then becomes main->projectX->featureY->jeff_local_branch_of_featureY. Or is is possible for a developer to work directory on a remote branch?

As Junio noted, you can't, because they're remote. What you have in your local repository is a remote _tracking branch_, which is a local ref that tracks your idea of what the remote's branches are. And git will feel free to overwrite the contents of that tracking branch whenever you do a fetch or pull. So if you make commits on it, they are subject to being overwritten (and we note this property of the refs by putting them in the refs/remotes hierarchy, rather than refs/heads).

Quoted 6 lines

So in the case of our developer Jeff, his local repository will have a "projectX/featureY" branch that he works on. And he will also have a remote tracking branch "origin/projectX/featureY" which indicates where his local repo thinks the remote repo points. And the remote repo will have a "projectX/featureY" branch, of course.

I'm just wondering at this point why git lets you checkout remote tracking branches if it's something you really shouldn't do. Unless it's something you want to be able to do in edge cases to fix screwups maybe?

Quoted 42 lines
Quoted 5 lines

Ah,I see... The light is beginning to come on somewhat here, though it's dimmed somewhat by the remote/local branch confusion I mention above, which tends to imply that rebase is only really useful in local branches since it is always the outer-most branch (assuming that my understanding on that is correct, which it may well not be).

Yes, although the important distinction is not so much "this is a local branch" but rather "this is a _published_ branch" which implies that other people are looking at (and possibly basing work on) it.

Quoted 4 lines

I actually quite like the idea of the freezing before re-basing in the sub-branches. However, to answer the question of which merge strategy would work best for us I think I need to actually set this up and have a play with it to see how it all pans out using the various options available.

Yes, it is easy to get into academic discussions of setups, but in practice you need to find a workflow that is smooth for your team.

On one web-based project I work on, we have a setup like this (which is very centralized):

- a central repo resides on a development server with two branches, "master" and "production"
- each developer clones the repo with a working tree
- some developers develop directly on 'master' if they have small changes, or only work on one thing at a time; other developers use topic branches to work on simultaneous changes
- any developer can push to master; it is expected that your code is in a reasonable state, since it will now be consumed by other developers. Anything that has made it into master is considered published and should not be rebased. It is up to developers whether they want to rebase their work before publishing or to simply merge.
- some developers communicate directly with each other: "hey, check out branch 'foo' in my repo" or "what do you think of this patch?"
- The live site has a repo cloned from the central repo, pointing at "production".
- there is a test site with a repo cloned from the master. Occasionally the master branch is pulled and tested. If it passes, it is pushed to the "production" branch. In addition, small immediate fixes can go onto "production", tested, and then pushed to the central repo's "production"

Quoted 5 lines

So this is not necessarily using the distributed nature of that much, but it allows those developers who aren't very comfortable with SCMs to stick to a "pull, hack, push" workflow. And those who want to can do more interesting things if it helps them.

Thanks for this, it's very useful to read examples of workflows in actual use. In fact, I was thinking the other day that it would be good to have a site that acts as a directory of many different workflows, including descriptions of how they work, how you actually go about setting it up and using it day to day (i.e. lists of commands for each role/task) and the pros/cons that it provides. I reckon that would help newbies out quite a bit (if only for the examples). I've seen a few individual examples of workflow but nothing like a comprehensive set of them.

Quoted 1 line

-Peff

Thanks!

Signature · 1 line
Russ
#12 lore raw
Quoted 4 lines · Russ Brown

Yes, this is very helpful indeed: thank you for that. /me bookmarks. I hadn't actually realised that rebase creates new commits and replaces your old ones: I'd thought they just got 'moved' (dunno how I thought it worked though!)

It's a necessity, since the commits are named by hash, and the hash encompasses _all_ of the history. So the same change at a different location in history will be a different commit.

And that is why rebases can make merging harder. Git can very quickly compare two commits by hash and say "these are the same commit", or look at them and say "one side has these changes, the other side has these other changes, and here is where they meet." Rebasing ruins that, since the same changes occur in two places with different names.

Quoted 3 lines

I'm just wondering at this point why git lets you checkout remote tracking branches if it's something you really shouldn't do. Unless it's something you want to be able to do in edge cases to fix screwups maybe?

Junio explained in much more detail, but I use it largely for read-only access ("oh, let me speed-test my branch against the upstream 'master'"; git-checkout master; test test test; git-checkout mybranch).

Quoted 9 lines

Thanks for this, it's very useful to read examples of workflows in actual use. In fact, I was thinking the other day that it would be good to have a site that acts as a directory of many different workflows, including descriptions of how they work, how you actually go about setting it up and using it day to day (i.e. lists of commands for each role/task) and the pros/cons that it provides. I reckon that would help newbies out quite a bit (if only for the examples). I've seen a few individual examples of workflow but nothing like a comprehensive set of them.

I agree. That sort of information is sprinkled throughout the mailing list, but it might be nice on a wiki. I have thought of it as a sort of "git cookbook" where you say "here is a recipe for accomplishing X". The user manual comes close to this for smaller tasks.

-Peff

#13 lore raw
In reply to Russ Brown
Quoted 3 lines · Russ Brown

I'm just wondering at this point why git lets you checkout remote tracking branches if it's something you really shouldn't do. Unless it's something you want to be able to do in edge cases to fix screwups maybe?

You actually never checkout "remote tracking branches".

You can be in two states, either you are on a branch (meaning, if you create a commit, the new commit will have the current tip commit of that branch as its first parent and will become the new tip commit of that branch), or you aren't on _any_ branch.

The latter state is often affectionately called "detached HEAD" state.

This is primarily useful for sightseeing. Sometimes people would want to check out a commit that is not a tip of any branch. The most typical one is "I want to have a checkout of version 2.6.17", and people call that (loosely) as "checking out a tag". In the same way, you can "checkout a remote tracking branch" (but if you want to be anal in terminology, you never "check out a tag" nor "check out a remote branch"---you are detaching your HEAD at the named commit (which could be the one pointed at by the tag, or the one at the tip of your remote tracking branch).

Detached HEAD state allows you to make further commits and merges. Because git allows you to create a new branch from the current commit (i.e. whatever HEAD points at, be it on any branch or detached) without losing local changes in the index nor the work tree, this is often handy for doing quick fixups and experiments --- you first start on detached HEAD and if it turns out not to be so "quick" fixup, at that point you can create a real branch so that you can continue working on it without losing track.

Quoted 9 lines

Thanks for this, it's very useful to read examples of workflows in actual use. In fact, I was thinking the other day that it would be good to have a site that acts as a directory of many different workflows, including descriptions of how they work, how you actually go about setting it up and using it day to day (i.e. lists of commands for each role/task) and the pros/cons that it provides. I reckon that would help newbies out quite a bit (if only for the examples). I've seen a few individual examples of workflow but nothing like a comprehensive set of them.

"Everyday" might be a good starting point for catalogs of workflows for people playing various roles.

#14 lore raw
In reply to Russ Brown
Quoted 5 lines · Russ Brown

I keep reading things similar to this and bit by bit I'm starting to get it. :) I suppose this is one case in which it's definitely a disadvantage to have a good understanding of svn before coming to git...

<yoda>You must unlearn what you have learned</yoda>

You do not have to unlearn; if Jeff truly unlearned he wouldn't have spotted you were trapped in SVN mentality. You just need to learn there could be other ways ;-).

Quoted 3 lines

If you delete a branch that has commits on it that aren't referenced by any other branches, will those commits be removed by something like git pack or git gc?

Yes, eventually.

Quoted 6 lines

I suppose what has me the most confused is how a developer works with a remote branch: I've come to understand that a developer should never check out and work on a remote branch, and always create a local one and work on that. If he does that using the above hierarchy, there then becomes main->projectX->featureY->jeff_local_branch_of_featureY. Or is is possible for a developer to work directory on a remote branch?

The statement in the last sentence does not make any sense. Remote is called remote because it is remote and supposed to be out of reach ;-)

More seriously, remotes are used as reference points so if you "work directly on them", you cannot use them as reference points any more; you defeat the sole purpose of existence of remotes.

You can work _without_ using remote tracking branches, but that is mostly for merge based workflow. It appears that you are leaning towards rebase-heavy workflow, so I do not think it is applicable to your project.

author
#15 lore raw
Quoted 6 lines · Junio C Hamano
Quoted 4 lines · Russ Brown

I keep reading things similar to this and bit by bit I'm starting to get it. :) I suppose this is one case in which it's definitely a disadvantage to have a good understanding of svn before coming to git...

Quoted 6 lines
Quoted 1 line

<yoda>You must unlearn what you have learned</yoda>

You do not have to unlearn; if Jeff truly unlearned he wouldn't have spotted you were trapped in SVN mentality. You just need to learn there could be other ways ;-).

I suppose what I really mean is you need to stop assuming what you've already learned. :)

Quoted 13 lines
Quoted 3 lines

If you delete a branch that has commits on it that aren't referenced by any other branches, will those commits be removed by something like git pack or git gc?

Yes, eventually.

Quoted 6 lines

I suppose what has me the most confused is how a developer works with a remote branch: I've come to understand that a developer should never check out and work on a remote branch, and always create a local one and work on that. If he does that using the above hierarchy, there then becomes main->projectX->featureY->jeff_local_branch_of_featureY. Or is is possible for a developer to work directory on a remote branch?

Quoted 4 lines

The statement in the last sentence does not make any sense. Remote is called remote because it is remote and supposed to be out of reach ;-)

Ah. I think I was a little confused by the fact that git does let you checkout remote branches, through I see that it does warn you about it when you do it.

Quoted 8 lines

More seriously, remotes are used as reference points so if you "work directly on them", you cannot use them as reference points any more; you defeat the sole purpose of existence of remotes.

You can work _without_ using remote tracking branches, but that is mostly for merge based workflow. It appears that you are leaning towards rebase-heavy workflow, so I do not think it is applicable to your project.

Right, I think we're going to be aiming for that, though as I say I'm going to be experimenting a bit to see how things work when using both approaches.

Thanks.

Signature · 1 line
Russ
#16 lore raw
In reply to Russ Brown
Quoted 34 lines · Russ Brown
Quoted 33 lines · Andreas Ericsson
Quoted 32 lines · Russ Brown

Hi,

I've been trying to think of a git workflow that we could use to replace our current svn/svk setup without simply using git in exactly the same way that we use svn/svk.

Basically, we develop, maintain and enhance a website. On the central repo is trunk which represents live, and any number of project branches. Developers don't use local branches: they check out the project branches they're working on and commit to those. Developers merge from trunk to project branch from time to time to keep them current, and when a project rolls out the branch is merged to trunk.

In addition to the obvious advantages that git would give us (such as properly tracking that code author as opposed to the person who did the merge), I'm wanting to gain the following benefits:

* The repository is very large (multiple gigabytes) and mirroring using
svk obviously takes a lot of time and space, so I'm keen to bring that down, most likely by the developer not needing to mirror branches he doesn't care about, or by being able to throw away branches he's done with.
* The repository is full of revisions that fail review (or break
things) and are fixed by subsequent revisions. We'd much rather be able to have the developer fix his revisions before they get committed 'upstream' (whatever that ends up meaning).

I asked earlier about the email-based model that git itself uses, and while it appears to work very well for a widely-dispersed open-source project, I think it will be too cumbersome and slow for a fast-paced internal development team who make a number of live releases every day.

Quoted 9 lines
Quoted 4 lines

We came to the same conclusion at our workplace. Email works great, but it's faster and better to just walk over to your colleague and ask what he/she thinks about something.

Very true, with the minor exception that at my place there are developers working at different sites, so the walk-over method will only work for specific subsets of the team. :)

So do what Peff suggested and use emails for code-review and git-pull/push for shoveling the code around. He's got some valid points. Where I work, there's mostly three or at most four developers at any one team, so paired programming or walking over works quite well. Otherwise we'd likely use the patch-emails-for-review thing.

Quoted 6 lines

Here's a question: is the creation and deletion of a branch also version controlled as it is in Subversion? In other words, if I create a branch, develop on it and delete it without merging it anywhere, will the revisions hang around and get pulled down by future people cloning the repository, or do they get thrown away?

They don't get thrown away unless you garbage-collect them, using git-gc, but they won't get propagated unless they can't be reached from a 'ref' or a ref's log. The refs that can be propagated between repos are tags and branches. There are others too, but they are less user-visible and oftentimes more for housekeeping than anything else.

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines

4. Developers create a local branch of the project they
are working on and commit to that.
5. Once they think they're done, they publish their branch to the
development repo and request for comments.

Quoted 8 lines

Using topic-branches is a much better strategy, usually, since that allows each feature to be evaluated and improved on on its own, rather than having to merge *all* of a particular developers changes just to get desirable feature X. Note that cherry-pick provides ways of doing that anyways, but in a much less elegant way, and your integrator/ release engineer will likely tear his hair out on a daily basis without topic branches.

Quoted 6 lines

I've seen the term 'topic-branch' used here quite a bit but it's unfamiliar to me. It is basically synonymous with what we call a 'project branch'? i.e. Management decide that feature X is required so we create a branch to develop it on which all developers on the project commit to.

Pretty much, yes. 'feature branch' and 'topic branch' are interchangeable.

Quoted 10 lines

Note that in step 4 above I mean the developer takes a local branch of the topic branch. For example, we start projectX/main, and create branch projectX on the shared repo. Developer 'jeff' works on the project and so creates local branch projectX/jeff and begins work. In step 5 they push this local branch to the shared repo so everyone can see it (alternative to the 'walk-over' method or emailing). Note that all changes jeff making in projectX/jeff are specific to the project branch, so he can rebase against other changes that get committed to that project branch.

Quoted 8 lines

If colleagues don't like his changes he can create projectXjeff2 and try again.

Jeff can also have other local branches to keep separate changes he is making on the other projects he is involved in.

That's how I'd thought of it happening...

Ye, that sounds about right.

Quoted 17 lines
Quoted 7 lines
Quoted 7 lines

6. If all is not well, the developer creates a new local branch and
moves good revisions from his previous one to the new one, modifying things as he goes, and republishes his new branch.
7. If all is well, their works gets merged or rebased onto the main
project branch, and once that's ready it gets pushed to the master and rolled to live. The developer's individual branches get deleted from the dev repo since they're no longer required.

Quoted 9 lines

Topic branches would work the same, basically, except they can be pushed up for review a lot faster.

If all the pushing gets cumbersome, it also makes it easy to send the patches
out as emails for discussion. It's usually easier to let git handle the actual code transmissions, but discussing patches in emails works quite well if it's intended for a wider audience.

Quoted 4 lines

Yes, I am going to experiment with this a little too to see just how much work it would involve for the developers (if it's too much they won't do it) :)

It's a slightly steeper starting cost, but the maintenance costs go way down, so the developers can spend more of their time doing fun and new stuff rather than figuring out how to get their already-ancient fixes merged to the master-branch.

Quoted 14 lines
Quoted 4 lines
Quoted 4 lines

8. From time to time the master branch gets merged to the project
branches. Developer's local branches can be rebased against the project branch as they please.

Quoted 9 lines

criss-cross merging can turn kinda nasty though, as you may have a hard time
finding *the* common point when you run into that rogue merge with conflict markers everywhere (it happens for everyone sooner or later).

I'd suggest you rebase the developer/topic branches onto master with regular
intervals instead.

Quoted 8 lines

Having been using git-svn for a while I really like the clean history result that rebase gives, however my understanding was that you should never rebase any published branch as it could screw up clones of that branch. In fact, this is what has me the most confused: how to rebase a project branch that is on a shared repository against master when everyone will have it cloned? Or is this something that I clearly don't understand properly?

Well, afaiu you want every developer to have his own topic-branch, in which case they can simply rebase those topic-branches onto master, which will make sure they always know they're conflict-free with the latest bugfixes and whatnot.

They don't really have to push their changes upstream until they're ready for review.

Quoted 2 lines

(Thanks for your answers BTW Andreas)

np. Happy to help :)

Signature · 3 lines
Andreas Ericsson                   [email protected]
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231