Skip to contentSkip to conversation

rethinking patch management with GIT / topgit

DISCUSSION Thomas Koch 3 participants 7 messages Jun 15, 2016 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hi,

I'd like to argue, that topgit (tg) fails to fullfill it's task and propose a different approach to the problem of patch management on top of git.

IMHO tg fails due to the following reasons:

- it's to complicate
- AFAIK nobody has solved the problem of managing different patchsets with tg
- it pollutes the patch branches with metadata (.topmsg, .topdeps)
- even after 1 1/2 years topgit isn't feature complete and development seems to stagnate
- there also aren't best practices documented
- it quickly fills the list of branches
- it's very easy to break something

Although the above is quite a harsh judgement, I'd like to note, that tg has had its merrit to promote one right idea: Patches should be managed in the form of branches by the means of the underlying VCS and not as simple patchfiles.

The new approach I'd like to propose is based on two core ideas:

1) merge commits to save history

git allows the free creation of merge commits with an arbitrary content tree. So we can create an octupus merge combining all patch-branches while the content of this merge can contain meta data about a patchset instead of the content of the merged commits.
Such a merge commit thus provides a pointer to all the history of all patches and can contain all metadata about the merged patch branches. Pushing only a branch or tag with this commit to a central repository thus pushes all the history of all contained patches.

2) collapse / expand branches

Managing a Debian package in stable, unstable and experimental can quickly doom you to manage at least three different patchsets with possibly three different roots. The list of branches grows in the douzens. Which branches belong to which patchset? Which branches are already pushed or pulled? It may be an advantage to see only some main branches and the branches of one patchset I'm currently working on.
The tool I propose would manage each patchset in one branch per patchset. This branch has two roles:
- keep the metadata of the patchset as files in the content tree
- keep pointers to the top of the patch-branches in the parent pointers of the commit
With the help of such a patchset-metadata-branch I can:
- delete the patch-branches of one patchset while the commits are kept save
- recreate the patch-branches of one patchset

For Debian packaging one last function is needed: export a patchset as quilt series.

Is my explanation understandable? Could this approach work or did I miss something? Who has time to implement it (GSOC?)?

Best regards,

Thomas Koch, http://www.koch.ro

#1 lore raw

Hi!

Quoted 6 lines · Thomas Koch

I'd like to argue, that topgit (tg) fails to fullfill it's task and propose a different approach to the problem of patch management on top of git.

IMHO tg fails due to the following reasons:

- it pollutes the patch branches with metadata (.topmsg, .topdeps)

I'd like to single this out - this is a very arbitrary decision and has its distinct pros and cons compared to alternative approaches, and was not taken lightly. It wouldn't be that difficult to go in a different way and it's not a fundamental limitation.

Quoted 1 line

- AFAIK nobody has solved the problem of managing different patchsets with tg

Why not just manage them in separate repositories? If you want to pick patches between patchsets, that should be easily posible through remotes.

Quoted 11 lines

- it's to complicate
- even after 1 1/2 years topgit isn't feature complete and development seems to stagnate
- there also aren't best practices documented
- it quickly fills the list of branches
- it's very easy to break something

Although the above is quite a harsh judgement, I'd like to note, that tg has had its merrit to promote one right idea: Patches should be managed in the form of branches by the means of the underlying VCS and not as simple patchfiles.

I agree that topgit really sucks in many areas. :) While it is usable
in practice if you know what you are doing, in some aspects it never got out of the proof-of-concept stage.

Quoted 4 lines

The new approach I'd like to propose is based on two core ideas:

1) merge commits to save history

Quoted 8 lines

git allows the free creation of merge commits with an arbitrary content tree. So we can create an octupus merge combining all patch-branches while the content of this merge can contain meta data about a patchset instead of the content of the merged commits.
Such a merge commit thus provides a pointer to all the history of all patches and can contain all metadata about the merged patch branches. Pushing only a branch or tag with this commit to a central repository thus pushes all the history of all contained patches.

But how is this different from TopGit's approach? TopGit doesn't use
octopus merges for various reasons (I mainly guess since I was too lazy to implement that), but that's more of an implementation detail than anything conceptual.

Quoted 16 lines

2) collapse / expand branches

Managing a Debian package in stable, unstable and experimental can quickly doom you to manage at least three different patchsets with possibly three different roots. The list of branches grows in the douzens. Which branches belong to which patchset? Which branches are already pushed or pulled? It may be an advantage to see only some main branches and the branches of one patchset I'm currently working on.
The tool I propose would manage each patchset in one branch per patchset. This branch has two roles:
- keep the metadata of the patchset as files in the content tree
- keep pointers to the top of the patch-branches in the parent pointers of the commit
With the help of such a patchset-metadata-branch I can:
- delete the patch-branches of one patchset while the commits are kept save
- recreate the patch-branches of one patchset

The disadvantage is that you will have to invent a lot of arbitrary
metadata and wide range of commands to manipulate and work with this metadata, all to accomplish something that _is actually one of the main Git functions to do_!

Wouldn't it be better to do the collapsing/expanding instead, e.g.
have a convention for patchset/stage branch tying up all patchset/* branches, and an alias that lists only */stage branches and another that lists only patchset/* minus patchset/stage branches.

Quoted 5 lines

For Debian packaging one last function is needed: export a patchset as quilt series.

Is my explanation understandable? Could this approach work or did I miss something? Who has time to implement it (GSOC?)?

It's especially not clear at all whether you propose to start over or
just make two improvements to TopGit, and if the former, how would your approach differ from TopGit in all the other aspects. It's all a bit ambiguous.

Signature · 2 lines
				Petr "Pasky" Baudis
http://pasky.or.cz/ | "Ars longa, vita brevis." -- Hippocrates
#2 lore raw

Petr Baudis:

Quoted 6 lines
Quoted 1 line

- it pollutes the patch branches with metadata (.topmsg, .topdeps)

I'd like to single this out - this is a very arbitrary decision and has its distinct pros and cons compared to alternative approaches, and was not taken lightly. It wouldn't be that difficult to go in a different way and it's not a fundamental limitation.

It's also the most minor problem I see, rather a matter of taste.

Quoted 5 lines
Quoted 2 lines

- AFAIK nobody has solved the problem of managing different patchsets with tg

Why not just manage them in separate repositories? If you want to pick patches between patchsets, that should be easily posible through remotes.

You mean having multiple local working directories per project? That's not nice. If you also work on the project with an IDE, it'd most probably mean setting up the same project multiple times.

Quoted 3 lines

I agree that topgit really sucks in many areas. :) While it is usable
in practice if you know what you are doing, in some aspects it never got out of the proof-of-concept stage.

Thank you for not taking it personaly! :-)

Quoted 11 lines
Quoted 10 lines

1) merge commits to save history

git allows the free creation of merge commits with an arbitrary content tree. So we can create an octupus merge combining all patch-branches while the content of this merge can contain meta data about a patchset instead of the content of the merged commits.
Such a merge commit thus provides a pointer to all the history of all patches and can contain all metadata about the merged patch branches. Pushing only a branch or tag with this commit to a central repository thus pushes all the history of all contained patches.

Quoted 4 lines

But how is this different from TopGit's approach? TopGit doesn't use
octopus merges for various reasons (I mainly guess since I was too lazy to implement that), but that's more of an implementation detail than anything conceptual.

Maybe I didn't make clear that the commits on the patchset branch should not necessarily contain the merged content of all patches. The role of the merges is solely to keep pointers to the commits making up the patch branches. Now that you're asking there may however be a similarity to topgit in that you can have one topgit controlled branch that combines all topgit branches belonging to the same patchset.
However there still remain some differences:

- in tg you need to push all tg branches while in my approach you'd need to push only one branch per patchset
- As far as I understand tg, all commands are repository global while for the Debian packaging use case you'd need patchset local commands:
- tg export <patchset>
export <patchset> as a quilt series

- tg summary <patchset>
gives summary information for the branches of <patchset>

- tg delete <patchset> <branch>
removes <branch> from <patchset>. However old history of <branch> is still
recorded in the history of the <patchset>-branch

- tg add <patchset> <branch>
adds <branch> to <patchset>
(The question is open, where to record the merge resolution of all
patches)

- tg recreate <patchset> <newbase> <new patchset name>
Creates a new patchset with root <newbase> by creating new patch branches
for each patch branch in <patchset>
This command is useful if you need to keep the old patchset to maintain an
older version of your Debian package.

Quoted 17 lines
Quoted 16 lines

2) collapse / expand branches

Managing a Debian package in stable, unstable and experimental can quickly doom you to manage at least three different patchsets with possibly three different roots. The list of branches grows in the douzens. Which branches belong to which patchset? Which branches are already pushed or pulled? It may be an advantage to see only some main branches and the branches of one patchset I'm currently working on. The tool I propose would manage each patchset in one branch per patchset. This branch has two roles:
- keep the metadata of the patchset as files in the content tree
- keep pointers to the top of the patch-branches in the parent pointers of the commit
With the help of such a patchset-metadata-branch I can:
- delete the patch-branches of one patchset while the commits are kept save - recreate the patch-branches of one patchset

Quoted 4 lines

The disadvantage is that you will have to invent a lot of arbitrary
metadata and wide range of commands to manipulate and work with this metadata, all to accomplish something that _is actually one of the main Git functions to do_!

I don't see this. What do I miss? All metadata I'd need to manage is:
- one file with the name of each branch, it's last commit and the names of its dependencies (the root of the patchset, if empty)
- one message file for each patch
- the root of the patchset

The example commands given above would manipulate or read the patchset branch in the background much like pristine-tar does it with its metadata branch.

Quoted 4 lines

Wouldn't it be better to do the collapsing/expanding instead, e.g.
have a convention for patchset/stage branch tying up all patchset/* branches, and an alias that lists only */stage branches and another that lists only patchset/* minus patchset/stage branches.

So you propose not to delete/recreate the patch branches but to provide extra commands to list only the desired subset of branches? This would still mean that I'd see douzens of patch branches in gitweb and that I't need to push douzens of branches to my co-packagers. - That doesn't solve it for me. It may also be of interest, that Guido Günther, the author of git- buildpackage[1] wrote a command git-pq to "maintain debian/patches on a patch queue branch". - There seems to be a need to have few branches and that it seems to be practical to create and destroy patch branches in your workflow.

[1] https://honk.sigxcpu.org/piki/projects/git-buildpackage/

Quoted 7 lines
Quoted 2 lines

Is my explanation understandable? Could this approach work or did I miss something? Who has time to implement it (GSOC?)?

It's especially not clear at all whether you propose to start over or
just make two improvements to TopGit, and if the former, how would your approach differ from TopGit in all the other aspects. It's all a bit ambiguous.

I hope I managed to make it clearer this time. I believe my proposals are incompatible to topgit and thus would require a new project from scratch. However it may well be that some code from topgit could be reused.

Best regards,

Thomas Koch, http://www.koch.ro

#4 lore raw
In reply to a message not in the archive
Quoted 6 lines · Thomas Koch

Petr Baudis:
- tg recreate <patchset> <newbase> <new patchset name>
Creates a new patchset with root <newbase> by creating new patch branches
for each patch branch in <patchset>
This command is useful if you need to keep the old patchset to maintain an
older version of your Debian package.

This means wiping out history again; in TopGit, you would ideally checkpoint all the branches within the patchset, then just tg update your branches. It's another matter that the former is now difficult to do easily.

Quoted 8 lines

I don't see this. What do I miss? All metadata I'd need to manage is:
- one file with the name of each branch, it's last commit and the names of its dependencies (the root of the patchset, if empty)
- one message file for each patch
- the root of the patchset

The example commands given above would manipulate or read the patchset branch in the background much like pristine-tar does it with its metadata branch.

Hmm, to a degree I misunderstood your idea. You would still need quirky commands to update the references when you make a new commit, to go to a certain patch (at which point git will start acting a bit annoyed since it's not on a branch), etc. Other than that, I can offer only my gut feeling. ;-)

Quoted 8 lines
Quoted 4 lines

Wouldn't it be better to do the collapsing/expanding instead, e.g.
have a convention for patchset/stage branch tying up all patchset/* branches, and an alias that lists only */stage branches and another that lists only patchset/* minus patchset/stage branches.

So you propose not to delete/recreate the patch branches but to provide extra commands to list only the desired subset of branches? This would still mean that I'd see douzens of patch branches in gitweb and that I't need to push douzens of branches to my co-packagers. - That doesn't solve it for me.

There are already some patches in the wild to make gitweb topgit-aware; I don't see why is the latter a problem.

Quoted 2 lines

I hope I managed to make it clearer this time. I believe my proposals are incompatible to topgit and thus would require a new project from scratch.

Yes, I finally understood what do you mean, sorry for being a bit dense.

Signature · 2 lines
				Petr "Pasky" Baudis
http://pasky.or.cz/ | "Ars longa, vita brevis." -- Hippocrates
#5 lore raw

Hi Petr,

Petr Baudis:

Quoted 8 lines · Thomas Koch
Quoted 6 lines

Petr Baudis:
- tg recreate <patchset> <newbase> <new patchset name> Creates a new patchset with root <newbase> by creating new patch
branches for each patch branch in <patchset>
This command is useful if you need to keep the old patchset to
maintain an older version of your Debian package.

Quoted 4 lines

This means wiping out history again; in TopGit, you would ideally checkpoint all the branches within the patchset, then just tg update your branches. It's another matter that the former is now difficult to do easily.

It's not necessary to lose history. Creating a new patchset from a given patchset could be done in at least three different ways. Given that you've a patchset named patches-x.y.z/* and you'd like to create patches-x.y.z+1/* from this, you could

- checkout new patch branches from the top of the old patch branches and merge upstream into each of them

- recreate (like rebase) the full history of the patch branches on top of the new upstream

- collapse the branch history and create one commit per patch branch on top of the new upstream

In all three cases you're free to either keep or throw away the old patchset.

Best regards,

Thomas Koch, http://www.koch.ro

#6 lore raw
Quoted 1 line · Thomas Koch

In all three cases you're free to either keep or throw away the old patchset.

Yes, but to the same degree e.g. with StGIT I'm free to keep the head of the old patch series. That does not mean the operation *preserves* the history, only that the history is still around somewhere in the repository, however it won't be around in other incarnations of the repository and it will not be connected in any way to the current version of the patchset.

Yes, if you are lucky you can figure out the name of the previous version, but it's like starting development of each new kernel version by a clean import of the sources.

Signature · 2 lines
				Petr "Pasky" Baudis
http://pasky.or.cz/ | "Ars longa, vita brevis." -- Hippocrates
#7 lore raw

Petr Baudis:

Quoted 11 lines · Thomas Koch
Quoted 2 lines

In all three cases you're free to either keep or throw away the old patchset.

Yes, but to the same degree e.g. with StGIT I'm free to keep the head of the old patch series. That does not mean the operation *preserves* the history, only that the history is still around somewhere in the repository, however it won't be around in other incarnations of the repository and it will not be connected in any way to the current version of the patchset.

Quoted 3 lines

Yes, if you are lucky you can figure out the name of the previous version, but it's like starting development of each new kernel version by a clean import of the sources.

Hi Petr,

let me copy the list of methods from my last mail:

1) checkout new patch branches from the top of the old patch branches and merge upstream into each of them

2) recreate (like rebase) the full history of the patch branches on top of the new upstream

3) collapse the branch history and create one commit per patch branch on top of the new upstream

Quoted 1 line

From these methods, 3) loses all history, 2) loses some history but preserves

the individual history of one patch branch on a new base and 1) preserves all history. Let me give an example for method 1).
You've got a patchset identified by the prefix debian/. Now you want to package a new upstream but need to retain the old patchset in case of security updates in Debian stable. Debian stable has version 0.1, new upstream is 0.2. So you
- rename the old patchset from debian/ to debian-0.1/
- clone/copy/recreate (pick a name) a new patchset debian/ on top of upstream/0.2. This is done by merging upstream/0.2 into each debian/* branch.
- Once you don't maintain version 0.1 anymore, you can delete the debian-0.1/* branches

After these steps, the debian/ branch still contains pointers to all commits from the debian-0.1/* branches.
It's an additional question, how to deal with commits that are done in debian-0.1/* after the new upstream merge.

Best regards,

Thomas Koch, http://www.koch.ro