Stgit - patch history / add extra parents

Hi,

I like stgit very much, but I feel there is still something missing:
stgit is very handy when you use it for patches which should be pushed to mainline rather quickly. But for pacthes which won't be pushed immediately to mainline, it would be usefull to have a history of the patches itself.

The patch below, together with the following script could be used to make snapshots of the patch stack (I call it freeze, as I thought snapshot was already going to be used for something else):

#!/bin/bash

# stg-freeze
set -ex
top=$(stg top)

# Freeze the current stack
branchname=$(basename $(readlink .git/HEAD))
while read patch
do
cp .git/patches/$branchname/$patch/top .git/patches/$branchname/$patch/parent
done < .git/patches/$branchname/applied

# Refresh the patches
stg pop -a
stg push -t $top

# end stg-freeze

Note: this is a proof of concept, when/if it would be incorporated, it should be implemented in stgit itself, and a bit more efficient, especially the refreshing of the patches ;-)

This is how it works: when not doing anything, stgit works as normal. Once stg-freeze is called, it creates .git/patches/$branchname/$patch/parent which contains the SHA1 id of the "frozen" patch. By refreshing the stack, all patches now include the corresponding frozen patch as a parent.

The following script is a test I use. Add stg-freeze to the path and source/run it in an empty directory, view with gitk afterwards:

###
echo "Initial commit" | cg-init
stg init

stg new a -mpatch-1
echo a > a
stg add a
stg refresh -mpatch-1
stg new b -mpatch-2
echo b > b
stg add b
stg refresh -mpatch-2
stg-freeze
stg pop
echo a2 >> a
stg refresh -mpatch-1-update
stg push
stg-freeze
echo b2 >> b
stg refresh -mpatch-2-update

stg pop -a
echo c > c
cg-add c
cg-commit -m"Mainline advance"
stg push -a

stg-freeze

stg pop -a
echo d > d
cg-add d
cg-commit -m"Mainline advance 2"
stg push -a

stg pop -a
echo e > e
cg-add e
cg-commit -m"Mainline advance 3"
stg push -a

###

Comments/remarks are very welcome.

---
 stgit/stack.py |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)

@@ -140,6 +140,13 @@ class Patch:
         elif os.path.isfile(fname):
             os.remove(fname)
 
+    def get_parents(self):
+        parents=[]
+        if (self.__get_field('parent') != None):
+                parents = parents + [self.__get_field('parent')]
+        parents = parents + [ self.get_bottom() ]
+        return parents
+
     def get_bottom(self):
         return self.__get_field('bottom')
 

@@ -362,7 +369,7 @@ class Series:
         if not committer_email:
             committer_email = patch.get_commemail()
 
-        commit_id = git.commit(message = descr, parents = [patch.get_bottom()],
+        commit_id = git.commit(message = descr, parents = patch.get_parents(),
                                allowempty = True,
                                author_name = author_name,
                                author_email = author_email,

The patch history feature was available in StGIT 0.1/0.2 releases where you should have run a 'stg commit' before 'stg refresh'. The commit was handling all the history changes, with separate commit messages and refresh was updating the main commit with 2 parents. I removed it in 0.3 after some people (I think it was Paul Jackson and Daniel Barkalow) convinced me that this would make it yet another SCM interface on top of GIT, which wasn't really my intention.

The main problem with having multiple parents for a commit object corresponding to a patch is the upstream merging via 'git pull'. In general you don't want a gatekeeper to pull the history of your patch but the patch only.

'snapshot' is not yet used for anything and I'm not sure how it is best to be implemented. I thought about simply saving the current HEAD into some .git/refs/heads/<file>, without preserving any history for the patch. A gitk on this file would show the patches as they were on the time of the snapshot creation. A new snapshot would remove this.

It might be best for a per-patch history to have a separate file in <branch>/<patch>/, maybe called freeze, which keeps this history information. The top one should remain unchanged. Its hash could be accessed with the 'stg id /freeze' command (implemented yesterday). This file would only be updated via the 'freeze' command and its parent would be the previous freeze value.

Would this be close to what you need?

Catalin

hmm, I must have misted those threads, I'll try to find and read them.

I agree that such history should not be imported into the mainline, but such history would still be very usefull when these patches won't be pushed to mailine immediately. Also, when pushing to mainline, this history can easily be removed by removing the branch/patch/parent files and refreshing (this should off course be automated)

hmm, not exactly, for example, when reordering the patches (including the top one), I would like to see this in gitk.
Or when a patch has been dropped (amongst a lot of patches), it should be easily spotted.

But even if the "stg-freeze" would not be incorporated into stgit, would it still be possible to include some sort of extra parents directory. So that the freeze can be implemented on top of stgit?

TIA

Best regards,
Jan

I tried your patch but the gitk image confused me. I will look again at it tomorrow (it's quite late in the UK now).

One immediate thing I noticed was that the commits directly accessible via .git/HEAD are shown as empty by gitk and you would need to follow the parents to see what they contain. For every freeze, the patches expand to the right in gitk and the graph could get very complex after several freeze commands.

Catalin

I've quickly reread the threads about stg commit. Am I right to assume that _all_ history was being recorded? Because this is not what I want. The person controlling the archive should specify when to record the history.

So for example, you only tag (freeze) the history when exporting the patches. When an error is being reported on that version, it's easy to view it and also view the progress that was already been made on those patches.

Best regards,
Jan

True, I was talking about the full history, but I don't find the idea much different. There was another mail on this issue -
http://article.gmane.org/gmane.comp.version-control.git/5191

I agree that it is a useful feature to be able to individually tag the patches. The problem is how to do this best. Your approach looks to me like it's not following the GIT DAG structure recommendation. Maybe the GIT designers could further comment on this but a commit object with multiple parents should be a result of a merge operation. A commit with a single parent should represent a transition of the tree from one state to another. With the freeze command you proposed, a commit with multiple parents is no longer a result of a merge operation, but just a convenience for tracking the patch history with gitk.

I tried your patch and test script but, if you try to reorder the patches, the DAG structure become quite hard to understand and that's only for a small set of frozen states.

I think the best approach would be to keep a list of tags for each patch. They won't be browsable with gitk but the DAG structure would be clean. We might get another tool like gquilt to be able to show all the saved states of a patch.

Sorry for not including your patch but, while I agree with the general idea of frozen patch states, I disagree with its implementation.

Catalin

My interpretation of parents is broader than only merges, and reading the README file, I believe it also the intension to do so (snippet from README file):

A "commit" object ties such directory hierarchies together into a DAG of revisions - each "commit" is associated with exactly one tree (the directory hierarchy at the time of the commit). In addition, a "commit" refers to one or more "parent" commit objects that describe the history of how we arrived at that directory hierarchy.

Another advantage of this approach is when pushing the top of your repository to others, less rebasing would be noticed:
When pushing the top of your stack out to others, the new one won't be a (possibly indirect) child of the previous push. This makes updating such a head more dificult. With this approach, the new head will be a (possibly indirect) child of the previous one (except when the top patch has been removed, but this corner case could also be solved).

Because of this less rebasing, this aproach also facilitates co-development on stgit patch stacks: by using a base:top id pair, the stgit patch stack could be initialized, including its exported history. By following the parent/child relations, 2 patch stacks could be compared and merged.

I acknowledge this problem, but the solution for this would be to give gitk another way of viewing such history: only the latest description of a patch should be visible and the history should become more horizontal (because the older descriptions wouldn't be shown, the history of those patches could be viewed on the line with the latest description)

But changing gitk would only be usefull when this approach would be taken.

I understand your objections when limiting parents only to merges, but I see them broader. So as you said: we indeed need a git designer to clear this up for us.

Best regards,
Jan

I understand I am counted as one of the "git designers"; I have been aware of this thread but I have to admit that I am way behind and have not caught up with the issues. I'll try to comment on them when I feel I know enough about the issues involved, but not right now. Sorry.

Please do not start explain them to me yet, I am behind due to lack of time and you would end up wasting your time.

One factor not mentioned there is that, as things move upstream, we often want to discard a lot of history; if someone commits constantly to deal with editor malfunction or something, we don't really want to take all of this junk into the project history when it is cleaned up and accepted.

So the point is that there are things which are, in fact, parents, but we don't want to list them, because it's not desired information.

Probably the right thing is to have two views of the stack: the internal view, showing what actually happened, and the external view, showing what would have happened if the developers had done everything right the first time. When you make changes to the series, this adds to the internal view and entirely replaces the external view.

I think that users will also want to discard the commits from the stack before rebasing in favor of the commits after, because (a) rebasing isn't all that interesting, especially if there's minimal merging, and (b) otherwise you'd get a ton of boring commits that obscure the interesting ones.

I think that the best rule would be that, when you modify a patch, the previous version is the new version's parent, and when you rebase a series, you include as a parent any parent of the input that isn't also in the input (but never include the input itself as a parent of the output; the point of rebasing is to pretend that it was the newer mainline that you modified). This should mean that the internal history of a patch consists of the present version, based on each version that was replaced due to changing the patch rather than rebasing it.

Of course, there's an interesting situation with the commits earlier in a series from a patch that was changed not being ancestors of the newer versions of those patches (because they weren't interesting in the development of those patches) but accessible as the commits that an interesting patch was based on.

A possible solution is just to consider the revision of any patch a significant event in the history of the whole stack, causing all of the patches to get a new retained version.

-Daniel
*This .sig left intentionally blank*

That's true but Jan's proposal is to choose which commits to preserve in the history via a 'freeze' command. That's a bit difficult to implement in a clean way since the patches are floating on top of the stack base and they change every time the base changes. It is possible that a previous frozen state might no longer apply on top of a new base.

What's the definition of a parent in GIT terms? What are the restriction for a commit object to be a parent? Can a parent be an arbitrarily chosen commit?

That's what I've been thinking. StGIT currently only implements the external view.

An StGIT patch is a represented by a top and bottom commit objects. The bottom one is the same as the parent of the top commit. The patch is the diff between the top's tree id and the bottom's tree id.

Jan's proposal is to allow a freeze command to save the current top hash and later be used as a second parent for the newly generated top. The problem I see with this approach is that (even for the internal view you described) the newly generated top will have two parents, new-bottom and old-top, but only the diff between new-top and new-bottom is meaningful. The diff between new-top and old-top (as a parent-child relation) wouldn't contain anything relevant to the patch but all the new changes to the base of the stack.

Is the above an acceptable usage of the GIT DAG structure?

StGIT does this currently. The old commit is no longer available and can be pruned (I still need to save the commits corresponding to the unpushed patches since prune would remove those as well).

Since you proposed this, my above usage of the DAG structure would be fine then.

That's another idea to think about.

Catalin


P.S. I'll be away until next week and not able to follow-up the
discussions.

Yes it can. GIT does not care if the commit ancestry does not make sense in contents terms (i.e. you can record one tree object in a commit object, and record another, completely unrelated tree object in a commit object that has the first commit object as its parent). The "git-diff-tree" output from comparing those two commits may not make _any_ sense at all to the human, though, but that is not a problem for GIT to do its work.

Surely, as long as it stays internal as StGIT patch stacks. Even when it is exported (i.e. you manage to have other non StGIT managed tree to pull and merge with such a commit), the development history _might_ look funny and complicated, but it should not be a problem for GIT to handle. After all, you are doing complicated thing in the StGIT patch stacks when you pop, push and freeze number of times, so your history is complicated. If you want to record that history at least for StGIT internal bookkeeping purposes, GIT should not prevent you from doing it.

Having said that, I would 100% agree with you that having a cleansed external view would be a good idea. It makes merging with StGIT managed repositories much more acceptable to ordinary GIT managed history.

One issue is later merges.

If you have incorrect parent information, trying to do a merge may prove impossible and/or ignore changes from the "already merged" stream. By marking another head as a parent, you basically say "I have merged _everything_ leading up to that other head", and if you have only actually done a partial merge (and gotten just a part of the changes, ignoring the others), you'll have to notice that yourself, and forward-port the rest by hand.

For stgit, this probably doesn't matter.

Linus

Something is legitimate as a parent if someone took that commit and did something to it to get the new commit. The operation which caused the change is not specified. But you only want to include it if anyone cares about the parent.

(For example, I often start with a chunk of work that does multiple things and is committed; I take mainline and generate a series of commits from there. It would be legitimate to list my development commit as a parent of each of these, since I did actually take it and strip out the unrelated changes. This would be a bit confusing in the log, but would make merges between something based on the "messy" version and something based on the "refined" version work well. On the other hand, I don't want to report the existance of the messy version, so I don't include it.)

Having a useful diff isn't really a requirement for a parent; the diff in the case of a merge is going to be the total of everything that happened elsewhere. The point is to be able to reach some commits between which there are interesting diffs.

This also depends on how exactly freeze is used; if you use it before commiting a modification to the patch without rebasing, you get:

old-top -> new-top

      ^    ^
       \  /
      bottom

bottom to old-top is the old patch
bottom to new-top is the new patch
old-top to new-top is the change to the patch

Then you want to keep new-top as a parent for rebasings until one of these is frozen. These links are not interesting to look at, but preserve the path to the old-top:new-top change, which is interesting.

Ignoring the links to the corresponding bottoms, the development therefore looks like:

local1 -> local2 -> merge -> local3 -> merge
^ ^ ^
mainline---->-->--------->------>-->----->

And this is how development is normally supposed to look. The trick is to only include a minimal number of merges.

-Daniel
*This .sig left intentionally blank*

This is indeed what I thought a parent should be used for. As an adition, I'll try to explain why I would sometimes want to care about some parents:

I want to track a mailine tree, but have quite a few changes, which shoudn't be commited to the mainline immediately (let's call it my development tree). This is why I would use stgit. But I would also want to colaborate with other developers on this development tree, so I sometimes want to make updates available of this development tree to the others. This is where current stgit falls short. To easily share this development tree, I want some history (not all, only the ones I choose) of this development tree included, so that the other developers can easily follow my development.

The parents which should be visible to the outside, will always be versions of my development tree, which I have previously pushed out. My way of working would become:
* make changes, all over the place, using stgit
* still make changes (none of these gets tracked, intermittent versions are lost)
* having a good day: changes looks good, I want to push this out:
* push my tree out
* stgit-free (which makes the pushed out commits, the new parents of my stgit patches)
* restart from top

[...]

my proposal does something like this, but a little more: not only does it keep track of the link between old-top and new-top, it also keeps track of the links between old-patch-in-between and new-patch-in-between. (This makes sense when the top is being removed or reordered)

I hope this kind of clarifies my intension.

Thank you for clarifying the meaning of parents.

Best regards,
Jan

I'm not sure how applicable to this situation stgit really is; I see stgit as optimized for the case of a patch set which is basically done, where you want to keep it applicable to the mainline as the mainline advances.

For your application, I'd just have a git branch full of various stuff, and then generate clean commits by branching mainline, diffing development against it, cutting the diff down to just what I want to push, and applying that. Then the clean patch goes into stgit.

I was thinking of this as being the top and bottom commits for a single tracked patch, not as a whole series. I think patches lower wouldn't be affected, and patches higher would see this as a rebase.

-Daniel
*This .sig left intentionally blank*

Daniel Barkalow wrote:

[...]

Maybe I forgot to mention this: I would also like to have my development tree split up in a patch stack. The separate patches makes tracking the mainline a lot easier (conflicts are a lot easier to solve)

But this would assume that once the patch goes into stgit, it won't change except when the parent gets updated. I think we will still change the patches quite a bit and simultanious by a couple of people.

ah, ok, I misunderstood that part

Best regards,
Jan

PS. sorry if my responses are sometimes a bit late, I'm trying to find more time to spend on this list ;-)

I just try to keep things in this state sufficiently briefly that it doesn't become a problem. I also split things up into a bunch of branches, rather than into a stack of patches, and only work on parallel development before I've actually got a candidate for a series.

The extension I had proposed to stgit should work for this; it would let you version control each patch just like other git projects. I just think it wouldn't work so well before the group has agreed on what patches there are.

-Daniel
*This .sig left intentionally blank*

StGIT has the 'refresh' command which allows a patch to be indefinitely modified (that's pretty much how I use StGIT). I use it for the case where I would like to keep the changes in separate logical entities (patches) but they are not independent enough to create separate branches.

Take for example a new platform port, you can have some of the existing code re-worked in a patch, some CPU support in the next patch, basic platform support in another patch, some device drivers specific to this platform in yet another patch. Subsequent patches are dependent on the previous ones. Branching from the mainline for each of these features might not make sense and keeping all of them in the same branch can make maintenance a bit more difficult.

Catalin

When the tree status looks good, you could tag it and the whole stack (the commits corresponding to the patches) would be available via this tag.

As I said, I agree with the idea of freezing a patch and even having multiple parents to a commit but this commit should not be visible via HEAD. We could have another id for a patch, 'frozen', which stores the last frozen state with the parents being the previous frozen states. Its SHA1 value could be accessed via 'stg id <patch>/frozen' (in the same way as /top and /bottom ids).

Catalin

Back from holiday. Thanks to all who replied to this thread.

This was my initial StGIT implementation (up to version 0.3), only that there was no freeze command. Since I want an StGIT tree to be clean to the outside world, I wouldn't keep multiple parents for the visible top of a patch.

As I understand from Junio's and Linus' e-mails (on the 23rd of August), there might be problems with merging the HEAD of an StGIT-managed tree if the above method is accessible via HEAD.

A merge occurs every time a patch is rebased. Anyway, having the bottoms in the graph (which is the main idea of StGIT) together with the old-top (or frozen state) parents make the graph pretty complicated.

Catalin

Catalin, Daniel,

thank you both for your input.
I'll try your recommendations (when time is available ;-) and see how it goes.

Best regards,
Jan

Right, you'd want a separate head which is what you ask people to merge; the rest is only visible to people who are working on preparing the patch. But you could keep both sets of stuff (sharing tree objects but not commits).

It should be possible to drop merges such that there's only one between any pair of local changes. That is, if you rebase at the end of the line above, it would get as parents local3 and the new bottom, not the last merge and the new bottom.

The mainline changes only come in through the bottoms, so higher levels should look the same, but with the lower levels in the place of mainline.

-Daniel
*This .sig left intentionally blank*