Skip to contentSkip to conversation

Metadata addendum to git commit

Ksummit-discuss Takashi Iwai 9 participants 25 messages May 13, 2014 – May 22, 2014

lore
Share & tools
lore
#0 lore raw

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit, or a bugzilla or ML link for the further discussion or debugging session. The regression markers would be especially helpful to avoid a pitfall through performing bisection.

IMO, it would be convenient if such information can be embedded in the published git tree, something like (infamous) git-notes. But, I'd like to hear any other options as well.

thanks,

Takashi

#1 lore raw
Quoted 11 lines · On Tue, 13 May 2014 15:25:57 +0200 Takashi Iwai <[email protected]> wrote:

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit, or a bugzilla or ML link for the further discussion or debugging session. The regression markers would be especially helpful to avoid a pitfall through performing bisection.

Quoted 4 lines

IMO, it would be convenient if such information can be embedded in the published git tree, something like (infamous) git-notes. But, I'd like to hear any other options as well.

+1

A particular practical issue is that when doing a git-bisect there might be a range of commits that only compile/run if some later commit is applied. If I'm bisecting in that range, I have to repeatedly apply that commit by hand.

If would really like it if a new commit could be marked as "a really important successor to the first parent" so that git could keep the extra metadata to easily find the parent->child link, and so that "git-bisect" could be told --always-include-important-children

This linkage would be enough to add anything else as "important successors".

Yes, I know I should send a patch rather than a suggestion. Sorry.

NeilBrown

#2 lore raw
Quoted 4 lines · NeilBrown

A particular practical issue is that when doing a git-bisect there might be a range of commits that only compile/run if some later commit is applied. If I'm bisecting in that range, I have to repeatedly apply that commit by hand.

You can just have it stored in a separate branch and perform the bisection using a script that will always first merge the branch, and reset after good/bad verification is done.

If the fix is already applied (because you are testing a state of the tree which has the commit already), git handles the merge of the already-applied patch with grace even if they don't have the same SHA-1.

Signature · 2 lines
Jiri Kosina
SUSE Labs
#3 lore raw
Quoted 7 lines · On Wed, 14 May 2014 01:29:29 +0200 (CEST) Jiri Kosina <[email protected]> wrote:
Quoted 4 lines · NeilBrown

A particular practical issue is that when doing a git-bisect there might be a range of commits that only compile/run if some later commit is applied. If I'm bisecting in that range, I have to repeatedly apply that commit by hand.

Quoted 8 lines

You can just have it stored in a separate branch and perform the bisection using a script that will always first merge the branch, and reset after good/bad verification is done.

If the fix is already applied (because you are testing a state of the tree which has the commit already), git handles the merge of the already-applied patch with grace even if they don't have the same SHA-1.

True. But every extra step I have to add to a script is an extra barrier to performing what is already a somewhat tedious task.

If fixes were always Strongly-Attached to the commit they fix, then I wouldn't even need to find these "fix-up" patches. One person commits the fix in the "right" way, and everyone benefits.

Thanks,
NeilBrown

#4 lore raw
In reply to Takashi Iwai
Quoted 11 lines · Takashi Iwai

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit, or a bugzilla or ML link for the further discussion or debugging session. The regression markers would be especially helpful to avoid a pitfall through performing bisection.

Yeah, this is very useful.

For example, instead of adding a tag in the changelog to point to the guilt commit that introduces the bug:

Fixes commit_id ("Subject")

It's much more convenient and useful if we can add metadata to the guilt commit that points to later fixes, and then we simply ask git to show us those fixes. This helps a lot to people who need to backport things to older kernels.

Quoted 4 lines

IMO, it would be convenient if such information can be embedded in the published git tree, something like (infamous) git-notes. But, I'd like to hear any other options as well.

#5 lore raw
In reply to Takashi Iwai

Takashi,

Quoted 7 lines · Takashi Iwai

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit,

For the stuff flying by me, I've been adding the:

Fixes <12-char hash>: ('Offending patch subject')

On patches fixing a regression. It helps the stable team (when Cc stable is also added) and in your scenario, you could grep the commits for the result of your bisect.

Quoted 1 line

or a bugzilla or

Sorry, I don't use it.

Quoted 1 line

ML link for the further discussion or debugging session.

We've also been autogenerating a tag:

For all patches that points to the email the patch came from. Not exactly what you were looking for, but nothing prevents someone from replying to that thread a year later with a regression report.

I've also been contemplating adding

Coverletter: https://lkml.kernel.org/r/<First Reference Msg-Id>

for large series where the patch submitter has done a thorough writeup in the coverletter.

thx,

Jason.

#6 lore raw

Hi,

From: Jason Cooper <[email protected]>

Quoted 33 lines

Takashi,

Quoted 7 lines · Takashi Iwai

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit,

For the stuff flying by me, I've been adding the:

Fixes <12-char hash>: ('Offending patch subject')

On patches fixing a regression. It helps the stable team (when Cc stable is also added) and in your scenario, you could grep the commits for the result of your bisect.

Quoted 1 line

or a bugzilla or

Sorry, I don't use it.

Quoted 1 line

ML link for the further discussion or debugging session.

We've also been autogenerating a tag:

For all patches that points to the email the patch came from. Not exactly what you were looking for, but nothing prevents someone from replying to that thread a year later with a regression report.

Quoted 6 lines

I've also been contemplating adding

Coverletter: https://lkml.kernel.org/r/<First Reference Msg-Id>

for large series where the patch submitter has done a thorough writeup in the coverletter.

I have been working on a new git command "git interpret-trailers" to help people add the above kind of trailer lines:

http://thread.gmane.org/gmane.comp.version-control.git/245874/

This was started by the following thread on the git mailing list:

http://thread.gmane.org/gmane.comp.version-control.git/236770/

which itself was started by decisions at Linux Kernel Summit 2013.

About bisecting with untestable commits, I suggest preparing fixed up branches where the bug that prevents testing is fixed and then using "git replace" to tell Git to use the fixed up branch instead of the original one. This way you can easily share these fixed up branches.

Here is a presentation I give about "git replace":

http://www.slideshare.net/ChristianCouder/new-views-onyourhistorywithgitr

where using fixed up branches to bisect is described starting at slide 19.

(By the way, I gave that presentation at LinuxTag last week in Berlin and I proposed it for LinuxCon North America next August.)

Best,
Christian.

#7 lore raw
In reply to Jason Cooper
Quoted 11 lines · Jason Cooper

Takashi,

Quoted 7 lines · Takashi Iwai

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit,

Quoted 8 lines

For the stuff flying by me, I've been adding the:

Fixes <12-char hash>: ('Offending patch subject')

On patches fixing a regression. It helps the stable team (when Cc stable is also added) and in your scenario, you could grep the commits for the result of your bisect.

Yeah, but the problem with this is you can't add this "Fixes" tag to git-log after the patch has been applied to the git tree.

#8 lore raw
In reply to Jason Cooper

At Thu, 15 May 2014 23:07:08 -0400,

Quoted 12 lines · Jason Cooper

Takashi,

Quoted 7 lines · Takashi Iwai

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit,

Quoted 7 lines

For the stuff flying by me, I've been adding the:

Fixes <12-char hash>: ('Offending patch subject')

On patches fixing a regression. It helps the stable team (when Cc stable is also added) and in your scenario, you could grep the commits for the result of your bisect.

Yeah, that was suggested in the last year's KS, and helps some cases.

However, the problem is that people(*1) often notice this too late after the tree has been already published. Also, some information (e.g. bug reports) can come only after commits.

(*1) statistics taken from one person :)

And, for bisection, we need some reverse mapping for efficiency. It'd take time to look all commit logs from each revlist, especially if the bisection is done for the early history.

I tried a hackish way once ago: keeping simple text files named with $SHAID in a separate branch, and refers to it at git log or bisect time. There must be much elegant way, I suppose, though.

Quoted 14 lines
Quoted 1 line

or a bugzilla or

Sorry, I don't use it.

Quoted 1 line

ML link for the further discussion or debugging session.

We've also been autogenerating a tag:

For all patches that points to the email the patch came from. Not exactly what you were looking for, but nothing prevents someone from replying to that thread a year later with a regression report.

Quoted 6 lines

I've also been contemplating adding

Coverletter: https://lkml.kernel.org/r/<First Reference Msg-Id>

for large series where the patch submitter has done a thorough writeup in the coverletter.

Yes, this kind of information is helpful for checking patches at later point, indeed.

thanks,

Takashi

#9 lore raw

From: Takashi Iwai <[email protected]>

Quoted 23 lines

At Thu, 15 May 2014 23:07:08 -0400,

Quoted 12 lines · Jason Cooper

Takashi,

Quoted 7 lines · Takashi Iwai

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit,

Quoted 7 lines

For the stuff flying by me, I've been adding the:

Fixes <12-char hash>: ('Offending patch subject')

On patches fixing a regression. It helps the stable team (when Cc stable is also added) and in your scenario, you could grep the commits for the result of your bisect.

Quoted 5 lines

Yeah, that was suggested in the last year's KS, and helps some cases.

However, the problem is that people(*1) often notice this too late after the tree has been already published. Also, some information (e.g. bug reports) can come only after commits.

It is possible to use "git notes" or even "git replace" to add information to existing commits.

Quoted 9 lines

(*1) statistics taken from one person :)

And, for bisection, we need some reverse mapping for efficiency. It'd take time to look all commit logs from each revlist, especially if the bisection is done for the early history.

I tried a hackish way once ago: keeping simple text files named with $SHAID in a separate branch, and refers to it at git log or bisect time. There must be much elegant way, I suppose, though.

Yeah, using "git replace" is more elegant. And there will be hopefully soon the --edit option that will make it very easy to use git replace.

Best,
Christian.

#10 lore raw
Quoted 1 line · Christian Couder

From: Takashi Iwai <[email protected]>

...

Quoted 6 lines
Quoted 3 lines

I tried a hackish way once ago: keeping simple text files named with $SHAID in a separate branch, and refers to it at git log or bisect time. There must be much elegant way, I suppose, though.

Yeah, using "git replace" is more elegant. And there will be hopefully soon the --edit option that will make it very easy to use git replace.

Yikes. Sorry, but I don't like that at all. I hope that feature is disabled by default (git replace).

I guess my first question is: Does the PGP signature verification of a signed tag fail if there are replaced commits in the history? And if so, is that a cryptographic failure, or a logical failure?

thx,

Jason.

#11 lore raw

From: Jason Cooper <[email protected]>

Quoted 7 lines · Christian Couder
Quoted 1 line

From: Takashi Iwai <[email protected]>

...

Quoted 4 lines
Quoted 3 lines

I tried a hackish way once ago: keeping simple text files named with $SHAID in a separate branch, and refers to it at git log or bisect time. There must be much elegant way, I suppose, though.

Quoted 5 lines
Quoted 2 lines

Yeah, using "git replace" is more elegant. And there will be hopefully soon the --edit option that will make it very easy to use git replace.

Yikes. Sorry, but I don't like that at all. I hope that feature is disabled by default (git replace).

By default replace refs are not fetched and not pushed. So you only have the replace refs you created yourself on your own repo. They are only fetched or pushed if you use or configure a refspec like 'refs/replace/*:refs/replace/*'.

You can also use the GIT_NO_REPLACE_OBJECTS environment variable or "alias git='git --no-replace-objects'", if/when you want to make sure that replace refs are not used.

A hook could also check that for example each replace ref you add to your repo replaces a commit with another commit that points to the same tree.

You could also use 2 different repos, one where you do most of your work and don't have any replace ref, and one where you do your bisecting and where you have replace refs. You can create, fetch and push replace refs in your bisecting repo.

Quoted 2 lines

I guess my first question is: Does the PGP signature verification of a signed tag fail if there are replaced commits in the history?

First any kind of Git object can be replaced, not just commits, even tag objects. Also I think the PGP signature verification only checks the content of the tag for signed tags or the content of the commit for signed commits.

So it might be a good idea to disable replace refs (for example using the GIT_NO_REPLACE_OBJECTS environment variable or --no-replace-objects) when you want to check signatures, and maybe Git could warn or error out if the commit or tags that are being verified are replaced.

But if you have replace refs and some blobs or trees are replaced, you have to trust these replace refs. (And maybe you are very right to trust them because for example you created them yourself and never fetched any replace ref.)

Anyway maybe we should cc Junio and other people on this because signing is not a part of Git that I know well.

Quoted 2 lines

And if
so, is that a cryptographic failure, or a logical failure?

I don't think there is any failure.

Some things are inherently risky. For example if you bisect kernels, you might compile and run kernels that have known security bugs, so you have to be careful.

Best,
Christian.

#12 lore raw
In reply to Christian Couder

At Sun, 18 May 2014 21:23:15 +0200 (CEST),

Quoted 25 lines · Christian Couder

From: Takashi Iwai <[email protected]>

Quoted 23 lines

At Thu, 15 May 2014 23:07:08 -0400,

Quoted 19 lines · Jason Cooper

Takashi,

Quoted 7 lines · Takashi Iwai

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit,

For the stuff flying by me, I've been adding the:

Fixes <12-char hash>: ('Offending patch subject')

On patches fixing a regression. It helps the stable team (when Cc stable is also added) and in your scenario, you could grep the commits for the result of your bisect.

Quoted 8 lines
Quoted 5 lines

Yeah, that was suggested in the last year's KS, and helps some cases.

However, the problem is that people(*1) often notice this too late after the tree has been already published. Also, some information (e.g. bug reports) can come only after commits.

It is possible to use "git notes" or even "git replace" to add information to existing commits.

Yes, I know of git-notes, as already mentioned in the first post. But the problem is that publishing and importing git-notes changes isn't well established for kernel git tree management. And, IIRC, Linus didn't like its usage somehow. Not sure whether it's because of git-notes technical design or its concept...

Quoted 6 lines
Quoted 6 lines

(*1) statistics taken from one person :)

And, for bisection, we need some reverse mapping for efficiency. It'd take time to look all commit logs from each revlist, especially if the bisection is done for the early history.

Quoted 6 lines
Quoted 3 lines

I tried a hackish way once ago: keeping simple text files named with $SHAID in a separate branch, and refers to it at git log or bisect time. There must be much elegant way, I suppose, though.

Yeah, using "git replace" is more elegant. And there will be hopefully soon the --edit option that will make it very easy to use git replace.

We don't want to change the history at all. A preferred option is just addendum on top of the existing commits, and the way to easily share the change (at best with the normal git pull). If git-replace provides such a good integration, I'd love to see it in our use case.

thanks,

Takashi

#13 lore raw

From: Takashi Iwai <[email protected]>

Quoted 33 lines

At Sun, 18 May 2014 21:23:15 +0200 (CEST),

Quoted 31 lines · Christian Couder

From: Takashi Iwai <[email protected]>

Quoted 28 lines

At Thu, 15 May 2014 23:07:08 -0400,

Quoted 12 lines · Jason Cooper

Takashi,

Quoted 7 lines · Takashi Iwai

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit,

Quoted 7 lines

For the stuff flying by me, I've been adding the:

Fixes <12-char hash>: ('Offending patch subject')

On patches fixing a regression. It helps the stable team (when Cc stable is also added) and in your scenario, you could grep the commits for the result of your bisect.

Yeah, that was suggested in the last year's KS, and helps some cases.

However, the problem is that people(*1) often notice this too late after the tree has been already published. Also, some information (e.g. bug reports) can come only after commits.

Quoted 8 lines
Quoted 2 lines

It is possible to use "git notes" or even "git replace" to add information to existing commits.

Yes, I know of git-notes, as already mentioned in the first post. But the problem is that publishing and importing git-notes changes isn't well established for kernel git tree management. And, IIRC, Linus didn't like its usage somehow. Not sure whether it's because of git-notes technical design or its concept...

Maybe Linus just doesn't want to be bothered with maintaining notes/replace refs in his git repos. But these refs could be maintained by other people in other repos.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

(*1) statistics taken from one person :)

And, for bisection, we need some reverse mapping for efficiency. It'd take time to look all commit logs from each revlist, especially if the bisection is done for the early history.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I tried a hackish way once ago: keeping simple text files named with $SHAID in a separate branch, and refers to it at git log or bisect time. There must be much elegant way, I suppose, though.

Yeah, using "git replace" is more elegant. And there will be hopefully soon the --edit option that will make it very easy to use git replace.

We don't want to change the history at all.

What I understood is that you want to be able to bisect on a history where many bugs that prevent testing can easily be fixed.

Using git replace, it can go like this:

- a developer learns that some part of the history can be fixed to bisect more easily by applying a patch,

- the developer recreates a part of the history where the patch has been applied (using git cherry-pick or git rebase -i for example)

- the developer uses git replace to create a replace ref so that the new part of the history is used instead of the old one

- the developer pushes this new replace ref to a common repo where replace refs are shared (this pushes the new part of the history too)

- other developers fetch the replace refs from the common repo before they bisect

- when these other developers bisect, the new parts of the history are automatically used, so bisecting is faster, easier and maybe safer too

I think that it is better than when everyone changes the history manually by applying patches in the middle of a bisection.

Quoted 4 lines

A preferred option is
just addendum on top of the existing commits, and the way to easily share the change (at best with the normal git pull). If git-replace provides such a good integration, I'd love to see it in our use case.

git notes is probably better if what you want is only addendum on top of the existing commits. git notes creates notes refs that you can easily share.

Best,
Christian.

#14 lore raw

At Tue, 20 May 2014 08:37:18 +0200 (CEST),

Quoted 45 lines · Christian Couder

From: Takashi Iwai <[email protected]>

Quoted 33 lines

At Sun, 18 May 2014 21:23:15 +0200 (CEST),

Quoted 31 lines · Christian Couder

From: Takashi Iwai <[email protected]>

Quoted 28 lines

At Thu, 15 May 2014 23:07:08 -0400,

Quoted 12 lines · Jason Cooper

Takashi,

Quoted 7 lines · Takashi Iwai

I think this has been already raised a few times, but I'm still dreaming one thing in our git management: having some metadata collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for tracking the information that has been revealed after commit, e.g. a regression the commit causes, the later fix commit,

Quoted 7 lines

For the stuff flying by me, I've been adding the:

Fixes <12-char hash>: ('Offending patch subject')

On patches fixing a regression. It helps the stable team (when Cc stable is also added) and in your scenario, you could grep the commits for the result of your bisect.

Yeah, that was suggested in the last year's KS, and helps some cases.

However, the problem is that people(*1) often notice this too late after the tree has been already published. Also, some information (e.g. bug reports) can come only after commits.

Quoted 8 lines
Quoted 2 lines

It is possible to use "git notes" or even "git replace" to add information to existing commits.

Yes, I know of git-notes, as already mentioned in the first post. But the problem is that publishing and importing git-notes changes isn't well established for kernel git tree management. And, IIRC, Linus didn't like its usage somehow. Not sure whether it's because of git-notes technical design or its concept...

Quoted 3 lines

Maybe Linus just doesn't want to be bothered with maintaining notes/replace refs in his git repos. But these refs could be maintained by other people in other repos.

Maybe. But just a guess at this point.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

(*1) statistics taken from one person :)

And, for bisection, we need some reverse mapping for efficiency. It'd take time to look all commit logs from each revlist, especially if the bisection is done for the early history.

I tried a hackish way once ago: keeping simple text files named with $SHAID in a separate branch, and refers to it at git log or bisect time. There must be much elegant way, I suppose, though.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Yeah, using "git replace" is more elegant. And there will be hopefully soon the --edit option that will make it very easy to use git replace.

We don't want to change the history at all.

What I understood is that you want to be able to bisect on a history where many bugs that prevent testing can easily be fixed.

Well, the bisection is just an example where the extra metadata would help. It's not the purpose itself.

Quoted 19 lines

Using git replace, it can go like this:

- a developer learns that some part of the history can be fixed to bisect more easily by applying a patch,

- the developer recreates a part of the history where the patch has been applied (using git cherry-pick or git rebase -i for example)

- the developer uses git replace to create a replace ref so that the new part of the history is used instead of the old one

- the developer pushes this new replace ref to a common repo where replace refs are shared (this pushes the new part of the history too)

- other developers fetch the replace refs from the common repo before they bisect

Quoted 6 lines

- when these other developers bisect, the new parts of the history are automatically used, so bisecting is faster, easier and maybe safer too

I think that it is better than when everyone changes the history manually by applying patches in the middle of a bisection.

This is nice, but I'm not sure whether it'll work / suit well with a tree like kernel, which is the results of tons of merges from various maintainers. I guess little maintainers would publish the corrected repo. I hope my guess is wrong.

Quoted 8 lines
Quoted 4 lines

A preferred option is
just addendum on top of the existing commits, and the way to easily share the change (at best with the normal git pull). If git-replace provides such a good integration, I'd love to see it in our use case.

git notes is probably better if what you want is only addendum on top of the existing commits. git notes creates notes refs that you can easily share.

IMO, it's not that easy to share git notes. At least, it's not in the same level as sharing tags. If it were as easy as tags, I see no reason to avoid using git-notes. That said, having a separate refs that requires the extra non-standard step looks like a big obstacle.

thanks,

Takashi

#15 lore raw

From: Takashi Iwai <[email protected]>

Quoted 37 lines

At Tue, 20 May 2014 08:37:18 +0200 (CEST),

Quoted 7 lines · Christian Couder

From: Takashi Iwai <[email protected]>

Quoted 1 line

We don't want to change the history at all.

What I understood is that you want to be able to bisect on a history where many bugs that prevent testing can easily be fixed.

Well, the bisection is just an example where the extra metadata would help. It's not the purpose itself.

Quoted 23 lines

Using git replace, it can go like this:

- a developer learns that some part of the history can be fixed to bisect more easily by applying a patch,

- the developer recreates a part of the history where the patch has been applied (using git cherry-pick or git rebase -i for example)

- the developer uses git replace to create a replace ref so that the new part of the history is used instead of the old one

- the developer pushes this new replace ref to a common repo where replace refs are shared (this pushes the new part of the history too)

- other developers fetch the replace refs from the common repo before they bisect

- when these other developers bisect, the new parts of the history are automatically used, so bisecting is faster, easier and maybe safer too

Quoted 7 lines
Quoted 2 lines

I think that it is better than when everyone changes the history manually by applying patches in the middle of a bisection.

This is nice, but I'm not sure whether it'll work / suit well with a tree like kernel, which is the results of tons of merges from various maintainers. I guess little maintainers would publish the corrected repo. I hope my guess is wrong.

Yeah, I don't know if it's easy enough to redo many merges with current Git. On the other hand, I don't see why it couldn't be made easy if it isn't already.

About maintainers publishing the corrected/improved repo, I think if a few developers publish one that provide some benefits, and come up with a way/workflow to do so quite easily, then other will follow.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

A preferred option is
just addendum on top of the existing commits, and the way to easily share the change (at best with the normal git pull). If git-replace provides such a good integration, I'd love to see it in our use case.

Quoted 8 lines
Quoted 3 lines

git notes is probably better if what you want is only addendum on top of the existing commits. git notes creates notes refs that you can easily share.

IMO, it's not that easy to share git notes. At least, it's not in the same level as sharing tags. If it were as easy as tags, I see no reason to avoid using git-notes. That said, having a separate refs that requires the extra non-standard step looks like a big obstacle.

I don't see why it is not easy to share git notes.

Basically I think that you just need to add refspecs like

	  push = +refs/notes/*:refs/notes/*

or

	  fetch = +refs/notes/*:refs/notes/*

in your config file, in the sections of the remotes you want to push or fetch notes.

But we should probably CC Johan Herland <[email protected]> or Junio when discussing this as I don't know notes well.

Best,
Christian.

#16 lore raw
In reply to Takashi Iwai

I told Junio and Johan about this discussion, and Junio, as he did not find a good way to subscribe to the list as a newcomer and still send a response to an existing thread, said that I can forward the following.

Johan agrees with Junio about this. I don't agree with their opinion about "replace".

From: Junio C Hamano <[email protected]>:

-- >8 --

Quoted 6 lines · in response to what Jason cooper
Quoted 6 lines

For the stuff flying by me, I've been adding the:

Fixes: <12-char hash>: ('Offending patch subject')
...
I've also been contemplating adding

Quoted 8 lines
Quoted 5 lines

Coverletter: https://lkml.kernel.org/r/<First Reference Msg-Id>

for large series where the patch submitter has done a thorough writeup
in the coverletter.

Yes, this kind of information is helpful for checking patches at later point, indeed.

I tend to think that:

* "replace" is too heavy-handed tool to use in general. Nobody sane should publish history full of replacements for public consumption, especially if that is done merely to help bisection.

* "notes" is very handy and may be an efficient mechanism to add information after the fact to existing commits, but merging two or more lines of notes histories together is cumbersome.

A good way forward to solve Iwai-san's original issue might be

* Establish the "Fixes:" mentioned above as a standard practice. Polishing Christian's interpret-trailers tool might be a good way to encourage developers to do so.

* Have an easy way for developers to scan incoming commits for these "Fixes:" footer, and record the reverse mapping locally, so that we can go from a commit whose brokenness is discovered later to the commit that fixes its breakage efficiently. "notes" may be a good mechanism to implement this mapping, and we do not have to worry about sharing the notes trees among developers.

* The information is visible with "log --show-notes" if it is stored in local notes. When an earlier commit that was later found to be broken is shown, the note that points at the commit that fixes it will be shown.

* Teach "bisect" to also take notice of this information, and temporarily cherry-pick while testing commits with fixes that were discovered later, in a way similar to what was suggested by Jiri Kosina in an earlier message.

-- >8 --

#17 lore raw

At Thu, 22 May 2014 06:49:10 +0200 (CEST),

Quoted 33 lines · Christian Couder

I told Junio and Johan about this discussion, and Junio, as he did not find a good way to subscribe to the list as a newcomer and still send a response to an existing thread, said that I can forward the following.

Johan agrees with Junio about this. I don't agree with their opinion about "replace".

From: Junio C Hamano <[email protected]>:

-- >8 --

Quoted 6 lines · in response to what Jason cooper
Quoted 6 lines

For the stuff flying by me, I've been adding the:

Fixes: <12-char hash>: ('Offending patch subject')
...
I've also been contemplating adding

Quoted 8 lines
Quoted 5 lines

Coverletter: https://lkml.kernel.org/r/<First Reference Msg-Id>

for large series where the patch submitter has done a thorough writeup
in the coverletter.

Yes, this kind of information is helpful for checking patches at later point, indeed.

I tend to think that:

Quoted 8 lines

* "replace" is too heavy-handed tool to use in general. Nobody sane should publish history full of replacements for public consumption, especially if that is done merely to help bisection.

* "notes" is very handy and may be an efficient mechanism to add information after the fact to existing commits, but merging two or more lines of notes histories together is cumbersome.

Do you mean that it's cumbersome from technical/performance viewpoint, or about the appearance of actual text outputs?

Quoted 18 lines

A good way forward to solve Iwai-san's original issue might be

* Establish the "Fixes:" mentioned above as a standard practice. Polishing Christian's interpret-trailers tool might be a good way to encourage developers to do so.

* Have an easy way for developers to scan incoming commits for these "Fixes:" footer, and record the reverse mapping locally, so that we can go from a commit whose brokenness is discovered later to the commit that fixes its breakage efficiently. "notes" may be a good mechanism to implement this mapping, and we do not have to worry about sharing the notes trees among developers.

* The information is visible with "log --show-notes" if it is stored in local notes. When an earlier commit that was later found to be broken is shown, the note that points at the commit that fixes it will be shown.

Quoted 4 lines

* Teach "bisect" to also take notice of this information, and temporarily cherry-pick while testing commits with fixes that were discovered later, in a way similar to what was suggested by Jiri Kosina in an earlier message.

These sound like a good plan, indeed.

But, one missing, and maybe often happening thing is: people forget to tag at the right time. In the scenario above, if a maintainer forgets to add Fixes: tag in the fix commit, it's all gone?

thanks,

Takashi

#18 lore raw
Quoted 24 lines · Takashi Iwai
Quoted 19 lines

From: Junio C Hamano <[email protected]>:
A good way forward to solve Iwai-san's original issue might be

* Establish the "Fixes:" mentioned above as a standard practice. Polishing Christian's interpret-trailers tool might be a good way to encourage developers to do so.

* Have an easy way for developers to scan incoming commits for these "Fixes:" footer, and record the reverse mapping locally, so that we can go from a commit whose brokenness is discovered later to the commit that fixes its breakage efficiently. "notes" may be a good mechanism to implement this mapping, and we do not have to worry about sharing the notes trees among developers.

* The information is visible with "log --show-notes" if it is stored in local notes. When an earlier commit that was later found to be broken is shown, the note that points at the commit that fixes it will be shown.

Quoted 4 lines

* Teach "bisect" to also take notice of this information, and temporarily cherry-pick while testing commits with fixes that were discovered later, in a way similar to what was suggested by Jiri Kosina in an earlier message.

Quoted 5 lines

These sound like a good plan, indeed.

But, one missing, and maybe often happening thing is: people forget to tag at the right time. In the scenario above, if a maintainer forgets to add Fixes: tag in the fix commit, it's all gone?

Yes, unless you add it (using git-notes) as an annotation to the fix commit, but then you're back to the (perceived) problem of sharing those notes.

Alternatively, you would have to find a different mechanism for developers to insert entries into their reverse mappings from post-facto/missing "Fixes:" tags.

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#19 lore raw

From: Johan Herland <[email protected]>

Quoted 27 lines · Takashi Iwai
Quoted 26 lines
Quoted 19 lines

From: Junio C Hamano <[email protected]>:
A good way forward to solve Iwai-san's original issue might be

* Establish the "Fixes:" mentioned above as a standard practice. Polishing Christian's interpret-trailers tool might be a good way to encourage developers to do so.

* Have an easy way for developers to scan incoming commits for these "Fixes:" footer, and record the reverse mapping locally, so that we can go from a commit whose brokenness is discovered later to the commit that fixes its breakage efficiently. "notes" may be a good mechanism to implement this mapping, and we do not have to worry about sharing the notes trees among developers.

* The information is visible with "log --show-notes" if it is stored in local notes. When an earlier commit that was later found to be broken is shown, the note that points at the commit that fixes it will be shown.

Quoted 4 lines

* Teach "bisect" to also take notice of this information, and temporarily cherry-pick while testing commits with fixes that were discovered later, in a way similar to what was suggested by Jiri Kosina in an earlier message.

These sound like a good plan, indeed.

Quoted 7 lines
Quoted 3 lines

But, one missing, and maybe often happening thing is: people forget to tag at the right time. In the scenario above, if a maintainer forgets to add Fixes: tag in the fix commit, it's all gone?

Yes, unless you add it (using git-notes) as an annotation to the fix commit, but then you're back to the (perceived) problem of sharing those notes.

By the way, I wonder if it might be possible to have signed "notes" or signed "replace"?

Best,
Christian.

#20 lore raw
Quoted 32 lines · Christian Couder

From: Johan Herland <[email protected]>

Quoted 31 lines · Takashi Iwai
Quoted 29 lines
Quoted 19 lines

From: Junio C Hamano <[email protected]>:
A good way forward to solve Iwai-san's original issue might be

* Establish the "Fixes:" mentioned above as a standard practice. Polishing Christian's interpret-trailers tool might be a good way to encourage developers to do so.

* Have an easy way for developers to scan incoming commits for these "Fixes:" footer, and record the reverse mapping locally, so that we can go from a commit whose brokenness is discovered later to the commit that fixes its breakage efficiently. "notes" may be a good mechanism to implement this mapping, and we do not have to worry about sharing the notes trees among developers.

* The information is visible with "log --show-notes" if it is stored in local notes. When an earlier commit that was later found to be broken is shown, the note that points at the commit that fixes it will be shown.

Quoted 4 lines

* Teach "bisect" to also take notice of this information, and temporarily cherry-pick while testing commits with fixes that were discovered later, in a way similar to what was suggested by Jiri Kosina in an earlier message.

These sound like a good plan, indeed.

But, one missing, and maybe often happening thing is: people forget to tag at the right time. In the scenario above, if a maintainer forgets to add Fixes: tag in the fix commit, it's all gone?

Quoted 6 lines
Quoted 3 lines

Yes, unless you add it (using git-notes) as an annotation to the fix commit, but then you're back to the (perceived) problem of sharing those notes.

By the way, I wonder if it might be possible to have signed "notes" or signed "replace"?

Isn't that merely a matter of creating a signed tag pointing at whatever the corresponding refs/notes/* or refs/replace/* ref is pointing at?

Of course that creates a new refs/tags/* ref that is separate from your notes/replace ref. If you also want the refs/notes/* or refs/replace/* ref to point at the signed tag (and have it automatically peel through to the tagged object) you might run into some problems. For notes, I'm not sure if the notes code peels through tags to find the notes commit/tree object. For replace refs, I guess it would look like the original object was not replaced by a different object of the same type, but was instead replaced by a signed tag object. Not sure how to solve that...

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#21 lore raw
In reply to Johan Herland

At Thu, 22 May 2014 08:28:37 +0200,

Quoted 32 lines · Johan Herland
Quoted 24 lines · Takashi Iwai
Quoted 19 lines

From: Junio C Hamano <[email protected]>:
A good way forward to solve Iwai-san's original issue might be

* Establish the "Fixes:" mentioned above as a standard practice. Polishing Christian's interpret-trailers tool might be a good way to encourage developers to do so.

* Have an easy way for developers to scan incoming commits for these "Fixes:" footer, and record the reverse mapping locally, so that we can go from a commit whose brokenness is discovered later to the commit that fixes its breakage efficiently. "notes" may be a good mechanism to implement this mapping, and we do not have to worry about sharing the notes trees among developers.

* The information is visible with "log --show-notes" if it is stored in local notes. When an earlier commit that was later found to be broken is shown, the note that points at the commit that fixes it will be shown.

Quoted 4 lines

* Teach "bisect" to also take notice of this information, and temporarily cherry-pick while testing commits with fixes that were discovered later, in a way similar to what was suggested by Jiri Kosina in an earlier message.

Quoted 5 lines

These sound like a good plan, indeed.

But, one missing, and maybe often happening thing is: people forget to tag at the right time. In the scenario above, if a maintainer forgets to add Fixes: tag in the fix commit, it's all gone?

Quoted 7 lines

Yes, unless you add it (using git-notes) as an annotation to the fix commit, but then you're back to the (perceived) problem of sharing those notes.

Alternatively, you would have to find a different mechanism for developers to insert entries into their reverse mappings from post-facto/missing "Fixes:" tags.

After reconsidering this, now I don't think we can rely only on Fixes tag. The addendum notes are useful for other purposes, e.g. finding external links. And, yet one useful thing by addendum is that you can mark the regression no matter whether it's already fixed or not. With Fixes-tag reverse mapping, you can find only fixed regressions, but not open regressions.

So, we seem back to square now :)

Like Christian mentioned, what about some way to push / pull git notes in a trusted way? If we apply the maintainer model also for managing such information, too, the similar mechanism would be required.

Or, better to manage such information in a totally different manner (e.g. a web API)?

Takashi

#22 lore raw
Quoted 8 lines · Takashi Iwai
Quoted 7 lines

Yes, unless you add it (using git-notes) as an annotation to the fix commit, but then you're back to the (perceived) problem of sharing those notes.

Alternatively, you would have to find a different mechanism for developers to insert entries into their reverse mappings from post-facto/missing "Fixes:" tags.

Quoted 6 lines

After reconsidering this, now I don't think we can rely only on Fixes tag. The addendum notes are useful for other purposes, e.g. finding external links. And, yet one useful thing by addendum is that you can mark the regression no matter whether it's already fixed or not. With Fixes-tag reverse mapping, you can find only fixed regressions, but not open regressions.

Do you want to import the whole bug tracking system, too? ;-)

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#23 lore raw

At Thu, 22 May 2014 09:49:44 +0200,

Quoted 10 lines · Geert Uytterhoeven
Quoted 8 lines · Takashi Iwai
Quoted 7 lines

Yes, unless you add it (using git-notes) as an annotation to the fix commit, but then you're back to the (perceived) problem of sharing those notes.

Alternatively, you would have to find a different mechanism for developers to insert entries into their reverse mappings from post-facto/missing "Fixes:" tags.

Quoted 8 lines
Quoted 6 lines

After reconsidering this, now I don't think we can rely only on Fixes tag. The addendum notes are useful for other purposes, e.g. finding external links. And, yet one useful thing by addendum is that you can mark the regression no matter whether it's already fixed or not. With Fixes-tag reverse mapping, you can find only fixed regressions, but not open regressions.

Do you want to import the whole bug tracking system, too? ;-)

Oh no. But just adding a regression tag to a commit would be no hard work for maintainers, I suppose.

Takashi

#24 lore raw
In reply to Christian Couder
Quoted 5 lines · Christian Couder

I told Junio and Johan about this discussion, and Junio, as he did not find a good way to subscribe to the list as a newcomer and still send a response to an existing thread, said that I can forward the following.

If someone sends a message who is not on the list, it goes into a moderation queue. When I approve such requests, I'll also whitelist that particular e-mail address, so the 4-8 hour latency should only apply for the first e-mail address from a particular e-mail address.

So just reply to the message; it will eventually make it out to the whole list. My approval time will generally be faster during US business hours.

Cheers,

- Ted