Skip to contentSkip to conversation

new -stable tag variant, Git workflow question

RFC In review Ingo Molnar 9 participants 32 messages Nov 10, 2009 – Nov 19, 2009

lore
Share & tools
lore
#0 lore raw

RFC 1/1 new -stable tag variant, Git workflow question

FYI, today i committed a scheduler performance fix that has a number of commit prerequisites for -stable integration. Those commits are not marked -stable.

Previously, in similar situations, i solved it by email-forwarding the prereq commits to [email protected]. (or by waiting for your 'it does not apply to -stable' email and figuring out the prereqs then.)

But we can move this into the Git commit space too, and minimize the work for the -stable team, via a new -stable tag variant. So with this new commit i started using a new tagging scheme in the commit itself:

Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
Cc: <[email protected]> # .32.x
LKML-Reference: <[email protected]>

The tag sequence has the meaning of:

git cherry-pick a1f84a3
git cherry-pick 1b9508f
git cherry-pick fd21073
git cherry-pick <this commit>

and i'm wondering whether this tagging scheme is fine with your -stable scripting, etc.

A further question is, i can see using this tagging scheme in the future in merge commits log messages too - will your scripts notice that too?

For example if there's a few commits left in tip:*/urgent branches (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time v2.6.32 is released, i will then merge them into tip:sched/core, tip:core/core, tip:x86/core, etc. - and i could use the merge commit as a notification area to 'activate' them for -stable backporting via a merge commit.

This is how such merge commits would look like:

Merge branch 'core/urgent' into core/rcu

Merge reason: Pick up urgent fixes that did not make it into .32.0

Cc <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race

This is not so rare of a situation as it might seem - for the trees i maintain it happens in almost every release cycle - i typically skip urgent branch merges after -rc8/-rc9, unless they are very-very-urgent fixes - but they'd still be eligible for -stable.

I've attached the full commit below. The prereq commits are not uptream yet, and they dont carry a -stable backporting tag as the -stable relevance was not anticipated at that point yet. They will all be upstream in the next merge window when Linus merges the relevant tree - and then all these tags become visible to the -stable team's scripts.

What do you think about this new -stable tagging variant? To me it looks quite intuitive, less error-prone and it is more informative as well. Furthermore, it gives us some freedom to mark commits as backport candidates later on. I kept them oneliners for the purpose of making them all self-sufficient tags.

( Sidenote: i wouldnt go as far as to generate null Git commits to mark
backports after the fact - this scheme is for a series of commits that get 'completed' - there's usually a final followup commit that can embedd this information. )

Ingo

---------------------------->

Quoted 1 line

From eae0c9dfb534cb3449888b9601228efa6480fdb5 Mon Sep 17 00:00:00 2001

From: Mike Galbraith <[email protected]>
Date: Tue, 10 Nov 2009 03:50:02 +0100
Subject: [PATCH] sched: Fix and clean up rate-limit newidle code

Commit 1b9508f, "Rate-limit newidle" has been confirmed to fix the netperf UDP loopback regression reported by Alex Shi.

This is a cleanup and a fix:

- moved to a more out of the way spot

- fix to ensure that balancing doesn't try to balance runqueues which haven't gone online yet, which can mess up CPU enumeration during boot.

Cc <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
Cc <[email protected]> # .32.x

LKML-Reference: <[email protected]>

---
 kernel/sched.c |   28 +++++++++++++++-------------
 1 files changed, 15 insertions(+), 13 deletions(-)
kernel/sched.c +15 -13
index 23e3535..ad37776 100644
@@ -2354,17 +2354,6 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state,
 	if (rq != orig_rq)
 		update_rq_clock(rq);
 
-	if (rq->idle_stamp) {
-		u64 delta = rq->clock - rq->idle_stamp;
-		u64 max = 2*sysctl_sched_migration_cost;
-
-		if (delta > max)
-			rq->avg_idle = max;
-		else
-			update_avg(&rq->avg_idle, delta);
-		rq->idle_stamp = 0;
-	}
-
 	WARN_ON(p->state != TASK_WAKING);
 	cpu = task_cpu(p);
 
@@ -2421,6 +2410,17 @@ out_running:
 #ifdef CONFIG_SMP
 	if (p->sched_class->task_wake_up)
 		p->sched_class->task_wake_up(rq, p);
+
+	if (unlikely(rq->idle_stamp)) {
+		u64 delta = rq->clock - rq->idle_stamp;
+		u64 max = 2*sysctl_sched_migration_cost;
+
+		if (delta > max)
+			rq->avg_idle = max;
+		else
+			update_avg(&rq->avg_idle, delta);
+		rq->idle_stamp = 0;
+	}
 #endif
 out:
 	task_rq_unlock(rq, &flags);
@@ -4098,7 +4098,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,
 	unsigned long flags;
 	struct cpumask *cpus = __get_cpu_var(load_balance_tmpmask);
 
-	cpumask_setall(cpus);
+	cpumask_copy(cpus, cpu_online_mask);
 
 	/*
 	 * When power savings policy is enabled for the parent domain, idle
@@ -4261,7 +4261,7 @@ load_balance_newidle(int this_cpu, struct rq *this_rq, struct sched_domain *sd)
 	int all_pinned = 0;
 	struct cpumask *cpus = __get_cpu_var(load_balance_tmpmask);
 
-	cpumask_setall(cpus);
+	cpumask_copy(cpus, cpu_online_mask);
 
 	/*
 	 * When power savings policy is enabled for the parent domain, idle
@@ -9522,6 +9522,8 @@ void __init sched_init(void)
 		rq->cpu = i;
 		rq->online = 0;
 		rq->migration_thread = NULL;
+		rq->idle_stamp = 0;
+		rq->avg_idle = 2*sysctl_sched_migration_cost;
 		INIT_LIST_HEAD(&rq->migration_queue);
 		rq_attach_root(rq, &def_root_domain);
 #endif
#1 lore raw
Quoted 21 lines · Ingo Molnar

FYI, today i committed a scheduler performance fix that has a number of commit prerequisites for -stable integration. Those commits are not marked -stable.

Previously, in similar situations, i solved it by email-forwarding the prereq commits to [email protected]. (or by waiting for your 'it does not apply to -stable' email and figuring out the prereqs then.)

But we can move this into the Git commit space too, and minimize the work for the -stable team, via a new -stable tag variant. So with this new commit i started using a new tagging scheme in the commit itself:

Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
Cc: <[email protected]> # .32.x
LKML-Reference: <[email protected]>

The tag sequence has the meaning of:

Quoted 7 lines

git cherry-pick a1f84a3
git cherry-pick 1b9508f
git cherry-pick fd21073
git cherry-pick <this commit>

and i'm wondering whether this tagging scheme is fine with your -stable scripting, etc.

It would work just fine.

I only rely on one main script right now, one that runs from James's directory on kernel.org that picks out the "Cc: <[email protected]>" lines and forwards the full commit message to [email protected].

Then I have a simple script that I just pass the git commit id and formats it properly for inclusion on the quilt tree for the stable queue. If you list the other git commit ids that are needed as a prerequesite as you did above, that's trivial to also pick out.

So I think this is good for me and my workflow.

Quoted 2 lines

A further question is, i can see using this tagging scheme in the future in merge commits log messages too - will your scripts notice that too?

Hm, I don't think we look at merges as there's nothing there to actually commit.

Quoted 13 lines

For example if there's a few commits left in tip:*/urgent branches (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time v2.6.32 is released, i will then merge them into tip:sched/core, tip:core/core, tip:x86/core, etc. - and i could use the merge commit as a notification area to 'activate' them for -stable backporting via a merge commit.

This is how such merge commits would look like:

Merge branch 'core/urgent' into core/rcu

Merge reason: Pick up urgent fixes that did not make it into .32.0

Quoted 7 lines
Cc <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race

This is not so rare of a situation as it might seem - for the trees i maintain it happens in almost every release cycle - i typically skip urgent branch merges after -rc8/-rc9, unless they are very-very-urgent fixes - but they'd still be eligible for -stable.

Ok, that would be good and fine with me.

James, would your script pick this up, or does it need to also pay attention to merge commits?

Quoted 11 lines

I've attached the full commit below. The prereq commits are not uptream yet, and they dont carry a -stable backporting tag as the -stable relevance was not anticipated at that point yet. They will all be upstream in the next merge window when Linus merges the relevant tree - and then all these tags become visible to the -stable team's scripts.

What do you think about this new -stable tagging variant? To me it looks quite intuitive, less error-prone and it is more informative as well. Furthermore, it gives us some freedom to mark commits as backport candidates later on. I kept them oneliners for the purpose of making them all self-sufficient tags.

I agree.

Quoted 4 lines

( Sidenote: i wouldnt go as far as to generate null Git commits to mark
backports after the fact - this scheme is for a series of commits that get 'completed' - there's usually a final followup commit that can embedd this information. )

That's fine, don't worry about this.

thanks,

greg k-h

#2 lore raw
Quoted 7 lines · Greg KH
Quoted 7 lines

I've attached the full commit below. The prereq commits are not uptream yet, and they dont carry a -stable backporting tag as the -stable relevance was not anticipated at that point yet. They will all be upstream in the next merge window when Linus merges the relevant tree - and then all these tags become visible to the -stable team's scripts.

Quoted 7 lines
Quoted 5 lines

What do you think about this new -stable tagging variant? To me it looks quite intuitive, less error-prone and it is more informative as well. Furthermore, it gives us some freedom to mark commits as backport candidates later on. I kept them oneliners for the purpose of making them all self-sufficient tags.

I agree.

Ok - thanks for the confirmation - i've pushed out the first such commit. (Let me know if there's any problem with it down the line - it will be a few weeks, in the next merge window, until it truly 'activates' for -stable.)

Ingo

#3 lore raw
Quoted 17 lines · Ingo Molnar
Quoted 7 lines · Greg KH
Quoted 7 lines

I've attached the full commit below. The prereq commits are not uptream yet, and they dont carry a -stable backporting tag as the -stable relevance was not anticipated at that point yet. They will all be upstream in the next merge window when Linus merges the relevant tree - and then all these tags become visible to the -stable team's scripts.

Quoted 7 lines
Quoted 5 lines

What do you think about this new -stable tagging variant? To me it looks quite intuitive, less error-prone and it is more informative as well. Furthermore, it gives us some freedom to mark commits as backport candidates later on. I kept them oneliners for the purpose of making them all self-sufficient tags.

I agree.

Quoted 4 lines

Ok - thanks for the confirmation - i've pushed out the first such commit. (Let me know if there's any problem with it down the line - it will be a few weeks, in the next merge window, until it truly 'activates' for -stable.)

Can you give me the commit id in your tree so I can run a few tests?

Thanks,

James

#4 lore raw
Quoted 18 lines · James Bottomley
Quoted 17 lines · Ingo Molnar
Quoted 14 lines · Greg KH
Quoted 7 lines

I've attached the full commit below. The prereq commits are not uptream yet, and they dont carry a -stable backporting tag as the -stable relevance was not anticipated at that point yet. They will all be upstream in the next merge window when Linus merges the relevant tree - and then all these tags become visible to the -stable team's scripts.

Quoted 5 lines

What do you think about this new -stable tagging variant? To me it looks quite intuitive, less error-prone and it is more informative as well. Furthermore, it gives us some freedom to mark commits as backport candidates later on. I kept them oneliners for the purpose of making them all self-sufficient tags.

I agree.

Quoted 6 lines
Quoted 4 lines

Ok - thanks for the confirmation - i've pushed out the first such commit. (Let me know if there's any problem with it down the line - it will be a few weeks, in the next merge window, until it truly 'activates' for -stable.)

Can you give me the commit id in your tree so I can run a few tests?

Let me re-quote what you quoted above:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I've attached the full commit below. [...]

Ingo

#5 lore raw
Quoted 8 lines · Ingo Molnar
Quoted 2 lines · James Bottomley

Can you give me the commit id in your tree so I can run a few tests?

Let me re-quote what you quoted above:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I've attached the full commit below. [...]

Let me re-quote James:

"Can you give me the commit id in your tree so I can run a few tests?"

He wanted the tree. So that he can run tests on his machinery. His machinery that does _not_ take some emailed commit description, but works on git trees.

So Ingo, if you want to be snarky about quoting people, at least be snarky with a reason.

Linus

#6 lore raw
Quoted 14 lines · Linus Torvalds
Quoted 8 lines · Ingo Molnar
Quoted 2 lines · James Bottomley

Can you give me the commit id in your tree so I can run a few tests?

Let me re-quote what you quoted above:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I've attached the full commit below. [...]

Let me re-quote James:

Quoted 8 lines

"Can you give me the commit id in your tree so I can run a few tests?"

He wanted the tree. So that he can run tests on his machinery. His machinery that does _not_ take some emailed commit description, but works on git trees.

So Ingo, if you want to be snarky about quoting people, at least be snarky with a reason.

The commit i quoted had the commit ID on its first line (i did that so because i anticipated someone wanting to check my tree), but here it is again plus the (well-known to James) URI of the -tip tree:

Quoted 1 line

From eae0c9dfb534cb3449888b9601228efa6480fdb5 Mon Sep 17 00:00:00 2001

git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git

Thanks,

Ingo

#7 lore raw
In reply to Greg KH
Quoted 36 lines · Greg KH
Quoted 21 lines · Ingo Molnar

FYI, today i committed a scheduler performance fix that has a number of commit prerequisites for -stable integration. Those commits are not marked -stable.

Previously, in similar situations, i solved it by email-forwarding the prereq commits to [email protected]. (or by waiting for your 'it does not apply to -stable' email and figuring out the prereqs then.)

But we can move this into the Git commit space too, and minimize the work for the -stable team, via a new -stable tag variant. So with this new commit i started using a new tagging scheme in the commit itself:

Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
Cc: <[email protected]> # .32.x
LKML-Reference: <[email protected]>

The tag sequence has the meaning of:

Quoted 7 lines

git cherry-pick a1f84a3
git cherry-pick 1b9508f
git cherry-pick fd21073
git cherry-pick <this commit>

and i'm wondering whether this tagging scheme is fine with your -stable scripting, etc.

It would work just fine.

I only rely on one main script right now, one that runs from James's directory on kernel.org that picks out the "Cc: <[email protected]>" lines and forwards the full commit message to [email protected].

Quoted 6 lines

Then I have a simple script that I just pass the git commit id and formats it properly for inclusion on the quilt tree for the stable queue. If you list the other git commit ids that are needed as a prerequesite as you did above, that's trivial to also pick out.

So I think this is good for me and my workflow.

So I take this to mean that I don't alter my script and you pick out the precursors with yours ...

Quoted 27 lines
Quoted 2 lines

A further question is, i can see using this tagging scheme in the future in merge commits log messages too - will your scripts notice that too?

Hm, I don't think we look at merges as there's nothing there to actually commit.

Quoted 13 lines

For example if there's a few commits left in tip:*/urgent branches (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time v2.6.32 is released, i will then merge them into tip:sched/core, tip:core/core, tip:x86/core, etc. - and i could use the merge commit as a notification area to 'activate' them for -stable backporting via a merge commit.

This is how such merge commits would look like:

Merge branch 'core/urgent' into core/rcu

Merge reason: Pick up urgent fixes that did not make it into .32.0

Quoted 7 lines
Cc <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race

This is not so rare of a situation as it might seem - for the trees i maintain it happens in almost every release cycle - i typically skip urgent branch merges after -rc8/-rc9, unless they are very-very-urgent fixes - but they'd still be eligible for -stable.

Quoted 4 lines

Ok, that would be good and fine with me.

James, would your script pick this up, or does it need to also pay attention to merge commits?

No ... because merge commits should effectively be empty (and when they're not you can't generate an applyable diff). If I understand the workflow, the desire is to have the whole branch sent to stable by tagging the merge commit. That's possible ... it's exactly the same logic I use in the commit scripts for the SCSI tree, so it should be possible to extract the logic.

By the looks of the above it's only a few commits, or is it the entire branch?

Quoted 12 lines
Quoted 11 lines

I've attached the full commit below. The prereq commits are not uptream yet, and they dont carry a -stable backporting tag as the -stable relevance was not anticipated at that point yet. They will all be upstream in the next merge window when Linus merges the relevant tree - and then all these tags become visible to the -stable team's scripts.

What do you think about this new -stable tagging variant? To me it looks quite intuitive, less error-prone and it is more informative as well. Furthermore, it gives us some freedom to mark commits as backport candidates later on. I kept them oneliners for the purpose of making them all self-sufficient tags.

Quoted 8 lines

I agree.

Quoted 4 lines

( Sidenote: i wouldnt go as far as to generate null Git commits to mark
backports after the fact - this scheme is for a series of commits that get 'completed' - there's usually a final followup commit that can embedd this information. )

That's fine, don't worry about this.

The question is, how important is this?

One of the assumptions behind the current setup is that I assume backports are independent (so the order of transmission doesn't matter that much). This isn't always true, but the exceptions tend to get handled manually. Part of what the above is requesting is an implementation that starts to care about ordering.

James

#8 lore raw

James Bottomley wrote:
[...]

Quoted 4 lines
Quoted 4 lines · Ingo Molnar
Quoted 3 lines

FYI, today i committed a scheduler performance fix that has a number of commit prerequisites for -stable integration. Those commits are not marked -stable.

[...]

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

we can move this into the Git commit space too, and minimize the work for the -stable team, via a new -stable tag variant.

[...]

Quoted 7 lines

The question is, how important is this?

One of the assumptions behind the current setup is that I assume backports are independent (so the order of transmission doesn't matter that much). This isn't always true, but the exceptions tend to get handled manually. Part of what the above is requesting is an implementation that starts to care about ordering.

More importantly, isn't this against the character of the -stable kernel branches as _safe and simple_ hotfix branches?

If a fix has a number of prerequisites which ar not -stable fixes themselves, then it is more than a hint that this fix is not really well suited for -stable.

Signature · 3 lines
Stefan Richter
-=====-==--= =-== -=-=-
http://arcgraph.de/sr/
#9 lore raw
Quoted 6 lines · Stefan Richter

More importantly, isn't this against the character of the -stable kernel branches as _safe and simple_ hotfix branches?

If a fix has a number of prerequisites which ar not -stable fixes themselves, then it is more than a hint that this fix is not really well suited for -stable.

Alternately, it's conceivable that the prerequisites were not in-and-of-themselves candidates for -stable (maybe they didn't do anything by themselves) but when combined with the final commit the overall change is suitable for inclusion in -stable.

Chris

#10 lore raw
Quoted 5 lines · Chris Friesen
Quoted 3 lines · Stefan Richter

More importantly, isn't this against the character of the -stable kernel branches as _safe and simple_ hotfix branches?

Quoted 8 lines
Quoted 3 lines

If a fix has a number of prerequisites which ar not -stable fixes themselves, then it is more than a hint that this fix is not really well suited for -stable.

Alternately, it's conceivable that the prerequisites were not in-and-of-themselves candidates for -stable (maybe they didn't do anything by themselves) but when combined with the final commit the overall change is suitable for inclusion in -stable.

Yeah.

The way i do it as a maintainer is that when i add a new commit that i realize as a -stable candidate, and i know that it has no semantic prereqs (such as a new API, etc.), i git-cherry-pick it into stable in a test branch. If that works fine i mark it -stable straight away.

If it conflicts, i figure out the prereqs, and look at those. If they are too big, or too risky, i often decide not to mark a patch for -stable backporting. (If it's not obvious to be in -stable then it should not be in -stable, almost by definition - a _LOT_ of people are using the stable kernels.)

If the prereqs look sane and are wanted for -stable, i end up with a list of 2 or at most 3 commits that will cherry-pick cleanly. (rarely more than that - the 4 commits here are really an exception - they are a string of prereqs that are also fixes)

I send that list of commits to [email protected].

( Sidenote: rarely does it make sense to port a conflicting commit to
-stable. The risks of introducing some regression are just too high. Cherry-picking of commits, (while not entirely risk-free of course), is far more robust in practice. (conflicting backmerges do happen too occasionally, for high-profile bug fixes that justify the cost.) )

So this 'send the list to [email protected]' step is simplified via these tags:

    Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
    Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
    Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
    Cc: <[email protected]> # .32.x

I'd have done that via a plain email in any case - so this scheme does not enable anything new - it just simplifies the process and makes it a bit more robust.

Ingo

#11 lore raw
In reply to Stefan Richter
Quoted 18 lines · Stefan Richter

James Bottomley wrote:
[...]

Quoted 4 lines
Quoted 4 lines · Ingo Molnar
Quoted 3 lines

FYI, today i committed a scheduler performance fix that has a number of commit prerequisites for -stable integration. Those commits are not marked -stable.

[...]

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

we can move this into the Git commit space too, and minimize the work for the -stable team, via a new -stable tag variant.

[...]

Quoted 7 lines

The question is, how important is this?

One of the assumptions behind the current setup is that I assume backports are independent (so the order of transmission doesn't matter that much). This isn't always true, but the exceptions tend to get handled manually. Part of what the above is requesting is an implementation that starts to care about ordering.

Quoted 6 lines

More importantly, isn't this against the character of the -stable kernel branches as _safe and simple_ hotfix branches?

If a fix has a number of prerequisites which ar not -stable fixes themselves, then it is more than a hint that this fix is not really well suited for -stable.

Not true, we have been doing this kind of thing for quite some time now. Sometimes it's just a simple "this patch cleans up the code, and the second one fixes it in an obvious manner" type thing. It is easier for me and everyone else for us to apply 2 commits to the -stable tree, instead of rewriting the second patch that actually does the fix and hope I got it all correct in doing so.

It's also easier to review stuff, which is the most important thing.

thanks,

greg k-h

#12 lore raw
Quoted 7 lines · Greg KH
Quoted 6 lines

More importantly, isn't this against the character of the -stable kernel branches as _safe and simple_ hotfix branches?

If a fix has a number of prerequisites which ar not -stable fixes themselves, then it is more than a hint that this fix is not really well suited for -stable.

Quoted 8 lines

Not true, we have been doing this kind of thing for quite some time now. Sometimes it's just a simple "this patch cleans up the code, and the second one fixes it in an obvious manner" type thing. It is easier for me and everyone else for us to apply 2 commits to the -stable tree, instead of rewriting the second patch that actually does the fix and hope I got it all correct in doing so.

It's also easier to review stuff, which is the most important thing.

Yeah. This new tagging scheme doesnt really allow anything 'new' per se
- it just helps the existing practice some more. All these commits were -stable candidates anyway, in exactly the same order - the only difference the new tagging scheme adds here is a more organized, in-upsream-Git way of communicating it to you.

This is also easier and less error prone for me than using email: i can do all the -stable tagging when i create a commit - or if i see that a commit has prereqs and those should be in -stable too. In those situations i check out the last stable kernel version, and cherry-pick the prereqs and the target commit, to see that it cherry-picks without conflicts.

But i cannot send you an email to [email protected] just yet: as i havent fully tested the last commit yet, and have not pushed it out yet. The commit ID is not stable yet.

So without the in-commit tagging, i'd have to remember to send you an email in an hour (or in a day - whenever testing is done) - and that is error prone and easy to forget. The prereqs might be lost, etc. It's better to do this all in one well-focused moment of time, gather the information and mention it in the changelog.

Ingo

#13 lore raw
Quoted 5 lines · Ingo Molnar

Yeah. This new tagging scheme doesnt really allow anything 'new' per se
- it just helps the existing practice some more. All these commits were -stable candidates anyway, in exactly the same order - the only difference the new tagging scheme adds here is a more organized, in-upsream-Git way of communicating it to you.

I am just a bystander, but if it were truly in-upstream-git way, wouldn't you be forking a branch from the tagged target release (the latest of 2.6.32.X), and queuing only the changes meant for -stable to it, and giving the name of the branch to git people and sending out patches from that branch for e-mailed review and application?

There won't be any special tagging required, only a dedicated branch.

Or am I missing something?

#14 lore raw
Quoted 14 lines · Junio C Hamano
Quoted 5 lines · Ingo Molnar

Yeah. This new tagging scheme doesnt really allow anything 'new' per se
- it just helps the existing practice some more. All these commits were -stable candidates anyway, in exactly the same order - the only difference the new tagging scheme adds here is a more organized, in-upsream-Git way of communicating it to you.

I am just a bystander, but if it were truly in-upstream-git way, wouldn't you be forking a branch from the tagged target release (the latest of 2.6.32.X), and queuing only the changes meant for -stable to it, and giving the name of the branch to git people and sending out patches from that branch for e-mailed review and application?

Quoted 3 lines

There won't be any special tagging required, only a dedicated branch.

Or am I missing something?

Yes, these are patches going to Linus's tree, which would be 2.6.33 at the time. I need to know the ids to add them back to the older .32.y tree.

hope this helps,

greg k-h

#15 lore raw
In reply to Junio C Hamano
Quoted 14 lines · Junio C Hamano
Quoted 5 lines · Ingo Molnar

Yeah. This new tagging scheme doesnt really allow anything 'new' per se
- it just helps the existing practice some more. All these commits were -stable candidates anyway, in exactly the same order - the only difference the new tagging scheme adds here is a more organized, in-upsream-Git way of communicating it to you.

I am just a bystander, but if it were truly in-upstream-git way, wouldn't you be forking a branch from the tagged target release (the latest of 2.6.32.X), and queuing only the changes meant for -stable to it, and giving the name of the branch to git people and sending out patches from that branch for e-mailed review and application?

Quoted 3 lines

There won't be any special tagging required, only a dedicated branch.

Or am I missing something?

There's no Git flow towards -stable. It's either forwarded emails, or tags in the upstream kernel. Also, _only_ commits that were pulled by Linus are eligible for -stable.

So the pull requests all first go to Linus - then can any commit flow to -stable.

But even if it was possible to send pull requests to Greg, marking commits as -stable candidates is more natural in the commit log itself.

That informs people ('hey, that's a dangerous patch, dont mark it for -stable!!' or 'hey, why isnt this commit tagged to stable??'), and it also ensures it that only commits from Linus's tree flow towards -stable.

Ingo

#16 lore raw
In reply to James Bottomley
Quoted 42 lines · James Bottomley
Quoted 41 lines · Greg KH
Quoted 21 lines · Ingo Molnar

FYI, today i committed a scheduler performance fix that has a number of commit prerequisites for -stable integration. Those commits are not marked -stable.

Previously, in similar situations, i solved it by email-forwarding the prereq commits to [email protected]. (or by waiting for your 'it does not apply to -stable' email and figuring out the prereqs then.)

But we can move this into the Git commit space too, and minimize the work for the -stable team, via a new -stable tag variant. So with this new commit i started using a new tagging scheme in the commit itself:

Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
Cc: <[email protected]> # .32.x
LKML-Reference: <[email protected]>

The tag sequence has the meaning of:

Quoted 7 lines

git cherry-pick a1f84a3
git cherry-pick 1b9508f
git cherry-pick fd21073
git cherry-pick <this commit>

and i'm wondering whether this tagging scheme is fine with your -stable scripting, etc.

It would work just fine.

I only rely on one main script right now, one that runs from James's directory on kernel.org that picks out the "Cc: <[email protected]>" lines and forwards the full commit message to [email protected].

Then I have a simple script that I just pass the git commit id and formats it properly for inclusion on the quilt tree for the stable queue. If you list the other git commit ids that are needed as a prerequesite as you did above, that's trivial to also pick out.

Quoted 4 lines
Quoted 1 line

So I think this is good for me and my workflow.

So I take this to mean that I don't alter my script and you pick out the precursors with yours ...

Exactly, it's easier for me that way, as I know the dependancy of what needs to go before what. And it's just so trivial to feed a git commit id to my script :)

Quoted 39 lines
Quoted 27 lines
Quoted 2 lines

A further question is, i can see using this tagging scheme in the future in merge commits log messages too - will your scripts notice that too?

Hm, I don't think we look at merges as there's nothing there to actually commit.

Quoted 13 lines

For example if there's a few commits left in tip:*/urgent branches (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time v2.6.32 is released, i will then merge them into tip:sched/core, tip:core/core, tip:x86/core, etc. - and i could use the merge commit as a notification area to 'activate' them for -stable backporting via a merge commit.

This is how such merge commits would look like:

Merge branch 'core/urgent' into core/rcu

Merge reason: Pick up urgent fixes that did not make it into .32.0

Quoted 7 lines
Cc <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race

This is not so rare of a situation as it might seem - for the trees i maintain it happens in almost every release cycle - i typically skip urgent branch merges after -rc8/-rc9, unless they are very-very-urgent fixes - but they'd still be eligible for -stable.

Quoted 4 lines

Ok, that would be good and fine with me.

James, would your script pick this up, or does it need to also pay attention to merge commits?

No ... because merge commits should effectively be empty (and when they're not you can't generate an applyable diff). If I understand the workflow, the desire is to have the whole branch sent to stable by tagging the merge commit. That's possible ... it's exactly the same logic I use in the commit scripts for the SCSI tree, so it should be possible to extract the logic.

Quoted 2 lines

By the looks of the above it's only a few commits, or is it the entire branch?

I'm thinking the commit would be the merge, right Ingo? So it would just be a single commit that has the marker in it.

Quoted 21 lines
Quoted 12 lines
Quoted 11 lines

I've attached the full commit below. The prereq commits are not uptream yet, and they dont carry a -stable backporting tag as the -stable relevance was not anticipated at that point yet. They will all be upstream in the next merge window when Linus merges the relevant tree - and then all these tags become visible to the -stable team's scripts.

What do you think about this new -stable tagging variant? To me it looks quite intuitive, less error-prone and it is more informative as well. Furthermore, it gives us some freedom to mark commits as backport candidates later on. I kept them oneliners for the purpose of making them all self-sufficient tags.

Quoted 8 lines

I agree.

Quoted 4 lines

( Sidenote: i wouldnt go as far as to generate null Git commits to mark
backports after the fact - this scheme is for a series of commits that get 'completed' - there's usually a final followup commit that can embedd this information. )

That's fine, don't worry about this.

Quoted 7 lines

The question is, how important is this?

One of the assumptions behind the current setup is that I assume backports are independent (so the order of transmission doesn't matter that much). This isn't always true, but the exceptions tend to get handled manually. Part of what the above is requesting is an implementation that starts to care about ordering.

No, I'll take care of the ordering myself. Heck, I already have to do that today with our current setup as we have dependant staging patches right now. I just want to make sure the merge commits will get picked up and sent to me so I can then pick the correct git commit ids out of them.

thanks,

greg k-h

#17 lore raw
Quoted 5 lines · Greg KH
Quoted 2 lines

By the looks of the above it's only a few commits, or is it the entire branch?

I'm thinking the commit would be the merge, right Ingo? So it would just be a single commit that has the marker in it.

Correct.

This is really a special case, a small variation of the commit eae0c9d -stable tagging scheme i outlined in the first mail.

When i merge */urgent branches into the for-linus branch in the merge window, i cannot change the commits anymore (it would amount to a rebase), but i have the opportunity to modify the merge commit message itself. (which is typically a regular merge commit and does not carry any -stable actionable change itself.)

I already annotate merge commits today - for example:

 | commit 43315956509ca6913764861ac7dec128b91eb1ec
 | Merge: 9bf4e7f 6beba7a
 | Author: Ingo Molnar <[email protected]>
 | Date:   Fri Oct 23 08:23:20 2009 +0200
 |
 |     Merge branch 'perf/core' into perf/probes
 |    
 |     Conflicts:
 |         tools/perf/Makefile
 |    
 |     Merge reason:
 |    
 |      - fix the conflict
 |      - pick up the pr_*() infrastructure to queue up dependent patch
 |    
 |     Signed-off-by: Ingo Molnar <[email protected]>

Note how i already put a SOB line into the merge commit - i treat every merge as something that 'had to be done' so they are never arbitrary and always carry real information.

So my idea was to potentially use the extended -stable notification scheme in certain merge commits too. Here's a mockup merge commit log:

Merge branch 'sched/urgent' into sched/core

Conflicts:
tools/perf/Makefile

Merge reason:

- resolve the conflict
- queue up urgent fixes for the next merge window

Cc <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc <[email protected]> # .32.x: fd21073: sched: Fix affinity logic

Note that the merge commit itself carries no action for -stable: there's no "Cc: <[email protected]>" line - only 'pointer' lines in the form of:

    Cc: <[email protected]> # .32.x: sha1: title

But ... if you or Linus dislikes this direction of tagging for some reason i can still do the manual approach as well. It seemed useful to me though and it would be a natural portion of my workflow.

Ingo

#18 lore raw
In reply to Greg KH
Quoted 41 lines · Greg KH
Quoted 40 lines · James Bottomley
Quoted 27 lines · Greg KH
Quoted 2 lines

A further question is, i can see using this tagging scheme in the future in merge commits log messages too - will your scripts notice that too?

Hm, I don't think we look at merges as there's nothing there to actually commit.

Quoted 13 lines

For example if there's a few commits left in tip:*/urgent branches (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time v2.6.32 is released, i will then merge them into tip:sched/core, tip:core/core, tip:x86/core, etc. - and i could use the merge commit as a notification area to 'activate' them for -stable backporting via a merge commit.

This is how such merge commits would look like:

Merge branch 'core/urgent' into core/rcu

Merge reason: Pick up urgent fixes that did not make it into .32.0

Quoted 7 lines
Cc <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race

This is not so rare of a situation as it might seem - for the trees i maintain it happens in almost every release cycle - i typically skip urgent branch merges after -rc8/-rc9, unless they are very-very-urgent fixes - but they'd still be eligible for -stable.

Quoted 4 lines

Ok, that would be good and fine with me.

James, would your script pick this up, or does it need to also pay attention to merge commits?

No ... because merge commits should effectively be empty (and when they're not you can't generate an applyable diff). If I understand the workflow, the desire is to have the whole branch sent to stable by tagging the merge commit. That's possible ... it's exactly the same logic I use in the commit scripts for the SCSI tree, so it should be possible to extract the logic.

Quoted 5 lines
Quoted 2 lines

By the looks of the above it's only a few commits, or is it the entire branch?

I'm thinking the commit would be the merge, right Ingo? So it would just be a single commit that has the marker in it.

OK, so I can make it send you this just by removing the --no-merge flag from the git rev-list the script uses to sift through what changed (which I've already done).

The slight problem is that further down, to generate the patch the script uses git format-patch -k --stdout commit^..commit. For a merge commit, this will generate a patch equivalent to the entire branch that was merged, even though the commit message will only pick out some of these ... is this OK?

If not, I can look at using git show instead to generate the patches (it will effectively generate null diffs for merge points with the stable tag, which is closer to what you want).

Alternatively, if you pick up the commits from Linus' tree anyway, I could just stop producing diffs, which will save email bandwidth and then be automatically correct whether the commit is a merge or not.

James

#19 lore raw
Quoted 6 lines · James Bottomley

The slight problem is that further down, to generate the patch the script uses git format-patch -k --stdout commit^..commit. For a merge commit, this will generate a patch equivalent to the entire branch that was merged, even though the commit message will only pick out some of these ... is this OK?

Don't do that.

Since you want to show a single commit, not a commit range, don't use format-patch, use something like

	git show --pretty=email -M --cc --stat $commit

instead. That will not show the difference between a commit and its first parent (which for a merge diff is more than one commit, and can be absolutely _huge_), but will show just the named commit (which for a merge commit will usually be an empty diff, but will show how conflicts were resolved if they weren't just taken from one or the other side).

(Of course, maybe you'll want to change the exact flags in question. For example, '--stat' for merge commits is often useful, but only if the merge was done by an upper-level maintainer. Anybody who does reverse merges will just get totally meaningless diffstats, since the diffstat is always done against the first parent, so anybody who does back-merges will see the stat of everthing _I_ have merged in the meantime, which is generally not useful)

Linus

#20 lore raw
Quoted 22 lines · Linus Torvalds
Quoted 6 lines · James Bottomley

The slight problem is that further down, to generate the patch the script uses git format-patch -k --stdout commit^..commit. For a merge commit, this will generate a patch equivalent to the entire branch that was merged, even though the commit message will only pick out some of these ... is this OK?

Don't do that.

Since you want to show a single commit, not a commit range, don't use format-patch, use something like

	git show --pretty=email -M --cc --stat $commit

instead. That will not show the difference between a commit and its first parent (which for a merge diff is more than one commit, and can be absolutely _huge_), but will show just the named commit (which for a merge commit will usually be an empty diff, but will show how conflicts were resolved if they weren't just taken from one or the other side).

Quoted 7 lines

(Of course, maybe you'll want to change the exact flags in question. For example, '--stat' for merge commits is often useful, but only if the merge was done by an upper-level maintainer. Anybody who does reverse merges will just get totally meaningless diffstats, since the diffstat is always done against the first parent, so anybody who does back-merges will see the stat of everthing _I_ have merged in the meantime, which is generally not useful)

Right ... that's why I was wondering if Greg still needs the diff. If not, I'll just send standard git log output with no diff. If he does, I'll use something like the above.

So, now that you're showing an interest in all of this, how about I send you the final update hook to add to your tree? It will produce less latency than the cron job I run on kernel.org to pull your tree into one I have here ever twenty minutes or so and run the hooks.

James

#21 lore raw
Quoted 5 lines · James Bottomley

So, now that you're showing an interest in all of this, how about I send you the final update hook to add to your tree? It will produce less latency than the cron job I run on kernel.org to pull your tree into one I have here ever twenty minutes or so and run the hooks.

Quite frankly, I'd much rather not run hooks on my tree directly. The less that can possibly get screwed up with my kernel.org tree, the happier I am. So in many ways, I really prefer the whole "done separately by others" approach.

But hey, send the hook to me, and I'll think about it.

Linus

#22 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · James Bottomley

So, now that you're showing an interest in all of this, how about I send you the final update hook to add to your tree? It will produce less latency than the cron job I run on kernel.org to pull your tree into one I have here ever twenty minutes or so and run the hooks.

Quoted 7 lines

Quite frankly, I'd much rather not run hooks on my tree directly. The less that can possibly get screwed up with my kernel.org tree, the happier I am. So in many ways, I really prefer the whole "done separately by others" approach.

But hey, send the hook to me, and I'll think about it.

One thing we might be able to do is to create an "exploder hook" on your tree - something that asynchronously broadcasts a notification that anyone can subscribe to.

-hpa

#23 lore raw
Quoted 11 lines · H. Peter Anvin
Quoted 7 lines

Quite frankly, I'd much rather not run hooks on my tree directly. The less that can possibly get screwed up with my kernel.org tree, the happier I am. So in many ways, I really prefer the whole "done separately by others" approach.

But hey, send the hook to me, and I'll think about it.

One thing we might be able to do is to create an "exploder hook" on your tree - something that asynchronously broadcasts a notification that anyone can subscribe to.

Yeah, some minimal hook just informing any interested party about "ok, Linus pushed to the main tree" (and in fact, perhaps generalized so that you can show that you are interested in some particular branch of _any_ tree that people can just add to their trees) certainly doesn't sound like a bad idea.

That way we can get rid of "polling by cron", without having to have the tree owners themselves care about who is interested in exactly what..

Linus

#24 lore raw
Quoted 21 lines · Linus Torvalds
Quoted 11 lines · H. Peter Anvin
Quoted 7 lines

Quite frankly, I'd much rather not run hooks on my tree directly. The less that can possibly get screwed up with my kernel.org tree, the happier I am. So in many ways, I really prefer the whole "done separately by others" approach.

But hey, send the hook to me, and I'll think about it.

One thing we might be able to do is to create an "exploder hook" on your tree - something that asynchronously broadcasts a notification that anyone can subscribe to.

Yeah, some minimal hook just informing any interested party about "ok, Linus pushed to the main tree" (and in fact, perhaps generalized so that you can show that you are interested in some particular branch of _any_ tree that people can just add to their trees) certainly doesn't sound like a bad idea.

Quoted 3 lines

That way we can get rid of "polling by cron", without having to have the tree owners themselves care about who is interested in exactly what..

That's already possible via the [email protected] email list. Anyone can subscribe to it and can procmail process it. It is also more secure (and less intrusive to the tree owner) than any post-push hook script running under the pusher's UID. Polling via cron is really not required i think.

Ingo

#25 lore raw
Quoted 7 lines · Ingo Molnar

That's already possible via the [email protected] email list. Anyone can subscribe to it and can procmail process it. It is also more secure (and less intrusive to the tree owner) than any post-push hook script running under the pusher's UID. Polling via cron is really not required i think.

There are a couple of disadvantages with this -- a hera -> vger -> hera bounce, in particular (those two machines are nowhere close to each other, and there can be huge latecy.) For real-time robots running on hera, a lower-latency mechanism would be useful.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#26 lore raw
Quoted 9 lines · H. Peter Anvin
Quoted 3 lines · Ingo Molnar

That's already possible via the [email protected] email list. Anyone can subscribe to it and can procmail process it. It is also

There are a couple of disadvantages with this -- a hera -> vger -> hera bounce, in particular (those two machines are nowhere close to each other, and there can be huge latecy.) For real-time robots running on hera, a lower-latency mechanism would be useful.

Change the hook on hera that sends the email to use a local alias that includes git-commits-header@vger and a local list that processes on hera can subscribe to? That could avoid the bounce...

#27 lore raw
Quoted 6 lines · David Dillow
Quoted 5 lines · H. Peter Anvin
Quoted 3 lines · Ingo Molnar

That's already possible via the [email protected] email list. Anyone can subscribe to it and can procmail process it. It is also

Quoted 8 lines
Quoted 4 lines

There are a couple of disadvantages with this -- a hera -> vger -> hera bounce, in particular (those two machines are nowhere close to each other, and there can be huge latecy.) For real-time robots running on hera, a lower-latency mechanism would be useful.

Change the hook on hera that sends the email to use a local alias that includes git-commits-header@vger and a local list that processes on hera can subscribe to? That could avoid the bounce...

Not really... a lot of the bounce delay is due to mail processing on hera itself.

-hpa

#28 lore raw
In reply to Linus Torvalds
Quoted 14 lines · Linus Torvalds
Quoted 11 lines · H. Peter Anvin
Quoted 7 lines

Quite frankly, I'd much rather not run hooks on my tree directly. The less that can possibly get screwed up with my kernel.org tree, the happier I am. So in many ways, I really prefer the whole "done separately by others" approach.

But hey, send the hook to me, and I'll think about it.

One thing we might be able to do is to create an "exploder hook" on your tree - something that asynchronously broadcasts a notification that anyone can subscribe to.

Quoted 8 lines

Yeah, some minimal hook just informing any interested party about "ok, Linus pushed to the main tree" (and in fact, perhaps generalized so that you can show that you are interested in some particular branch of _any_ tree that people can just add to their trees) certainly doesn't sound like a bad idea.

That way we can get rid of "polling by cron", without having to have the tree owners themselves care about who is interested in exactly what..

Sure, I'd like to hook into this rather than use a polling cron job.

James

#29 lore raw
In reply to James Bottomley
Quoted 58 lines · James Bottomley
Quoted 41 lines · Greg KH
Quoted 40 lines · James Bottomley
Quoted 27 lines · Greg KH
Quoted 2 lines

A further question is, i can see using this tagging scheme in the future in merge commits log messages too - will your scripts notice that too?

Hm, I don't think we look at merges as there's nothing there to actually commit.

Quoted 13 lines

For example if there's a few commits left in tip:*/urgent branches (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time v2.6.32 is released, i will then merge them into tip:sched/core, tip:core/core, tip:x86/core, etc. - and i could use the merge commit as a notification area to 'activate' them for -stable backporting via a merge commit.

This is how such merge commits would look like:

Merge branch 'core/urgent' into core/rcu

Merge reason: Pick up urgent fixes that did not make it into .32.0

Quoted 7 lines
Cc <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race

This is not so rare of a situation as it might seem - for the trees i maintain it happens in almost every release cycle - i typically skip urgent branch merges after -rc8/-rc9, unless they are very-very-urgent fixes - but they'd still be eligible for -stable.

Quoted 4 lines

Ok, that would be good and fine with me.

James, would your script pick this up, or does it need to also pay attention to merge commits?

No ... because merge commits should effectively be empty (and when they're not you can't generate an applyable diff). If I understand the workflow, the desire is to have the whole branch sent to stable by tagging the merge commit. That's possible ... it's exactly the same logic I use in the commit scripts for the SCSI tree, so it should be possible to extract the logic.

Quoted 5 lines
Quoted 2 lines

By the looks of the above it's only a few commits, or is it the entire branch?

I'm thinking the commit would be the merge, right Ingo? So it would just be a single commit that has the marker in it.

OK, so I can make it send you this just by removing the --no-merge flag from the git rev-list the script uses to sift through what changed (which I've already done).

The slight problem is that further down, to generate the patch the script uses git format-patch -k --stdout commit^..commit. For a merge commit, this will generate a patch equivalent to the entire branch that was merged, even though the commit message will only pick out some of these ... is this OK?

Quoted 7 lines

If not, I can look at using git show instead to generate the patches (it will effectively generate null diffs for merge points with the stable tag, which is closer to what you want).

Alternatively, if you pick up the commits from Linus' tree anyway, I could just stop producing diffs, which will save email bandwidth and then be automatically correct whether the commit is a merge or not.

No, I'd like to keep diffs, and use Linus's suggestion to get the correct diff, I have had that same bug in some scripts I have written as well.

thanks,

greg k-h

#30 lore raw
Quoted 63 lines · Greg KH
Quoted 62 lines · James Bottomley
Quoted 41 lines · Greg KH
Quoted 40 lines · James Bottomley
Quoted 27 lines · Greg KH
Quoted 2 lines

A further question is, i can see using this tagging scheme in the future in merge commits log messages too - will your scripts notice that too?

Hm, I don't think we look at merges as there's nothing there to actually commit.

Quoted 13 lines

For example if there's a few commits left in tip:*/urgent branches (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time v2.6.32 is released, i will then merge them into tip:sched/core, tip:core/core, tip:x86/core, etc. - and i could use the merge commit as a notification area to 'activate' them for -stable backporting via a merge commit.

This is how such merge commits would look like:

Merge branch 'core/urgent' into core/rcu

Merge reason: Pick up urgent fixes that did not make it into .32.0

Quoted 7 lines
Cc <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race

This is not so rare of a situation as it might seem - for the trees i maintain it happens in almost every release cycle - i typically skip urgent branch merges after -rc8/-rc9, unless they are very-very-urgent fixes - but they'd still be eligible for -stable.

Quoted 4 lines

Ok, that would be good and fine with me.

James, would your script pick this up, or does it need to also pay attention to merge commits?

No ... because merge commits should effectively be empty (and when they're not you can't generate an applyable diff). If I understand the workflow, the desire is to have the whole branch sent to stable by tagging the merge commit. That's possible ... it's exactly the same logic I use in the commit scripts for the SCSI tree, so it should be possible to extract the logic.

Quoted 5 lines
Quoted 2 lines

By the looks of the above it's only a few commits, or is it the entire branch?

I'm thinking the commit would be the merge, right Ingo? So it would just be a single commit that has the marker in it.

OK, so I can make it send you this just by removing the --no-merge flag from the git rev-list the script uses to sift through what changed (which I've already done).

The slight problem is that further down, to generate the patch the script uses git format-patch -k --stdout commit^..commit. For a merge commit, this will generate a patch equivalent to the entire branch that was merged, even though the commit message will only pick out some of these ... is this OK?

If not, I can look at using git show instead to generate the patches (it will effectively generate null diffs for merge points with the stable tag, which is closer to what you want).

Quoted 7 lines
Quoted 3 lines

Alternatively, if you pick up the commits from Linus' tree anyway, I could just stop producing diffs, which will save email bandwidth and then be automatically correct whether the commit is a merge or not.

No, I'd like to keep diffs, and use Linus's suggestion to get the correct diff, I have had that same bug in some scripts I have written as well.

OK, so git show --pretty=email -u --stat -M --cc <commit>

seems to be what works well (the options are -u to generate diff plus stats, -M to detect renames and --cc to try to generate a diff for merge points showing if there have been non trivial fix ups).

I've attached a sample below of what it will look like for a stable tagged merge point. As you can see, because this is a simple merge, the diff is empty.

James

---

From: James Bottomley <[email protected]>
To: [email protected]
Subject: Patch Upstream: Merge branch 'master'
of /pub/scm/linux/kernel/git/jejb/scsi-rc-fixes-2.6
Date: 12/11/09 10:07:58 (Thu, 12 Nov 2009 16:07:58 GMT)

commit: f2d5c3c1e34a7cc006de8e3cf94fe11f68dad894
From: James Bottomley <[email protected]>
Date: Thu, 12 Nov 2009 15:51:38 +0000
Subject: [PATCH] Merge branch 'master' of /pub/scm/linux/kernel/git/jejb/scsi-rc-fixes-2.6

Cc Stable Tree <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race
 drivers/scsi/bfa/bfad_im.c         |    2 +-
 drivers/scsi/ipr.c                 |   42 ++++++++++++++++++++++++++++-------
 drivers/scsi/ipr.h                 |    1 +
 drivers/scsi/libsas/sas_expander.c |    1 -
 drivers/scsi/pmcraid.c             |   10 ++++----
 drivers/scsi/scsi_transport_fc.c   |    3 ++
 include/scsi/scsi_host.h           |   29 +++++++++---------------
 7 files changed, 54 insertions(+), 34 deletions(-)
#31 lore raw
Quoted 74 lines · James Bottomley
Quoted 63 lines · Greg KH
Quoted 62 lines · James Bottomley
Quoted 41 lines · Greg KH
Quoted 40 lines · James Bottomley
Quoted 27 lines · Greg KH
Quoted 2 lines

A further question is, i can see using this tagging scheme in the future in merge commits log messages too - will your scripts notice that too?

Hm, I don't think we look at merges as there's nothing there to actually commit.

Quoted 13 lines

For example if there's a few commits left in tip:*/urgent branches (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time v2.6.32 is released, i will then merge them into tip:sched/core, tip:core/core, tip:x86/core, etc. - and i could use the merge commit as a notification area to 'activate' them for -stable backporting via a merge commit.

This is how such merge commits would look like:

Merge branch 'core/urgent' into core/rcu

Merge reason: Pick up urgent fixes that did not make it into .32.0

Quoted 7 lines
Cc <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race

This is not so rare of a situation as it might seem - for the trees i maintain it happens in almost every release cycle - i typically skip urgent branch merges after -rc8/-rc9, unless they are very-very-urgent fixes - but they'd still be eligible for -stable.

Quoted 4 lines

Ok, that would be good and fine with me.

James, would your script pick this up, or does it need to also pay attention to merge commits?

No ... because merge commits should effectively be empty (and when they're not you can't generate an applyable diff). If I understand the workflow, the desire is to have the whole branch sent to stable by tagging the merge commit. That's possible ... it's exactly the same logic I use in the commit scripts for the SCSI tree, so it should be possible to extract the logic.

Quoted 5 lines
Quoted 2 lines

By the looks of the above it's only a few commits, or is it the entire branch?

I'm thinking the commit would be the merge, right Ingo? So it would just be a single commit that has the marker in it.

OK, so I can make it send you this just by removing the --no-merge flag from the git rev-list the script uses to sift through what changed (which I've already done).

The slight problem is that further down, to generate the patch the script uses git format-patch -k --stdout commit^..commit. For a merge commit, this will generate a patch equivalent to the entire branch that was merged, even though the commit message will only pick out some of these ... is this OK?

If not, I can look at using git show instead to generate the patches (it will effectively generate null diffs for merge points with the stable tag, which is closer to what you want).

Quoted 7 lines
Quoted 3 lines

Alternatively, if you pick up the commits from Linus' tree anyway, I could just stop producing diffs, which will save email bandwidth and then be automatically correct whether the commit is a merge or not.

No, I'd like to keep diffs, and use Linus's suggestion to get the correct diff, I have had that same bug in some scripts I have written as well.

OK, so git show --pretty=email -u --stat -M --cc <commit>

Quoted 7 lines

seems to be what works well (the options are -u to generate diff plus stats, -M to detect renames and --cc to try to generate a diff for merge points showing if there have been non trivial fix ups).

I've attached a sample below of what it will look like for a stable tagged merge point. As you can see, because this is a simple merge, the diff is empty.

Looks great to me, thanks for doing this.

greg k-h