Fix WARN_ON / WARN_ON_ONCE regression
PATCH 1/1 Fix WARN_ON / WARN_ON_ONCE regression
Hi Herbet,
The patch "Let WARN_ON/WARN_ON_ONCE return the condition"
http://kernel.org/git/?
p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=684f978347deb42d180373ac4c427f82ef963171
introduced 40% more 2nd level cache miss to tbench workload being run in a loop back mode on a Core 2 machine. I think the introduction of the local variables to WARN_ON and WARN_ON_ONCE
typeof(x) __ret_warn_on = (x);
typeof(condition) __ret_warn_once = (condition);
results in the extra cache misses. In our test workload profile, we see heavily used functions like do_softirq and local_bh_enable takes a lot longer to execute.
The modification below helps fix the problem. I made a slight modification to sched.c to get around a gcc bug.
include/asm-generic/bug.h +0 -0
@@ -17,13 +17,12 @@ #endif#ifndef HAVE_ARCH_WARN_ON #define WARN_ON (condition)
({ \
- typeof(condition) __ret_warn_on =
(condition); \
- if (unlikely(__ret_warn_on))
{ \
+ if (unlikely(condition))
{ \
printk("BUG: warning at %s:%d/%s()\n",
__FILE__, \
__LINE__,
__FUNCTION__); \
dump_stack
(); \
} \- unlikely
(__ret_warn_on); \
+ unlikely
(condition); \
})
#endif
@@ -43,12 +42,16 @@ #endif
#define WARN_ON_ONCE(condition) ({ \
static int __warn_once = 1; \
- typeof(condition) __ret_warn_once = (condition);\
\
- if (likely(__warn_once)) \
- if (WARN_ON(__ret_warn_once)) \
- __warn_once = 0; \
- unlikely(__ret_warn_once); \
+ if (unlikely(condition)){ \
+ if (likely(__warn_once)){ \
+ __warn_once=0; \
+ printk("BUG: warning at %s:%d/%s()\n", __FILE__,
\
+ __LINE__, __FUNCTION__);\
+ dump_stack(); \
+ } \
+ } \
+ unlikely(condition); \})
#ifdef CONFIG_SMP
kernel/sched.c +2 -1
@@ -5629,7 +5629,8 @@ static unsigned long domain_distance(intstruct sched_domain *sd;for_each_domain (cpu1, sd) { -WARN_ON (!cpu_isset (cpu1, sd->span)); +if (unlikely (!cpu_isset (cpu1, sd->span))) +WARN_ON (1 );if (cpu_isset (cpu2, sd->span))return distance; distance++;
Quoted 10 lines · Tim Chen
Hi Herbet,
The patch "Let WARN_ON/WARN_ON_ONCE return the condition"
http://kernel.org/git/?
p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=684f978347deb42d180373ac4c427f82ef963171introduced 40% more 2nd level cache miss to tbench workload being run in a loop back mode on a Core 2 machine. I think the introduction of the local variables to
WARN_ONandWARN_ON_ONCE
Quoted 9 lines
typeof(x)
__ret_warn_on= (x);
typeof(condition)__ret_warn_once= (condition);results in the extra cache misses. In our test workload profile, we see heavily used functions like
do_softirqandlocal_bh_enabletakes a lot longer to execute.The modification below helps fix the problem. I made a slight modification to
sched.cto get around a gcc bug.
How does the generated code change? Doesn't evaluating the condition multiple times have the potential to cause problems?
J
Quoted 3 lines · Jeremy Fitzhardinge
How does the generated code change? Doesn't evaluating the condition multiple times have the potential to cause problems?
I think if the condition changes between two evaluations, we do have a problem with my fix. I don't have a better idea to avoid using a local variable to store the condition. I think we should at least reverse the WARN_ON/WARN_ON_ONCE patch if a better way cannot be found.
Tim
Quoted 2 lines · Tim Chen
I think if the condition changes between two evaluations, we do have a problem with my fix.
That's not the problem; one hopes that the WARN_ON predicate has no side-effects (though I know there are places which have side effects in BUG_ON). The point is that the vast majority of WARN_ONs *don't* have their values used, so in the current code the variable reference is dead code and will be removed. But if gcc can't prove the predicate is side-effect free (call to an external function, for example), then gcc will have to generate two calls to it, regardless of whether the second value is used.
And since the condition variable will - at worst - be stored on the stack on a hot cache line, I don't see how there could be any extra cache misses.
Quoted 3 lines
I don't have a better idea to avoid using a local
variable to store the condition. I think we should at least reverse the WARN_ON/WARN_ON_ONCE patch if a better way cannot be found.
I don't think you've proved your case here. Do you *know* there are extra cache misses (ie, measuring them), or is it just your theory to explain a performance regression?
The other question is whether WARN_ON should return a value. Where does it get used? It doesn't seem very valuable.
J
Quoted 5 lines · Jeremy Fitzhardinge
I don't think you've proved your case here. Do you *know* there are extra cache misses (ie, measuring them), or is it just your theory to explain a performance regression?
I have measured the cache miss with tool. So it is not just my theory.
Tim
Quoted 10 lines · On Wed, 04 Oct 2006 06:21:35 -0700 Tim Chen <[email protected]> wrote:
Quoted 5 lines · Jeremy Fitzhardinge
I don't think you've proved your case here. Do you *know* there are extra cache misses (ie, measuring them), or is it just your theory to explain a performance regression?
I have measured the cache miss with tool. So it is not just my theory.
And what did that tool tell you?
Guys. Please. Help us out here. None of this makes sense, and it's possible that we have an underlying problem in there which we need to know about.
Please don't just ignore my questions. *why* are we getting a cache miss rate on that integer which is causing measurable performance changes? If we're reading it that frequently then the variable should be in cache(!).
Again: do you know which callsite is causing the problem? I assume one of the ones in softirq.c? Do you know what the cache miss frequency is? etc.
Because if we don't answer these questions there's an excellent chance that the problem (whatever it is) will come back and bite us again.
Quoted 4 lines · Andrew Morton
Quoted 2 lines
I have measured the cache miss with tool. So it is not just my theory.
And what did that tool tell you?
I am using emon. Measuring a 20 second stretch of tbench run saw the L2 cache miss go from 14 million to 25 million on each of the cpu core.
Quoted 5 lines
Please don't just ignore my questions. *why* are we getting a cache miss rate on that integer which is causing measurable performance changes? If we're reading it that frequently then the variable should be in cache(!).
The point is valid, __warn_once should be in cache, unless something evicts it. What I have found so far is with patch by Andrew and Leonid that avoid looking up the __warn_once integer, the cache miss rate is reduced to the level before.
Quoted 3 lines
Again: do you know which callsite is causing the problem? I assume one of the ones in
softirq.c? Do you know what the cache miss frequency is? etc.
Unfortunately emon does not directly give the callsite. Oprofile data shows a marked increase in time spent in do_softirq and local_bh_enable. What I could do is to individually turn off WARN_ON_ONCE at these sites and see if they are responsible for the cache miss. Will let you know what I found.
Oprofile data --
Before WARN_ON_ONCE patch:
117767 thread_return
106651 local_bh_enable
83767 tcp_v4_rcv
72266 copy_user_generic_unrolled
47136 do_softirq
41100 tcp_recvmsg
39394 tcp_sendmsg
118383 thread_return
88171 copy_user_generic
..
8281 local_bh_enable
6790 do_softirq
After WARN_ON_ONCE patch:
117767 thread_return
106651 local_bh_enable
83767 tcp_v4_rcv
72266 copy_user_generic_unrolled
47136 do_softirq
Tim
Quoted 20 lines · On Wed, 04 Oct 2006 09:22:09 -0700 Tim Chen <[email protected]> wrote:
Quoted 4 lines · Andrew Morton
Quoted 2 lines
I have measured the cache miss with tool. So it is not just my theory.
And what did that tool tell you?
I am using emon. Measuring a 20 second stretch of tbench run saw the L2 cache miss go from 14 million to 25 million on each of the cpu core.
Quoted 5 lines
Please don't just ignore my questions. *why* are we getting a cache miss rate on that integer which is causing measurable performance changes? If we're reading it that frequently then the variable should be in cache(!).
The point is valid,
__warn_onceshould be in cache, unless something evicts it. What I have found so far is with patch by Andrew and Leonid that avoid looking up the__warn_onceinteger, the cache miss rate is reduced to the level before.
Quoted 8 lines
Quoted 3 lines
Again: do you know which callsite is causing the problem? I assume one of the ones in
softirq.c? Do you know what the cache miss frequency is? etc.Unfortunately emon does not directly give the callsite. Oprofile data shows a marked increase in time spent in
do_softirqandlocal_bh_enable. What I could do is to individually turn offWARN_ON_ONCEat these sites and see if they are responsible for the cache miss. Will let you know what I found.
I see, thanks. How very peculiar.
I wonder if we just got unlucky and that particular benchmark with that particular kernel build just happens to reach the cache system's associativity threshold, and this one extra cacheline took it over the edge. Or something.
But if that was the case, one would expect even a small change to the kernel (say, a different compiler version, or different config) would improve things a lot, or would worsen things a lot.
I think the change is right. But, again, I worry that any teeny change we make in there in the future will bring the problem back.
Quoted 6 lines · Andrew Morton
I wonder if we just got unlucky and that particular benchmark with that particular kernel build just happens to reach the cache system's associativity threshold, and this one extra cacheline took it over the edge. Or something.
I took a look at the "L2 cache lines evicted" data from emon. It increases by 17 million per cpu core during the 20 second tbench run with the WARN_ON_ONCE patch, along with 11 million increase in "L2 cache miss". So perhaps the __warn_once variable got evicted. No theory at least point on how it got evicted.
Also did an experiment to locate the callsite causing cache miss with the original WARN_ON_ONCE patch. I turned off WARN_ON_ONCE one by one in kernel/softirq.c where almost all call for WARN_ON_ONCE originated. I found that with each turnoff of WARN_ON_ONCE, the L2 cache miss go down by 1 to 2 million, L2 cache line eviction goes down by 2 to 3 million.
One particular WARN_ON_ONCE in local_bh_enable contributed the most: 6 million L2 cache miss and 10 million L2 cache line evictions. That particular WARN_ON_ONCE also has the most effect on tbench throughput (accounting for 70% of the tbench throughput reduction).
-- Code snippet --
void local_bh_enable(void)
{
#ifdef CONFIG_TRACE_IRQFLAGS
unsigned long flags;
WARN_ON_ONCE(in_irq());
#endif
WARN_ON_ONCE(irqs_disabled()); <------- most degradation here.
CONFIG_TRACE_IRQFLAGS off#ifdef CONFIG_TRACE_IRQFLAGS
local_irq_save(flags);
#endif
/*Thanks.
Tim
Quoted 11 lines · Andrew Morton
Quoted 9 lines
Quoted 4 lines
Please don't just ignore my questions. *why* are we getting a cache miss rate on that integer which is causing measurable performance changes? If we're reading it that frequently then the variable should be in cache(!).
The point is valid,
__warn_onceshould be in cache, unless something evicts it. What I have found so far is with patch by Andrew and Leonid that avoid looking up the__warn_onceinteger, the cache miss rate is reduced to the level before.
Quoted 7 lines
I see, thanks. How very peculiar.
I wonder if we just got unlucky and that particular benchmark with that particular kernel build just happens to reach the cache system's associativity threshold, and this one extra cacheline took it over the edge. Or something.
We believe we found the real cause of the performance regression:
it is precisely a self-inflicted cache line conflict on __warn_once global variable thanks to the older gcc (v3.4.5) backend assembly code optimizer. The newer version of gcc (v4.1.0) does not suffer the same problem.
We fall in the trap of thinking the __warn_once variable
is truly read mostly and is written only once in the very very unlikely case of bug triggering. But the compiler is doing something which turns the following innocent read only looking code into write always assembly code. The compiler is doing so to avoid a conditional jump.
The original "C" code looks very innocent:
if (WARN_ON(__ret_warn_once));
__warn_once = 0;The equivalent asm code generated by gcc looks like:
temp = 0;
if (!WARN_ON(__ret_warn_once))
temp = __warn_once;
__warn_once = temp;As a result, a global variable is being written from all CPUs everywhere and caused excessive cache line bouncing on SMP. We measured that HITM event increased by 75% and read-for-ownership event increased by 50%. Adding a __read_mostly directive to __warn_once didn't help because gcc still generate assembly code that write to that global variable.
Thanks.
Tim
Tim,
Great work on finding this!
Quoted 15 lines · Tim Chen
The original "C" code looks very innocent:
if (WARN_ON(__ret_warn_once)); __warn_once = 0;The equivalent asm code generated by gcc looks like:
temp = 0; if (!WARN_ON(__ret_warn_once)) temp = __warn_once; __warn_once = temp;
Quoted 7 lines
As a result, a global variable is being written from all CPUs everywhere and caused excessive cache line bouncing on SMP. We measured that HITM event increased by 75% and read-for-ownership event increased by 50%. Adding a
__read_mostlydirective to__warn_oncedidn't help because gcc still generate assembly code that write to that global variable.
Holy crap! I wonder where else in the kernel gcc is doing this. (of course I'm using gcc4 so I don't know). Is there another gcc attribute to actually tell gcc that a variable is really mostly read only (besides placing it in a mostly read only elf section)?
What was wrong with the original WARN_ON_ONCE with
if (unlikely(condition) && __warn_once)
This didn't have the cache crash problem too, did it? I don't have a gcc3 around to test.
-- Steve
Quoted 6 lines · Steven Rostedt
Holy crap! I wonder where else in the kernel gcc is doing this. (of course I'm using gcc4 so I don't know). Is there another gcc attribute to actually tell gcc that a variable is really mostly read only (besides placing it in a mostly read only elf section)?
Quoted 6 lines
What was wrong with the original
WARN_ON_ONCEwithif (unlikely(condition) && __warn_once)This didn't have the cache crash problem too, did it? I don't have a gcc3 around to test.
In the original WARN_ON_ONCE, gcc3 only writes to __warn_once when (condition) is true. So it does not cause cache bouncing by always writing to __warn_once.
Tim
Quoted 5 lines · Steven Rostedt
Holy crap! I wonder where else in the kernel gcc is doing this. (of course I'm using gcc4 so I don't know). Is there another gcc attribute to actually tell gcc that a variable is really mostly read only (besides placing it in a mostly read only elf section)?
That would be nice, but I don't know of one (apart from "volatile", which has its own downsides). Once could imagine an annotation which makes gcc consider writes to the variable relatively expensive, so that it avoids generating unnecessary/excessive writes.
J
Sorry my mailer screwed up the line-wrap. Here's a resent of the patch.
Tim
The patch "Let WARN_ON/WARN_ON_ONCE return the condition"
http://kernel.org/git/?
p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=684f978347deb42d180373ac4c427f82ef963171
introduced 40% more 2nd level cache miss to tbench workload being run in a loop back mode on a Core 2 machine. I think the introduction of the local variables to WARN_ON and WARN_ON_ONCE
typeof(x) __ret_warn_on = (x);
typeof(condition) __ret_warn_once = (condition);
results in the extra cache misses. In our test workload profile, we see heavily used functions like do_softirq and local_bh_enable takes a lot longer to execute.
The modification below helps fix the problem. I made a slight modification to sched.c to get around a gcc bug.
include/asm-generic/bug.h +11 -8
@@ -17,13 +17,12 @@ #endif#ifndef HAVE_ARCH_WARN_ON #define WARN_ON (condition) ({ \ - typeof(condition) __ret_warn_on = (condition); \ - if (unlikely(__ret_warn_on)) { \ + if (unlikely(condition)) { \ printk("BUG: warning at %s:%d/%s()\n", __FILE__, \ __LINE__, __FUNCTION__); \ dump_stack(); \ } \ - unlikely(__ret_warn_on); \ + unlikely(condition); \ })#endif
@@ -43,12 +42,16 @@ #endif#define WARN_ON_ONCE (condition) ({ \ static int __warn_once = 1; \ - typeof(condition) __ret_warn_once = (condition);\ \ - if (likely(__warn_once)) \ - if (WARN_ON(__ret_warn_once)) \ - __warn_once = 0; \ - unlikely(__ret_warn_once); \ + if (unlikely(condition)){ \ + if (likely(__warn_once)){ \ + __warn_once=0; \ + printk("BUG: warning at %s:%d/%s()\n", __FILE__, \ + __LINE__, __FUNCTION__);\ + dump_stack(); \ + } \ + } \ + unlikely(condition); \ })#ifdef CONFIG_SMP
kernel/sched.c +2 -1
@@ -5629,7 +5629,8 @@ static unsigned long domain_distance(intstruct sched_domain *sd;for_each_domain (cpu1, sd) { -WARN_ON (!cpu_isset (cpu1, sd->span)); +if (unlikely (!cpu_isset (cpu1, sd->span))) +WARN_ON (1 );if (cpu_isset (cpu2, sd->span))return distance; distance++;
Quoted 6 lines · On Tue, 03 Oct 2006 16:04:04 -0700 Tim Chen <[email protected]> wrote:
Hi Herbet,
The patch "Let WARN_ON/WARN_ON_ONCE return the condition"
http://kernel.org/git/?
p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=684f978347deb42d180373ac4c427f82ef963171
Quoted 8 lines
introduced 40% more 2nd level cache miss to tbench workload being run in a loop back mode on a Core 2 machine. I think the introduction of the local variables to
WARN_ONandWARN_ON_ONCEtypeof(x)
__ret_warn_on= (x);
typeof(condition)__ret_warn_once= (condition);results in the extra cache misses.
I don't see why it should.
Perhaps the `static int __warn_once' is getting put in the same cacheline
as some frequently-modified thing. Perhaps try marking that as __read_mostly?
Quoted 15 lines
In our test workload profile, we see
heavily used functions likedo_softirqandlocal_bh_enabletakes a lot longer to execute.The modification below helps fix the problem. I made a slight modification to
sched.cto get around a gcc bug.Signed-off-by Tim Chen <[email protected]>include/asm-generic/bug.h +0
-0@@ -17,13 +17,12 @@ #endif#ifndef HAVE_ARCH_WARN_ON
include/asm-generic/bug.h #endif · line 26
Quoted 10 lines
include/asm-generic/bug.h +1
-2#define WARN_ON (condition) ({ \ - typeof(condition) __ret_warn_on = (condition); \ - if (unlikely(__ret_warn_on)) { \ + if (unlikely(condition)) { \ printk("BUG: warning at %s:%d/%s()\n", __FILE__, \
Your patch is extremely wordwrapped.
Quoted 5 lines · Andrew Morton
Quoted 5 lines
introduced 40% more 2nd level cache miss to tbench workload being run in a loop back mode on a Core 2 machine. I think the introduction of the local variables to
WARN_ONandWARN_ON_ONCE
Quoted 7 lines
Quoted 4 lines
typeof(x)
__ret_warn_on= (x);
typeof(condition)__ret_warn_once= (condition);results in the extra cache misses.
I don't see why it should.
Before the WARN_ON/WARN_ON_ONCE patch, the condition given to WARN_ON/WARN_ON_ONCE is evaluated once and that's it. But after the patch, the condition is stored in a variable and returned later. I think that accessing this variable causes cache misses.
Quoted 3 lines
Perhaps the `static int
__warn_once' is getting put in the same cacheline
as some frequently-modified thing. Perhaps try marking that as__read_mostly?
I'll give that a try to see if it will improve things.
Tim
Quoted 4 lines · Andrew Morton
Perhaps the `static int
__warn_once' is getting put in the same cacheline
as some frequently-modified thing. Perhaps try marking that as__read_mostly?
I've tried marking static int __warn_once as __read_mostly. However, it did not help with reducing the cache miss :(
So I would suggest reversing the "Let WARN_ON/WARN_ON_ONCE return the condition" patch. It has just been added 3 days ago so reversing it should not be a problem.
Tim
Quoted 7 lines · On Tue, 03 Oct 2006 17:09:29 -0700 Tim Chen <[email protected]> wrote:
Quoted 4 lines · Andrew Morton
Perhaps the `static int
__warn_once' is getting put in the same cacheline
as some frequently-modified thing. Perhaps try marking that as__read_mostly?
Quoted 7 lines
I've tried marking static int
__warn_onceas__read_mostly. However, it did not help with reducing the cache miss :(So I would suggest reversing the "Let WARN_ON/WARN_ON_ONCE return the condition" patch. It has just been added 3 days ago so reversing it should not be a problem.
Not yet, please. This is presently a mystery, and we need to work out what's going on.
First up, is it due to WARN_ON, or WARN_ON_ONCE? Please try reverting each one separately.
Let's look at WARN_ON.
Before:
#define WARN_ON(condition) do { \
if (unlikely((condition)!=0)) { \
printk("BUG: warning at %s:%d/%s()\n", __FILE__, __LINE__, __FUNCTION__); \
dump_stack(); \
} \
} while (0)After:
#define WARN_ON(condition) ({ \
typeof(condition) __ret_warn_on = (condition); \
if (unlikely(__ret_warn_on)) { \
printk("BUG: warning at %s:%d/%s()\n", __FILE__, \
__LINE__, __FUNCTION__); \
dump_stack(); \
} \
unlikely(__ret_warn_on); \
})There's no difference, except we return the temporary.
And WARN_ON_ONCE.
Before:
#define WARN_ON_ONCE(condition) \
({ \
static int __warn_once = 1; \
int __ret = 0; \
\
if (unlikely((condition) && __warn_once)) { \
__warn_once = 0; \
WARN_ON(1); \
__ret = 1; \
} \
__ret; \
})After:
#define WARN_ON_ONCE(condition) ({ \
static int __warn_once = 1; \
typeof(condition) __ret_warn_once = (condition);\
\
if (likely(__warn_once)) \
if (WARN_ON(__ret_warn_once)) \
__warn_once = 0; \
unlikely(__ret_warn_once); \
})There are changes here: in the old code we'll avoid reading the static variable. In the new code we'll read the static variable, but we'll avoid evaluating the condition.
Why would that make a measurable difference?
Do you know which WARN_ON (or is it WARN_ON_ONCE?) callsite is causing a problem?
Quoted 14 lines · Andrew Morton
Quoted 10 lines · On Tue, 03 Oct 2006 17:09:29 -0700 Tim Chen <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
Perhaps the `static int
__warn_once' is getting put in the same cacheline
as some frequently-modified thing. Perhaps try marking that as__read_mostly?I've tried marking static int
__warn_onceas__read_mostly. However, it did not help with reducing the cache miss :(
Quoted 8 lines
Quoted 4 lines
So I would suggest reversing the "Let WARN_ON/WARN_ON_ONCE return the condition" patch. It has just been added 3 days ago so reversing it should not be a problem.
Not yet, please. This is presently a mystery, and we need to work out what's going on.
Still, it seems kind of odd to add this IMO. Especially the WARN_ON_ONCE makes the if statement less readable.
if (WARN_ON_ONCE(blah)) {
}What does that mean? Without looking at the implementation, that says the condition is true at most once, when the warning is printed.
What's wrong with adding WARN and WARN_ONCE, and eating the single extra line? You're always telling people to do that with assignments (which I agree with, but are _more_ readable than this WARN_ON_ONCE thing).
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Quoted 3 lines · Andrew Morton
There are changes here: in the old code we'll avoid reading the static variable. In the new code we'll read the static variable, but we'll avoid evaluating the condition.
Tim Chen's patch goes back to the old behaviour. I suspect the cache misses on __warn_once is what he is measuring. If so, the (untested) patch below should reduce the cache misses back to those of the old code.
include/asm-generic/bug.h +3 -2
@@ -45,9 +45,10 @@static int __warn_once =1 ; \typeof (condition) __ret_warn_once = (condition);\ \ -if (likely (__warn_once)) \ -if (WARN_ON (__ret_warn_once)) \ +if (unlikely (__ret_warn_once) && __warn_once) { \ __warn_once =0 ; \ +WARN_ON (1 ); \ + }; \unlikely (__ret_warn_once); \ })
Quoted 16 lines · On Tue, 3 Oct 2006 23:24:27 -0400 Andrew James Wade <[email protected]>…
Quoted 3 lines · Andrew Morton
There are changes here: in the old code we'll avoid reading the static variable. In the new code we'll read the static variable, but we'll avoid evaluating the condition.
Tim Chen's patch goes back to the old behaviour. I suspect the cache misses on
__warn_onceis what he is measuring. If so, the (untested) patch below should reduce the cache misses back to those of the old code.Signed-off-by Andrew Wade <[email protected]>diff -rupN
a/include/asm-generic/bug.hb/include/asm-generic/bug.hinclude/asm-generic/bug.h +0
-0@@ -45,9 +45,10 @@static int __warn_once =1 ; \
include/asm-generic/bug.h line 53
Quoted 10 lines
include/asm-generic/bug.h +3
-2typeof (condition) __ret_warn_once = (condition);\ \ -if (likely (__warn_once)) \ -if (WARN_ON (__ret_warn_once)) \ +if (unlikely (__ret_warn_once) && __warn_once) { \ __warn_once =0 ; \ +WARN_ON (1 ); \ + }; \unlikely (__ret_warn_once); \ })
It might help, but we still don't know what's going on (I think).
I mean, if cache misses against __warn_once were sufficiently high for it to affect performance, then __warn_once would be, err, in cache?
Quoted 4 lines · Andrew Morton
It might help, but we still don't know what's going on (I think).
I mean, if cache misses against
__warn_oncewere sufficiently high for it to affect performance, then__warn_oncewould be, err, in cache?
Yes, of course. I'm embarrassed.
I took a look at the generated code, and GCC is having difficulty optimizing WARN_ON_ONCE. Here is the start of __local_bh_enable:
00000130 <__local_bh_enable>:
130: 83 ec 10 sub $0x10,%esp 133: 8b 15 04 00 00 00 mov 0x4,%edx <-+ 139: 89 e0 mov %esp,%eax | 13b: 25 00 e0 ff ff and $0xffffe000,%eax | !!! 140: 8b 40 14 mov 0x14(%eax),%eax | 143: 25 00 00 ff 0f and $0xfff0000,%eax | 148: 85 d2 test %edx,%edx <-+ 14a: 74 04 je 150 <__local_bh_enable+0x20> 14c: 85 c0 test %eax,%eax 14e: 75 35 jne 185 <__local_bh_enable+0x55> 150: 89 e0 mov %esp,%eax 152: 8b 0d 00 00 00 00 mov 0x0,%ecx <-+ 158: 25 00 e0 ff ff and $0xffffe000,%eax | 15d: 8b 40 14 mov 0x14(%eax),%eax | !!! 160: 25 00 ff 00 00 and $0xff00,%eax | 165: 3d 00 01 00 00 cmp $0x100,%eax | 16a: 0f 94 c0 sete %al | 16d: 85 c9 test %ecx,%ecx <-+ 16f: 0f b6 c0 movzbl %al,%eax 172: 74 04 je 178 <__local_bh_enable+0x48> 174: 85 c0 test %eax,%eax 176: 75 42 jne 1ba <__local_bh_enable+0x8a> 178: b8 00 01 00 00 mov $0x100,%eax 17d: 83 c4 10 add $0x10,%esp 180: e9 fc ff ff ff jmp 181 <__local_bh_enable+0x51> 185: c7 44 24 0c 3e 00 00 movl $0x3e,0xc(%esp)
...
WARN_ON is better, but still has problems:
000011a0 <do_exit>:
11a0: 55 push %ebp
11a1: 57 push %edi
11a2: 56 push %esi
11a3: 53 push %ebx
11a4: 83 ec 30 sub $0x30,%esp
11a7: 89 44 24 18 mov %eax,0x18(%esp)
11ab: 89 e0 mov %esp,%eax
11ad: 25 00 e0 ff ff and $0xffffe000,%eax
11b2: 8b 30 mov (%eax),%esi
11b4: 8b 86 58 0a 00 00 mov 0xa58(%esi),%eax
11ba: 89 44 24 2c mov %eax,0x2c(%esp) <-+
11be: 8b 44 24 2c mov 0x2c(%esp),%eax <-+
11c2: 85 c0 test %eax,%eax | !!!
11c4: 0f 85 65 07 00 00 jne 192f <do_exit+0x78f> |
11ca: 8b 44 24 2c mov 0x2c(%esp),%eax <-+
11ce: 89 e0 mov %esp,%eax
...This is gcc (GCC) 4.0.3 (Ubuntu 4.0.3-1ubuntu5), -O2.
I don't know why this would show up as cache misses, but it does look like a compiler bug is being tickled.
Andrew Wade
Quoted 12 lines · On Wed, 4 Oct 2006 12:47:00 -0400 Andrew James Wade <[email protected]>…
Quoted 4 lines · Andrew Morton
It might help, but we still don't know what's going on (I think).
I mean, if cache misses against
__warn_oncewere sufficiently high for it to affect performance, then__warn_oncewould be, err, in cache?Yes, of course. I'm embarrassed.
I took a look at the generated code, and GCC is having difficulty optimizing
WARN_ON_ONCE. Here is the start of__local_bh_enable:
Quoted 7 lines
00000130 <
__local_bh_enable>:130: 83 ec 10 sub $0x10,%esp 133: 8b 15 04 00 00 00 mov 0x4,%edx <-+ 139: 89 e0 mov %esp,%eax | 13b: 25 00 e0 ff ff and $0xffffe000,%eax | !!! 140: 8b 40 14 mov 0x14(%eax),%eax | 143: 25 00 00 ff 0f and $0xfff0000,%eax |
This is the evaluation of in_irq(): calculate `current', grab current->thread_info->preempt_count.
Normally gcc does manage to CSE the value of current.
Quoted 7 lines
148: 85 d2 test %edx,%edx <-+ 14a: 74 04 je 150 <__local_bh_enable+0x20> 14c: 85 c0 test %eax,%eax 14e: 75 35 jne 185 <__local_bh_enable+0x55> 150: 89 e0 mov %esp,%eax 152: 8b 0d 00 00 00 00 mov 0x0,%ecx <-+ 158: 25 00 e0 ff ff and $0xffffe000,%eax |
but this time it went and reevaluated it.
Quoted 27 lines
15d: 8b 40 14 mov 0x14(%eax),%eax | !!! 160: 25 00 ff 00 00 and $0xff00,%eax | 165: 3d 00 01 00 00 cmp $0x100,%eax | 16a: 0f 94 c0 sete %al | 16d: 85 c9 test %ecx,%ecx <-+ 16f: 0f b6 c0 movzbl %al,%eax 172: 74 04 je 178 <__local_bh_enable+0x48> 174: 85 c0 test %eax,%eax 176: 75 42 jne 1ba <__local_bh_enable+0x8a> 178: b8 00 01 00 00 mov $0x100,%eax 17d: 83 c4 10 add $0x10,%esp 180: e9 fc ff ff ff jmp 181 <__local_bh_enable+0x51> 185: c7 44 24 0c 3e 00 00 movl $0x3e,0xc(%esp)...
WARN_ONis better, but still has problems:
000011a0<do_exit>:11a0: 55 push %ebp 11a1: 57 push %edi 11a2: 56 push %esi 11a3: 53 push %ebx 11a4: 83 ec 30 sub $0x30,%esp 11a7: 89 44 24 18 mov %eax,0x18(%esp) 11ab: 89 e0 mov %esp,%eax 11ad: 25 00 e0 ff ff and $0xffffe000,%eax 11b2: 8b 30 mov (%eax),%esi
Quoted 8 lines
11b4: 8b 86 58 0a 00 00 mov 0xa58(%esi),%eax
11ba: 89 44 24 2c mov %eax,0x2c(%esp) <-+
11be: 8b 44 24 2c mov 0x2c(%esp),%eax <-+
11c2: 85 c0 test %eax,%eax | !!!
11c4: 0f 85 65 07 00 00 jne 192f <do_exit+0x78f> |
11ca: 8b 44 24 2c mov 0x2c(%esp),%eax <-+
11ce: 89 e0 mov %esp,%eax
...No, that's pretty much the same. In the __local_bh_enable case we have the evaluation of in_irq() as well as softirq_count(). do_exit() just has a single WARN_ON().
Quoted 4 lines
This is gcc (GCC) 4.0.3 (Ubuntu 4.0.3-1ubuntu5), -O2.
I don't know why this would show up as cache misses, but it does look like a compiler bug is being tickled.
The code could be better, but nope, there are no additional cache misses there.
Still weird.
Quoted 23 lines · Andrew Morton
Quoted 12 lines · On Wed, 4 Oct 2006 12:47:00 -0400 Andrew James Wade <[email protected]>…
Quoted 4 lines · Andrew Morton
It might help, but we still don't know what's going on (I think).
I mean, if cache misses against
__warn_oncewere sufficiently high for it to affect performance, then__warn_oncewould be, err, in cache?Yes, of course. I'm embarrassed.
I took a look at the generated code, and GCC is having difficulty optimizing
WARN_ON_ONCE. Here is the start of__local_bh_enable:Quoted 7 lines
00000130 <
__local_bh_enable>:130: 83 ec 10 sub $0x10,%esp 133: 8b 15 04 00 00 00 mov 0x4,%edx <-+ 139: 89 e0 mov %esp,%eax | 13b: 25 00 e0 ff ff and $0xffffe000,%eax | !!! 140: 8b 40 14 mov 0x14(%eax),%eax | 143: 25 00 00 ff 0f and $0xfff0000,%eax |
Quoted 2 lines
This is the evaluation of
in_irq(): calculate `current', grabcurrent->thread_info->preempt_count.
Actually I was confusing "mov 0x4,%edx" with "mov $0x4,%edx". That code's fine (albeit unlinked). There are stupid inefficiencies in some of the other code, but nothing really stood out at me in __local_bh_enable, _local_bh_enable, or local_bh_Enable.
(from earlier)
Quoted 2 lines
Perhaps the `static int
__warn_once' is getting put in the same cacheline as some frequently-modified thing.
hmm:
00000460 l O .data 00000044 task_exit_notifier 000004c0 l O .data 0000002c task_free_notifier 000004ec l O .data 00000004 warnlimit.15904 000004f0 l O .data 00000004 firsttime.15774 000004f4 l O .data 00000004 __warn_once.15180 000004f8 l O .data 00000004 __warn_once.15174 000004fc l O .data 00000004 __warn_once.15213 00000500 l O .data 00000004 __warn_once.15207 00000504 l O .data 00000004 __warn_once.15145 00000508 l O .data 00000004 __warn_once.15309 0000050c l O .data 00000004 __warn_once.15256 00000510 l O .data 00000004 __warn_once.15250 000005a0 l O .data 0000006c proc_iomem_operations
(extracted from objdump -t kernel/built-in.o)
warnlimit and firsttime are fine, and proc_iomem_operations is presumably fine as well. But I'm not so sure task_free_notifier is infrequently modified. But that's just my .config and I'm out of my depth.
Andrew Wade
Quoted 19 lines · On Thu, 5 Oct 2006 04:13:07 -0400 Andrew James Wade <[email protected]>…
(from earlier)
Quoted 2 lines
Perhaps the `static int
__warn_once' is getting put in the same cacheline as some frequently-modified thing.hmm:
00000460 l O .data 00000044 task_exit_notifier 000004c0 l O .data 0000002c task_free_notifier 000004ec l O .data 00000004 warnlimit.15904 000004f0 l O .data 00000004 firsttime.15774 000004f4 l O .data 00000004 __warn_once.15180 000004f8 l O .data 00000004 __warn_once.15174 000004fc l O .data 00000004 __warn_once.15213 00000500 l O .data 00000004 __warn_once.15207 00000504 l O .data 00000004 __warn_once.15145 00000508 l O .data 00000004 __warn_once.15309 0000050c l O .data 00000004 __warn_once.15256 00000510 l O .data 00000004 __warn_once.15250 000005a0 l O .data 0000006c proc_iomem_operations
Quoted 1 line
(extracted from objdump -t
kernel/built-in.o)
That all looks OK (by sheer luck).
Well. What's the cache line size on that machine? Every exit() will cause a down_read() on task_exit_notifier's lock which might affect things. And I think you snipped the above list a bit short (depending on that line size).
But still, we know that moving those things into __read_mostly didn't fix it, yes?
Quoted 8 lines · Andrew Morton
Quoted 1 line · On Thu, 5 Oct 2006 04:13:07 -0400 Andrew James Wade <[email protected]>…
(...)
That all looks OK (by sheer luck).
Quoted 8 lines
Well. What's the cache line size on that machine? Every
exit()will cause adown_read()on task_exit_notifier's lock which might affect things. And I think you snipped the above list a bit short (depending on that line size).But still, we know that moving those things into
__read_mostlydidn't fix it, yes?
No. To my knowledge Tim Chen hasn't tried __read_mostly, and I have not attempted to replicate the test case. (I only have a uniprocessor machine.) Core 2 machines have a cache line size of 64 bytes, but Tim Chen is likely using a different kernel/.config than I am so my objdump isn't definitive.
Tim, perhaps you can try the __read_mostly marking as Andrew suggests?
upN a/include/asm-generic/bug.h b/include/asm-generic/bug.h
include/asm-generic/bug.h +1 -1
@@ -42,7 +42,7 @@#endif #define WARN_ON_ONCE (condition) ({ \ - static int __warn_once = 1; \ + static int __warn_once __read_mostly = 1; \ typeof(condition) __ret_warn_once = (condition);\ \ if (likely(__warn_once)) \
Quoted 18 lines · Andrew James Wade
Quoted 8 lines · Andrew Morton
Quoted 1 line · On Thu, 5 Oct 2006 04:13:07 -0400 Andrew James Wade <[email protected]>…
(...)
That all looks OK (by sheer luck).
Quoted 8 lines
Well. What's the cache line size on that machine? Every
exit()will cause adown_read()on task_exit_notifier's lock which might affect things. And I think you snipped the above list a bit short (depending on that line size).But still, we know that moving those things into
__read_mostlydidn't fix it, yes?
Quoted 8 lines
No. To my knowledge Tim Chen hasn't tried
__read_mostly, and I have not attempted to replicate the test case. (I only have a uniprocessor machine.) Core 2 machines have a cache line size of 64 bytes, but Tim Chen is likely using a differentkernel/.configthan I am so my objdump isn't definitive.Tim, perhaps you can try the
__read_mostlymarking as Andrew suggests?
I have run the workload with __read_mostly marking. But it didn't make a difference. By the way, the cache line size of my machine is 64 bytes.
Thanks.
Tim