Lots of "rescheduling IPIs" in powertop
Hi,
Recap: powertop shows between 200-400 wakeups/second with the description "<kernel IPI>: Rescheduling interrupts" when all processors have load (e.g. I need to run two busy-loops on my 2-CPU system for this to show up).
The bisect resulted in this commit:
commit 0c07ee38c9d4eb081758f5ad14bbffa7197e1aec
Author: Andi Kleen <[email protected]> Date: Wed Jan 30 13:33:16 2008 +0100
x86: use the correct cpuid method to detect MWAIT support for C states
Previously there was a AMD specific quirk to handle the case of AMD Fam10h MWAIT not supporting any C states. But it turns out that CPUID already has ways to detectly detect that without using special quirks.
The new code simply checks if MWAIT supports at least C1 and doesn't use it if it doesn't. No more vendor specific code.
Note this is does not simply clear MWAIT because MWAIT can be still useful even without C states.
Credit goes to Ben Serebrin for pointing out the (nearly) obvious.
I was not able to revert this without conflicts, so I will leave that part to you. Here is the bisect log in case it is useful:
git-bisect start 'v2.6.26-rc1' 'v2.6.24' git-bisect bad 3f85d63ea4ff922f6abdb509f4aaf6993b3273a3 git-bisect bad f77bc6a420eba845605ff1d53cadf55f94c5e8b7 git-bisect good c76c04758b8fd24a1c38b19742e3437e954e945b git-bisect good 0ba6c33bcddc64a54b5f1c25a696c4767dc76292 git-bisect good fde1b3fa947c2512e3715962ebb1d3a6a9b9bb7d git-bisect bad d690b2afd5a7a02816386aa704c8c0b1aca8d2de git-bisect good 773221f46f82dc2f277dacc331d9d2ef2c690cb6 git-bisect bad f8f76481bc2803aea03ff213c7e1405b53f7e488 git-bisect bad 9042219cd8d43b81322b826d463dd6e52acae6cf git-bisect bad ca74a6f84e68b44867022f4a4f3ec17c087c864e git-bisect good 064954761254ef17fae2b84fb5a034d48a769143 git-bisect good 30d432dfab2bcfd021d352e2058fae6b9405caeb git-bisect bad 27cc2a812eb504f4aadff5baa862da715fb0f886 git-bisect bad 0c07ee38c9d4eb081758f5ad14bbffa7197e1aec
This initially showed up on my 2.6.24.5-85.fc8 (Fedora) kernel.
I have two different systems (one laptop, one desktop) which both show the same symptoms, please do tell me if you need any more info.
Vegard
Quoted 5 lines · Vegard Nossum
This initially showed up on my 2.6.24.5-85.fc8 (Fedora) kernel.
I have two different systems (one laptop, one desktop) which both show the same symptoms, please do tell me if you need any more info.
What CPUs do they have? And does the problem really go away when you boot with idle=mwait ? And what does the following program output?
#include <asm/msr.h>
main()
{
printf("%x\n", cpuid_edx(5));
}-Andi
Quoted 10 lines · Andi Kleen
Quoted 5 lines · Vegard Nossum
This initially showed up on my 2.6.24.5-85.fc8 (Fedora) kernel.
I have two different systems (one laptop, one desktop) which both show the same symptoms, please do tell me if you need any more info.
What CPUs do they have? And does the problem really go away when you boot with idle=mwait ? And what does the following program output?
Yes, the problem goes away with idle=mwait.
The desktop is a P4:
processor : 0vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Intel(R) Pentium(R) 4 CPU 3.00GHz
stepping : 5
cpu MHz : 2992.624
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1fdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 6
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pebs bts pni monitor ds_cpl est tm2 cid cx16 xtpr lahf_lm
bogomips : 5990.81
clflush size : 64
(similar for processor 1)
# msr 0
The laptop is a Pentium Dual-Core:
processor : 0vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Pentium(R) Dual CPU T2310 @ 1.46GHz
stepping : 13
cpu MHz : 800.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2fdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc arch_perfmon pebs bts pni monitor ds_cpl est tm2 ssse3 cx16 xtpr lahf_lm
bogomips : 2930.23
clflush size : 64
(similar for processor 1)
# ./msr 1110
Vegard
Signature · 4 lines
"The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer's own creation." -- E. W. Dijkstra, EWD1036
Quoted 19 lines
The desktop is a P4:
processor : 0
vendor_id: GenuineIntel
cpu family : 15
model : 6
model name : Intel(R) Pentium(R) 4 CPU 3.00GHz
stepping : 5
cpu MHz : 2992.624
cache size : 2048 KBflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lmconstant_tscpebs bts pni monitords_cplest tm2 cid cx16 xtprlahf_lm
bogomips : 5990.81
clflush size : 64
Quoted 4 lines
(similar for processor 1)
# msr 0
Ok the CPU reports it doesn't support any C states in MWAIT. If that is correct then it would be correct to not use MWAIT idle and might actually save more power to not use it.
I don't know if that's true or not. Do you have a power meter perhaps? If yes can you measure if there's a difference between mwait=idle / default on your box when it is idle?
[cc Arjan he might now if that CPU is supposed to support C1 in MWAIT]
Quoted 11 lines
The laptop is a Pentium Dual-Core:
processor : 0
vendor_id: GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Pentium(R) Dual CPU T2310 @ 1.46GHz
stepping : 13
cpu MHz : 800.000
cache size : 1024 KB
...
Quoted 11 lines
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lmconstant_tscarch_perfmonpebs bts pni monitords_cplest tm2 ssse3 cx16 xtprlahf_lm
bogomips : 2930.23
clflush size : 64(similar for processor 1)
# ./msr 1110
CPU reports it supports C1/C2/C3. Are you sure there is a difference on that box? The code should have kept using MWAIT because it checks C1. Please double check.
-Andi
Quoted 4 lines · On Tue, 13 May 2008 23:19:47 +0200 Andi Kleen <[email protected]> wrote:
Ok the CPU reports it doesn't support any C states in MWAIT. If that is correct then it would be correct to not use MWAIT idle and might actually save more power to not use it.
what does the current SVN powertop say on this cpu?
Quoted 6 lines
I don't know if that's true or not. Do you have a power meter perhaps? If yes can you measure if there's a difference between mwait=idle / default on your box when it is idle?
[cc Arjan he might now if that CPU is supposed to support C1 in MWAIT]
I wasn't aware that P4's supported mwait in this way; I thought it was core and later.
Quoted 3 lines
CPU reports it supports C1/C2/C3. Are you sure there is a difference on that box? The code should have kept using MWAIT because it checks C1. Please double check.
The check is .. dubious I suspect... because the cpuid bits are not actually the prime source of information, the BIOS is.
If the bios says mwait is usable, we need to use it with the values IT gives us.
Arjan van de Ven wrote:
[cc Len]
Quoted 9 lines
Quoted 3 lines · On Tue, 13 May 2008 23:19:47 +0200 Andi Kleen <[email protected]> wrote:
Ok the CPU reports it doesn't support any C states in MWAIT. If that is correct then it would be correct to not use MWAIT idle and might actually save more power to not use it.
what does the current SVN powertop say on this cpu?
Quoted 8 lines
Quoted 5 lines
I don't know if that's true or not. Do you have a power meter perhaps? If yes can you measure if there's a difference between mwait=idle / default on your box when it is idle?
[cc Arjan he might now if that CPU is supposed to support C1 in MWAIT]
I wasn't aware that P4's supported mwait in this way; I thought it was core and later.
Not even C1? I generally consider MWAIT without C1 to be unusable.
Anyways if C1 doesn't work then it would be correct to not use MWAIT.
Quoted 6 lines
Quoted 4 lines
CPU reports it supports C1/C2/C3. Are you sure there is a difference on that box? The code should have kept using MWAIT because it checks C1. Please double check.
The check is .. dubious I suspect...
I don't think so.
Quoted 2 lines
because the cpuid bits are not
actually the prime source of information, the BIOS is.
Hmmm? What BIOS information are you refering to?
Normally it's my experience that CPUID is more reliable than the BIOS.
Quoted 2 lines
If the bios says mwait is usable, we need to use it with the values IT gives us.
At least to my knowledge the ACPI FADT just says what C states are available, not if they are implemented with MWAIT or using IO ports.
-Andi
On Wed, 14 May 2008 08:58:05 +0200
Quoted 5 lines
Quoted 2 lines
If the bios says mwait is usable, we need to use it with the values IT gives us.
At least to my knowledge the ACPI FADT just says what C states are available, not if they are implemented with MWAIT or using IO ports.
_CST does; which is what the mwait C-state selection code uses.
(and we've seen quirky systems where C1 mwait didn't work, SMM did something wrong or something, because the other OS doesn't seem to use it for that, so making it the default is not really an option)
Quoted 10 lines · Arjan van de Ven
On Wed, 14 May 2008 08:58:05 +0200
Quoted 4 lines
Quoted 2 lines
If the bios says mwait is usable, we need to use it with the values IT gives us.
At least to my knowledge the ACPI FADT just says what C states are available, not if they are implemented with MWAIT or using IO ports.
_CSTdoes; which is what the mwait C-state selection code uses.(and we've seen quirky systems where C1 mwait didn't work, SMM did something wrong or something,
What do you mean with didn't work? And they did not report it in CPUID? And are they wide spread? I haven't heard of any such problems before.
Even the P4 problem is not really that serious btw.
because the other OS doesn't seem to use
Quoted 1 line
it for that, so making it the default is not really an option)
But it has been default for practically forever. It is strange
if you declare Linux default behavior for years suddenly as "not really an option"
I think it's a sane default. And MWAIT is a valuable optimization.
I don't think we should stop sanely architected interfaces on the first BIOS quirk someone reports. I'm sure for nearly every feature there's some broken BIOS out there that gets it wrong somehow. If we did this in general soon we couldn't use anything.
-Andi
Quoted 29 lines · Andi Kleen
Quoted 20 lines
The desktop is a P4:
processor : 0
vendor_id: GenuineIntel
cpu family : 15
model : 6
model name : Intel(R) Pentium(R) 4 CPU 3.00GHz
stepping : 5
cpu MHz : 2992.624
cache size : 2048 KBflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lmconstant_tscpebs bts pni monitords_cplest tm2 cid cx16 xtprlahf_lm
bogomips : 5990.81
clflush size : 64Quoted 4 lines
(similar for processor 1)
# msr 0Ok the CPU reports it doesn't support any C states in MWAIT. If that is correct then it would be correct to not use MWAIT idle and might actually save more power to not use it.
Quoted 5 lines
I don't know if that's true or not. Do you have a power meter perhaps? If yes can you measure if there's a difference between mwait=idle / default on your box when it is idle?
[cc Arjan he might now if that CPU is supposed to support C1 in MWAIT]
No, sorry, no power meter :-/
Quoted 19 lines
Quoted 10 lines
The laptop is a Pentium Dual-Core:
processor : 0
vendor_id: GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Pentium(R) Dual CPU T2310 @ 1.46GHz
stepping : 13
cpu MHz : 800.000
cache size : 1024 KB...
Quoted 7 lines
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lmconstant_tscarch_perfmonpebs bts pni monitords_cplest tm2 ssse3 cx16 xtprlahf_lm
bogomips : 2930.23
clflush size : 64
Quoted 8 lines
Quoted 4 lines
(similar for processor 1)
# ./msr 1110CPU reports it supports C1/C2/C3. Are you sure there is a difference on that box? The code should have kept using MWAIT because it checks C1. Please double check.
Yes, sorry, you are correct. I tested the idle=mwait only on the desktop machine (P4, msr = 0), and it improved the IPI problem. (I even rechecked right now, and it really does.)
Now I tested it on the laptop as well, and here it makes no difference.
Vegard
Signature · 4 lines
"The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer's own creation." -- E. W. Dijkstra, EWD1036
Quoted 6 lines · Vegard Nossum
Hi,
Recap: powertop shows between 200-400 wakeups/second with the description "<kernel IPI>: Rescheduling interrupts" when all processors have load (e.g. I need to run two busy-loops on my 2-CPU system for this to show up).
ok, could you try the fix below? It was a mistake to make mwait use dependent on power considerations - on a desktop CPU it is unlikely to use more power than a simple HLT - and the IPIs are extra scheduling latency and extra power used.
Ingo
-------------->
Subject: x86: remove mwait C-state capability
From: Ingo Molnar <[email protected]>
Date: Wed May 14 08:47:40 CEST 2008
--- arch/x86/kernel/process.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
arch/x86/kernel/process.c +1 -10
@@ -99,15 +99,6 @@ static void mwait_idle(void)local_irq_enable (); } - -static int __cpuinit mwait_usable (const struct cpuinfo_x86 *c) -{ -if (force_mwait) -return 1 ; -/* Any C1 states supported? */ -return c->cpuid_level >=5 && ((cpuid_edx (5 ) >>4 ) &0xf ) >0 ; -} -/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the
@@ -131,7 +122,7 @@ void __cpuinit select_idle_routine(const " performance may degrade.\n"); }#endif -if (cpu_has (c,X86_FEATURE_MWAIT ) &&mwait_usable (c)) { +if (cpu_has (c,X86_FEATURE_MWAIT )) {/* * Skip, if setup has overridden idle. * One CPU supports mwait => All CPUs supports mwait
Hi,
Quoted 10 lines · Ingo Molnar
Quoted 6 lines · Vegard Nossum
Hi,
Recap: powertop shows between 200-400 wakeups/second with the description "<kernel IPI>: Rescheduling interrupts" when all processors have load (e.g. I need to run two busy-loops on my 2-CPU system for this to show up).
Quoted 4 lines
ok, could you try the fix below? It was a mistake to make mwait use dependent on power considerations - on a desktop CPU it is unlikely to use more power than a simple HLT - and the IPIs are extra scheduling latency and extra power used.
This fixes it for the desktop machine at least. I guess I should try on the laptop as well, but that will have to wait. Time for school :-)
Thanks.
Vegard
Quoted 41 lines
-------------->
Subject: x86: remove mwait C-state capability
From: Ingo Molnar <[email protected]>
Date: Wed May 14 08:47:40 CEST 2008Signed-off-by Ingo Molnar <[email protected]>---
arch/x86/kernel/process.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)Index:
linux/arch/x86/kernel/process.c
===================================================================
---linux.orig/arch/x86/kernel/process.c
+++linux/arch/x86/kernel/process.c
@@ -99,15 +99,6 @@ static voidmwait_idle(void)local_irq_enable();}
-
-static int__cpuinitmwait_usable(const struct cpuinfo_x86 *c)-{
- if (force_mwait)
- return 1;
- /* Any C1 states supported? */
- returnc->cpuid_level>= 5 && ((cpuid_edx(5)>> 4) &0xf) > 0;
-}
-
/*
* On SMP it's slightly faster (but much more power-consuming!)
* to poll the ->work.need_resched flag instead of waiting for the
@@ -131,7 +122,7 @@ void__cpuinitselect_idle_routine(const" performance may degrade.\n"); }#endif
- if (cpu_has(c, X86_FEATURE_MWAIT)&&mwait_usable(c)) {
+ if (cpu_has(c, X86_FEATURE_MWAIT)) {
/*
* Skip, if setup has overridden idle.
* One CPU supports mwait => All CPUs supports mwait
Signature · 4 lines
"The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer's own creation." -- E. W. Dijkstra, EWD1036
Quoted 10 lines · Ingo Molnar
Quoted 6 lines · Vegard Nossum
Hi,
Recap: powertop shows between 200-400 wakeups/second with the description "<kernel IPI>: Rescheduling interrupts" when all processors have load (e.g. I need to run two busy-loops on my 2-CPU system for this to show up).
Quoted 4 lines
ok, could you try the fix below? It was a mistake to make mwait use dependent on power considerations - on a desktop CPU it is unlikely to use more power than a simple HLT - and the IPIs are extra scheduling latency and extra power used.
It depends on the CPU. For AMD CPUs that support MWAIT this is wrong. Family 0x10 and 0x11 CPUs will enter C1 on HLT. Powersavings then depend on a clock divisor and current Pstate of the core.
If all cores of a processor are in halt state (C1) the processor can enter the C1E (C1 enhanced) state. If mwait is used this will never happen.
Thus HLT saves more power than MWAIT here.
It might be best to switch off the mwait flag for these AMD CPU families like it was introduced with commit f039b754714a422959027cb18bb33760eb8153f0 (x86: Don't use MWAIT on AMD Family 10)
Andreas
Quoted 3 lines · Andreas Herrmann
It depends on the CPU. For AMD CPUs that support MWAIT this is wrong. Family
0x10and0x11CPUs will enter C1 on HLT. Powersavings then
^ not
Quoted 4 lines
It might be best to switch off the mwait flag for these AMD CPU families like it was introduced with commit
f039b754714a422959027cb18bb33760eb8153f0(x86: Don't use MWAIT on AMD Family 10)
Then you have to special case everything again. We still need to work out if the P4 is even correct here or not, but if it's not i would rather quirk the cpuid reporting on it.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 3 lines · Andreas Herrmann
It depends on the CPU. For AMD CPUs that support MWAIT this is wrong. Family
0x10and0x11CPUs will enter C1 on HLT. Powersavings then^ not
Not sure what you meant by your comment.
Maybe you should re-read the paragraph.
It's as simple as that:
If OS executes Halt the core enters C1.
The core exits C1 if an interrupt is received.
Quoted 8 lines
Quoted 4 lines
It might be best to switch off the mwait flag for these AMD CPU families like it was introduced with commit
f039b754714a422959027cb18bb33760eb8153f0(x86: Don't use MWAIT on AMD Family 10)Then you have to special case everything again. We still need to work out if the P4 is even correct here or not, but if it's not i would rather quirk the cpuid reporting on it.
I just want to ensure that for AMD family 0x10 and 0x11 halt and not mwait is executed when the core is idle.
Andreas
Quoted 18 lines · Andreas Herrmann
Quoted 9 lines
Quoted 4 lines
Recap: powertop shows between 200-400 wakeups/second with the description "<kernel IPI>: Rescheduling interrupts" when all processors have load (e.g. I need to run two busy-loops on my 2-CPU system for this to show up).
ok, could you try the fix below? It was a mistake to make mwait use dependent on power considerations - on a desktop CPU it is unlikely to use more power than a simple HLT - and the IPIs are extra scheduling latency and extra power used.
It depends on the CPU. For AMD CPUs that support MWAIT this is wrong. Family
0x10and0x11CPUs will enter C1 on HLT. Powersavings then depend on a clock divisor and current Pstate of the core.If all cores of a processor are in halt state (C1) the processor can enter the C1E (C1 enhanced) state. If mwait is used this will never happen.
Quoted 6 lines
Thus HLT saves more power than MWAIT here.
It might be best to switch off the mwait flag for these AMD CPU families like it was introduced with commit
f039b754714a422959027cb18bb33760eb8153f0(x86: Don't use MWAIT on AMD Family 10)
agreed, that looks like the cleanest solution. Could you please send a patch for that, ontop of my patch?
Ingo