New: CPU is waking up too often
http://bugzilla.kernel.org/show_bug.cgi?id=9745
Summary: CPU is waking up too often
Product: Power Management
Version: 2.5
KernelVersion: 2.6.24-rc7
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: cpufreq
AssignedTo: [email protected]
ReportedBy: [email protected]
Latest working kernel version: 2.6.22
Earliest failing kernel version: 2.6.24-rc7
Distribution: Debian GNU/Linux Sid (Unstable)
Hardware Environment:
CPU: Mobile AMD Sempron(tm) Processor 3300+
System is a Fujitsu Siemens K7610 mobile system.
lspci:
00:00.0 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge 00:00.1 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge 00:00.2 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge 00:00.3 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge 00:00.4 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge 00:00.7 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge 00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI bridge [K8T800/K8T890 South]
00:09.0 CardBus bridge: ENE Technology Inc CB1410 Cardbus Controller (rev 01) 00:0a.0 Ethernet controller: Atheros Communications, Inc. AR5005G 802.11abg NIC (rev 01)
00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 80)
00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 80)
00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 80)
00:10.3 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 82) 00:11.0 ISA bridge: VIA Technologies, Inc. VT8235 ISA Bridge 00:11.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06) 00:11.5 Multimedia audio controller: VIA Technologies, Inc. VT8233/A/8235/8237 AC97 Audio Controller (rev 50)
00:11.6 Communication controller: VIA Technologies, Inc. AC'97 Modem Controller (rev 80)
00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 74) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: VIA Technologies, Inc. S3 Unichrome Pro VGA Adapter (rev 01)
Software Environment:
Debian Sid, Xorg with openchrome drivers from openchrome.org, Software typical for a Desktop system
Problem Description:
With 2.6.22 according to powertop my system is waking up between 55times a second and 200 times a second with an average at about 80 times a second. When I now boot 2.6.24rc7 this number increases to an average of 2500 wakeups/second with (seldom but happening) tops above the 10000 wake-ups/second mark (to top values of 28000 wake-ups/second!!). This way the CPU is always running at C0 (99,2%).
Top causes for wakeups (according to powertop):
42.6% ( 17.6) <interrupt> : extra timer interrupt 21.8% ( 9.0) <interrupt> : ide1 4.8% ( 2.0) <interrupt> : acpi 4.1% ( 1.7) xfsbufd : schedule_timeout (process_timeout) 3.4% ( 1.4) Xorg : do_setitimer (it_real_fn) 2.4% ( 1.0) dhcdbd : schedule_timeout (process_timeout) 2.4% ( 1.0) nm-applet : schedule_timeout (process_timeout) 1.5% ( 0.6) <kernel core> : neigh_table_init_no_netlink (neigh_periodic_timer) 1.2% ( 0.5) ifconfig : __netdev_watchdog_up (dev_watchdog) 1.2% ( 0.5) hald-addon-stor : schedule_timeout (process_timeout) 1.2% ( 0.5) NetworkManager : schedule_timeout (process_timeout) 1.2% ( 0.5) gnome-panel : schedule_timeout (process_timeout) 1.2% ( 0.5) <kernel module> : neigh_table_init_no_netlink (neigh_periodic_timer) 1.2% ( 0.5) <kernel core> : queue_delayed_work_on (delayed_work_timer_fn) 0.7% ( 0.3) <interrupt> : eth0 0.7% ( 0.3) <kernel core> : neigh_update (neigh_timer_handler) 0.7% ( 0.3) <kernel module> : register_sound_midi (irlmp_discovery_timer_expired) 0.7% ( 0.3) gnome-cups-icon : schedule_timeout (process_timeout) 0.7% ( 0.3) gnome-power-man : schedule_timeout (process_timeout) 0.5% ( 0.2) <interrupt> : ide0 0.5% ( 0.2) mapping-daemon : schedule_timeout (process_timeout) 0.5% ( 0.2) gnome-cups-icon : futex_wait (hrtimer_wakeup) 0.5% ( 0.2) init : schedule_timeout (process_timeout) 0.5% ( 0.2) cupsd : schedule_timeout (process_timeout) 0.5% ( 0.2) nautilus : schedule_timeout (process_timeout) 0.5% ( 0.2) gnome-settings- : schedule_timeout (process_timeout) 0.5% ( 0.2) <kernel module> : ide_do_rw_disk (ledtrig_ide_timerfunc) 0.2% ( 0.1) urxvtd : schedule_timeout (process_timeout) 0.2% ( 0.1) syslogd : do_setitimer (it_real_fn) 0.2% ( 0.1) gconfd-2 : schedule_timeout (process_timeout) 0.2% ( 0.1) <kernel core> : page_writeback_init (wb_timer_fn)
Steps to reproduce:
Boot system with 2.6.24-rc7
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
http://bugzilla.kernel.org/show_bug.cgi?id=9745
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected], | |[email protected], [email protected]
------- Comment #1 from [email protected] 2008-01-14 05:12 ------- This has nothing to do with cpufreq, but with timers. I also saw a similar bugreport somewhere else.
I cannot help here (but would like to learn how to debug this...), if you have some luck Ingo Molnar or Thomas Gleixner can give you some hints how to debug this.
I expect this has been introduced (just a guess, git should help) by the clockevents/highres/tickless timer patches. If you could prove this with git, you should get some help from the author of the patchsets...
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #2 from [email protected] 2008-01-14 05:32 ------- Patrick, can you please upload your .config, a full boot log and the output of /proc/timer_list ?
Thanks,
tglx
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #3 from [email protected] 2008-01-14 05:45 ------- Created an attachment (id=14450)
--> (http://bugzilla.kernel.org/attachment.cgi?id=14450&action=view)
kernel config
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #4 from [email protected] 2008-01-14 05:45 ------- Created an attachment (id=14451)
--> (http://bugzilla.kernel.org/attachment.cgi?id=14451&action=view)
bootlog
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #5 from [email protected] 2008-01-14 05:46 ------- Created an attachment (id=14452)
--> (http://bugzilla.kernel.org/attachment.cgi?id=14452&action=view)
output of cat /proc/timer_list
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #6 from [email protected] 2008-01-14 05:48 ------- Thomas Renninger,
yeah, you are right. It first looked like cpufreq scaling wouldn't happen and I didn't really knew which category would fit. Neither do I do really know now. (Subsections of Timers is quiet confusing for me).
Thomas Gleixner,
I'm not exactly sure what do you mean by "full boot log"? A full boot log of the problematic kernel or a log including boots of working kernels? I've added the part of /var/log/kern.log where I booted the kernel the last time.
Hope that helps.
Regards,
Patrick
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #7 from [email protected] 2008-01-14 06:01 -------
Quoted 3 lines
I'm not exactly sure what do you mean by "full boot log"? A full boot log of the problematic kernel or a log including boots of working kernels? I've added the part of
/var/log/kern.logwhere I booted the kernel the last time.
boot log of the problematic kernel. kern.log is fine. The output of dmesg is easier to read though.
According to the output of /proc/timer_list there is no fast rearming timer, so it's likely to be something else.
Can you please do the following:
# cat /proc/interrupts >irqs.txt; sleep 10; cat /proc/interrupts >>irqs.txt
and upload irqs.txt
Thanks,
tglx
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #8 from [email protected] 2008-01-14 06:06 ------- Created an attachment (id=14453)
--> (http://bugzilla.kernel.org/attachment.cgi?id=14453&action=view)
output file of at /proc/interrupts >irqs.txt; sleep 10; cat /proc/interrupts
Quoted 1 line
Quoted 1 line
irqs.txt
Here you are.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
------- Comment #9 from [email protected] 2008-01-14 06:27 ------- Hmm. The deltas are:
0: 742
14: 4
15: 90
18: 1
LOC: 724
-----------
1561
1561/10 ~= 156 interrupts/sec
Not amazing, but not really in the range of 2500+.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #10 from [email protected] 2008-01-14 06:38 ------- Hmm. So probably this really is a powertop issue. However why isn't it triggered by 2.6.22? Btw. you can see [1] for an example on a powertop output, where the value is beyond 16000.
[1] http://just-imho.net/16000wakeupspersecond.png
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #11 from [email protected] 2008-01-14 06:46 ------- Maybe this is related to:
https://bugzilla.novell.com/show_bug.cgi?id=335086
and
https://bugzilla.novell.com/show_bug.cgi?id=334170 (there only after s2ram case)
and
https://bugs.launchpad.net/ubuntu/+bug/145377
there also powertop shows extrem interrupt activity and blacklisting yenta_socket driver helps.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #12 from [email protected] 2008-01-14 06:51 ------- Yeah, indeed it seems related to 335086 @ Novell Bugzilla. If I do rmmod processor (i need to force it but that just besides) then the situation is relative normal with an average of 100-200 wakeups. Thats better then 2500, but still worse as with 2.6.22 (there the average is 80 and sometimes less then 50)
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #13 from [email protected] 2008-01-14 07:01 ------- Then a workaround for you should be processor.max_cstate=2 boot param, if this does not help processor.max_cstate=1.
But it would be great to know more about the root cause (and get this fixed to fully support the C-states).
Could you try to keep processor driver without max_cstate workaround and blacklist yenta_socket, e.g. add a line:
install yenta_socket /bin/echo "Yenta socket blacklisted, it will not get loaded"
to /etc/modprobe.conf
I already asked Arjan about the yenta_socket problem and he said it would be the yenta driver's fault... As there seem to be quite some people affected, it would be great to bring them together and find out a bit more about the problem (Did this machine ever worked correctly with C-states? If yes, one might want to start a git bisect...).
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #14 from [email protected] 2008-01-14 07:19 ------- Thomas Renninger,
the current workaround for me is to use 2.6.22 which seems not to be affected by the problem. However in my case it does help nothing to blacklist yenta_socket.
Well, I haven't yet tried the max_cstate thing as it requires me to reboot again.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #15 from [email protected] 2008-01-14 07:40 -------
Quoted 5 lines
the current workaround for me is to use 2.6.22 which seems not to be affected by the problem. However in my case it does help nothing to blacklist
yenta_socket.
Well, I haven't yet tried themax_cstatething as it requires me to reboot again.
If you decide to reboot, could you please try to capture something, which shows us the large number of interrupts (via the shell command I gave you before).
I looked at your .config again. That's the .22 one. Is the .23 significantly different? Can you please upload it as well.
Thanks,
tglx
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #16 from [email protected] 2008-01-14 08:42 ------- (In reply to comment #15)
Quoted 3 lines
If you decide to reboot, could you please try to capture something, which shows us the large number of interrupts (via the shell command I gave you before).
You mean With those above stated workarounds?
Quoted 2 lines
I looked at your .config again. That's the .22 one. Is the .23 significantly different? Can you please upload it as well.
Oh. You are right. I'll attach the one for .24. Sorry, my mistake. However I don't have a .23 version ;)
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #17 from [email protected] 2008-01-14 08:43 ------- Created an attachment (id=14454)
--> (http://bugzilla.kernel.org/attachment.cgi?id=14454&action=view)
kernel config (the right one :o)
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #14450|0 |1
is obsolete| |
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #18 from [email protected] 2008-01-14 23:58 ------- Okay I did some additional testing. With the cstate options from above I experience the following result (NOK = NOT OK)
max_cstate = 2 = NOKmax_cstate = 1 = NOK (increased (!) value in powertop!)
I have created files with the cat /proc interupts.. command again and will attach them in a minute.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #19 from [email protected] 2008-01-14 23:59 ------- Created an attachment (id=14460)
--> (http://bugzilla.kernel.org/attachment.cgi?id=14460&action=view)
ircqs with max_cstate=1
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #20 from [email protected] 2008-01-15 00:00 ------- Created an attachment (id=14461)
--> (http://bugzilla.kernel.org/attachment.cgi?id=14461&action=view)
irqs with max_cstate=2
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #21 from [email protected] 2008-01-15 00:08 ------- BTW. interesting: If I do cat /proc.. on a 2.6.22 system the interrupts/sec is even higher. So is this probably really a bug in powertop?
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
------- Comment #22 from [email protected] 2008-01-15 06:35 ------- max_cstate=1 is a red herring btw
the kernel doesn't count wakeups for C1 so powertop doesnt get to see them either; that makes it a counting game not something reflecting what is actually happening on the system
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
------- Comment #23 from [email protected] 2008-06-03 22:28 ------- What is the status on this problem, still exists with recent kernel? Thanks.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
[Bug 9745] CPU is waking up too often
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=9745
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
AssignedTo|[email protected] |platform_i386@kernel- | |bugs.osdl.org Component|cpufreq |i386 Product|Power Management |Platform Specific/Hardware
Signature · 4 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee. You are the assignee for the bug, or are watching the assignee.