2.6.33 & 2.6.34-rc6: Huge number of Load Balancing tick wakeups.
Hi all.
I'm seeing a huge number of load balancing tick wakeups, with both 2.6.33 and current git (f1c448e0a9e99c76f4ece368714fb35a40a8daba).
This has been reported as a bug in some distros:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281
Regards,
Nigel
PowerTOP 1.12 (C) 2007, 2008 Intel Corporation
Collecting data for 15 seconds
Your CPU supports the following C-states : C1 C2 C3 C4 Your BIOS reports the following C-states : C1 C2 C4
Cn Avg residency
C0 (cpu running) (40.8%)
polling 0.7ms ( 0.1%) C1 mwait 0.0ms ( 0.0%) C2 mwait 0.1ms ( 3.2%) C4 mwait 0.4ms (56.0%) P-states (frequencies) Turbo Mode 7.3% 2.21 Ghz 0.7% 1.60 Ghz 0.8% 1200 Mhz 0.9%
800 Mhz 90.3%
Wakeups-from-idle per second : 1727.3 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
53.0% (1983.1) [kernel scheduler] Load balancing tick
27.7% (1035.1) vmware-vmx
8.8% (330.6) [extra timer interrupt] 2.3% ( 85.5) [TLB shootdowns] <kernel IPI> 1.6% ( 61.3) totem 1.2% ( 45.9) [Rescheduling interrupts] <kernel IPI> 1.1% ( 41.1) [iwl3945] <interrupt> 1.0% ( 36.2) firefox-bin 0.4% ( 16.2) [ata_piix] <interrupt> 0.3% ( 11.6) vmware 0.3% ( 11.3) pulseaudio 0.3% ( 11.1) vmware-unity-he 0.3% ( 10.7) skype 0.2% ( 8.0) [kernel core] usb_hcd_poll_rh_status (rh_timer_func) 0.1% ( 5.4) [hda_intel] <interrupt> 0.1% ( 5.3) gnome-do 0.1% ( 5.1) compiz 0.1% ( 4.1) python 0.1% ( 4.0) avant-window-na 0.1% ( 2.3) liferea 0.1% ( 2.1) vmware-tray 0.1% ( 2.1) thunderbird-bin 0.1% ( 2.0) mysqld 0.1% ( 2.0) emerald 0.1% ( 1.9) gnome-terminal 0.0% ( 1.3) [kernel core] sk_reset_timer (tcp_delack_timer) 0.0% ( 1.3) [ahci] <interrupt> 0.0% ( 1.2) cc1 0.0% ( 1.0) [nvidia] <interrupt> 0.0% ( 1.0) vmware-usbarbit 0.0% ( 1.0) vmnet-natd 0.0% ( 1.0) gvfs-afc-volume 0.0% ( 1.0) [kernel core] enqueue_task_rt (sched_rt_period_timer) 0.0% ( 1.0) [kernel module] HostIF_InitUptime (HostIFUptimeResyncMono) 0.0% ( 0.8) cli 0.0% ( 0.5) events/1 0.0% ( 0.5) events/0 0.0% ( 1.0) vmware-usbarbit 0.0% ( 1.0) vmnet-natd 0.0% ( 1.0) gvfs-afc-volume 0.0% ( 1.0) [kernel core] enqueue_task_rt (sched_rt_period_timer) 0.0% ( 1.0) [kernel module] HostIF_InitUptime (HostIFUptimeResyncMono) 0.0% ( 0.8) cli 0.0% ( 0.5) events/1 0.0% ( 0.5) events/0 0.0% ( 0.5) hald-addon-stor 0.0% ( 0.5) udisks-daemon 0.0% ( 0.3) tomboy 0.0% ( 0.3) awn-applet 0.0% ( 0.2) rtkit-daemon 0.0% ( 0.2) Thunar 0.0% ( 0.2) padevchooser 0.0% ( 0.2) xfdesktop 0.0% ( 0.1) gnome-power-man 0.0% ( 0.1) ssh-agent 0.0% ( 0.1) mono 0.0% ( 0.1) NetworkManager 0.0% ( 0.1) hald
An audio device is active 100.0% of the time:
hwC0D2 SigmaTel STAC9228
A USB device is active 0.0% of the time:
USB device 2-2 : Biometric Coprocessor (STMicroelectronics)
Suggestion: Enable SATA ALPM link power management via:
echo min_power > /sys/class/scsi_host/host0/link_power_management_policy
or press the S key.
Suggestion: Enable wireless power saving mode by executing the following command:
iwconfig wlan0 power timeout 500ms
This will sacrifice network performance slightly to save power.
Suggestion: Enable the CONFIG_USB_SUSPEND kernel configuration option. This option will automatically disable UHCI USB when not in use, and may save approximately 1 Watt of power.
Recent USB suspend statistics
Active Device name 0.0% USB device 5-2 : Generic RNDIS (HTC) 0.0% USB device 2-2 : Biometric Coprocessor (STMicroelectronics) 0.0% USB device 1-1 : Laptop Integrated Webcam (OmniVision Technologies, Inc. -2640-07.07.20.3) 0.0% USB device usb7 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd) 0.0% USB device usb6 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd) 0.0% USB device usb5 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd) 0.0% USB device usb4 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd) 0.0% USB device usb3 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd) 0.0% USB device usb2 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd) 0.0% USB device usb1 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd)
Recent audio activity statistics
Active Device name
100.0% hwC0D2 SigmaTel STAC9228
Recent SATA AHCI link activity statistics
Active Partial Slumber Device name
Hi again.
Forgot to include my kernel config.
This is a Core 2 Duo laptop (Dell XPS M1530), currently running with a 32 bit kernel.
Regards,
Nigel
(linux-pm cc'd)
Quoted 7 lines
I'm seeing a huge number of load balancing tick wakeups, with both 2.6.33 and current git (
f1c448e0a9e99c76f4ece368714fb35a40a8daba).This has been reported as a bug in some distros:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281
Unclear if your sighting is the same as those two.
What do you see if vmware is not present?
What do you see if you boot single user mode,
how about with maxcpusx=1?
cheers,
Len Brown, Intel Open Source Technology Center
Quoted 121 lines
PowerTOP 1.12 (C) 2007, 2008 Intel Corporation
Collecting data for 15 seconds
Your CPU supports the following C-states : C1 C2 C3 C4 Your BIOS reports the following C-states : C1 C2 C4
Cn Avg residency
C0 (cpu running) (40.8%)polling 0.7ms ( 0.1%) C1 mwait 0.0ms ( 0.0%) C2 mwait 0.1ms ( 3.2%) C4 mwait 0.4ms (56.0%) P-states (frequencies) Turbo Mode 7.3% 2.21 Ghz 0.7% 1.60 Ghz 0.8% 1200 Mhz 0.9%800 Mhz 90.3%
Wakeups-from-idle per second : 1727.3 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
53.0% (1983.1) [kernel scheduler] Load balancing tick
27.7% (1035.1) vmware-vmx8.8% (330.6) [extra timer interrupt] 2.3% ( 85.5) [TLB shootdowns] <kernel IPI> 1.6% ( 61.3) totem 1.2% ( 45.9) [Rescheduling interrupts] <kernel IPI> 1.1% ( 41.1) [iwl3945] <interrupt> 1.0% ( 36.2) firefox-bin 0.4% ( 16.2) [ata_piix] <interrupt> 0.3% ( 11.6) vmware 0.3% ( 11.3) pulseaudio 0.3% ( 11.1) vmware-unity-he 0.3% ( 10.7) skype 0.2% ( 8.0) [kernel core] usb_hcd_poll_rh_status (rh_timer_func) 0.1% ( 5.4) [hda_intel] <interrupt> 0.1% ( 5.3) gnome-do 0.1% ( 5.1) compiz 0.1% ( 4.1) python 0.1% ( 4.0) avant-window-na 0.1% ( 2.3) liferea 0.1% ( 2.1) vmware-tray 0.1% ( 2.1) thunderbird-bin 0.1% ( 2.0) mysqld 0.1% ( 2.0) emerald 0.1% ( 1.9) gnome-terminal 0.0% ( 1.3) [kernel core] sk_reset_timer (tcp_delack_timer) 0.0% ( 1.3) [ahci] <interrupt> 0.0% ( 1.2) cc1 0.0% ( 1.0) [nvidia] <interrupt> 0.0% ( 1.0) vmware-usbarbit 0.0% ( 1.0) vmnet-natd 0.0% ( 1.0) gvfs-afc-volume 0.0% ( 1.0) [kernel core] enqueue_task_rt (sched_rt_period_timer) 0.0% ( 1.0) [kernel module] HostIF_InitUptime (HostIFUptimeResyncMono) 0.0% ( 0.8) cli 0.0% ( 0.5) events/1 0.0% ( 0.5) events/0 0.0% ( 1.0) vmware-usbarbit 0.0% ( 1.0) vmnet-natd 0.0% ( 1.0) gvfs-afc-volume 0.0% ( 1.0) [kernel core] enqueue_task_rt (sched_rt_period_timer) 0.0% ( 1.0) [kernel module] HostIF_InitUptime (HostIFUptimeResyncMono) 0.0% ( 0.8) cli 0.0% ( 0.5) events/1 0.0% ( 0.5) events/0 0.0% ( 0.5) hald-addon-stor 0.0% ( 0.5) udisks-daemon 0.0% ( 0.3) tomboy 0.0% ( 0.3) awn-applet 0.0% ( 0.2) rtkit-daemon 0.0% ( 0.2) Thunar 0.0% ( 0.2) padevchooser 0.0% ( 0.2) xfdesktop 0.0% ( 0.1) gnome-power-man 0.0% ( 0.1) ssh-agent 0.0% ( 0.1) mono 0.0% ( 0.1) NetworkManager 0.0% ( 0.1) haldAn audio device is active 100.0% of the time:
hwC0D2 SigmaTel STAC9228A USB device is active 0.0% of the time:
USB device 2-2 : Biometric Coprocessor (STMicroelectronics)Suggestion: Enable SATA ALPM link power management via:
echomin_power>/sys/class/scsi_host/host0/link_power_management_policy
or press the S key.Suggestion: Enable wireless power saving mode by executing the following command:
iwconfig wlan0 power timeout 500ms
This will sacrifice network performance slightly to save power.Suggestion: Enable the
CONFIG_USB_SUSPENDkernel configuration option. This option will automatically disable UHCI USB when not in use, and may save approximately 1 Watt of power.Recent USB suspend statistics
Active Device name 0.0% USB device 5-2 : Generic RNDIS (HTC) 0.0% USB device 2-2 : Biometric Coprocessor (STMicroelectronics) 0.0% USB device 1-1 : Laptop Integrated Webcam (OmniVision Technologies, Inc. -2640-07.07.20.3) 0.0% USB device usb7 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd) 0.0% USB device usb6 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd) 0.0% USB device usb5 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd) 0.0% USB device usb4 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd) 0.0% USB device usb3 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd) 0.0% USB device usb2 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd) 0.0% USB device usb1 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd)Recent audio activity statistics
Active Device name
100.0% hwC0D2 SigmaTel STAC9228Recent SATA AHCI link activity statistics
Active Partial Slumber Device name
Hi Len.
Quoted 5 lines · Len Brown
(linux-pm cc'd)
Quoted 3 lines
I'm seeing a huge number of load balancing tick wakeups, with both 2.6.33 and current git (
f1c448e0a9e99c76f4ece368714fb35a40a8daba).
Quoted 8 lines
Quoted 4 lines
This has been reported as a bug in some distros:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281Unclear if your sighting is the same as those two.
What do you see if vmware is not present?
I still see a huge number, just not as huge. The number of load balancing ticks seems to increase with the number of ticks from other sources. Without vmware running I got:
Cn Avg residency
C0 (cpu running) (14.5%)
polling 0.2ms ( 0.0%) C1 mwait 0.0ms ( 0.0%) C2 mwait 0.1ms ( 1.7%) C4 mwait 1.1ms (83.8%) P-states (frequencies) Turbo Mode 1.2% 2.21 Ghz 0.0% 1.60 Ghz 0.0% 1200 Mhz 0.1%
800 Mhz 98.7%
Wakeups-from-idle per second : 907.4 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
42.1% (435.6) [kernel scheduler] Load balancing tick 21.2% (219.6) [extra timer interrupt] 10.2% (105.6) plugin-containe 9.5% ( 98.1) pulseaudio 1.7% ( 17.9) [Rescheduling interrupts] <kernel IPI> 1.7% ( 17.1) [iwl3945] <interrupt> 1.4% ( 14.8) [kernel core] hrtimer_start (tick_sched_timer) 1.4% ( 14.3) [ata_piix] <interrupt>
Quoted 2 lines
What do you see if you boot single user mode,
how about with maxcpusx=1?
I'll try those later today and get back to you with more info.
Regards,
Nigel
Quoted 29 lines
cheers,
Len Brown, Intel Open Source Technology CenterQuoted 26 lines
PowerTOP 1.12 (C) 2007, 2008 Intel Corporation
Collecting data for 15 seconds
Your CPU supports the following C-states : C1 C2 C3 C4 Your BIOS reports the following C-states : C1 C2 C4
Cn Avg residency
C0 (cpu running) (40.8%)polling 0.7ms ( 0.1%) C1 mwait 0.0ms ( 0.0%) C2 mwait 0.1ms ( 3.2%) C4 mwait 0.4ms (56.0%) P-states (frequencies) Turbo Mode 7.3% 2.21 Ghz 0.7% 1.60 Ghz 0.8% 1200 Mhz 0.9% 800 Mhz 90.3% Wakeups-from-idle per second : 1727.3 interval: 15.0s no ACPI power usage estimate available Top causes for wakeups: 53.0% (1983.1) [kernel scheduler] Load balancing tick 27.7% (1035.1) vmware-vmx 8.8% (330.6) [extra timer interrupt] 2.3% ( 85.5) [TLB shootdowns]<kernel IPI>
Hey,
Quoted 34 lines · Nigel Cunningham
Quoted 5 lines · Len Brown
(linux-pm cc'd)
Quoted 3 lines
I'm seeing a huge number of load balancing tick wakeups, with both 2.6.33 and current git (
f1c448e0a9e99c76f4ece368714fb35a40a8daba).Quoted 8 lines
Quoted 4 lines
This has been reported as a bug in some distros:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281Unclear if your sighting is the same as those two.
What do you see if vmware is not present?
I still see a huge number, just not as huge. The number of load balancing ticks seems to increase with the number of ticks from other sources. Without vmware running I got:
Cn Avg residency
C0 (cpu running) (14.5%)polling 0.2ms ( 0.0%) C1 mwait 0.0ms ( 0.0%) C2 mwait 0.1ms ( 1.7%) C4 mwait 1.1ms (83.8%) P-states (frequencies) Turbo Mode 1.2% 2.21 Ghz 0.0% 1.60 Ghz 0.0% 1200 Mhz 0.1% 800 Mhz 98.7%Wakeups-from-idle per second : 907.4 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
Quoted 8 lines
42.1% (435.6) [kernel scheduler] Load balancing tick
21.2% (219.6) [extra timer interrupt]
10.2% (105.6) plugin-containe
9.5% ( 98.1) pulseaudio
1.7% ( 17.9) [Rescheduling interrupts] <kernel IPI>
1.7% ( 17.1) [iwl3945] <interrupt>
1.4% ( 14.8) [kernel core] hrtimer_start (tick_sched_timer)
1.4% ( 14.3) [ata_piix] <interrupt>Seems to be exactly what I'm seeing here, too. See my description and possible patch at
http://lkml.org/lkml/2010/4/26/249
Suresh Siddha reported he's working on getting some (more advanced) patches ready for submission.
Best,
Dominik
Hi again.
Quoted 3 lines · Len Brown
What do you see if vmware is not present?
What do you see if you boot single user mode,
how about with maxcpusx=1?
Repeated the tests, just booting into X (xfce) - except for the init S test. No programs were started and wifi was turned off.
1) No vmware (etc)
Cn Avg residency
C0 (cpu running) ( 3.7%)
polling 0.0ms ( 0.0%) C1 mwait 0.0ms ( 0.0%) C2 mwait 0.1ms ( 0.3%) C4 mwait 5.4ms (96.0%) P-states (frequencies) Turbo Mode 1.4% 2.21 Ghz 0.0% 1.60 Ghz 0.0% 1200 Mhz 0.0%
800 Mhz 98.6%
Wakeups-from-idle per second : 202.7 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
56.1% (128.5) [kernel scheduler] Load balancing tick 13.1% ( 30.1) [extra timer interrupt] 6.9% ( 15.9) [ata_piix] <interrupt>
2) Above + maxcpus=1
Cn Avg residency
C0 (cpu running) ( 4.0%)
polling 0.0ms ( 0.0%) C1 mwait 0.0ms ( 0.0%) C2 mwait 0.4ms ( 0.2%) C4 mwait 16.0ms (95.9%) P-states (frequencies) Turbo Mode 6.0% 2.21 Ghz 0.0% 1.60 Ghz 0.0% 1200 Mhz 0.0%
800 Mhz 94.0%
Wakeups-from-idle per second : 64.5 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
39.6% ( 46.3) [kernel scheduler] Load balancing tick
12.3% ( 14.3) [ata_piix] <interrupt>
(No idea why you need a load balancing tick with one core!)
3) Init S, without maxcpus=1
Cn Avg residency
C0 (cpu running) ( 0.0%)
polling 0.0ms ( 0.0%) C1 mwait 0.0ms ( 0.0%) C2 mwait 0.1ms ( 0.0%) C4 mwait 48.9ms (100.0%) P-states (frequencies) Turbo Mode 0.0% 2.21 Ghz 0.0% 1.60 Ghz 0.0% 1200 Mhz 0.0%
800 Mhz 100.0%
Wakeups-from-idle per second : 21.4 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
33.5% ( 10.3) [kernel core] hrtimer_start (tick_sched_timer) 16.1% ( 4.9) [kernel core] cursor_timer_handler (cursor_timer_handler) 15.9% ( 4.9) [kernel scheduler] Load balancing tick 12.8% ( 3.9) [kernel core] usb_hcd_poll_rh_status (rh_timer_func) 9.3% ( 2.9) [ahci] <interrupt>
I also tried the patch Dominik provided, and it didn't seem to make a difference:
P-states (frequencies)
Turbo Mode 3.7% 2.21 Ghz 0.0% 1.60 Ghz 0.0% 1200 Mhz 0.6%
800 Mhz 95.7%
Wakeups-from-idle per second : 236.7 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
55.4% (119.7) [kernel scheduler] Load balancing tick 9.7% ( 20.9) skype 7.7% ( 16.5) [ata_piix] <interrupt>
Regards,
Nigel