Skip to contentSkip to conversation

2.6.33 & 2.6.34-rc6: Huge number of Load Balancing tick wakeups.

DISCUSSION Nigel Cunningham 4 participants 6 messages May 8, 2010 – May 9, 2010

lore
Share & tools
lore
#0 lore raw

Hi all.

I'm seeing a huge number of load balancing tick wakeups, with both 2.6.33 and current git (f1c448e0a9e99c76f4ece368714fb35a40a8daba).

This has been reported as a bug in some distros:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281

Regards,

Nigel

PowerTOP 1.12 (C) 2007, 2008 Intel Corporation

Collecting data for 15 seconds

Your CPU supports the following C-states : C1 C2 C3 C4 Your BIOS reports the following C-states : C1 C2 C4
Cn Avg residency
C0 (cpu running) (40.8%)

polling           0.7ms ( 0.1%)
C1 mwait          0.0ms ( 0.0%)
C2 mwait          0.1ms ( 3.2%)
C4 mwait          0.4ms (56.0%)
P-states (frequencies)
Turbo Mode     7.3%
  2.21 Ghz     0.7%
  1.60 Ghz     0.8%
  1200 Mhz     0.9%

800 Mhz 90.3%
Wakeups-from-idle per second : 1727.3 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
53.0% (1983.1) [kernel scheduler] Load balancing tick
27.7% (1035.1) vmware-vmx

   8.8% (330.6)   [extra timer interrupt]
   2.3% ( 85.5)   [TLB shootdowns] <kernel IPI>
   1.6% ( 61.3)   totem
   1.2% ( 45.9)   [Rescheduling interrupts] <kernel IPI>
   1.1% ( 41.1)   [iwl3945] <interrupt>
   1.0% ( 36.2)   firefox-bin
   0.4% ( 16.2)   [ata_piix] <interrupt>
   0.3% ( 11.6)   vmware
   0.3% ( 11.3)   pulseaudio
   0.3% ( 11.1)   vmware-unity-he
   0.3% ( 10.7)   skype
   0.2% (  8.0)   [kernel core] usb_hcd_poll_rh_status (rh_timer_func)
   0.1% (  5.4)   [hda_intel] <interrupt>
   0.1% (  5.3)   gnome-do
   0.1% (  5.1)   compiz
   0.1% (  4.1)   python
   0.1% (  4.0)   avant-window-na
   0.1% (  2.3)   liferea
   0.1% (  2.1)   vmware-tray
   0.1% (  2.1)   thunderbird-bin
   0.1% (  2.0)   mysqld
   0.1% (  2.0)   emerald
   0.1% (  1.9)   gnome-terminal
   0.0% (  1.3)   [kernel core] sk_reset_timer (tcp_delack_timer)
   0.0% (  1.3)   [ahci] <interrupt>
   0.0% (  1.2)   cc1
   0.0% (  1.0)   [nvidia] <interrupt>
   0.0% (  1.0)   vmware-usbarbit
   0.0% (  1.0)   vmnet-natd
   0.0% (  1.0)   gvfs-afc-volume
   0.0% (  1.0)   [kernel core] enqueue_task_rt (sched_rt_period_timer)
   0.0% (  1.0)   [kernel module] HostIF_InitUptime 
(HostIFUptimeResyncMono)
   0.0% (  0.8)   cli
   0.0% (  0.5)   events/1
   0.0% (  0.5)   events/0
   0.0% (  1.0)   vmware-usbarbit
   0.0% (  1.0)   vmnet-natd
   0.0% (  1.0)   gvfs-afc-volume
   0.0% (  1.0)   [kernel core] enqueue_task_rt (sched_rt_period_timer)
   0.0% (  1.0)   [kernel module] HostIF_InitUptime 
(HostIFUptimeResyncMono)
   0.0% (  0.8)   cli
   0.0% (  0.5)   events/1
   0.0% (  0.5)   events/0
   0.0% (  0.5)   hald-addon-stor
   0.0% (  0.5)   udisks-daemon
   0.0% (  0.3)   tomboy
   0.0% (  0.3)   awn-applet
   0.0% (  0.2)   rtkit-daemon
   0.0% (  0.2)   Thunar
   0.0% (  0.2)   padevchooser
   0.0% (  0.2)   xfdesktop
   0.0% (  0.1)   gnome-power-man
   0.0% (  0.1)   ssh-agent
   0.0% (  0.1)   mono
   0.0% (  0.1)   NetworkManager
   0.0% (  0.1)   hald

An audio device is active 100.0% of the time:
hwC0D2 SigmaTel STAC9228

A USB device is active 0.0% of the time:
USB device 2-2 : Biometric Coprocessor (STMicroelectronics)

Suggestion: Enable SATA ALPM link power management via:
echo min_power > /sys/class/scsi_host/host0/link_power_management_policy
or press the S key.

Suggestion: Enable wireless power saving mode by executing the following command:
iwconfig wlan0 power timeout 500ms
This will sacrifice network performance slightly to save power.

Suggestion: Enable the CONFIG_USB_SUSPEND kernel configuration option. This option will automatically disable UHCI USB when not in use, and may save approximately 1 Watt of power.

Recent USB suspend statistics

Active  Device name
  0.0%  USB device  5-2 : Generic RNDIS (HTC)
  0.0%  USB device  2-2 : Biometric Coprocessor (STMicroelectronics)
  0.0%  USB device  1-1 : Laptop Integrated Webcam (OmniVision 
Technologies, Inc. -2640-07.07.20.3)
  0.0%  USB device usb7 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
  0.0%  USB device usb6 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
  0.0%  USB device usb5 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
  0.0%  USB device usb4 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd)
  0.0%  USB device usb3 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
  0.0%  USB device usb2 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
  0.0%  USB device usb1 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd)

Recent audio activity statistics
Active Device name
100.0% hwC0D2 SigmaTel STAC9228

Recent SATA AHCI link activity statistics
Active Partial Slumber Device name

#1 lore raw

Hi again.

Forgot to include my kernel config.

This is a Core 2 Duo laptop (Dell XPS M1530), currently running with a 32 bit kernel.

Regards,

Nigel

#2 lore raw
In reply to Nigel Cunningham

(linux-pm cc'd)

Quoted 7 lines

I'm seeing a huge number of load balancing tick wakeups, with both 2.6.33 and current git (f1c448e0a9e99c76f4ece368714fb35a40a8daba).

This has been reported as a bug in some distros:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281

Unclear if your sighting is the same as those two.

What do you see if vmware is not present?
What do you see if you boot single user mode,
how about with maxcpusx=1?

cheers,
Len Brown, Intel Open Source Technology Center

Quoted 121 lines

PowerTOP 1.12 (C) 2007, 2008 Intel Corporation

Collecting data for 15 seconds

Your CPU supports the following C-states : C1 C2 C3 C4 Your BIOS reports the following C-states : C1 C2 C4
Cn Avg residency
C0 (cpu running) (40.8%)

polling           0.7ms ( 0.1%)
C1 mwait          0.0ms ( 0.0%)
C2 mwait          0.1ms ( 3.2%)
C4 mwait          0.4ms (56.0%)
P-states (frequencies)
Turbo Mode     7.3%
  2.21 Ghz     0.7%
  1.60 Ghz     0.8%
  1200 Mhz     0.9%

800 Mhz 90.3%
Wakeups-from-idle per second : 1727.3 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
53.0% (1983.1) [kernel scheduler] Load balancing tick
27.7% (1035.1) vmware-vmx

   8.8% (330.6)   [extra timer interrupt]
   2.3% ( 85.5)   [TLB shootdowns] <kernel IPI>
   1.6% ( 61.3)   totem
   1.2% ( 45.9)   [Rescheduling interrupts] <kernel IPI>
   1.1% ( 41.1)   [iwl3945] <interrupt>
   1.0% ( 36.2)   firefox-bin
   0.4% ( 16.2)   [ata_piix] <interrupt>
   0.3% ( 11.6)   vmware
   0.3% ( 11.3)   pulseaudio
   0.3% ( 11.1)   vmware-unity-he
   0.3% ( 10.7)   skype
   0.2% (  8.0)   [kernel core] usb_hcd_poll_rh_status (rh_timer_func)
   0.1% (  5.4)   [hda_intel] <interrupt>
   0.1% (  5.3)   gnome-do
   0.1% (  5.1)   compiz
   0.1% (  4.1)   python
   0.1% (  4.0)   avant-window-na
   0.1% (  2.3)   liferea
   0.1% (  2.1)   vmware-tray
   0.1% (  2.1)   thunderbird-bin
   0.1% (  2.0)   mysqld
   0.1% (  2.0)   emerald
   0.1% (  1.9)   gnome-terminal
   0.0% (  1.3)   [kernel core] sk_reset_timer (tcp_delack_timer)
   0.0% (  1.3)   [ahci] <interrupt>
   0.0% (  1.2)   cc1
   0.0% (  1.0)   [nvidia] <interrupt>
   0.0% (  1.0)   vmware-usbarbit
   0.0% (  1.0)   vmnet-natd
   0.0% (  1.0)   gvfs-afc-volume
   0.0% (  1.0)   [kernel core] enqueue_task_rt (sched_rt_period_timer)
   0.0% (  1.0)   [kernel module] HostIF_InitUptime (HostIFUptimeResyncMono)
   0.0% (  0.8)   cli
   0.0% (  0.5)   events/1
   0.0% (  0.5)   events/0
   0.0% (  1.0)   vmware-usbarbit
   0.0% (  1.0)   vmnet-natd
   0.0% (  1.0)   gvfs-afc-volume
   0.0% (  1.0)   [kernel core] enqueue_task_rt (sched_rt_period_timer)
   0.0% (  1.0)   [kernel module] HostIF_InitUptime (HostIFUptimeResyncMono)
   0.0% (  0.8)   cli
   0.0% (  0.5)   events/1
   0.0% (  0.5)   events/0
   0.0% (  0.5)   hald-addon-stor
   0.0% (  0.5)   udisks-daemon
   0.0% (  0.3)   tomboy
   0.0% (  0.3)   awn-applet
   0.0% (  0.2)   rtkit-daemon
   0.0% (  0.2)   Thunar
   0.0% (  0.2)   padevchooser
   0.0% (  0.2)   xfdesktop
   0.0% (  0.1)   gnome-power-man
   0.0% (  0.1)   ssh-agent
   0.0% (  0.1)   mono
   0.0% (  0.1)   NetworkManager
   0.0% (  0.1)   hald

An audio device is active 100.0% of the time:
hwC0D2 SigmaTel STAC9228

A USB device is active 0.0% of the time:
USB device 2-2 : Biometric Coprocessor (STMicroelectronics)

Suggestion: Enable SATA ALPM link power management via:
echo min_power > /sys/class/scsi_host/host0/link_power_management_policy
or press the S key.

Suggestion: Enable wireless power saving mode by executing the following command:
iwconfig wlan0 power timeout 500ms
This will sacrifice network performance slightly to save power.

Suggestion: Enable the CONFIG_USB_SUSPEND kernel configuration option. This option will automatically disable UHCI USB when not in use, and may save approximately 1 Watt of power.

Recent USB suspend statistics

Active  Device name
  0.0%  USB device  5-2 : Generic RNDIS (HTC)
  0.0%  USB device  2-2 : Biometric Coprocessor (STMicroelectronics)
  0.0%  USB device  1-1 : Laptop Integrated Webcam (OmniVision Technologies,
Inc. -2640-07.07.20.3)
  0.0%  USB device usb7 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
  0.0%  USB device usb6 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
  0.0%  USB device usb5 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
  0.0%  USB device usb4 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd)
  0.0%  USB device usb3 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
  0.0%  USB device usb2 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
  0.0%  USB device usb1 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd)

Recent audio activity statistics
Active Device name
100.0% hwC0D2 SigmaTel STAC9228

Recent SATA AHCI link activity statistics
Active Partial Slumber Device name

#3 lore raw

Hi Len.

Quoted 5 lines · Len Brown

(linux-pm cc'd)

Quoted 3 lines

I'm seeing a huge number of load balancing tick wakeups, with both 2.6.33 and current git (f1c448e0a9e99c76f4ece368714fb35a40a8daba).

Quoted 8 lines
Quoted 4 lines

This has been reported as a bug in some distros:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281

Unclear if your sighting is the same as those two.

What do you see if vmware is not present?

I still see a huge number, just not as huge. The number of load balancing ticks seems to increase with the number of ticks from other sources. Without vmware running I got:

Cn Avg residency
C0 (cpu running) (14.5%)

polling		  0.2ms ( 0.0%)
C1 mwait	  0.0ms ( 0.0%)
C2 mwait	  0.1ms ( 1.7%)
C4 mwait	  1.1ms (83.8%)
P-states (frequencies)
Turbo Mode     1.2%
  2.21 Ghz     0.0%
  1.60 Ghz     0.0%
  1200 Mhz     0.1%

800 Mhz 98.7%
Wakeups-from-idle per second : 907.4 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:

  42.1% (435.6)   [kernel scheduler] Load balancing tick
  21.2% (219.6)   [extra timer interrupt]
  10.2% (105.6)   plugin-containe
   9.5% ( 98.1)   pulseaudio
   1.7% ( 17.9)   [Rescheduling interrupts] <kernel IPI>
   1.7% ( 17.1)   [iwl3945] <interrupt>
   1.4% ( 14.8)   [kernel core] hrtimer_start (tick_sched_timer)
   1.4% ( 14.3)   [ata_piix] <interrupt>
Quoted 2 lines

What do you see if you boot single user mode,
how about with maxcpusx=1?

I'll try those later today and get back to you with more info.

Regards,

Nigel

Quoted 29 lines

cheers,
Len Brown, Intel Open Source Technology Center

Quoted 26 lines

PowerTOP 1.12 (C) 2007, 2008 Intel Corporation

Collecting data for 15 seconds

Your CPU supports the following C-states : C1 C2 C3 C4 Your BIOS reports the following C-states : C1 C2 C4
Cn Avg residency
C0 (cpu running) (40.8%)

polling           0.7ms ( 0.1%)
C1 mwait          0.0ms ( 0.0%)
C2 mwait          0.1ms ( 3.2%)
C4 mwait          0.4ms (56.0%)
P-states (frequencies)
Turbo Mode     7.3%
   2.21 Ghz     0.7%
   1.60 Ghz     0.8%
   1200 Mhz     0.9%
    800 Mhz    90.3%
Wakeups-from-idle per second : 1727.3   interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
   53.0% (1983.1)   [kernel scheduler] Load balancing tick
   27.7% (1035.1)   vmware-vmx
    8.8% (330.6)   [extra timer interrupt]
    2.3% ( 85.5)   [TLB shootdowns]<kernel IPI>
#4 lore raw

Hey,

Quoted 34 lines · Nigel Cunningham
Quoted 5 lines · Len Brown

(linux-pm cc'd)

Quoted 3 lines

I'm seeing a huge number of load balancing tick wakeups, with both 2.6.33 and current git (f1c448e0a9e99c76f4ece368714fb35a40a8daba).

Quoted 8 lines
Quoted 4 lines

This has been reported as a bug in some distros:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281

Unclear if your sighting is the same as those two.

What do you see if vmware is not present?

I still see a huge number, just not as huge. The number of load balancing ticks seems to increase with the number of ticks from other sources. Without vmware running I got:

Cn Avg residency
C0 (cpu running) (14.5%)

polling		  0.2ms ( 0.0%)
C1 mwait	  0.0ms ( 0.0%)
C2 mwait	  0.1ms ( 1.7%)
C4 mwait	  1.1ms (83.8%)
P-states (frequencies)
Turbo Mode     1.2%
   2.21 Ghz     0.0%
   1.60 Ghz     0.0%
   1200 Mhz     0.1%
    800 Mhz    98.7%

Wakeups-from-idle per second : 907.4 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:

Quoted 8 lines
   42.1% (435.6)   [kernel scheduler] Load balancing tick
   21.2% (219.6)   [extra timer interrupt]
   10.2% (105.6)   plugin-containe
    9.5% ( 98.1)   pulseaudio
    1.7% ( 17.9)   [Rescheduling interrupts] <kernel IPI>
    1.7% ( 17.1)   [iwl3945] <interrupt>
    1.4% ( 14.8)   [kernel core] hrtimer_start (tick_sched_timer)
    1.4% ( 14.3)   [ata_piix] <interrupt>

Seems to be exactly what I'm seeing here, too. See my description and possible patch at

	http://lkml.org/lkml/2010/4/26/249

Suresh Siddha reported he's working on getting some (more advanced) patches ready for submission.

Best,
Dominik

#5 lore raw
In reply to Len Brown

Hi again.

Quoted 3 lines · Len Brown

What do you see if vmware is not present?
What do you see if you boot single user mode,
how about with maxcpusx=1?

Repeated the tests, just booting into X (xfce) - except for the init S test. No programs were started and wifi was turned off.

1) No vmware (etc)

Cn Avg residency
C0 (cpu running) ( 3.7%)

polling           0.0ms ( 0.0%)
C1 mwait          0.0ms ( 0.0%)
C2 mwait          0.1ms ( 0.3%)
C4 mwait          5.4ms (96.0%)
P-states (frequencies)
Turbo Mode     1.4%
  2.21 Ghz     0.0%
  1.60 Ghz     0.0%
  1200 Mhz     0.0%

800 Mhz 98.6%
Wakeups-from-idle per second : 202.7 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:

  56.1% (128.5)   [kernel scheduler] Load balancing tick
  13.1% ( 30.1)   [extra timer interrupt]
   6.9% ( 15.9)   [ata_piix] <interrupt>

2) Above + maxcpus=1

Cn Avg residency
C0 (cpu running) ( 4.0%)

polling           0.0ms ( 0.0%)
C1 mwait          0.0ms ( 0.0%)
C2 mwait          0.4ms ( 0.2%)
C4 mwait         16.0ms (95.9%)
P-states (frequencies)
Turbo Mode     6.0%
  2.21 Ghz     0.0%
  1.60 Ghz     0.0%
  1200 Mhz     0.0%

800 Mhz 94.0%
Wakeups-from-idle per second : 64.5 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
39.6% ( 46.3) [kernel scheduler] Load balancing tick
12.3% ( 14.3) [ata_piix] <interrupt>

(No idea why you need a load balancing tick with one core!)

3) Init S, without maxcpus=1

Cn Avg residency
C0 (cpu running) ( 0.0%)

polling           0.0ms ( 0.0%)
C1 mwait          0.0ms ( 0.0%)
C2 mwait          0.1ms ( 0.0%)
C4 mwait         48.9ms (100.0%)
P-states (frequencies)
Turbo Mode     0.0%
  2.21 Ghz     0.0%
  1.60 Ghz     0.0%
  1200 Mhz     0.0%

800 Mhz 100.0%
Wakeups-from-idle per second : 21.4 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:

  33.5% ( 10.3)   [kernel core] hrtimer_start (tick_sched_timer)
  16.1% (  4.9)   [kernel core] cursor_timer_handler (cursor_timer_handler)
  15.9% (  4.9)   [kernel scheduler] Load balancing tick
  12.8% (  3.9)   [kernel core] usb_hcd_poll_rh_status (rh_timer_func)
   9.3% (  2.9)   [ahci] <interrupt>

I also tried the patch Dominik provided, and it didn't seem to make a difference:

P-states (frequencies)

Turbo Mode     3.7%
  2.21 Ghz     0.0%
  1.60 Ghz     0.0%
  1200 Mhz     0.6%

800 Mhz 95.7%
Wakeups-from-idle per second : 236.7 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:

  55.4% (119.7)   [kernel scheduler] Load balancing tick
   9.7% ( 20.9)   skype
   7.7% ( 16.5)   [ata_piix] <interrupt>

Regards,

Nigel