A few questions and issues with dynticks, NOHZ and powertop
Subsystems (2)
Hey!
Before I'm off hiding some Easter eggs, here are some questions and issues related to "dynticks", NOHZ, and powertop:
1) single-CPU systems, SMP-capable kernel and RCU
2) dual-core CPU[*] and select_nohz_load_balancer()
3) USB, autosuspend failure, excessive ticks
4) SynPS/2 touchpad and hundreds of IRQs per second
5) powertop: 1 + 1 = 1
1) single-CPU systems, SMP-capable kernel and RCU
CONFIG_TREE_RCU=yCONFIG_RCU_FANOUT=64CONFIG_RCU_FAST_NO_HZ=y
Booting a SMP-capable kernel with "nosmp", or manually offlining one CPU (or -- though I haven't tested it -- booting a SMP-capable kernel on a system with merely one CPU) means that in up to about half of the calls to tick_nohz_stop_sched_tick() are aborted due to rcu_needs_cpu(). This is quite strange to me: AFAIK, RCU is an excellent tool for SMP, but not really needed for UP? And all updates seem to be local to the CPU anyway. Therefore, I'd presume that rcu_needs_cpu() should return 0 on one-CPU-systems. Or could RCU switch between TINY_RCU on UP and TREE_RCU on SMP (using alternatives or whatever)?
2) dual-core CPU[*] and select_nohz_load_balancer() [*] (Intel(R) Core(TM)2 Duo CPU T7250)
# CONFIG_SCHED_SMT is not setCONFIG_SCHED_MC=yCONFIG_SCHED_HRTICK=y
CONFIG_SCHED_MC is igored, as mc_capable() returns 0 on a one-socket, dual-core system. Quite surprisingly, even under moderate load (~98.0% idle) while writing this bugreport, up to half of the calls to tick_nohz_stop_sched_tick() are aborted due to select_nohz_load_balancer(1):
if (atomic_read(&nohz.load_balancer) == -1) {
/* make me the ilb owner */
if (atomic_cmpxchg(&nohz.load_balancer, -1, cpu) == -1)
return 1;I'm not really sure, but I guess this is caused by the following phenomenon under minor load but still, every once in a while, parallel work for both CPUs:
CPU #0 CPU #1
<active> <active>
<idle> <active>
tick_nohz_stop_sched_tick(1) <active>
select_nohz_load_balancer(1) <active>
=> becomes ilb owner <idle>
=> tick is not stopped tick_nohz_stop_sched_tick(1)
=> CPU goes to sleep for 1 tick => as it isn't the ILB owner, tick
<sleep for 1 tick> is stopped .
---> scheduler_tick() <sleeeeeeeep>
tick_nohz_stop_sched_tick(0)
<still idle>
tick_nohz_stop_sched_tick(1)
select_nohz_load_balancer(1)
=> is ilb owner, all CPUs idle,
may go to sleep.
If both CPUs have hardly anything to do, letting the _active_ CPU do ilb allows us to enter deep sleep states earlier, and longer:
current ILB model (* = ILB)
tick ---------- tick -------- tick ----- IRQ CPU0: active|IDLE(C2)--|*|IDLE (C3) | CPU1: active....| IDLE (C3) | core: .......???| C2 | C3 |
ILB-by-active-CPU-on-light-load:
tick ---------- tick -------- tick ----- IRQ CPU0: active|IDLE(C3) | CPU1: active....*| IDLE (C3) | core: .......????| C3 |
3) USB: built-in UHCI and a built-in 0a5c:2101 Broadcom Corp. A-Link BlueUsbA2 Bluetooth module; built-in EHCI and a built-in 0ac8:c302 Z-Star Microelectronics Corp. Vega USB 2.0 Camera.
usbcore.autosuspend is enabled (= 2), of course.
Recent USB suspend statistics
Active Device name
100.0% USB device 7-1 : BCM92045NMD (Broadcom Corp)
100.0% USB device 1-2 : Vega USB 2.0 Camera. (Vimicro Corp.)
100.0% USB device usb7 : UHCI Host Controller (Linux 2.6.34-rc3 uhci_hcd)
100.0% USB device usb1 : EHCI Host Controller (Linux 2.6.34-rc3 ehci_hcd)
Booting into /bin/bash on a SMP kernel booted with "nosmp" leads to ~ 10 wakeups per second; disabling the cursor helps halfway (~ 5 wakeups); and manually unbinding the USB host drivers from the USB host devices finally lead to ~ 1.1 wakeups per second. What's keeping USB from suspending these unused devices here?
4) SynPS/2 touchpad:
Why does moving the touchpad lead to sooo many IRQs? I can't look as fast as the mouse pointer seems to get new data:
62,5% (473,1) <interrupt> : PS/2 keyboard/mouse/touchpad
5) powertop and hrtimer_start_range_ns (tick_sched_timer) on a SMP kernel booted with "nosmp":
Wakeups-from-idle per second : 9.9 interval: 15.0s
...
48.5% ( 9.4) <kernel core> : hrtimer_start (tick_sched_timer) 26.1% ( 5.1) <kernel core> : cursor_timer_handler (cursor_timer_handler) 20.6% ( 4.0) <kernel core> : usb_hcd_poll_rh_status (rh_timer_func) 1.0% ( 0.2) <kernel core> : arm_supers_timer (sync_supers_timer_fn) 0.7% ( 0.1) <interrupt> : ata_piix
...
Accoding to http://www.linuxpowertop.org , the count in the brackets is how many wakeups per seconds were caused by one source. Adding all _except_
48.5% ( 9.4) <kernel core> : hrtimer_start (tick_sched_timer)
up leads to the 9.9; adding also the 9.4 leads to 19.3 wakeups-from-idle per second. However, http://www.linuxpowertop.org says:
Quoted 5 lines
"Should "Wakeups-from-idle per second" equal the sum of the wakeups/second/core listed on the "Top causes for wakeups" list?
It should be higher, since there are some causes for wakeups that are nearly impossible to detect by software."
Best, and Happy Easter,
Dominik
Dmitry Torokhov
maintainerHi Dominik,
Quoted 6 lines · Dominik Brodowski
4) SynPS/2 touchpad:
Why does moving the touchpad lead to sooo many IRQs? I can't look as fast as the mouse pointer seems to get new data:
62,5% (473,1) <interrupt> : PS/2 keyboard/mouse/touchpad
80 pps @ 6 bytes/packet = 480 interrupts/sec.
You can try using psmouse.rate=40 to limit it to 40 pps which should bring it to the rate of standard PS/2 mouse at the expense of sensitivity...
Signature · 1 line
Dmitry
Hi Dmitry,
Quoted 8 lines · Dmitry Torokhov
Quoted 6 lines · Dominik Brodowski
4) SynPS/2 touchpad:
Why does moving the touchpad lead to sooo many IRQs? I can't look as fast as the mouse pointer seems to get new data:
62,5% (473,1) <interrupt> : PS/2 keyboard/mouse/touchpad
Quoted 5 lines
80 pps @ 6 bytes/packet = 480 interrupts/sec.
You can try using psmouse.rate=40 to limit it to 40 pps which should bring it to the rate of standard PS/2 mouse at the expense of sensitivity...
Excellent. Maybe this could be added to the Tips&Tricks section at the Powertop website?
I guess obtaining all 6 bytes at once is not really possible? It seems a new byte only appears ~1.75 ms after the last one, at least on my notebook, so waiting for this is not an option...
Best,
Dominik
PS: Dmitry, got something small for you in return:
[PATCH] i8042: spelling fix
drivers/input/serio/i8042.c +1 -1
@@ -38,7 +38,7 @@ MODULE_PARM_DESC(noaux, "Do not probe or use AUX (mouse) port.");static bool i8042_nomux;module_param_named (nomux, i8042_nomux, bool,0 ); -MODULE_PARM_DESC (nomux,"Do not check whether an active multiplexing conrtoller is present." ); +MODULE_PARM_DESC (nomux,"Do not check whether an active multiplexing controller is present." );static bool i8042_unlock;module_param_named (unlock, i8042_unlock, bool,0 );
Dmitry Torokhov
maintainerQuoted 11 lines · Dominik Brodowski
Hi Dmitry,
Quoted 8 lines · Dmitry Torokhov
Quoted 6 lines · Dominik Brodowski
4) SynPS/2 touchpad:
Why does moving the touchpad lead to sooo many IRQs? I can't look as fast as the mouse pointer seems to get new data:
62,5% (473,1) <interrupt> : PS/2 keyboard/mouse/touchpad
Quoted 9 lines
Quoted 5 lines
80 pps @ 6 bytes/packet = 480 interrupts/sec.
You can try using psmouse.rate=40 to limit it to 40 pps which should bring it to the rate of standard PS/2 mouse at the expense of sensitivity...
Excellent. Maybe this could be added to the Tips&Tricks section at the Powertop website?
Do not see the point - the touchpad does not generate interrupts while you are not touching it and while you are touching it the machine is not going to sleep. The only reason I see for lowering the rate is if your keyboard contrioller can not handle it (some Toshibas couldn't).
Quoted 1 line
I guess obtaining all 6 bytes at once is not really possible?
No. i8042 is byte-oriented.
Quoted 9 lines
It
seems a new byte only appears ~1.75 ms after the last one, at least on my notebook, so waiting for this is not an option...Best,
DominikPS: Dmitry, got something small for you in return:
Thanks, will apply.
Quoted 19 lines
[PATCH] i8042: spelling fix
Signed-off-by Dominik Brodowski <[email protected]>drivers/input/serio/i8042.c +1
-1@@ -38,7 +38,7 @@ MODULE_PARM_DESC(noaux, "Do not probe or use AUX (mouse) port.");static bool i8042_nomux;module_param_named (nomux, i8042_nomux, bool,0 ); -MODULE_PARM_DESC (nomux,"Do not check whether an active multiplexing conrtoller is present." ); +MODULE_PARM_DESC (nomux,"Do not check whether an active multiplexing controller is present." );static bool i8042_unlock;module_param_named (unlock, i8042_unlock, bool,0 );
Signature · 1 line
Dmitry
Hey,
Quoted 8 lines · Dmitry Torokhov
Quoted 6 lines · Dominik Brodowski
4) SynPS/2 touchpad:
Why does moving the touchpad lead to sooo many IRQs? I can't look as fast as the mouse pointer seems to get new data:
62,5% (473,1) <interrupt> : PS/2 keyboard/mouse/touchpad
Quoted 5 lines
80 pps @ 6 bytes/packet = 480 interrupts/sec.
You can try using psmouse.rate=40 to limit it to 40 pps which should bring it to the rate of standard PS/2 mouse at the expense of sensitivity...
as a sidenote: if we know -- like here -- that the next IRQ will be issued soon, in approximately 1.75 ms (well, at least on my system), might it make sense to make tick_nohz_get_sleep_length() smarter to know about this?
Best,
Dominik
Quoted 13 lines · Dominik Brodowski
Hey,
Quoted 10 lines · Dmitry Torokhov
Quoted 6 lines · Dominik Brodowski
4) SynPS/2 touchpad:
Why does moving the touchpad lead to sooo many IRQs? I can't look as fast as the mouse pointer seems to get new data:62,5% (473,1)<interrupt> : PS/2 keyboard/mouse/touchpad80 pps @ 6 bytes/packet = 480 interrupts/sec.
Quoted 7 lines
Quoted 3 lines
You can try using psmouse.rate=40 to limit it to 40 pps which should bring it to the rate of standard PS/2 mouse at the expense of sensitivity...
as a sidenote: if we know -- like here -- that the next IRQ will be issued soon, in approximately 1.75 ms (well, at least on my system), might it make sense to make
tick_nohz_get_sleep_length()smarter to know about this?
yes and no.
if you are very sure (95%+ or so) then absolutely it needs to know about this so that the C state selection code can make a better decision. Right now it tries to look at history to guess this delay.
Unfortunately we do not currently have such a concept in the code to make this work... but it'd be really nice to have.
Quoted 22 lines · Arjan van de Ven
Quoted 13 lines · Dominik Brodowski
Hey,
Quoted 10 lines · Dmitry Torokhov
Quoted 6 lines · Dominik Brodowski
4) SynPS/2 touchpad:
Why does moving the touchpad lead to sooo many IRQs? I can't look as fast as the mouse pointer seems to get new data:
62,5% (473,1)<interrupt> : PS/2 keyboard/mouse/touchpad80 pps @ 6 bytes/packet = 480 interrupts/sec.
Quoted 7 lines
Quoted 3 lines
You can try using psmouse.rate=40 to limit it to 40 pps which should bring it to the rate of standard PS/2 mouse at the expense of sensitivity...
as a sidenote: if we know -- like here -- that the next IRQ will be issued soon, in approximately 1.75 ms (well, at least on my system), might it make sense to make
tick_nohz_get_sleep_length()smarter to know about this?
Quoted 8 lines
yes and no.
if you are very sure (95%+ or so) then absolutely it needs to know about this so that the C state selection code can make a better decision. Right now it tries to look at history to guess this delay.
Unfortunately we do not currently have such a concept in the code to make this work... but it'd be really nice to have.
Dmitry, are we "very sure" in this touchpad case?
Best,
Dominik
Dmitry Torokhov
maintainerQuoted 25 lines · Dominik Brodowski
Quoted 24 lines · Arjan van de Ven
Quoted 13 lines · Dominik Brodowski
Hey,
Quoted 10 lines · Dmitry Torokhov
Quoted 6 lines · Dominik Brodowski
4) SynPS/2 touchpad:
Why does moving the touchpad lead to sooo many IRQs? I can't look as fast as the mouse pointer seems to get new data:
62,5% (473,1)<interrupt> : PS/2 keyboard/mouse/touchpad80 pps @ 6 bytes/packet = 480 interrupts/sec.
Quoted 7 lines
Quoted 3 lines
You can try using psmouse.rate=40 to limit it to 40 pps which should bring it to the rate of standard PS/2 mouse at the expense of sensitivity...
as a sidenote: if we know -- like here -- that the next IRQ will be issued soon, in approximately 1.75 ms (well, at least on my system), might it make sense to make
tick_nohz_get_sleep_length()smarter to know about this?yes and no.
Quoted 9 lines
Quoted 6 lines
if you are very sure (95%+ or so) then absolutely it needs to know about this so that the C state selection code can make a better decision. Right now it tries to look at history to guess this delay.
Unfortunately we do not currently have such a concept in the code to make this work... but it'd be really nice to have.
Dmitry, are we "very sure" in this touchpad case?
Psmouse driver tries to not rely on any timing data really... But yes, we do expect the next interrupt to arrive "shortly" and I guess the driver could do some data gathering to collect average time between interrupts. The question is - is it worth it?
Signature · 1 line
Dmitry
Quoted 6 lines · Dominik Brodowski
Booting a SMP-capable kernel with "nosmp", or manually offlining one CPU (or -- though I haven't tested it -- booting a SMP-capable kernel on a system with merely one CPU) means that in up to about half of the calls to
tick_nohz_stop_sched_tick()are aborted due torcu_needs_cpu(). This is quite strange to me: AFAIK, RCU is an excellent tool for SMP, but not really needed for UP?
I can't answer the real question here, not knowing enough about the RCU implementation. However, your impression is wrong: RCU very definitely _is_ useful and needed on UP systems. It coordinates among processes (and interrupt handlers) as well as among processors.
Quoted 13 lines
3) USB: built-in UHCI and a built-in 0a5c:2101 Broadcom Corp. A-Link BlueUsbA2 Bluetooth module; built-in EHCI and a built-in 0ac8:c302 Z-Star Microelectronics Corp. Vega USB 2.0 Camera.
usbcore.autosuspend is enabled (= 2), of course.
Recent USB suspend statistics
Active Device name
100.0% USB device 7-1 : BCM92045NMD (Broadcom Corp)
100.0% USB device 1-2 : Vega USB 2.0 Camera. (Vimicro Corp.)
100.0% USB device usb7 : UHCI Host Controller (Linux 2.6.34-rc3uhci_hcd)
100.0% USB device usb1 : EHCI Host Controller (Linux 2.6.34-rc3ehci_hcd)
Quoted 5 lines
Booting into
/bin/bashon a SMP kernel booted with "nosmp" leads to ~ 10 wakeups per second; disabling the cursor helps halfway (~ 5 wakeups); and manually unbinding the USB host drivers from the USB host devices finally lead to ~ 1.1 wakeups per second. What's keeping USB from suspending these unused devices here?
Either the drivers don't support autosuspend or the devices aren't enabled for autosuspend. By default, autosuspend is disabled for (almost) all non-hub devices. You or your distribution must enable it manually by doing:
echo auto >/sys/bus/usb/devices/.../power/level
If the driver doesn't support autosuspend then enabling it won't be enough; you'll also have to unbind the driver from the device. The easiest way to do this is to unconfigure the device:
echo 0 >/sys/bus/usb/devices/.../bConfigurationValue
Alan Stern
Quoted 9 lines · Alan Stern
Quoted 6 lines · Dominik Brodowski
Booting a SMP-capable kernel with "nosmp", or manually offlining one CPU (or -- though I haven't tested it -- booting a SMP-capable kernel on a system with merely one CPU) means that in up to about half of the calls to
tick_nohz_stop_sched_tick()are aborted due torcu_needs_cpu(). This is quite strange to me: AFAIK, RCU is an excellent tool for SMP, but not really needed for UP?
Quoted 4 lines
I can't answer the real question here, not knowing enough about the RCU implementation. However, your impression is wrong: RCU very definitely _is_ useful and needed on UP systems. It coordinates among processes (and interrupt handlers) as well as among processors.
Okay, but still: can't this be sped up by much on UP (especially if CONFIG_RCU_FAST_NO_HZ is set), so that we can go to sleep right away?
Quoted 24 lines
Quoted 13 lines
3) USB: built-in UHCI and a built-in 0a5c:2101 Broadcom Corp. A-Link BlueUsbA2 Bluetooth module; built-in EHCI and a built-in 0ac8:c302 Z-Star Microelectronics Corp. Vega USB 2.0 Camera.
usbcore.autosuspend is enabled (= 2), of course.
Recent USB suspend statistics
Active Device name
100.0% USB device 7-1 : BCM92045NMD (Broadcom Corp)
100.0% USB device 1-2 : Vega USB 2.0 Camera. (Vimicro Corp.)
100.0% USB device usb7 : UHCI Host Controller (Linux 2.6.34-rc3uhci_hcd)
100.0% USB device usb1 : EHCI Host Controller (Linux 2.6.34-rc3ehci_hcd)Quoted 5 lines
Booting into
/bin/bashon a SMP kernel booted with "nosmp" leads to ~ 10 wakeups per second; disabling the cursor helps halfway (~ 5 wakeups); and manually unbinding the USB host drivers from the USB host devices finally lead to ~ 1.1 wakeups per second. What's keeping USB from suspending these unused devices here?Either the drivers don't support autosuspend or the devices aren't enabled for autosuspend. By default, autosuspend is disabled for (almost) all non-hub devices. You or your distribution must enable it manually by doing:
Quoted 7 lines
echo auto >/sys/bus/usb/devices/.../power/level
If the driver doesn't support autosuspend then enabling it won't be enough; you'll also have to unbind the driver from the device. The easiest way to do this is to unconfigure the device:
echo 0 >/sys/bus/usb/devices/.../bConfigurationValue
Thanks! This way, it works, even without manually unbinding the host drivers.
Best,
Dominik
Quoted 10 lines · Dominik Brodowski
Quoted 9 lines · Alan Stern
Quoted 6 lines · Dominik Brodowski
Booting a SMP-capable kernel with "nosmp", or manually offlining one CPU (or -- though I haven't tested it -- booting a SMP-capable kernel on a system with merely one CPU) means that in up to about half of the calls to
tick_nohz_stop_sched_tick()are aborted due torcu_needs_cpu(). This is quite strange to me: AFAIK, RCU is an excellent tool for SMP, but not really needed for UP?
Quoted 7 lines
Quoted 4 lines
I can't answer the real question here, not knowing enough about the RCU implementation. However, your impression is wrong: RCU very definitely _is_ useful and needed on UP systems. It coordinates among processes (and interrupt handlers) as well as among processors.
Okay, but still: can't this be sped up by much on UP (especially if
CONFIG_RCU_FAST_NO_HZis set), so that we can go to sleep right away?
One situation that will prevent CONFIG_RCU_FAST_NO_HZ from putting the machine to sleep right away is if there is an RCU callback posted that spawns another RCU callback, and so on. CONFIG_RCU_FAST_NO_HZ will handle one callback that spawns another, but it gives up if the second callback spawns a third.
Might this be what is happening to you?
If so, would you be willing to patch your kernel? RCU_NEEDS_CPU_FLUSHES is currently set to 5, and might be set to (say) 8. This is defined in kernel/rcutree_plugin.h, near line 990.
Another thing to try would be running with TINY_RCU, at least if it is OK that RCU be non-preemptible.
Thanx, Paul
Quoted 42 lines
Quoted 24 lines
Quoted 13 lines
3) USB: built-in UHCI and a built-in 0a5c:2101 Broadcom Corp. A-Link BlueUsbA2 Bluetooth module; built-in EHCI and a built-in 0ac8:c302 Z-Star Microelectronics Corp. Vega USB 2.0 Camera.
usbcore.autosuspend is enabled (= 2), of course.
Recent USB suspend statistics
Active Device name
100.0% USB device 7-1 : BCM92045NMD (Broadcom Corp)
100.0% USB device 1-2 : Vega USB 2.0 Camera. (Vimicro Corp.)
100.0% USB device usb7 : UHCI Host Controller (Linux 2.6.34-rc3uhci_hcd)
100.0% USB device usb1 : EHCI Host Controller (Linux 2.6.34-rc3ehci_hcd)Quoted 5 lines
Booting into
/bin/bashon a SMP kernel booted with "nosmp" leads to ~ 10 wakeups per second; disabling the cursor helps halfway (~ 5 wakeups); and manually unbinding the USB host drivers from the USB host devices finally lead to ~ 1.1 wakeups per second. What's keeping USB from suspending these unused devices here?Either the drivers don't support autosuspend or the devices aren't enabled for autosuspend. By default, autosuspend is disabled for (almost) all non-hub devices. You or your distribution must enable it manually by doing:
Quoted 7 lines
echo auto >/sys/bus/usb/devices/.../power/level
If the driver doesn't support autosuspend then enabling it won't be enough; you'll also have to unbind the driver from the device. The easiest way to do this is to unconfigure the device:
echo 0 >/sys/bus/usb/devices/.../bConfigurationValue
Thanks! This way, it works, even without manually unbinding the host drivers.
Best,
Dominik
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 25 lines · Paul E. McKenney
Quoted 10 lines · Dominik Brodowski
Quoted 9 lines · Alan Stern
Quoted 6 lines · Dominik Brodowski
Booting a SMP-capable kernel with "nosmp", or manually offlining one CPU (or -- though I haven't tested it -- booting a SMP-capable kernel on a system with merely one CPU) means that in up to about half of the calls to
tick_nohz_stop_sched_tick()are aborted due torcu_needs_cpu(). This is quite strange to me: AFAIK, RCU is an excellent tool for SMP, but not really needed for UP?Quoted 7 lines
Quoted 4 lines
I can't answer the real question here, not knowing enough about the RCU implementation. However, your impression is wrong: RCU very definitely _is_ useful and needed on UP systems. It coordinates among processes (and interrupt handlers) as well as among processors.
Okay, but still: can't this be sped up by much on UP (especially if
CONFIG_RCU_FAST_NO_HZis set), so that we can go to sleep right away?One situation that will prevent
CONFIG_RCU_FAST_NO_HZfrom putting the machine to sleep right away is if there is an RCU callback posted that spawns another RCU callback, and so on.CONFIG_RCU_FAST_NO_HZwill handle one callback that spawns another, but it gives up if the second callback spawns a third.
Quoted 8 lines
Might this be what is happening to you?
If so, would you be willing to patch your kernel?
RCU_NEEDS_CPU_FLUSHESis currently set to 5, and might be set to (say) 8. This is defined inkernel/rcutree_plugin.h, near line 990.Another thing to try would be running with
TINY_RCU, at least if it is OK that RCU be non-preemptible.
And you did mention offlining some CPUs above. The folloiwng patch (from Lai Jiangshan) is needed to handle this case.
Thanx, Paul
------------------------------------------------------------------------
commit 6a2ae79877827355b747c0b91133a963b74ed396
Author: Lai Jiangshan <[email protected]> Date: Tue Mar 30 18:40:36 2010 +0800
rcu: ignore offline CPUs in last non-dyntick-idle CPU check
Offline CPUs are not in nohz_cpu_mask, but can be ignored when checking for the last non-dyntick-idle CPU. This patch therefore only checks online CPUs for not being dyntick idle, allowing fast entry into full-system dyntick-idle state even when there are some offline CPUs.
kernel/rcutree_plugin.h +1 -1
@@ -1016,7 +1016,7 @@ int rcu_needs_cpu(int cpu)/* Don't bother unless we are the last non-dyntick-idle CPU. */ for_each_cpu_not (thatcpu, nohz_cpu_mask) -if (thatcpu != cpu) { +if (cpu_online (thatcpu) && thatcpu != cpu) {per_cpu (rcu_dyntick_drain, cpu) =0 ;per_cpu (rcu_dyntick_holdoff, cpu) = jiffies -1 ;return rcu_needs_cpu_quick_check (cpu);
Quoted 2 lines
And you did mention offlining some CPUs above. The folloiwng patch (from Lai Jiangshan) is needed to handle this case.
btw on x86... don't offline CPUs if you want to save power.. it doesn't.
(at least not during idle.. and when you're busy it might save power, but it won't save you energy normally)
Quoted 6 lines · Arjan van de Ven
Quoted 2 lines
And you did mention offlining some CPUs above. The folloiwng patch (from Lai Jiangshan) is needed to handle this case.
btw on x86... don't offline CPUs if you want to save power.. it doesn't. (at least not during idle.. and when you're busy it might save power, but it won't save you energy normally)
Hmmm... The fact that offlining CPUs doesn't save power could form the basis of an interesting rationalization for my having ignored offlined CPUs in my original patch, I suppose. ;-)
So the proper approach is to affinity everything away from the CPUs in question so that they stay in dyntick-idle mode? I must confess that I find this quite counter-intuitive -- and I suspect that I am not the only one who would expect offlined CPUs to drop to the lowest possible power consumption.
Thanx, Paul
Quoted 8 lines · Paul E. McKenney
Quoted 6 lines · Arjan van de Ven
Quoted 2 lines
And you did mention offlining some CPUs above. The folloiwng patch (from Lai Jiangshan) is needed to handle this case.
btw on x86... don't offline CPUs if you want to save power.. it doesn't. (at least not during idle.. and when you're busy it might save power, but it won't save you energy normally)
Quoted 6 lines
Hmmm... The fact that offlining CPUs doesn't save power could form the basis of an interesting rationalization for my having ignored offlined CPUs in my original patch, I suppose. ;-)
So the proper approach is to affinity everything away from the CPUs in question so that they stay in dyntick-idle mode?
that is actually equivalently bad ;)
Quoted 4 lines
I must confess
that I find this quite counter-intuitive -- and I suspect that I am not the only one who would expect offlined CPUs to drop to the lowest possible power consumption.
on x86 (other archs might be different), nowadays idle is VERY efficient. (and the way to offline a cpu is to put it into the same deep C state as idle would)
the "offline is same power as idle" is only the part where you don't win energy. the part where you lose energy is the part where you realize that you can only really put the memory controller and memory in power saving state if all cpus are in idle.
If you offline a cpu (versus leaving it idle), and you have several tasks to run during an activity burst (which are more common the more we group activity to save power), it can happen that there are so many things to do that all remaining cpus can't handle it without the scheduler delaying some tasks. At which point you delay putting the memory (controllers) in low power state and you lose. In general, the winning strategy seems to be to finish things as quickly as you can, eg on as many cpus as you have in the system to then let them all go idle.
(this is not always true if you're picking cpu frequency/voltages, but for idle states it tends to be true. there's always cases where it isn't.... power is like that)
Quoted 36 lines · Arjan van de Ven
Quoted 8 lines · Paul E. McKenney
Quoted 6 lines · Arjan van de Ven
Quoted 2 lines
And you did mention offlining some CPUs above. The folloiwng patch (from Lai Jiangshan) is needed to handle this case.
btw on x86... don't offline CPUs if you want to save power.. it doesn't. (at least not during idle.. and when you're busy it might save power, but it won't save you energy normally)
Quoted 6 lines
Hmmm... The fact that offlining CPUs doesn't save power could form the basis of an interesting rationalization for my having ignored offlined CPUs in my original patch, I suppose. ;-)
So the proper approach is to affinity everything away from the CPUs in question so that they stay in dyntick-idle mode?
that is actually equivalently bad ;)
Quoted 4 lines
I must confess
that I find this quite counter-intuitive -- and I suspect that I am not the only one who would expect offlined CPUs to drop to the lowest possible power consumption.on x86 (other archs might be different), nowadays idle is VERY efficient. (and the way to offline a cpu is to put it into the same deep C state as idle would)
the "offline is same power as idle" is only the part where you don't win energy. the part where you lose energy is the part where you realize that you can only really put the memory controller and memory in power saving state if all cpus are in idle.
If you offline a cpu (versus leaving it idle), and you have several tasks to run during an activity burst (which are more common the more we group activity to save power), it can happen that there are so many things to do that all remaining cpus can't handle it without the scheduler delaying some tasks. At which point you delay putting the memory (controllers) in low power state and you lose. In general, the winning strategy seems to be to finish things as quickly as you can, eg on as many cpus as you have in the system to then let them all go idle.
Quoted 2 lines
(this is not always true if you're picking cpu frequency/voltages, but for idle states it tends to be true. there's always cases where it isn't.... power is like that)
So the main issue is that for many workloads, it is best to run full bore and get done quickly, thus allowing the entire machine to be powered down?
If so, it seems likely that there would be some workloads that were sometimes unable to use all the CPUs, in which case shutting down (idling, offlining, dyntick-idling, whatever) the excess CPUs might nevertheless be the right thing to do.
Or am I missing your point?
Thanx, Paul
Quoted 2 lines · Paul E. McKenney
So the main issue is that for many workloads, it is best to run full bore and get done quickly, thus allowing the entire machine to be powered down?
yep
Quoted 5 lines
If so, it seems likely that there would be some workloads that were sometimes unable to use all the CPUs, in which case shutting down (idling, offlining, dyntick-idling, whatever) the excess CPUs might nevertheless be the right thing to do.
but the point is that the normal scheduler + idle behavior gives you exactly that in a natural way !
If you don't have enough work (tasks) to keep all cores busy, the others are and stay idle.
Quoted 6 lines · Arjan van de Ven
Quoted 2 lines · Paul E. McKenney
So the main issue is that for many workloads, it is best to run full bore and get done quickly, thus allowing the entire machine to be powered down?
yep
Quoted 8 lines
Quoted 4 lines
If so, it seems likely that there would be some workloads that were sometimes unable to use all the CPUs, in which case shutting down (idling, offlining, dyntick-idling, whatever) the excess CPUs might nevertheless be the right thing to do.
but the point is that the normal scheduler + idle behavior gives you exactly that in a natural way !
If you don't have enough work (tasks) to keep all cores busy, the others are and stay idle.
So your earlier objection was not to dyntick-idle as such, but rather to artificially constraining the scheduler to induce dyntick-idle?
Thanx, Paul
Quoted 12 lines · Paul E. McKenney
Quoted 11 lines · Arjan van de Ven
Quoted 2 lines · Paul E. McKenney
So the main issue is that for many workloads, it is best to run full bore and get done quickly, thus allowing the entire machine to be powered down?
yep
Quoted 4 lines
If so, it seems likely that there would be some workloads that were sometimes unable to use all the CPUs, in which case shutting down (idling, offlining, dyntick-idling, whatever) the excess CPUs might nevertheless be the right thing to do.
Quoted 6 lines
Quoted 3 lines
but the point is that the normal scheduler + idle behavior gives you exactly that in a natural way !
If you don't have enough work (tasks) to keep all cores busy, the others are and stay idle.So your earlier objection was not to dyntick-idle as such, but rather to artificially constraining the scheduler to induce dyntick-idle?
my objection was against the notion that offlining cpus helps power/energy ;-)
Quoted 13 lines · Arjan van de Ven
Quoted 12 lines · Paul E. McKenney
Quoted 11 lines · Arjan van de Ven
Quoted 2 lines · Paul E. McKenney
So the main issue is that for many workloads, it is best to run full bore and get done quickly, thus allowing the entire machine to be powered down?
yep
Quoted 4 lines
If so, it seems likely that there would be some workloads that were sometimes unable to use all the CPUs, in which case shutting down (idling, offlining, dyntick-idling, whatever) the excess CPUs might nevertheless be the right thing to do.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
but the point is that the normal scheduler + idle behavior gives you exactly that in a natural way !
If you don't have enough work (tasks) to keep all cores busy, the others are and stay idle.So your earlier objection was not to dyntick-idle as such, but rather to artificially constraining the scheduler to induce dyntick-idle?
my objection was against the notion that offlining cpus helps power/energy ;-)
Fair enough, at least in general. I should hasten to add that Lai's patch also helps in the case where NR_CPUS is greater than the number of CPUs on the system.
Thanx, Paul
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines · Paul E. McKenney
So the main issue is that for many workloads, it is best to run full bore and get done quickly, thus allowing the entire machine to be powered down?
yep
Race To Idle works extremely well in a batch type situation where there is not going to be any work to do after you finish what you have.
It doesn't work quite as well if you are going to have new work to do in the near future.
You cannot power down the entire machine if you have to look for user input.
It takes time (and power) to shut down and start back up, if you are going to have more work to do before you can make the complete cycle (and save more power than it costs to make the transitions), it's best to stay at full power, even if you are idle.
As an example, video/audio playback.
This requires relativly little cpu, but it needs it frequently (to keep the hardware buffers filled), and you cannot power down even when the cpu is idle.
But you could save power by disabling cores, switching to a slower clock speed, etc while still having one core remaining awake all the time.
The key is to look at what you are waiting for. If you are just waiting for the processing to finish, race to idle is great. However if you are waiting for the outside world or for a clock tick you need to look into the exact situation more closely.
David Lang
Quoted 14 lines
Quoted 4 lines
If so, it seems likely that there would be some workloads that were sometimes unable to use all the CPUs, in which case shutting down (idling, offlining, dyntick-idling, whatever) the excess CPUs might nevertheless be the right thing to do.
but the point is that the normal scheduler + idle behavior gives you exactly that in a natural way !
If you don't have enough work (tasks) to keep all cores busy, the others are and stay idle. --
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 16 lines · On 4/5/2010 11:44, [email protected] wrote:
Quoted 7 lines · Arjan van de Ven
Quoted 4 lines · Paul E. McKenney
So the main issue is that for many workloads, it is best to run full bore
and get done quickly, thus allowing the entire machine to be powered down?yep
Race To Idle works extremely well in a batch type situation where there is not going to be any work to do after you finish what you have.
It doesn't work quite as well if you are going to have new work to do in the near future.
Quoted 7 lines
You cannot power down the entire machine if you have to look for user input.
It takes time (and power) to shut down and start back up, if you are going to have more work to do before you can make the complete cycle (and save more power than it costs to make the transitions), it's best to stay at full power, even if you are idle.
for the things we're talking about here (memory controllers etc) we're talking about single to low double digit microseconds (or even less) of time to go up and down. Many of the things you talk about are in the millisecond timeframe.
Quoted 20 lines · Arjan van de Ven
Quoted 19 lines · On 4/5/2010 11:44, [email protected] wrote:
Quoted 7 lines · Arjan van de Ven
Quoted 4 lines · Paul E. McKenney
So the main issue is that for many workloads, it is best to run full bore
and get done quickly, thus allowing the entire machine to be powered down?yep
Race To Idle works extremely well in a batch type situation where there is not going to be any work to do after you finish what you have.
It doesn't work quite as well if you are going to have new work to do in the near future.
You cannot power down the entire machine if you have to look for user input.
Quoted 8 lines
Quoted 4 lines
It takes time (and power) to shut down and start back up, if you are going to have more work to do before you can make the complete cycle (and save more power than it costs to make the transitions), it's best to stay at full power, even if you are idle.
for the things we're talking about here (memory controllers etc) we're talking about single to low double digit microseconds (or even less) of time to go up and down. Many of the things you talk about are in the millisecond timeframe.
So the decision will depend not only on the workload itself, but also on the hardware that the workload uses.
Thanx, Paul
Paul,
I really appreaciate your reply -- thanks! I've done some more testing in the meantime:
Quoted 16 lines · Paul E. McKenney
Quoted 15 lines · Dominik Brodowski
Quoted 13 lines · Alan Stern
Quoted 6 lines · Dominik Brodowski
Booting a SMP-capable kernel with "nosmp", or manually offlining one CPU (or -- though I haven't tested it -- booting a SMP-capable kernel on a system with merely one CPU) means that in up to about half of the calls to
tick_nohz_stop_sched_tick()are aborted due torcu_needs_cpu(). This is quite strange to me: AFAIK, RCU is an excellent tool for SMP, but not really needed for UP?I can't answer the real question here, not knowing enough about the RCU implementation. However, your impression is wrong: RCU very definitely _is_ useful and needed on UP systems. It coordinates among processes (and interrupt handlers) as well as among processors.
Quoted 8 lines
Quoted 2 lines
Okay, but still: can't this be sped up by much on UP (especially if
CONFIG_RCU_FAST_NO_HZis set), so that we can go to sleep right away?One situation that will prevent
CONFIG_RCU_FAST_NO_HZfrom putting the machine to sleep right away is if there is an RCU callback posted that spawns another RCU callback, and so on.CONFIG_RCU_FAST_NO_HZwill handle one callback that spawns another, but it gives up if the second callback spawns a third.
Will the remaining callbacks be executed immediately afterwards (due to a need_resched() etc.), or only after the next tick?
Quoted 5 lines
Might this be what is happening to you?
If so, would you be willing to patch your kernel?
RCU_NEEDS_CPU_FLUSHESis currently set to 5, and might be set to (say) 8. This is defined inkernel/rcutree_plugin.h, near line 990.
Applied the patch by Lai Jiangshan, and tested 5 and 8:
5: Wakeups-from-idle: 33.4 (hrtimer_sched_timer: 78 %)
34% of calls to tick_nohz_stop_sched_tick fail due to rcu_needs_cpu() 8: Wakeups-from-idle: 36.5 (hrtimer_sched_timer: 83 %) 37% of calls to tick_nohz_stop_sched_tick fail due to rcu_needs_cpu()
Quoted 2 lines
Another thing to try would be running with
TINY_RCU, at least if it is OK that RCU be non-preemptible.
tick_nohz_stop_sched_tick() doesn't fail in this case because of rcu_needs_cpu(). However, the improvements are hardly recognizable:
TINY_RCU: Wakeups-from-idle: 33.9 (hrtimer_sched_timer: 53 %)
Quoted 1 line
And you did mention offlining some CPUs above.
... just for testing how NOHZ works on UP systems ;)
Best,
Dominik
Quoted 25 lines · Dominik Brodowski
Paul,
I really appreaciate your reply -- thanks! I've done some more testing in the meantime:
Quoted 19 lines · Paul E. McKenney
Quoted 17 lines · Dominik Brodowski
Quoted 9 lines · Alan Stern
Quoted 6 lines · Dominik Brodowski
Booting a SMP-capable kernel with "nosmp", or manually offlining one CPU (or -- though I haven't tested it -- booting a SMP-capable kernel on a system with merely one CPU) means that in up to about half of the calls to
tick_nohz_stop_sched_tick()are aborted due torcu_needs_cpu(). This is quite strange to me: AFAIK, RCU is an excellent tool for SMP, but not really needed for UP?Quoted 4 lines
I can't answer the real question here, not knowing enough about the RCU implementation. However, your impression is wrong: RCU very definitely _is_ useful and needed on UP systems. It coordinates among processes (and interrupt handlers) as well as among processors.
Okay, but still: can't this be sped up by much on UP (especially if
CONFIG_RCU_FAST_NO_HZis set), so that we can go to sleep right away?
Quoted 8 lines
Quoted 5 lines
One situation that will prevent
CONFIG_RCU_FAST_NO_HZfrom putting the machine to sleep right away is if there is an RCU callback posted that spawns another RCU callback, and so on.CONFIG_RCU_FAST_NO_HZwill handle one callback that spawns another, but it gives up if the second callback spawns a third.Will the remaining callbacks be executed immediately afterwards (due to a
need_resched()etc.), or only after the next tick?
Only after the next tick. To see why, imagine an RCU callback that re-registers itself -- which is a perfectly legal thing to do. The only thing that will happen if we run through grace periods faster is that we will have more invocations of that same callback to deal with.
So we try for a bit, and if that doesn't get rid of all of the callbacks, we hold off until the next jiffy.
Quoted 6 lines
Quoted 5 lines
Might this be what is happening to you?
If so, would you be willing to patch your kernel?
RCU_NEEDS_CPU_FLUSHESis currently set to 5, and might be set to (say) 8. This is defined inkernel/rcutree_plugin.h, near line 990.
Quoted 8 lines
Applied the patch by Lai Jiangshan, and tested 5 and 8:
5: Wakeups-from-idle: 33.4 (
hrtimer_sched_timer: 78 %)34% of calls to tick_nohz_stop_sched_tick fail due to rcu_needs_cpu() 8: Wakeups-from-idle: 36.5 (hrtimer_sched_timer: 83 %) 37% of calls to tick_nohz_stop_sched_tick fail due to rcu_needs_cpu()
I don't recall your posting wakeups-from-idle for the original -- did we get improvement? You did say "roughly 50%", but...
OK, I see what is happening...
What happens in the CONFIG_RCU_FAST_NO_HZ case is as follows:
o Check to see if the holdoff period is in effect, and if so,
just check to see if RCU needs the CPU for later processing without attempting to accelerate grace periods.
o Check to see if there is some other non-dyntick-idle CPU.
If there is, reset holdoff state and just check to see if RCU needs the CPU for later processing without attempting to accelerate grace periods.
o Check for initialization and hitting the RCU_NEEDS_CPU_FLUSHES
limit, again doing the "just check" thing if we hit the limit.
o For each of RCU-sched and RCU-bh, note a quiescent state
and force the grace-period machinery, noting in each case whether or not there are callbacks left to invoke.
o If there are callbacks left to invoke, raise RCU_SOFTIRQ.
This softirq will process the callbacks. (Why not just invoke
the softirq function directly? Because lockdep yells at you
and I do not believe that this is a false positive.)
o If there are callbacks left to invoke, tell the caller that
this CPU cannot yet enter dyntick-idle state.
But if we told the caller that this CPU cannot yet enter dyntick-idle state, then we also raised RCU_SOFTIRQ. Once the softirq returns, we should once again try to enter dyntick-idle state.
So a significant fraction of calls to rcu_needs_cpu() saying "no" does not necessarily mean that we are taking significant time to get the grace periods and callbacks out of the way. The funny loop involving softirq is required due to locking-design issues.
Or are you seeing significant delays between successive calls to rcu_needs_cpu() on your setup?
Quoted 7 lines
Quoted 2 lines
Another thing to try would be running with
TINY_RCU, at least if it is OK that RCU be non-preemptible.
tick_nohz_stop_sched_tick()doesn't fail in this case because ofrcu_needs_cpu(). However, the improvements are hardly recognizable:
TINY_RCU: Wakeups-from-idle: 33.9 (hrtimer_sched_timer: 53 %)
TINY_RCU is set up to automatically do CONFIG_RCU_FAST_NO_HZ, and do the same softirq dance, or that is the theory, anyway. Again, are you seeing significant delays between successive calls to rcu_needs_cpu()?
Quoted 3 lines
Quoted 1 line
And you did mention offlining some CPUs above.
... just for testing how NOHZ works on UP systems ;)
;-)
Thanx, Paul
Quoted 58 lines · Paul E. McKenney
Quoted 25 lines · Dominik Brodowski
Paul,
I really appreaciate your reply -- thanks! I've done some more testing in the meantime:
Quoted 19 lines · Paul E. McKenney
Quoted 17 lines · Dominik Brodowski
Quoted 9 lines · Alan Stern
Quoted 6 lines · Dominik Brodowski
Booting a SMP-capable kernel with "nosmp", or manually offlining one CPU (or -- though I haven't tested it -- booting a SMP-capable kernel on a system with merely one CPU) means that in up to about half of the calls to
tick_nohz_stop_sched_tick()are aborted due torcu_needs_cpu(). This is quite strange to me: AFAIK, RCU is an excellent tool for SMP, but not really needed for UP?Quoted 4 lines
I can't answer the real question here, not knowing enough about the RCU implementation. However, your impression is wrong: RCU very definitely _is_ useful and needed on UP systems. It coordinates among processes (and interrupt handlers) as well as among processors.
Okay, but still: can't this be sped up by much on UP (especially if
CONFIG_RCU_FAST_NO_HZis set), so that we can go to sleep right away?Quoted 8 lines
Quoted 5 lines
One situation that will prevent
CONFIG_RCU_FAST_NO_HZfrom putting the machine to sleep right away is if there is an RCU callback posted that spawns another RCU callback, and so on.CONFIG_RCU_FAST_NO_HZwill handle one callback that spawns another, but it gives up if the second callback spawns a third.Will the remaining callbacks be executed immediately afterwards (due to a
need_resched()etc.), or only after the next tick?Only after the next tick. To see why, imagine an RCU callback that re-registers itself -- which is a perfectly legal thing to do. The only thing that will happen if we run through grace periods faster is that we will have more invocations of that same callback to deal with.
So we try for a bit, and if that doesn't get rid of all of the callbacks, we hold off until the next jiffy.
Quoted 6 lines
Quoted 5 lines
Might this be what is happening to you?
If so, would you be willing to patch your kernel?
RCU_NEEDS_CPU_FLUSHESis currently set to 5, and might be set to (say) 8. This is defined inkernel/rcutree_plugin.h, near line 990.Quoted 8 lines
Applied the patch by Lai Jiangshan, and tested 5 and 8:
5: Wakeups-from-idle: 33.4 (
hrtimer_sched_timer: 78 %)34% of calls to tick_nohz_stop_sched_tick fail due to rcu_needs_cpu() 8: Wakeups-from-idle: 36.5 (hrtimer_sched_timer: 83 %) 37% of calls to tick_nohz_stop_sched_tick fail due to rcu_needs_cpu()
Quoted 2 lines
I don't recall your posting wakeups-from-idle for the original -- did we get improvement? You did say "roughly 50%", but...
Actually, no. I'd say the 5-to-8 change has no significant effect at all; for the Patch by Lai Jiangshan, I'd need to re-run the test.
Quoted 32 lines
OK, I see what is happening...
What happens in the
CONFIG_RCU_FAST_NO_HZcase is as follows:o Check to see if the holdoff period is in effect, and if so,
just check to see if RCU needs the CPU for later processing without attempting to accelerate grace periods.o Check to see if there is some other non-dyntick-idle CPU.
If there is, reset holdoff state and just check to see if RCU needs the CPU for later processing without attempting to accelerate grace periods.o Check for initialization and hitting the
RCU_NEEDS_CPU_FLUSHES
limit, again doing the "just check" thing if we hit the limit.o For each of RCU-sched and RCU-bh, note a quiescent state
and force the grace-period machinery, noting in each case whether or not there are callbacks left to invoke.o If there are callbacks left to invoke, raise
RCU_SOFTIRQ.
This softirq will process the callbacks. (Why not just invoke
the softirq function directly? Because lockdep yells at you
and I do not believe that this is a false positive.)o If there are callbacks left to invoke, tell the caller that
this CPU cannot yet enter dyntick-idle state.But if we told the caller that this CPU cannot yet enter dyntick-idle state, then we also raised
RCU_SOFTIRQ. Once the softirq returns, we should once again try to enter dyntick-idle state.
Quoted 7 lines
So a significant fraction of calls to
rcu_needs_cpu()saying "no" does not necessarily mean that we are taking significant time to get the grace periods and callbacks out of the way. The funny loop involving softirq is required due to locking-design issues.Or are you seeing significant delays between successive calls to
rcu_needs_cpu()on your setup?
Will check this, but all the data I'm seeing points to rcu_needs_cpu() not leading to additional wakeups. It might just be wrong reports by powertop, after all, for the UP case. Quoting my original mail:
Quoted 14 lines
5) powertop and
hrtimer_start_range_ns(tick_sched_timer) on a SMP kernel booted with "nosmp":Wakeups-from-idle per second : 9.9 interval: 15.0s
...48.5% ( 9.4) <kernel core> : hrtimer_start (tick_sched_timer) 26.1% ( 5.1) <kernel core> : cursor_timer_handler (cursor_timer_handle 20.6% ( 4.0) <kernel core> : usb_hcd_poll_rh_status (rh_timer_func) 1.0% ( 0.2) <kernel core> : arm_supers_timer(
sync_supers_timer_fn)
0.7% ( 0.1) <interrupt> :ata_piix
...
Quoted 5 lines
Accoding to http://www.linuxpowertop.org , the count in the brackets is how
many wakeups per seconds were caused by one source. Adding all _except_
48.5% ( 9.4) <kernel core> :hrtimer_start(tick_sched_timer)
up leads to the 9.9.
Back to your mail:
Quoted 8 lines
Quoted 4 lines
tick_nohz_stop_sched_tick()doesn't fail in this case because ofrcu_needs_cpu(). However, the improvements are hardly recognizable:
TINY_RCU: Wakeups-from-idle: 33.9 (hrtimer_sched_timer: 53 %)
TINY_RCUis set up to automatically doCONFIG_RCU_FAST_NO_HZ, and do the same softirq dance, or that is the theory, anyway. Again, are you seeing significant delays between successive calls torcu_needs_cpu()?
Actually, rcu_needs_cpu() is statically defined to return 0 on TINY_RCU in include/linux/rcutiny.h .
Best,
Dominik
Quoted 102 lines · Dominik Brodowski
Quoted 58 lines · Paul E. McKenney
Quoted 25 lines · Dominik Brodowski
Paul,
I really appreaciate your reply -- thanks! I've done some more testing in the meantime:
Quoted 19 lines · Paul E. McKenney
Quoted 17 lines · Dominik Brodowski
Quoted 9 lines · Alan Stern
Quoted 6 lines · Dominik Brodowski
Booting a SMP-capable kernel with "nosmp", or manually offlining one CPU (or -- though I haven't tested it -- booting a SMP-capable kernel on a system with merely one CPU) means that in up to about half of the calls to
tick_nohz_stop_sched_tick()are aborted due torcu_needs_cpu(). This is quite strange to me: AFAIK, RCU is an excellent tool for SMP, but not really needed for UP?Quoted 4 lines
I can't answer the real question here, not knowing enough about the RCU implementation. However, your impression is wrong: RCU very definitely _is_ useful and needed on UP systems. It coordinates among processes (and interrupt handlers) as well as among processors.
Okay, but still: can't this be sped up by much on UP (especially if
CONFIG_RCU_FAST_NO_HZis set), so that we can go to sleep right away?Quoted 8 lines
Quoted 5 lines
One situation that will prevent
CONFIG_RCU_FAST_NO_HZfrom putting the machine to sleep right away is if there is an RCU callback posted that spawns another RCU callback, and so on.CONFIG_RCU_FAST_NO_HZwill handle one callback that spawns another, but it gives up if the second callback spawns a third.Will the remaining callbacks be executed immediately afterwards (due to a
need_resched()etc.), or only after the next tick?Only after the next tick. To see why, imagine an RCU callback that re-registers itself -- which is a perfectly legal thing to do. The only thing that will happen if we run through grace periods faster is that we will have more invocations of that same callback to deal with.
So we try for a bit, and if that doesn't get rid of all of the callbacks, we hold off until the next jiffy.
Quoted 6 lines
Quoted 5 lines
Might this be what is happening to you?
If so, would you be willing to patch your kernel?
RCU_NEEDS_CPU_FLUSHESis currently set to 5, and might be set to (say) 8. This is defined inkernel/rcutree_plugin.h, near line 990.Quoted 8 lines
Applied the patch by Lai Jiangshan, and tested 5 and 8:
5: Wakeups-from-idle: 33.4 (
hrtimer_sched_timer: 78 %)34% of calls to tick_nohz_stop_sched_tick fail due to rcu_needs_cpu() 8: Wakeups-from-idle: 36.5 (hrtimer_sched_timer: 83 %) 37% of calls to tick_nohz_stop_sched_tick fail due to rcu_needs_cpu()Quoted 2 lines
I don't recall your posting wakeups-from-idle for the original -- did we get improvement? You did say "roughly 50%", but...
Actually, no. I'd say the 5-to-8 change has no significant effect at all; for the Patch by Lai Jiangshan, I'd need to re-run the test.
Quoted 37 lines
OK, I see what is happening...
What happens in the
CONFIG_RCU_FAST_NO_HZcase is as follows:o Check to see if the holdoff period is in effect, and if so,
just check to see if RCU needs the CPU for later processing without attempting to accelerate grace periods.o Check to see if there is some other non-dyntick-idle CPU.
If there is, reset holdoff state and just check to see if RCU needs the CPU for later processing without attempting to accelerate grace periods.o Check for initialization and hitting the
RCU_NEEDS_CPU_FLUSHES
limit, again doing the "just check" thing if we hit the limit.o For each of RCU-sched and RCU-bh, note a quiescent state
and force the grace-period machinery, noting in each case whether or not there are callbacks left to invoke.o If there are callbacks left to invoke, raise
RCU_SOFTIRQ.
This softirq will process the callbacks. (Why not just invoke
the softirq function directly? Because lockdep yells at you
and I do not believe that this is a false positive.)o If there are callbacks left to invoke, tell the caller that
this CPU cannot yet enter dyntick-idle state.But if we told the caller that this CPU cannot yet enter dyntick-idle state, then we also raised
RCU_SOFTIRQ. Once the softirq returns, we should once again try to enter dyntick-idle state.So a significant fraction of calls to
rcu_needs_cpu()saying "no" does not necessarily mean that we are taking significant time to get the grace periods and callbacks out of the way. The funny loop involving softirq is required due to locking-design issues.
Quoted 6 lines
Quoted 2 lines
Or are you seeing significant delays between successive calls to
rcu_needs_cpu()on your setup?Will check this, but all the data I'm seeing points to
rcu_needs_cpu()not leading to additional wakeups. It might just be wrong reports by powertop, after all, for the UP case.
OK, for all I know, powertop might need some adjustment to allow for the presence of CONFIG_RCU_FAST_NO_HZ.
Quoted 21 lines
Quoting my original mail:
Quoted 19 lines
5) powertop and
hrtimer_start_range_ns(tick_sched_timer) on a SMP kernel booted with "nosmp":Wakeups-from-idle per second : 9.9 interval: 15.0s
...48.5% ( 9.4) <kernel core> : hrtimer_start (tick_sched_timer) 26.1% ( 5.1) <kernel core> : cursor_timer_handler (cursor_timer_handle 20.6% ( 4.0) <kernel core> : usb_hcd_poll_rh_status (rh_timer_func) 1.0% ( 0.2) <kernel core> : arm_supers_timer(
sync_supers_timer_fn)
0.7% ( 0.1) <interrupt> :ata_piix
...Accoding to http://www.linuxpowertop.org , the count in the brackets is how
many wakeups per seconds were caused by one source. Adding all _except_
48.5% ( 9.4) <kernel core> :hrtimer_start(tick_sched_timer)
up leads to the 9.9.
OK, so you further instrumented the hrtimer_sched_timer (or was it tick_sched_timer?) to find the number that you were attributing to rcu_needs_cpu()?
Quoted 5 lines
Back to your mail:
Quoted 3 lines
Quoted 3 lines
tick_nohz_stop_sched_tick()doesn't fail in this case because ofrcu_needs_cpu(). However, the improvements are hardly recognizable:
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
TINY_RCU: Wakeups-from-idle: 33.9 (hrtimer_sched_timer: 53 %)
TINY_RCUis set up to automatically doCONFIG_RCU_FAST_NO_HZ, and do the same softirq dance, or that is the theory, anyway. Again, are you seeing significant delays between successive calls torcu_needs_cpu()?Actually,
rcu_needs_cpu()is statically defined to return 0 onTINY_RCUininclude/linux/rcutiny.h.
Exactly! ;-)
Thanx, Paul
Quoted 20 lines · Paul E. McKenney
Quoted 14 lines
Quoted 14 lines
5) powertop and
hrtimer_start_range_ns(tick_sched_timer) on a SMP kernel booted with "nosmp":Wakeups-from-idle per second : 9.9 interval: 15.0s
...48.5% ( 9.4) <kernel core> : hrtimer_start (tick_sched_timer) 26.1% ( 5.1) <kernel core> : cursor_timer_handler (cursor_timer_handle 20.6% ( 4.0) <kernel core> : usb_hcd_poll_rh_status (rh_timer_func) 1.0% ( 0.2) <kernel core> : arm_supers_timer(
sync_supers_timer_fn)
0.7% ( 0.1) <interrupt> :ata_piix
...Quoted 5 lines
Quoted 5 lines
Accoding to http://www.linuxpowertop.org , the count in the brackets is how
many wakeups per seconds were caused by one source. Adding all _except_
48.5% ( 9.4) <kernel core> :hrtimer_start(tick_sched_timer)
up leads to the 9.9.
Quoted 3 lines
OK, so you further instrumented the
hrtimer_sched_timer(or was ittick_sched_timer?) to find the number that you were attributing torcu_needs_cpu()?
That's what I did -- to tick_nohz_stop_sched_tick(), to see why the tick_sched_timer did not get stopped. Or why powertop thinks it did not get stopped...
Patch below (works only for 1 or 2 CPUs) for all interested parties, _NOT_ intended for submission, though ;)
0x0001: CPU #10x0002: inidle0x0004: not inidle0x0008: INACTIVE0x0010: need_resched0x0020: softirq0x0040: rcu_needs_0x0080: printk_needs_0x0100: arch_needs_0x0200: off-by-one0x0400: skip0x0800: loadbalance:0x1000: cancel timer (SUCCESS)0x2000: timer was already in the past0x4000: hrtimer_start() (SUCCESS)0x8000: stop_tick (SUCCESS)
Fixing USB-autosuspend and disabling the fbcon cursor, I'm down to ~0.9 wakeups-per-second booting with "nosmp".
kernel/time/tick-sched.c +47 -7
@@ -210,19 +210,35 @@ void tick_nohz_stop_sched_tick(int inidle)struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev;u64 time_delta;int cpu; +static int testrun_counter =0 ; +int testrun =0 ; +u64 debugdata =0 ; + + testrun_counter++; + +if ((testrun_counter %123 ) ==0 ) + testrun =1 ;local_irq_save (flags); cpu =smp_processor_id (); ts = &per_cpu (tick_cpu_sched, cpu); + debugdata = cpu; + +if (inidle) + debugdata |=0x2 ; + +/* * Call to tick_nohz_start_idle stops the last_update_time from being * updated. Thus, it must not be called in the event we are called from * irq_exit() with the prior state different than idle. */ -if (!inidle && !ts->inidle) +if (!inidle && !ts->inidle) { + debugdata |=0x4 ; goto end; + }/* * Set ts->inidle unconditionally. Even if the system did not
@@ -245,11 +261,15 @@ void tick_nohz_stop_sched_tick(int inidle) tick_do_timer_cpu =TICK_DO_TIMER_NONE ; } -if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) +if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) { + debugdata |=0x8 ; goto end; + } -if (need_resched ()) +if (need_resched ()) { + debugdata |=0x10 ; goto end; + }if (unlikely (local_softirq_pending () &&cpu_online (cpu))) {static int ratelimit;
@@ -259,6 +279,7 @@ void tick_nohz_stop_sched_tick(int inidle) (unsigned int) local_softirq_pending()); ratelimit++; } + debugdata |=0x20 ; goto end; }
@@ -275,6 +296,12 @@ void tick_nohz_stop_sched_tick(int inidle) arch_needs_cpu(cpu)) { next_jiffies = last_jiffies +1 ; delta_jiffies =1 ; +if (rcu_needs_cpu (cpu)) + debugdata |=0x40 ; +if (printk_needs_cpu (cpu)) + debugdata |=0x80 ; +if (arch_needs_cpu (cpu)) + debugdata |=0x100 ; } else {/* Get the next timer wheel timer */ next_jiffies =get_next_timer_interrupt (last_jiffies);
@@ -284,8 +311,10 @@ void tick_nohz_stop_sched_tick(int inidle)* Do not stop the tick, if we are only one off * or if the cpu is required for rcu */ -if (!ts->tick_stopped && delta_jiffies ==1 ) +if (!ts->tick_stopped && delta_jiffies ==1 ) { + debugdata |=0x200 ; goto out; + }/* Schedule the tick, if we are at least one jiffie off */ if ((long )delta_jiffies >=1 ) {
@@ -341,8 +370,10 @@ void tick_nohz_stop_sched_tick(int inidle)cpumask_set_cpu (cpu, nohz_cpu_mask);/* Skip reprogram of event if its not changed */ -if (ts->tick_stopped &&ktime_equal (expires, dev->next_event)) +if (ts->tick_stopped &&ktime_equal (expires, dev->next_event)) { + debugdata |=0x400 ; goto out; + }/* * nohz_stop_sched_tick can be called several times before
@@ -357,8 +388,10 @@ void tick_nohz_stop_sched_tick(int inidle)* sched tick not stopped! */ cpumask_clear_cpu (cpu, nohz_cpu_mask); + debugdata |=0x800 ; goto out; } + debugdata |=0x8000 ; ts->idle_tick =hrtimer_get_expires (&ts->sched_timer); ts->tick_stopped =1 ;
@@ -375,7 +408,8 @@ void tick_nohz_stop_sched_tick(int inidle)* If the expiration time == KTIME_MAX, then * in this case we simply stop the tick timer. */ -if (unlikely (expires.tv64 ==KTIME_MAX )) { +if (unlikely (expires.tv64 ==KTIME_MAX )) { + debugdata |=0x1000 ;if (ts->nohz_mode ==NOHZ_MODE_HIGHRES )hrtimer_cancel (&ts->sched_timer); goto out;
@@ -384,9 +418,13 @@ void tick_nohz_stop_sched_tick(int inidle)if (ts->nohz_mode ==NOHZ_MODE_HIGHRES ) {hrtimer_start (&ts->sched_timer, expires,HRTIMER_MODE_ABS_PINNED ); + debugdata |=0x2000 ;/* Check, if the timer was already in the past */ -if (hrtimer_active (&ts->sched_timer)) +if (hrtimer_active (&ts->sched_timer)) { + debugdata &= ~(0x2000 ); + debugdata |=0x4000 ; goto out; + } }else if (!tick_program_event (expires,0 )) goto out;/*
@@ -403,6 +441,8 @@ out: ts->last_jiffies = last_jiffies; ts->sleep_length =ktime_sub (dev->next_event, now); end: +if (testrun) +printk (KERN_DEBUG "0x%0llx\n" , debugdata);local_irq_restore (flags); }
Quoted 25 lines · Dominik Brodowski
Quoted 20 lines · Paul E. McKenney
Quoted 14 lines
Quoted 14 lines
5) powertop and
hrtimer_start_range_ns(tick_sched_timer) on a SMP kernel booted with "nosmp":Wakeups-from-idle per second : 9.9 interval: 15.0s
...48.5% ( 9.4) <kernel core> : hrtimer_start (tick_sched_timer) 26.1% ( 5.1) <kernel core> : cursor_timer_handler (cursor_timer_handle 20.6% ( 4.0) <kernel core> : usb_hcd_poll_rh_status (rh_timer_func) 1.0% ( 0.2) <kernel core> : arm_supers_timer(
sync_supers_timer_fn)
0.7% ( 0.1) <interrupt> :ata_piix
...Quoted 5 lines
Quoted 5 lines
Accoding to http://www.linuxpowertop.org , the count in the brackets is how
many wakeups per seconds were caused by one source. Adding all _except_
48.5% ( 9.4) <kernel core> :hrtimer_start(tick_sched_timer)
up leads to the 9.9.Quoted 3 lines
OK, so you further instrumented the
hrtimer_sched_timer(or was ittick_sched_timer?) to find the number that you were attributing torcu_needs_cpu()?
Quoted 6 lines
That's what I did -- to
tick_nohz_stop_sched_tick(), to see why thetick_sched_timerdid not get stopped. Or why powertop thinks it did not get stopped...Patch below (works only for 1 or 2 CPUs) for all interested parties, _NOT_ intended for submission, though ;)
Cool!
If you ever do want to submit such a patch, I should give you multiple non-zero return values from rcu_needs_cpu() to allow distinguishing between the following:
1. RCU needs the CPU, but is trying to get done with it quickly.
2. RCU needs the CPU, and has given up on trying to get done with
it quickly.
Unless you tell me otherwise, I will assume that you do -not- need this.
Thanx, Paul
Quoted 169 lines
0x0001: CPU #10x0002: inidle0x0004: not inidle0x0008: INACTIVE0x0010:need_resched0x0020: softirq0x0040:rcu_needs_0x0080:printk_needs_0x0100:arch_needs_0x0200: off-by-one0x0400: skip0x0800: loadbalance:0x1000: cancel timer (SUCCESS)0x2000: timer was already in the past0x4000:hrtimer_start()(SUCCESS)0x8000:stop_tick(SUCCESS)Fixing USB-autosuspend and disabling the fbcon cursor, I'm down to ~0.9 wakeups-per-second booting with "nosmp".
kernel/time/tick-sched.c +47
-7@@ -210,19 +210,35 @@ void tick_nohz_stop_sched_tick(int inidle)struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev;u64 time_delta;int cpu; +static int testrun_counter =0 ; +int testrun =0 ; +u64 debugdata =0 ; + + testrun_counter++; + +if ((testrun_counter %123 ) ==0 ) + testrun =1 ;local_irq_save (flags); cpu =smp_processor_id (); ts = &per_cpu (tick_cpu_sched, cpu); + debugdata = cpu; + +if (inidle) + debugdata |=0x2 ; + +/* * Call to tick_nohz_start_idle stops the last_update_time from being * updated. Thus, it must not be called in the event we are called from * irq_exit() with the prior state different than idle. */ -if (!inidle && !ts->inidle) +if (!inidle && !ts->inidle) { + debugdata |=0x4 ; goto end; + }/* * Set ts->inidle unconditionally. Even if the system did not @@ -245,11 +261,15 @@ void tick_nohz_stop_sched_tick(int inidle) tick_do_timer_cpu =TICK_DO_TIMER_NONE ; } -if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) +if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) { + debugdata |=0x8 ; goto end; + } -if (need_resched ()) +if (need_resched ()) { + debugdata |=0x10 ; goto end; + }if (unlikely (local_softirq_pending () &&cpu_online (cpu))) {static int ratelimit;@@ -259,6 +279,7 @@ void tick_nohz_stop_sched_tick(int inidle) (unsigned int) local_softirq_pending()); ratelimit++; } + debugdata |=0x20 ; goto end; }@@ -275,6 +296,12 @@ void tick_nohz_stop_sched_tick(int inidle) arch_needs_cpu(cpu)) { next_jiffies = last_jiffies +1 ; delta_jiffies =1 ; +if (rcu_needs_cpu (cpu)) + debugdata |=0x40 ; +if (printk_needs_cpu (cpu)) + debugdata |=0x80 ; +if (arch_needs_cpu (cpu)) + debugdata |=0x100 ; } else {/* Get the next timer wheel timer */ next_jiffies =get_next_timer_interrupt (last_jiffies);@@ -284,8 +311,10 @@ void tick_nohz_stop_sched_tick(int inidle)* Do not stop the tick, if we are only one off * or if the cpu is required for rcu */ -if (!ts->tick_stopped && delta_jiffies ==1 ) +if (!ts->tick_stopped && delta_jiffies ==1 ) { + debugdata |=0x200 ; goto out; + }/* Schedule the tick, if we are at least one jiffie off */ if ((long )delta_jiffies >=1 ) {@@ -341,8 +370,10 @@ void tick_nohz_stop_sched_tick(int inidle)cpumask_set_cpu (cpu, nohz_cpu_mask);/* Skip reprogram of event if its not changed */ -if (ts->tick_stopped &&ktime_equal (expires, dev->next_event)) +if (ts->tick_stopped &&ktime_equal (expires, dev->next_event)) { + debugdata |=0x400 ; goto out; + }/* * nohz_stop_sched_tick can be called several times before @@ -357,8 +388,10 @@ void tick_nohz_stop_sched_tick(int inidle)* sched tick not stopped! */ cpumask_clear_cpu (cpu, nohz_cpu_mask); + debugdata |=0x800 ; goto out; } + debugdata |=0x8000 ; ts->idle_tick =hrtimer_get_expires (&ts->sched_timer); ts->tick_stopped =1 ;@@ -375,7 +408,8 @@ void tick_nohz_stop_sched_tick(int inidle)* If the expiration time == KTIME_MAX, then * in this case we simply stop the tick timer. */ -if (unlikely (expires.tv64 ==KTIME_MAX )) { +if (unlikely (expires.tv64 ==KTIME_MAX )) { + debugdata |=0x1000 ;if (ts->nohz_mode ==NOHZ_MODE_HIGHRES )hrtimer_cancel (&ts->sched_timer); goto out;@@ -384,9 +418,13 @@ void tick_nohz_stop_sched_tick(int inidle)if (ts->nohz_mode ==NOHZ_MODE_HIGHRES ) {hrtimer_start (&ts->sched_timer, expires,HRTIMER_MODE_ABS_PINNED ); + debugdata |=0x2000 ;/* Check, if the timer was already in the past */ -if (hrtimer_active (&ts->sched_timer)) +if (hrtimer_active (&ts->sched_timer)) { + debugdata &= ~(0x2000 ); + debugdata |=0x4000 ; goto out; + } }else if (!tick_program_event (expires,0 )) goto out;/* @@ -403,6 +441,8 @@ out: ts->last_jiffies = last_jiffies; ts->sleep_length =ktime_sub (dev->next_event, now); end: +if (testrun) +printk (KERN_DEBUG "0x%0llx\n" , debugdata);local_irq_restore (flags); }
RFC PATCH 1/1 nohz/sched: disable ilb on !mc_capable()
In reply to Dominik BrodowskiQuoted 50 lines · Dominik Brodowski
2) dual-core CPU[*] and
select_nohz_load_balancer()[*] (Intel(R) Core(TM)2 Duo CPU T7250)#
CONFIG_SCHED_SMTis not setCONFIG_SCHED_MC=yCONFIG_SCHED_HRTICK=y
CONFIG_SCHED_MCis igored, asmc_capable()returns 0 on a one-socket, dual-core system. Quite surprisingly, even under moderate load (~98.0% idle) while writing this bugreport, up to half of the calls totick_nohz_stop_sched_tick()are aborted due toselect_nohz_load_balancer(1):if (atomic_read(&nohz.load_balancer) == -1) { /* make me the ilb owner */ if (atomic_cmpxchg(&nohz.load_balancer, -1, cpu) == -1) return 1;I'm not really sure, but I guess this is caused by the following phenomenon under minor load but still, every once in a while, parallel work for both CPUs:
CPU #0 CPU #1
<active> <active>
<idle> <active>
tick_nohz_stop_sched_tick(1)<active>
select_nohz_load_balancer(1)<active>
=> becomes ilb owner <idle>
=> tick is not stoppedtick_nohz_stop_sched_tick(1)
=> CPU goes to sleep for 1 tick => as it isn't the ILB owner, tick
<sleep for 1 tick> is stopped .
--->scheduler_tick()<sleeeeeeeep>
tick_nohz_stop_sched_tick(0)
<still idle>
tick_nohz_stop_sched_tick(1)
select_nohz_load_balancer(1)
=> is ilb owner, all CPUs idle,
may go to sleep.If both CPUs have hardly anything to do, letting the _active_ CPU do ilb allows us to enter deep sleep states earlier, and longer:
current ILB model (* = ILB)
tick ---------- tick -------- tick ----- IRQ CPU0: active|IDLE(C2)--|*|IDLE (C3) | CPU1: active....| IDLE (C3) | core: .......???| C2 | C3 |
Quoted 6 lines
ILB-by-active-CPU-on-light-load:
tick ---------- tick -------- tick ----- IRQ CPU0: active|IDLE(C3) | CPU1: active....*| IDLE (C3) | core: .......????| C3 |
Tested this a bit further, and thought about it a bit further:
On systems like my laptop, which has one physical CPUs with two cores ( = SMP, !mc_capable() ), the "idle load balancing" seems to be _not_ necessary at all:
- if both cores are active, ilb is inactive anyway.
- if no core is active, ilb was inactive anyway
- if only one core is active and busy, it seems to attempt to balance its load on each tick anyway. ilb wouldn't act quicker anyways.
The attached patch decreases the amount of wakeups on my completely idle notebook ( init=/bin/bash ) from ~2 wakeups-per-second[*] to ~0.7. During normal system usage, the amount of wakeups-per-second seems to decrease as well, but is less easy to detect. More importantly, over 80 % of all calls to tick_nohz_stop_sched_tick() succeed immediately[**].
[*] needs an USB-autosuspend bugfix, manual enabling of USB autosuspend, and
disabling of the blinking fb cursor.
[**] about 10% return due to rcu_needs_cpu(), which often means the CPU can
go to sleep pretty soon afterwards.
The remaining reports of "tick_sched_timer" in powertop(1) seems to be related to timer ticks when one CPU is active for at least one jiffy. So this is probably not a real "wakeup" at all.
Best,
Dominik
From: Dominik Brodowski <[email protected]>
Date: Thu, 8 Apr 2010 21:51:18 +0200
Subject: [PATCH] nohz/sched: disable ilb on !mc_capable()
kernel/sched_fair.c +6 -0
@@ -3290,6 +3290,9 @@ int select_nohz_load_balancer(int stop_tick)if (stop_tick) {cpu_rq (cpu)->in_nohz_recently =1 ; +if (!mc_capable ()) +return 0 ; +if (!cpu_active (cpu)) {if (atomic_read (&nohz.load_balancer) != cpu)return 0 ;
@@ -3339,6 +3342,9 @@ int select_nohz_load_balancer(int stop_tick)if (!cpumask_test_cpu (cpu, nohz.cpu_mask))return 0 ; +if (!mc_capable ()) +return 0 ; +cpumask_clear_cpu (cpu, nohz.cpu_mask);if (atomic_read (&nohz.load_balancer) == cpu)