Skip to contentSkip to conversation

high power consumption in recent kernels

DISCUSSION Shi, Alex 4 participants 14 messages Sep 10, 2010 – Sep 30, 2010 5 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 1283840425.26157.6486.camel@debian

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): EA929A9653AAE14F841771FB1DE5A1366008738EC2@rrsmsx501.amr.corp.intel.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 1284107099.402.30.camel@laptop

Shi, Alex

author
#5 lore raw

CC to lkml since maybe other guys also have interesting on this.
Norbert:
In the powertop source code, the "extra timer interrupt" come from /proc/timer_stats. If my understanding is correct, so many "extra timer interrupt" and the nohz_ratemlimit checking make your system can not go to deep C states. but don't know what the effect for "load balancing tick" come from "extra timer interrupt". So maybe reduce the "extra timer interrupt" is helpful on your system. Could you like to try this?

Regards!
Alex

Quoted 36 lines

-----Original Message-----
From: Peter Zijlstra [mailto:[email protected]]
Sent: Friday, September 10, 2010 4:25 PM
To: Norbert Preining
Cc: Chen, Tim C; Shi, Alex; [email protected]; [email protected]; Li, Shaohua; tglx
Subject: Re: high power consumption in recent kernels

Quoted 6 lines · Norbert Preining
Quoted 1 line · Tim C

Have you tried to run powertop to see the causes of the wakeups

Of course, in the very first email of the thread I posted the results, here again for your convenience:
Top causes for wakeups:

Quoted 8 lines
  34.2% (185.3)   [kernel scheduler] Load balancing tick
  23.9% (129.6)   [extra timer interrupt]
  10.8% ( 58.6)   firefox-bin
   9.2% ( 49.7)   [iwlagn] <interrupt>
   7.2% ( 39.1)   [kernel core] hrtimer_start (tick_sched_timer)
   3.9% ( 20.9)   PS/2 keyboard/mouse/touchpad interrupt

The problem is the "Load balancing tick" that is far far to high.

Right, which makes perfect sense, delaying going into nohz state simply costs power.

One thing we could do is hand over nohz control to the power governor (I'm sure Arjan will be thrilled), it could compare the cost of nohz enter/exit against the projected sleep time and make an informed decision.

Because from what I understand, the nohz enter/exit cycle is what is costing you performance on this workload, its simply very expensive to prod at timer hardware (which is of course totally intel's own fault for giving us crummy timers to begin with).

#6 lore raw
Quoted 1 line · Alex

In the powertop source code, the "extra timer interrupt" come from /proc/timer_stats. If my understanding is correct, so many "extra timer interrupt" and the nohz_ratemlimit checking make your system can not go to deep C states. but don't know what the effect for "load balancing tick" come from "extra timer interrupt". So maybe reduce the "extra timer interrupt" is helpful on your system. Could you like to try this?

Can you be more specific how to do that?

- revert the nohz patch (can you send me a patch for that)
- how to reduce the "extra timer interrupt"?

Thanks

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan TeX Live & Debian Developer
DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
CAIRNPAT (n.)
A large piece of dried dung found in mountainous terrain above the cowline which leads the experienced tracker to believe that hikers have recently passed.
--- Douglas Adams, The Meaning of Liff

author
#7 lore raw
Quoted 6 lines · Norbert Preining
Quoted 1 line · Alex

In the powertop source code, the "extra timer interrupt" come from /proc/timer_stats. If my understanding is correct, so many "extra timer interrupt" and the nohz_ratemlimit checking make your system can not go to deep C states. but don't know what the effect for "load balancing tick" come from "extra timer interrupt". So maybe reduce the "extra timer interrupt" is helpful on your system. Could you like to try this?

Can you be more specific how to do that?

- revert the nohz patch (can you send me a patch for that)

include/linux/sched.h +6 -0
index 1e2a6db..e66c52e 100644
@@ -274,8 +274,14 @@ extern cpumask_var_t nohz_cpu_mask;
 #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
 extern void select_nohz_load_balancer(int stop_tick);
 extern int get_nohz_timer_target(void);
+extern int nohz_ratelimit(int cpu);
 #else
 static inline void select_nohz_load_balancer(int stop_tick) { }
+static inline int nohz_ratelimit(int cpu)
+{
+       return 0;
+}
+
 #endif
 
 /*
kernel/sched.c +11 -0
index ed09d4f..25399e7 100644
@@ -1182,6 +1182,17 @@ static void resched_task(struct task_struct *p)
 		smp_send_reschedule(cpu);
 }
 
+int nohz_ratelimit(int cpu)
+{
+       struct rq *rq = cpu_rq(cpu);
+       u64 diff = rq->clock - rq->nohz_stamp;
+
+       rq->nohz_stamp = rq->clock;
+
+       return diff < (NSEC_PER_SEC / HZ) >> 1;
+}
+
+
 static void resched_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
kernel/time/tick-sched.c +1 -1
index 3e216e0..d6bca17 100644
@@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
 	} while (read_seqretry(&xtime_lock, seq));
 
 	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
-	    arch_needs_cpu(cpu)) {
+	    arch_needs_cpu(cpu)|| nohz_ratelimit(cpu)) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
 	} else {
Quoted 1 line

- how to reduce the "extra timer interrupt"?

The timer was listed in /proc/timer_stats, you can post its contents, and let's see what is abnormal here.

Quoted 16 lines

Thanks

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan TeX Live & Debian Developer
DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
CAIRNPAT (n.)
A large piece of dried dung found in mountainous terrain above the cowline which leads the experienced tracker to believe that hikers have recently passed.
--- Douglas Adams, The Meaning of Liff

author
#8 lore raw
Quoted 5 lines
Quoted 2 lines

- how to reduce the "extra timer interrupt"?

The timer was listed in /proc/timer_stats, you can post its contents, and let's see what is abnormal here.

Norbert, did you try the patch?
We are still thinking the too many "extra timer interrupts" on your system, maybe the following message can give some help. So, could you like to check if your system is using pit timer?
#cat /proc/timer_list
Or some APP/system services in your system is using lots of timer?

===
http://www.linuxpowertop.org/faq.php

Even with a tickless kernel, there are some timer ticks still happening:

1.) For userspace events (also shown in the lower "top 10 events" list). 2.) The kernel still has timer ticks when userspace is executing code, to sample and get data for the "top" program.
3.) Hardware timers have a maximum time they can be set for, for the PIT timer (the one used if you don't have HPET active). This is around 27 milliseconds. If you would have longer sleep times than the maximum, the kernel has no choice but to set the timer to the maximum repeatedly until the actual event is approaching.

The "extra timer interrupts" data is a combination fo 1 and 3.

#9 lore raw

Hi Alex, hi all,

sorry for the long silence, I was on business travel and mountaineering for some weeks.

Quoted 1 line · Alex,Shi
Quoted 1 line

- revert the nohz patch (can you send me a patch for that)

I am now running 2.6.36-rc5 with this patch:

Quoted 46 lines
include/linux/sched.h +6 -0
index 1e2a6db..e66c52e 100644
@@ -274,8 +274,14 @@ extern cpumask_var_t nohz_cpu_mask;
 #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
 extern void select_nohz_load_balancer(int stop_tick);
 extern int get_nohz_timer_target(void);
+extern int nohz_ratelimit(int cpu);
 #else
 static inline void select_nohz_load_balancer(int stop_tick) { }
+static inline int nohz_ratelimit(int cpu)
+{
+       return 0;
+}
+
 #endif
 
 /*
kernel/sched.c +11 -0
index ed09d4f..25399e7 100644
@@ -1182,6 +1182,17 @@ static void resched_task(struct task_struct *p)
 		smp_send_reschedule(cpu);
 }
 
+int nohz_ratelimit(int cpu)
+{
+       struct rq *rq = cpu_rq(cpu);
+       u64 diff = rq->clock - rq->nohz_stamp;
+
+       rq->nohz_stamp = rq->clock;
+
+       return diff < (NSEC_PER_SEC / HZ) >> 1;
+}
+
+
 static void resched_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
kernel/time/tick-sched.c +0 -0
index 3e216e0..d6bca17 100644

kernel/time/tick-sched.c void tick_nohz_stop_sched_tick(int inidle) · line 331

Quoted 8 lines
kernel/time/tick-sched.c +1 -1
@@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
 	} while (read_seqretry(&xtime_lock, seq));
 
 	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
-	    arch_needs_cpu(cpu)) {
+	    arch_needs_cpu(cpu)|| nohz_ratelimit(cpu)) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
 	} else {

and it seems that still works nicely. I am now running on battery power, connected to the internet over bluetooth to my mobile, and get:
Wakeups-from-idle per second : 452.2 interval: 10.0s
Power usage (ACPI estimate): 12.4W (3.6 hours) (long term: 8.0W,/5.6h)

Top causes for wakeups:

  16.1% ( 79.5)   [yenta, ehci_hcd:usb2, uhci_hcd:usb6, uhci_hcd:usb7, uhci_hcd:
  14.3% ( 70.7)   [kernel scheduler] Load balancing tick
  14.1% ( 69.8)   [extra timer interrupt]
  13.1% ( 64.9)   kworker/0:0
  10.7% ( 53.0)   USB device  8-2 : BCM2046 Bluetooth Device (Broadcom Corp)
   8.4% ( 41.8)   [kernel core] hrtimer_start (tick_sched_timer)
   3.8% ( 19.0)   gnome-terminal
   2.9% ( 14.6)   icedove-bin
   2.2% ( 10.7)   PS/2 keyboard/mouse/touchpad interrupt

Which shows that the "Load balancing tick" and "extra timer interrupt" are down to halfway normal levels. I guess when I shutdown the usb/bluetooth connection both of them will drop even further.

Quoted 3 lines
Quoted 1 line

- how to reduce the "extra timer interrupt"?

The timer was listed in /proc/timer_stats, you can post its contents, and let's see what is abnormal here.

Here is the output of /proc/timer_stats:

Timer Stats Version: v0.2
Sample period: 7.828 s

   66,  4378 firefox-bin      hrtimer_start_range_ns (hrtimer_wakeup)
   15, 12527 aptitude         hrtimer_start_range_ns (hrtimer_wakeup)
    8,  3629 cpufreq-applet   hrtimer_start_range_ns (hrtimer_wakeup)
  612,     0 swapper          hrtimer_start_range_ns (tick_sched_timer)
 483D,  6082 kworker/0:0      do_dbs_timer (delayed_work_timer_fn)
   17, 12527 aptitude         queue_delayed_work (delayed_work_timer_fn)
  376,     0 kworker/0:0      hrtimer_start_range_ns (tick_sched_timer)
 237D,  6810 kworker/1:1      do_dbs_timer (delayed_work_timer_fn)
   51,  7843 sakura           hrtimer_start_range_ns (hrtimer_wakeup)
    8,  2665 Xorg             queue_delayed_work (delayed_work_timer_fn)
  387,     0 swapper          hrtimer_start (tick_sched_timer)
   39,  3510 syndaemon        hrtimer_start_range_ns (hrtimer_wakeup)
    8,  2705 gpsd             hrtimer_start_range_ns (hrtimer_wakeup)
   32,     0 swapper          __mod_timer (rh_timer_func)
    8,  7572 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8, 12916 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8, 12949 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8,  2080 apache2          hrtimer_start_range_ns (hrtimer_wakeup)
    8,  2795 ntpd             hrtimer_start_range_ns (posix_timer_fn)
   26,  3555 yarssr           hrtimer_start_range_ns (hrtimer_wakeup)
   91, 12790 sakura           hrtimer_start_range_ns (hrtimer_wakeup)
   15,  3583 multiload-apple  hrtimer_start_range_ns (hrtimer_wakeup)
    8,  4376 firefox-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8,  7579 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
   8D,  6810 kworker/1:1      queue_delayed_work (delayed_work_timer_fn)
   4D,  6810 kworker/1:1      queue_delayed_work (delayed_work_timer_fn)
   8D,  6082 kworker/0:0      queue_delayed_work (delayed_work_timer_fn)
   4D,  6082 kworker/0:0      queue_delayed_work (delayed_work_timer_fn)
    8, 12895 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    4,  3516 udisks-daemon    hrtimer_start_range_ns (hrtimer_wakeup)
    4,  2642 hald-addon-stor  hrtimer_start_range_ns (hrtimer_wakeup)
    8,  7540 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8,  7777 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    9,     0 kworker/0:0      hrtimer_start (tick_sched_timer)
    6,  2665 Xorg             __mod_timer (i915_hangcheck_elapsed)
    8,  7550 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8,  3531 gvfs-afc-volume  hrtimer_start_range_ns (hrtimer_wakeup)
    7,  7551 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    7,  7549 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    7,  7578 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    1,     0 swapper          __mod_timer (uhci_fsbr_timeout)
    7,  7527 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    7, 12890 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    7,  7536 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
   18, 12790 sakura           hrtimer_start_range_ns (hrtimer_wakeup)
    4,  3582 sensors-applet   hrtimer_start_range_ns (hrtimer_wakeup)
    4,     0 swapper          __mod_timer (laptop_mode_timer_fn)
    4,     0 swapper          __mod_timer (dev_watchdog)
   11,  2868 tor              __mod_timer (process_timeout)
    5,  7528 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    4,  4807 dropbox          hrtimer_start_range_ns (hrtimer_wakeup)
    1, 12537 http             __mod_timer (tcp_write_timer)
    5,  2825 nmbd             hrtimer_start_range_ns (hrtimer_wakeup)
    1,  3439 gconfd-2         hrtimer_start_range_ns (hrtimer_wakeup)
    4,  7403 kbnepd bnep0     __mod_timer (tcp_delack_timer)
    1,  3422 ssh-agent        hrtimer_start_range_ns (hrtimer_wakeup)
    1,  2195 hald             hrtimer_start_range_ns (hrtimer_wakeup)
    1,  2195 hald             __mod_timer (process_timeout)
    1,     7 watchdog/0       hrtimer_start (watchdog_timer_fn)
    1,  3444 gnome-power-man  hrtimer_start_range_ns (hrtimer_wakeup)
    3,   816 flush-8:0        __mod_timer (blk_unplug_timeout)
    1,     1 init             hrtimer_start_range_ns (hrtimer_wakeup)
    1,     1 swapper          enqueue_task_rt (sched_rt_period_timer)
   16, 12790 sakura           queue_delayed_work (delayed_work_timer_fn)
   14, 12791 bash             queue_delayed_work (delayed_work_timer_fn)
    5,  2665 Xorg             hrtimer_start_range_ns (hrtimer_wakeup)
    2,  3582 sensors-applet   __mod_timer (process_timeout)
    1,  3569 gnome-volume-ma  hrtimer_start_range_ns (hrtimer_wakeup)
    1,  2247 NetworkManager   hrtimer_start_range_ns (hrtimer_wakeup)
   1D,  6082 kworker/0:0      queue_delayed_work (delayed_work_timer_fn)
    1,  7403 kbnepd bnep0     __mod_timer (tcp_delack_timer)
    1,  2642 hald-addon-stor  __mod_timer (blk_rq_timed_out_timer)
    1,  4804 dropbox          __mod_timer (tcp_write_timer)
    1,  3511 gnome-panel      hrtimer_start_range_ns (hrtimer_wakeup)

2772 total events, 354.113 events/sec

Hope you can do something with it!

Let me know if I should do more testing.

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan TeX Live & Debian Developer
DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
BERKHAMSTED
The massive three-course midmorning blow-out enjoyed by a dieter who has already done his or her slimming duty by having a teaspoonful of cottage cheese for breakfast.
--- Douglas Adams, The Meaning of Liff

author
#10 lore raw
Quoted 17 lines

and it seems that still works nicely. I am now running on battery power, connected to the internet over bluetooth to my mobile, and get:
Wakeups-from-idle per second : 452.2 interval: 10.0s
Power usage (ACPI estimate): 12.4W (3.6 hours) (long term: 8.0W,/5.6h)

Top causes for wakeups:

  16.1% ( 79.5)   [yenta, ehci_hcd:usb2, uhci_hcd:usb6, uhci_hcd:usb7, uhci_hcd:
  14.3% ( 70.7)   [kernel scheduler] Load balancing tick
  14.1% ( 69.8)   [extra timer interrupt]
  13.1% ( 64.9)   kworker/0:0
  10.7% ( 53.0)   USB device  8-2 : BCM2046 Bluetooth Device (Broadcom Corp)
   8.4% ( 41.8)   [kernel core] hrtimer_start (tick_sched_timer)
   3.8% ( 19.0)   gnome-terminal
   2.9% ( 14.6)   icedove-bin
   2.2% ( 10.7)   PS/2 keyboard/mouse/touchpad interrupt
Quoted 8 lines

Which shows that the "Load balancing tick" and "extra timer interrupt" are down to halfway normal levels. I guess when I shutdown the usb/bluetooth connection both of them will drop even further.

Hope you can do something with it!

Let me know if I should do more testing.

Seems in .36 kernel the extra timer reduced. Guess the following patch worked. http://lkml.org/lkml/2010/6/9/109

Tim,Yakui and I talked with the issue. We thought the nohz_ratelimit() only hart the power when lots of interrupts with light load in system. Since the tick_nohz_stop_sched_tick() only be checked in cpu_idle() and in irq_exit(), we remove the nohz checking in irq_exit(). The TCP/UDP RR loop back testing of netperf also benefit from this. But it have no effect on real network testing.

Could you like to try the following patch on your system and see how many power it consumption now?

include/linux/sched.h +5 -0
index 1e2a6db..a4dbb37 100644
@@ -274,8 +274,13 @@ extern cpumask_var_t nohz_cpu_mask;
 #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
 extern void select_nohz_load_balancer(int stop_tick);
 extern int get_nohz_timer_target(void);
+extern int nohz_ratelimit(int cpu);
 #else
 static inline void select_nohz_load_balancer(int stop_tick) { }
+static inline int nohz_ratelimit(int cpu)
+{
+	return 0;
+}
 #endif
 
 /*
kernel/sched.c +10 -0
index dc85ceb..132a21c 100644
@@ -1182,6 +1182,16 @@ static void resched_task(struct task_struct *p)
 		smp_send_reschedule(cpu);
 }
 
+int nohz_ratelimit(int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+	u64 diff = rq->clock - rq->nohz_stamp;
+
+	rq->nohz_stamp = rq->clock;
+
+	return diff < (NSEC_PER_SEC / HZ) >> 1;
+}
+
 static void resched_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
kernel/time/tick-sched.c +1 -1
index 3e216e0..0b5b186 100644
@@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
 	} while (read_seqretry(&xtime_lock, seq));
 
 	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
-	    arch_needs_cpu(cpu)) {
+		arch_needs_cpu(cpu)|| (inidle && nohz_ratelimit(cpu))) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
 	} else {
#11 lore raw

Hi Alex,

Quoted 2 lines

Could you like to try the following patch on your system and see how many power it consumption now?

But this patch is excately what you send me already and what I tried and I answered in the last email?

Do I miss something here?

All the best

Norbert

#12 lore raw

Norbert,

Can you try with the no-hz ratelimit now code REMOVED, and see if that will reduce your system power and cpu wakeups further. Your last testing has the no-hz ratelimit code and we will like to compare the effect with and without the code.

Sorry for the confusion.

Tim

Quoted 23 lines

-----Original Message-----
From: Norbert Preining [mailto:[email protected]]
Sent: Wednesday, September 29, 2010 7:39 AM
To: Shi, Alex
Cc: Peter Zijlstra; Chen, Tim C; [email protected]; [email protected]; Li, Shaohua; tglx; [email protected]; Zhao, Yakui
Subject: Re: high power consumption in recent kernels

Hi Alex,

Quoted 2 lines

Could you like to try the following patch on your system and see how many power it consumption now?

But this patch is excately what you send me already and what I tried and I answered in the last email?

Do I miss something here?

All the best

Norbert

author
#13 lore raw
In reply to Norbert Preining
Quoted 5 lines · Norbert Preining

Hi Alex,

Quoted 2 lines

Could you like to try the following patch on your system and see how many power it consumption now?

Quoted 8 lines

But this patch is excately what you send me already and what I tried and I answered in the last email?

Do I miss something here?

All the best

Norbert

The following line is different, the new code remove the nohz_ratelimit from irq_exit(). That may resolve your system too many interrupts issue.

+ arch_needs_cpu(cpu)|| (inidle && nohz_ratelimit(cpu))) {

#14 lore raw
Quoted 1 line · Alex,Shi

+ arch_needs_cpu(cpu)|| (inidle && nohz_ratelimit(cpu))) {

Sorry right, the
inidle
is new, missed that one. I will rebuild in a minute and test.

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan TeX Live & Debian Developer
DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
AITH (n.)
The single bristle that sticks out sideways on a cheap paintbrush.
--- Douglas Adams, The Meaning of Liff

#15 lore raw
In reply to Alex,Shi
Quoted 5 lines

The following line is different, the new code remove the nohz_ratelimit from irq_exit(). That may resolve your system too many interrupts issue.

+ arch_needs_cpu(cpu)|| (inidle && nohz_ratelimit(cpu))) {

That is a very nice one ..., first time I came down with inactive laptop, lowest brightness, down to below 8W:
Cn Avg residency P-states (frequencies)

C0 (cpu running)        ( 1.1%)       Turbo Mode     0.0%
C0                0.0ms ( 0.0%)         2.54 Ghz     0.0%
C1 mwait          0.0ms ( 0.0%)         1.60 Ghz     0.0%

C2 mwait 0.3ms ( 0.8%) 800 Mhz 99.9%
C6 mwait 16.7ms (98.2%)

Wakeups-from-idle per second : 81.3 interval: 15.0s
Power usage (ACPI estimate): 7.9W (11.0 hours)

Top causes for wakeups:

  23.8% ( 18.1)   kworker/0:0
  14.0% ( 10.7)   [ahci] <interrupt>
  13.4% ( 10.2)   [kernel scheduler] Load balancing tick
   6.6% (  5.0)   syndaemon
   5.5% (  4.2)   Xorg
   5.5% (  4.2)   [extra timer interrupt]
   4.6% (  3.5)   [acpi] <interrupt>

Very nice, with activity and bluetooth and internet over bluetooth I am back at 10.3W, but that is also fine.

Anything else you need to know (kernel was latest git, -rc6 AFAIR).

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan TeX Live & Debian Developer
DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
The main reception foyer was almost empty but Ford nevertheless weaved his way through it.
--- Ford making his way out of Milliways whilst under the --- influence of enough alchol to make a rhino sing. --- Douglas Adams, The Hitchhikers Guide to the Galaxy

author Tested-by Norbert Preining
#16 lore raw

Thanks Norbert!
Mike&Peter:
would you like to add signed-off for the following patch?

---
sched: nohz_ratelimit function refresh

The nohz_ratelimit() function that written by Mike Galbraith can bring about more than 10% throughput for netperf TCP/UDP RR when scheduling cross-cpu. It did this by reducing down to nohz mode chance.
But the patch also reduce CPU chance to nohz mode after interrupt processed, that cause Norbert's system have 4 watts power increase(the system have about 100 int/sec and with a light load). That is not acceptable for a laptop.

So, I remove the nohz_ratelimit from irq_exit(). and then the Norbert's system back to low power consumption.

include/linux/sched.h +5 -0
index 1e2a6db..a4dbb37 100644
@@ -274,8 +274,13 @@ extern cpumask_var_t nohz_cpu_mask;
 #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
 extern void select_nohz_load_balancer(int stop_tick);
 extern int get_nohz_timer_target(void);
+extern int nohz_ratelimit(int cpu);
 #else
 static inline void select_nohz_load_balancer(int stop_tick) { }
+static inline int nohz_ratelimit(int cpu)
+{
+	return 0;
+}
 #endif
 
 /*
kernel/sched.c +10 -0
index dc85ceb..132a21c 100644
@@ -1182,6 +1182,16 @@ static void resched_task(struct task_struct *p)
 		smp_send_reschedule(cpu);
 }
 
+int nohz_ratelimit(int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+	u64 diff = rq->clock - rq->nohz_stamp;
+
+	rq->nohz_stamp = rq->clock;
+
+	return diff < (NSEC_PER_SEC / HZ) >> 1;
+}
+
 static void resched_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
kernel/time/tick-sched.c +1 -1
index 3e216e0..19a7914 100644
@@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
 	} while (read_seqretry(&xtime_lock, seq));
 
 	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
-	    arch_needs_cpu(cpu)) {
+		arch_needs_cpu(cpu) || (inidle && nohz_ratelimit(cpu))) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
 	} else {
maintainer
#17 lore raw
Quoted 3 lines · Alex,Shi

Thanks Norbert!
Mike&Peter:
would you like to add signed-off for the following patch?

I would really rather see the nohz decision move into the whole cpuidle governor thing, which can make a much better cost vs benefit decision.

Thomas, Arjan?

Quoted 72 lines

---
sched: nohz_ratelimit function refresh

The nohz_ratelimit() function that written by Mike Galbraith can bring about more than 10% throughput for netperf TCP/UDP RR when scheduling cross-cpu. It did this by reducing down to nohz mode chance.
But the patch also reduce CPU chance to nohz mode after interrupt processed, that cause Norbert's system have 4 watts power increase(the system have about 100 int/sec and with a light load). That is not acceptable for a laptop.

So, I remove the nohz_ratelimit from irq_exit(). and then the Norbert's system back to low power consumption.

include/linux/sched.h +5 -0
index 1e2a6db..a4dbb37 100644
@@ -274,8 +274,13 @@ extern cpumask_var_t nohz_cpu_mask;
 #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
 extern void select_nohz_load_balancer(int stop_tick);
 extern int get_nohz_timer_target(void);
+extern int nohz_ratelimit(int cpu);
 #else
 static inline void select_nohz_load_balancer(int stop_tick) { }
+static inline int nohz_ratelimit(int cpu)
+{
+	return 0;
+}
 #endif
 
 /*
kernel/sched.c +10 -0
index dc85ceb..132a21c 100644
@@ -1182,6 +1182,16 @@ static void resched_task(struct task_struct *p)
 		smp_send_reschedule(cpu);
 }
 
+int nohz_ratelimit(int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+	u64 diff = rq->clock - rq->nohz_stamp;
+
+	rq->nohz_stamp = rq->clock;
+
+	return diff < (NSEC_PER_SEC / HZ) >> 1;
+}
+
 static void resched_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
kernel/time/tick-sched.c +1 -1
index 3e216e0..19a7914 100644
@@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
 	} while (read_seqretry(&xtime_lock, seq));
 
 	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
-	    arch_needs_cpu(cpu)) {
+		arch_needs_cpu(cpu) || (inidle && nohz_ratelimit(cpu))) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
 	} else {
author
#18 lore raw
Quoted 7 lines
Quoted 3 lines

Thanks Norbert!
Mike&Peter:
would you like to add signed-off for the following patch?

I would really rather see the nohz decision move into the whole cpuidle governor thing, which can make a much better cost vs benefit decision.

See!