Skip to contentSkip to conversation

CFS idle injection

PATCH In review Jacob Pan 11 participants 78 messages Nov 13, 2015 – Jan 19, 2016

Subsystems (4)
lore
Share & tools
lore

Jacob Pan

author
#0 lore raw

cover CFS idle injection

We are entering a very power and thermal constrained environment. Often we have more horsepower than we can use due to these limits. But on the other side we all demand performance when needed. The reserved performance headroom does not come free. To conserve energy, more and more SoC blocks can be power gated at runtime. However, randomly scheduled idle time on individual CPU may not result in good power saving in that the common circuits such as memory controller can only be in the low power state when all cores are in idle at the same time.

Frequency-Voltage scaling presents a good solution but its efficiency is limited to certain range.

In general, only synchronized idle will allow SoC to enter the deepest power states. For most modern processors, deep idle power is nearly negligible to the peak running power. This implies if we can duty cycle the CPU between running and the deepest idle state, we can scale performance and power almost linearly. Combined with the most efficient frequency point, idle injection presents a way to cap power efficiently.

Intel powerclamp driver was introduced a while ago to address the problem but is broken in the sense of turning off idle ticks in the forced idle period. https://lkml.org/lkml/2014/12/18/369

It was suggested to replace the current kthread play idle loop with a timer based runqueue throttling scheme. I finally got around to implement this and code looks much simpler and more effective.

Test results were presented at LinuxCon where data/graph can be seen from slides #18 and later.
http://events.linuxfoundation.org/sites/events/files/slides/LinuxCon_Japan_2015_idle_injection1_0.pdf

RFC discussions are here.
https://lkml.org/lkml/2015/11/2/756

Thanks,

Jacob

Jacob Pan (4):
ktime: add a roundup function
timer: relax tick stop in idle entry
sched: introduce synchronized idle injection
sched: add trace event for idle injection

 include/linux/ktime.h        |  10 ++
 include/linux/sched.h        |  16 ++
 include/linux/sched/sysctl.h |   5 +
 include/trace/events/sched.h |  25 +++
 init/Kconfig                 |  10 ++
 kernel/sched/fair.c          | 356 ++++++++++++++++++++++++++++++++++++++++++-
 kernel/sched/sched.h         |  54 ++++++-
 kernel/sysctl.c              |  21 +++
 kernel/time/tick-sched.c     |   2 +-
 9 files changed, 493 insertions(+), 6 deletions(-)
Signature · 1 line
1.9.1
author
#1 lore raw

PATCH 1/4 ktime: add a roundup function

ktime roundup function can be used to keep timer aligned and prevent drift for recurring timeouts.

---
 include/linux/ktime.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
include/linux/ktime.h +10 -0
index 2b6a204..2e293fa 100644
@@ -233,6 +233,16 @@ static inline ktime_t ktime_sub_us(const ktime_t kt, const u64 usec)
 
 extern ktime_t ktime_add_safe(const ktime_t lhs, const ktime_t rhs);
 
+static inline ktime_t ktime_roundup(ktime_t x, ktime_t y)
+{
+	u64 temp_tv64;
+
+	temp_tv64 = x.tv64 + y.tv64 - 1;
+	temp_tv64 = div64_u64(temp_tv64, y.tv64);
+	x.tv64 = temp_tv64 * y.tv64;
+
+	return x;
+}
 /**
  * ktime_to_timespec_cond - convert a ktime_t variable to timespec
  *			    format only if the variable contains data
Signature · 1 line
1.9.1
#2 lore raw

Re: [PATCH 1/4] ktime: add a roundup function

Quoted 16 lines · Jacob Pan

ktime roundup function can be used to keep timer aligned and prevent drift for recurring timeouts.

---
 include/linux/ktime.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
include/linux/ktime.h +0 -0
index 2b6a204..2e293fa 100644
@@ -233,6 +233,16 @@ static inline ktime_t ktime_sub_us(const ktime_t kt, const u64 usec)
 
 extern ktime_t ktime_add_safe(const ktime_t lhs, const ktime_t rhs);
 

include/linux/ktime.h static inline ktime_t ktime_sub_us(const ktime_t kt, const u64 usec) · line 245

Quoted 10 lines
include/linux/ktime.h +10 -0
+static inline ktime_t ktime_roundup(ktime_t x, ktime_t y)
+{
+       u64 temp_tv64;
+
+       temp_tv64 = x.tv64 + y.tv64 - 1;
+       temp_tv64 = div64_u64(temp_tv64, y.tv64);
+       x.tv64 = temp_tv64 * y.tv64;
+
+       return x;
+}

Could you add a comment as to what the function does, and use some better variable names here to make it more immediately obvious what is being done here?

Something like:
/**
* ktime_roundup - Rounds value up to interval chunk
* @ value: Value to be rounded up
* @ interval: interval size to round up to
*
* Rounds a value up to the next higher multiple of an interval size */
static inline ktime ktime_roundup(ktime_t value, ktime_t interval)

thanks
-john

author
#3 lore raw

Re: [PATCH 1/4] ktime: add a roundup function

Quoted 4 lines · On Fri, 13 Nov 2015 12:11:01 -0800 John Stultz <[email protected]> wrote:

Could you add a comment as to what the function does, and use some better variable names here to make it more immediately obvious what is being done here?

Quoted 9 lines

Something like:
/**
* ktime_roundup - Rounds value up to interval chunk
* @ value: Value to be rounded up
* @ interval: interval size to round up to
*
* Rounds a value up to the next higher multiple of an interval size */
static inline ktime ktime_roundup(ktime_t value, ktime_t interval)

will do. thank you for taking the time.

#4 lore raw

Re: [PATCH 1/4] ktime: add a roundup function

In reply to Jacob Pan
Quoted 2 lines · Jacob Pan

ktime roundup function can be used to keep timer aligned and prevent drift for recurring timeouts.

That tells me a use case, but not WHAT the function does and WHY we need it.

Quoted 2 lines

+static inline ktime_t ktime_roundup(ktime_t x, ktime_t y)

Kerneldoc comment of this function would be appreciated.

Thanks,

tglx

author
#5 lore raw

Re: [PATCH 1/4] ktime: add a roundup function

Quoted 4 lines · On Fri, 13 Nov 2015 15:13:45 -0500 (EST) Thomas Gleixner <[email protected]>…
Quoted 2 lines

+static inline ktime_t ktime_roundup(ktime_t x, ktime_t y)

Kerneldoc comment of this function would be appreciated.

will do. Plan to reuse John's comment.

Thanks,

Jacob

author
#6 lore raw

PATCH 2/4 timer: relax tick stop in idle entry

In reply to Jacob Pan

Upon entering idle, we can turn off tick if the next timeout is exactly one tick away. Otherwise, we could enter inner idle loop with tick still enabled, without resched set, the tick will continue during idle therefore less optimal in terms of energy savings.

---
 kernel/time/tick-sched.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/tick-sched.c +1 -1
index 7c7ec45..bcadaab 100644
@@ -606,7 +606,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,
 	 * restart it proper.
 	 */
 	delta = next_tick - basemono;
-	if (delta <= (u64)TICK_NSEC) {
+	if (delta < (u64)TICK_NSEC) {
 		tick.tv64 = 0;
 		if (!ts->tick_stopped)
 			goto out;
Signature · 1 line
1.9.1
#7 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 4 lines · Jacob Pan

Upon entering idle, we can turn off tick if the next timeout is exactly one tick away. Otherwise, we could enter inner idle loop with tick still enabled, without resched set, the tick will continue during idle therefore less optimal in terms of energy savings.

This does not make any sense at all.

next_tick is the next required tick event. If it's exactly ONE tick away why should we go through the hassle of stopping it? Just to cancel the timer and then set it to the same value again? Oh well.

Thanks,

tglx

author
#8 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 9 lines · On Fri, 13 Nov 2015 15:22:16 -0500 (EST) Thomas Gleixner <[email protected]>…
Quoted 4 lines · Jacob Pan

Upon entering idle, we can turn off tick if the next timeout is exactly one tick away. Otherwise, we could enter inner idle loop with tick still enabled, without resched set, the tick will continue during idle therefore less optimal in terms of energy savings.

Quoted 6 lines

This does not make any sense at all.

next_tick is the next required tick event. If it's exactly ONE tick away why should we go through the hassle of stopping it? Just to cancel the timer and then set it to the same value again? Oh well.

I have been trying to understand this code, please help. Here is my theory and the ftrace of an injection period where tick did not stop. (sorry about the long lines). My comments are after [JP]

             cat-1993  [000]    30.093405: sched_cfs_idle_inject: action:0 throttled:1

[JP] injection timer expired, set forced idle flag, call scheduler

             cat-1993  [000]    30.093406: hrtimer_expire_exit:  hrtimer=0xffff88003de0cc20
             cat-1993  [000]    30.093406: hrtimer_start:        hrtimer=0xffff88003de0cc20 function=idle_inject_timer_fn/0x0 expires=29993055250 softexpires=29993055250
             cat-1993  [000]    30.093407: hrtimer_cancel:       hrtimer=0xffff88003dfce400
             cat-1993  [000]    30.093407: hrtimer_expire_entry: hrtimer=0xffff88003dfce400 now=29988042960 function=tick_sched_timer/0x0
             cat-1993  [000]    30.093407: function:             tick_sched_timer
             cat-1993  [000]    30.093422: function:                tick_sched_do_timer
             cat-1993  [000]    30.093422: function:                   tick_do_update_jiffies64
             cat-1993  [000]    30.093433: function:                tick_sched_handle.isra.15
             cat-1993  [000]    30.093447: sched_stat_runtime:   comm=cat pid=1993 runtime=1058498 [ns] vruntime=6695549826 [ns]
             cat-1993  [000]    30.093449: hrtimer_expire_exit:  hrtimer=0xffff88003dfce400
             cat-1993  [000]    30.093449: hrtimer_start:        hrtimer=0xffff88003dfce400 function=tick_sched_timer/0x0 expires=29989000000 softexpires=29989000000
             cat-1993  [000]    30.093450: function:             tick_program_event
             cat-1993  [000]    30.093460: sched_waking:         comm=rcu_preempt pid=7 prio=120 target_cpu=002
             cat-1993  [000]    30.093461: sched_wake_idle_without_ipi: cpu=2
             cat-1993  [000]    30.093463: sched_cfs_idle_inject: action:1 throttled:1

[JP] CFS pick_next_task_fair sees forced idle, pick no task to run.

             cat-1993  [000]    30.093463: sched_stat_runtime:   comm=cat pid=1993 runtime=16122 [ns] vruntime=6695565948 [ns]
             cat-1993  [000]    30.093464: sched_switch:         cat:1993 [120] R ==> swapper/0:0 [120]
          <idle>-0     [000]    30.093465: function:             tick_nohz_idle_enter
          <idle>-0     [000]    30.093473: bprint:               __tick_nohz_idle_enter: JPAN: __tick_nohz_idle_enter 803
          <idle>-0     [000]    30.093473: bprint:               __tick_nohz_idle_enter: JPAN: can_stop_idle_tick 743

[JP] can_stop_idle_tick() checks ok to stop tick

          <idle>-0     [000]    30.093474: bprint:               __tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta 1000000

[JP] but sees delta is exactly 1 tick away. didn't stop tick.

          <idle>-0     [000]    30.093475: function:             tick_check_broadcast_expired
          <idle>-0     [000]    30.094366: function:             tick_irq_enter
          <idle>-0     [000]    30.094367: function:                tick_check_oneshot_broadcast_this_cpu
          <idle>-0     [000]    30.094372: function:                tick_nohz_stop_idle
          <idle>-0     [000]    30.094387: hrtimer_cancel:
          hrtimer=0xffff88003dfce400

[JP] enter repeated tick sched in inner idle loop since !need_resched()

Quoted 5 lines

Thanks,

tglx

[Jacob Pan]

#9 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 6 lines · Jacob Pan
          <idle>-0     [000]    30.093473: bprint:               __tick_nohz_idle_enter: JPAN: __tick_nohz_idle_enter 803
          <idle>-0     [000]    30.093473: bprint:               __tick_nohz_idle_enter: JPAN: can_stop_idle_tick 743

[JP] can_stop_idle_tick() checks ok to stop tick

          <idle>-0     [000]    30.093474: bprint:               __tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta 1000000

[JP] but sees delta is exactly 1 tick away. didn't stop tick.

If the delta is 1 tick then it is not supposed to stop it. Did you ever try to figure out WHY it is 1 tick?

There are two code pathes which can set it to basemono + TICK_NSEC:

        if (rcu_needs_cpu(basemono, &next_rcu) ||
            arch_needs_cpu() || irq_work_needs_cpu()) {
                next_tick = basemono + TICK_NSEC;
        } else {
                next_tmr = get_next_timer_interrupt(basejiff, basemono);
                ts->next_timer = next_tmr;
                /* Take the next rcu event into account */
                next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
        }

Can you please figure out WHY the tick is requested to continue instead of blindly wreckaging the logic in that code?

Thanks,

tglx

author
#10 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 20 lines · On Mon, 16 Nov 2015 16:06:57 +0100 (CET) Thomas Gleixner <[email protected]>…
Quoted 4 lines
          <idle>-0     [000]    30.093474: bprint:

__tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop tick.

If the delta is 1 tick then it is not supposed to stop it. Did you ever try to figure out WHY it is 1 tick?

There are two code pathes which can set it to basemono + TICK_NSEC:

        if (rcu_needs_cpu(basemono, &next_rcu) ||
            arch_needs_cpu() || irq_work_needs_cpu()) {
                next_tick = basemono + TICK_NSEC;
        } else {
                next_tmr = get_next_timer_interrupt(basejiff,
basemono); ts->next_timer = next_tmr;
                /* Take the next rcu event into account */
                next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
        }
Quoted 2 lines

Can you please figure out WHY the tick is requested to continue instead of blindly wreckaging the logic in that code?

Looks like the it hits in both cases during forced idle.
+ Josh
+ Paul

For the first case, it is always related to RCU. I found there are two CONFIG options to avoid this undesired tick in idle loop.
1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)

Either one works but my concern is that users may not realize the intricate CONFIG_ options and how they translate into energy savings. Consulted with Josh, it seems we could add a check here to recognize the forced idle state and relax rcu_needs_cpu() to return false even it has callbacks. Since we are blocking everybody for a short time (5 ticks default). It should not impact synchronize and kfree rcu.

For the second case, which is much more rare, I think we do have next timer exactly one tick away. Just don't know why tick will continue into idle loop.

#11 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 3 lines · Jacob Pan

For the second case, which is much more rare, I think we do have next timer exactly one tick away. Just don't know why tick will continue into idle loop.

Well, it should not be hard to figure that out. There are not so many checks involved when tick_nohz_irq_exit() is called.

Thanks,

tglx

author
#12 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 6 lines · On Mon, 16 Nov 2015 23:01:12 +0100 (CET) Thomas Gleixner <[email protected]>…
Quoted 3 lines

For the second case, which is much more rare, I think we do have next timer exactly one tick away. Just don't know why tick will continue into idle loop.

Well, it should not be hard to figure that out. There are not so many checks involved when tick_nohz_irq_exit() is called.

Thanks for the tip, I found the cause is in
int idle_cpu(int cpu)

{
	if (rq->nr_running)
		return 0;

Since we only take into account of cfs_rq runnable taking over cfs_rq->nr_running when forced_idle is set.

Jacob

author
#13 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 6 lines · On Mon, 16 Nov 2015 16:09:10 -0800 Jacob Pan <[email protected]>…
Quoted 6 lines
Quoted 3 lines

For the second case, which is much more rare, I think we do have next timer exactly one tick away. Just don't know why tick will continue into idle loop.

Well, it should not be hard to figure that out. There are not so many checks involved when tick_nohz_irq_exit() is called.

Quoted 8 lines

Thanks for the tip, I found the cause is in
int idle_cpu(int cpu)

{
	if (rq->nr_running)
		return 0;

Since we only take into account of cfs_rq runnable taking over cfs_rq->nr_running when forced_idle is set.

I am not sure what is the best solution. It seems I can add additional checks like this.

kernel/sched/core.c +2 -3
@@ -3520,9 +3520,14 @@ int idle_cpu(int cpu)
        if (rq->curr != rq->idle)
                return 0;
 
-       if (rq->nr_running)
-               return 0;
-
+       if (rq->nr_running) {
+               /* if cfs_rq is in forced idle, nr_running could be

nonzero but still in idle */
+ if ((rq->nr_running != rq->cfs.h_nr_running) ||
+ cfs_rq_runnable(&rq->cfs))
+ return 0;
+ }

To recap the problem statement.
1. When entering idle loop tick_nohz_stop_sched_tick() checks if the next timer interrupt is exactly one tick away. if so, it will not stop it to avoid threshing timer disable and enable.
2. so it relies on the next round tick_nohz_irq_exit() to have another chance to stop the tick
3. with idle injection rq->nr_running could be nonzero when in idle
4. tick_nohz_irq_exit() will not be called if !idle_cpu()

However, idle_cpu() is used by many other callers, e.g. load balance. Do we want to consider forced idle in those cases? Or we can forgo this case and ignore it?

Jacob

maintainer
#14 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

In reply to Jacob Pan
Quoted 7 lines · Jacob Pan
Quoted 4 lines · On Mon, 16 Nov 2015 23:01:12 +0100 (CET) Thomas Gleixner <[email protected]>…
Quoted 3 lines

For the second case, which is much more rare, I think we do have next timer exactly one tick away. Just don't know why tick will continue into idle loop.

Quoted 5 lines
Quoted 2 lines

Well, it should not be hard to figure that out. There are not so many checks involved when tick_nohz_irq_exit() is called.

Thanks for the tip, I found the cause is in
int idle_cpu(int cpu)

Which one? That is, I cannot find a idle_cpu() call from tick_nohz_irq_exit().

author
#15 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 2 lines · On Thu, 19 Nov 2015 20:06:30 +0100 Peter Zijlstra <[email protected]> wrote:

Which one? That is, I cannot find a idle_cpu() call from tick_nohz_irq_exit().

it is tick_irq_exit(void), i will not call tick_nohz_irq_exit() if !idle_cpu()

please see my other email for complete story. pasted below.

I am not sure what is the best solution. It seems I can add additional checks like this.

kernel/sched/core.c +2 -3
@@ -3520,9 +3520,14 @@ int idle_cpu(int cpu)
        if (rq->curr != rq->idle)
                return 0;
 
-       if (rq->nr_running)
-               return 0;
-
+       if (rq->nr_running) {
+               /* if cfs_rq is in forced idle, nr_running could be

nonzero but still in idle */
+ if ((rq->nr_running != rq->cfs.h_nr_running) ||
+ cfs_rq_runnable(&rq->cfs))
+ return 0;
+ }

To recap the problem statement.
1. When entering idle loop tick_nohz_stop_sched_tick() checks if the next timer interrupt is exactly one tick away. if so, it will not stop it to avoid threshing timer disable and enable.
2. so it relies on the next round tick_nohz_irq_exit() to have another chance to stop the tick
3. with idle injection rq->nr_running could be nonzero when in idle
4. tick_nohz_irq_exit() will not be called if !idle_cpu()

However, idle_cpu() is used by many other callers, e.g. load balance. Do we want to consider forced idle in those cases? Or we can forgo this case and ignore it?

maintainer
#16 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 7 lines · Jacob Pan
Quoted 2 lines · On Thu, 19 Nov 2015 20:06:30 +0100 Peter Zijlstra <[email protected]> wrote:

Which one? That is, I cannot find a idle_cpu() call from tick_nohz_irq_exit().

it is tick_irq_exit(void), i will not call tick_nohz_irq_exit() if !idle_cpu()

Ah, but I think we really only need to test for the idle task there, the && need_resched() part of that function takes care of the rest.

Once we have need_resched() set, we'll be on our way to tick_nohz_idle_exit() to restart the tick again.

---
 kernel/softirq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/softirq.c +1 -1
index 479e4436f787..3bd30404f2ee 100644
@@ -367,7 +367,7 @@ static inline void tick_irq_exit(void)
 	int cpu = smp_processor_id();
 
 	/* Make sure that timer wheel updates are propagated */
-	if ((idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
+	if ((is_idle_task(current) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
 		if (!in_interrupt())
 			tick_nohz_irq_exit();
 	}
author
#17 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 6 lines · On Thu, 19 Nov 2015 20:59:05 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 5 lines · Jacob Pan
Quoted 2 lines · On Thu, 19 Nov 2015 20:06:30 +0100 Peter Zijlstra <[email protected]> wrote:

Which one? That is, I cannot find a idle_cpu() call from tick_nohz_irq_exit().

Quoted 8 lines
Quoted 2 lines

it is tick_irq_exit(void), i will not call tick_nohz_irq_exit() if !idle_cpu()

Ah, but I think we really only need to test for the idle task there, the && need_resched() part of that function takes care of the rest.

Once we have need_resched() set, we'll be on our way to tick_nohz_idle_exit() to restart the tick again.

Tested, it did the trick also. has less impact than changing idle_cpu().

My other point is the callers of idle_cpu() will see it returns false but in forced idle. Can't see anything bad since we skip load balance.

Thanks,

Jacob

#18 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

In reply to Jacob Pan
Quoted 35 lines · Jacob Pan
Quoted 20 lines · On Mon, 16 Nov 2015 16:06:57 +0100 (CET) Thomas Gleixner <[email protected]>…
Quoted 4 lines
          <idle>-0     [000]    30.093474: bprint:

__tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop tick.

If the delta is 1 tick then it is not supposed to stop it. Did you ever try to figure out WHY it is 1 tick?

There are two code pathes which can set it to basemono + TICK_NSEC:

        if (rcu_needs_cpu(basemono, &next_rcu) ||
            arch_needs_cpu() || irq_work_needs_cpu()) {
                next_tick = basemono + TICK_NSEC;
        } else {
                next_tmr = get_next_timer_interrupt(basejiff,
basemono); ts->next_timer = next_tmr;
                /* Take the next rcu event into account */
                next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
        }
Quoted 2 lines

Can you please figure out WHY the tick is requested to continue instead of blindly wreckaging the logic in that code?

Looks like the it hits in both cases during forced idle.
+ Josh
+ Paul

For the first case, it is always related to RCU. I found there are two CONFIG options to avoid this undesired tick in idle loop.
1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)

Quoted 6 lines

Either one works but my concern is that users may not realize the intricate CONFIG_ options and how they translate into energy savings. Consulted with Josh, it seems we could add a check here to recognize the forced idle state and relax rcu_needs_cpu() to return false even it has callbacks. Since we are blocking everybody for a short time (5 ticks default). It should not impact synchronize and kfree rcu.

Or we could just set things up so that whatever Kconfig you are using to enable this state causes CONFIG_RCU_NOCB_CPU_ALL to also be enabled. Or that causes CONFIG_RCU_FAST_NO_HZ to also be enabled, if that works better for you.

Just out of curiosity, what is the purpose of the forced idle state? Thermal control or some such?

Thanx, Paul

Quoted 4 lines

For the second case, which is much more rare, I think we do have next timer exactly one tick away. Just don't know why tick will continue into idle loop.

author
#19 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 2 lines · On Mon, 16 Nov 2015 14:31:17 -0800 "Paul E. McKenney"…

Just out of curiosity, what is the purpose of the forced idle state? Thermal control or some such?

yes. for thermal control or power capping. I have some description in cover letter.
https://lwn.net/Articles/664405/

Thanks,
Jacob

author
#20 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

In reply to Paul E. McKenney
Quoted 8 lines · On Mon, 16 Nov 2015 14:31:17 -0800 "Paul E. McKenney"…
Quoted 7 lines

Either one works but my concern is that users may not realize the intricate CONFIG_ options and how they translate into energy savings. Consulted with Josh, it seems we could add a check here to recognize the forced idle state and relax rcu_needs_cpu() to return false even it has callbacks. Since we are blocking everybody for a short time (5 ticks default). It should not impact synchronize and kfree rcu.

Quoted 4 lines

Or we could just set things up so that whatever Kconfig you are using to enable this state causes CONFIG_RCU_NOCB_CPU_ALL to also be enabled. Or that causes CONFIG_RCU_FAST_NO_HZ to also be enabled, if that works better for you.

That would be great, we can work this out once the patch is finalized. This is not a hard dependency in that it only affects the efficiency of idle injection.

#21 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 11 lines · Jacob Pan
Quoted 8 lines · On Mon, 16 Nov 2015 14:31:17 -0800 "Paul E. McKenney"…
Quoted 7 lines

Either one works but my concern is that users may not realize the intricate CONFIG_ options and how they translate into energy savings. Consulted with Josh, it seems we could add a check here to recognize the forced idle state and relax rcu_needs_cpu() to return false even it has callbacks. Since we are blocking everybody for a short time (5 ticks default). It should not impact synchronize and kfree rcu.

Quoted 8 lines
Quoted 4 lines

Or we could just set things up so that whatever Kconfig you are using to enable this state causes CONFIG_RCU_NOCB_CPU_ALL to also be enabled. Or that causes CONFIG_RCU_FAST_NO_HZ to also be enabled, if that works better for you.

That would be great, we can work this out once the patch is finalized. This is not a hard dependency in that it only affects the efficiency of idle injection.

Is this mostly an special-purpose embedded thing, or do you expect distros to be enabling this? If the former, I suggest CONFIG_RCU_NOCB_CPU_ALL, but if distros are doing this for general-purpose workloads, I instead suggest CONFIG_RCU_FAST_NO_HZ.

But as you say, we can work this out later. Figured I should ask now, though, just to get people thinking about it.

Thanx, Paul

#22 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 4 lines · Paul E. McKenney

Is this mostly an special-purpose embedded thing, or do you expect distros to be enabling this? If the former, I suggest CONFIG_RCU_NOCB_CPU_ALL, but if distros are doing this for general-purpose workloads, I instead suggest CONFIG_RCU_FAST_NO_HZ.

thermal overload happens a lot on small devices, but sadly also in big datacenters where it is not uncommon to underprovision cooling capacity by a bit (it's one of those "99% of the time you only need THIS much, the 1% you need 30% more" and that more is expensive or even impractical)

author
#23 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 7 lines · On Mon, 16 Nov 2015 15:32:38 -0800 Arjan van de Ven <[email protected]>…
Quoted 4 lines · Paul E. McKenney

Is this mostly an special-purpose embedded thing, or do you expect distros to be enabling this? If the former, I suggest
, but if distros are doing this for
general-purpose workloads, I instead suggest CONFIG_RCU_FAST_NO_HZ.

Quoted 5 lines

thermal overload happens a lot on small devices, but sadly also in big datacenters where it is not uncommon to underprovision cooling capacity by a bit (it's one of those "99% of the time you only need THIS much, the 1% you need 30% more" and that more is expensive or even impractical)

CONFIG_RCU_FAST_NO_HZ is more inline with idle injection in that both are targeted energy efficiency.
Checking ubuntu and fedora, seems both have CONFIG_RCU_NOCB_CPU_ALL=y.

#24 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 10 lines · Jacob Pan
Quoted 7 lines · On Mon, 16 Nov 2015 15:32:38 -0800 Arjan van de Ven <[email protected]>…
Quoted 4 lines · Paul E. McKenney

Is this mostly an special-purpose embedded thing, or do you expect distros to be enabling this? If the former, I suggest
, but if distros are doing this for
general-purpose workloads, I instead suggest CONFIG_RCU_FAST_NO_HZ.

Quoted 5 lines
Quoted 5 lines

thermal overload happens a lot on small devices, but sadly also in big datacenters where it is not uncommon to underprovision cooling capacity by a bit (it's one of those "99% of the time you only need THIS much, the 1% you need 30% more" and that more is expensive or even impractical)

Then CONFIG_RCU_FAST_NO_HZ it is.

Quoted 3 lines

CONFIG_RCU_FAST_NO_HZ is more inline with idle injection in that both are targeted energy efficiency.
Checking ubuntu and fedora, seems both have CONFIG_RCU_NOCB_CPU_ALL=y.

Ah, that is right -- they both do CONFIG_NO_HZ_FULL=y, which does imply CONFIG_RCU_NOCB_CPU_ALL=y. But SUSE does not, last I knew.

But it should be easy to set up Kconfig for this. Make your new Kconfig option select CONFIG_RCU_FAST_NO_HZ unless CONFIG_RCU_NOCB_CPU_ALL is already set. Alternatively, CONFIG_RCU_FAST_NO_HZ could be set up something like the following:

	depends on (NO_HZ_COMMON && SMP && RCU_EXPERT) || CONFIG_THERM_THROT
	default CONFIG_THERM_THROT

But the "select RCU_FAST_NO_HZ if !RCU_NOCB_CPU_ALL" is probably cleaner.

Anyway, again, the details can be settled later.

Thanx, Paul

#25 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

In reply to Jacob Pan
Quoted 35 lines · Jacob Pan
Quoted 20 lines · On Mon, 16 Nov 2015 16:06:57 +0100 (CET) Thomas Gleixner <[email protected]>…
Quoted 4 lines
          <idle>-0     [000]    30.093474: bprint:

__tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop tick.

If the delta is 1 tick then it is not supposed to stop it. Did you ever try to figure out WHY it is 1 tick?

There are two code pathes which can set it to basemono + TICK_NSEC:

        if (rcu_needs_cpu(basemono, &next_rcu) ||
            arch_needs_cpu() || irq_work_needs_cpu()) {
                next_tick = basemono + TICK_NSEC;
        } else {
                next_tmr = get_next_timer_interrupt(basejiff,
basemono); ts->next_timer = next_tmr;
                /* Take the next rcu event into account */
                next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
        }
Quoted 2 lines

Can you please figure out WHY the tick is requested to continue instead of blindly wreckaging the logic in that code?

Looks like the it hits in both cases during forced idle.
+ Josh
+ Paul

For the first case, it is always related to RCU. I found there are two CONFIG options to avoid this undesired tick in idle loop.
1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)

Quoted 6 lines

Either one works but my concern is that users may not realize the intricate CONFIG_ options and how they translate into energy savings. Consulted with Josh, it seems we could add a check here to recognize the forced idle state and relax rcu_needs_cpu() to return false even it has callbacks. Since we are blocking everybody for a short time (5 ticks default). It should not impact synchronize and kfree rcu.

Right; as long as you're blocking *everybody*, and RCU priority boosting doesn't come into play (meaning a real-time task is waiting on RCU callbacks), then I don't see any harm in blocking RCU callbacks for a while. You'd block completion of synchronize_rcu() and similar, as well as memory reclamation, but since you've blocked *every* CPU systemwide then that doesn't cause a problem.

#26 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 43 lines · Josh Triplett
Quoted 35 lines · Jacob Pan
Quoted 20 lines · On Mon, 16 Nov 2015 16:06:57 +0100 (CET) Thomas Gleixner <[email protected]>…
Quoted 4 lines
          <idle>-0     [000]    30.093474: bprint:

__tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop tick.

If the delta is 1 tick then it is not supposed to stop it. Did you ever try to figure out WHY it is 1 tick?

There are two code pathes which can set it to basemono + TICK_NSEC:

        if (rcu_needs_cpu(basemono, &next_rcu) ||
            arch_needs_cpu() || irq_work_needs_cpu()) {
                next_tick = basemono + TICK_NSEC;
        } else {
                next_tmr = get_next_timer_interrupt(basejiff,
basemono); ts->next_timer = next_tmr;
                /* Take the next rcu event into account */
                next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
        }
Quoted 2 lines

Can you please figure out WHY the tick is requested to continue instead of blindly wreckaging the logic in that code?

Looks like the it hits in both cases during forced idle.
+ Josh
+ Paul

For the first case, it is always related to RCU. I found there are two CONFIG options to avoid this undesired tick in idle loop.
1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)

Quoted 6 lines

Either one works but my concern is that users may not realize the intricate CONFIG_ options and how they translate into energy savings. Consulted with Josh, it seems we could add a check here to recognize the forced idle state and relax rcu_needs_cpu() to return false even it has callbacks. Since we are blocking everybody for a short time (5 ticks default). It should not impact synchronize and kfree rcu.

Quoted 6 lines

Right; as long as you're blocking *everybody*, and RCU priority boosting doesn't come into play (meaning a real-time task is waiting on RCU callbacks), then I don't see any harm in blocking RCU callbacks for a while. You'd block completion of synchronize_rcu() and similar, as well as memory reclamation, but since you've blocked *every* CPU systemwide then that doesn't cause a problem.

True enough. But how does RCU distinguish between this being a normal idle cycle that might last indefinitely on the one hand and the five-jiffy system-wide throttling on the other? OK, maybe there is a global variable that says that the just-now-starting idle period is system-wide throttling. But then what about the CPU that just went idle 10 microseconds ago, and therefore left its timer tick running? Fine and well, we could IPI it to wake it up and let it see that we are now doing thermal throttling. But then we presumably also have to IPI it at the end of the thermal-throttling interval in order for it to re-evaluate whether or not it should have the tick going. :-/

On the one hand, I am sure that all of this can be made to work, but simply having systems using thermal throttling enable either CONFIG_RCU_NOCB_CPU_ALL or CONFIG_RCU_FAST_NO_HZ seems -way- simpler. CONFIG_RCU_FAST_NO_HZ is probably the better choice for generic workloads, but CONFIG_RCU_NOCB_CPU_ALL is the better choice for embedded workloads where it is less likely that RCU callbacks will be posted with continuous wild abandon.

Or am I missing something subtle here?

Thanx, Paul

#27 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 70 lines · Paul E. McKenney
Quoted 43 lines · Josh Triplett
Quoted 35 lines · Jacob Pan
Quoted 20 lines · On Mon, 16 Nov 2015 16:06:57 +0100 (CET) Thomas Gleixner <[email protected]>…
Quoted 4 lines
          <idle>-0     [000]    30.093474: bprint:

__tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop tick.

If the delta is 1 tick then it is not supposed to stop it. Did you ever try to figure out WHY it is 1 tick?

There are two code pathes which can set it to basemono + TICK_NSEC:

        if (rcu_needs_cpu(basemono, &next_rcu) ||
            arch_needs_cpu() || irq_work_needs_cpu()) {
                next_tick = basemono + TICK_NSEC;
        } else {
                next_tmr = get_next_timer_interrupt(basejiff,
basemono); ts->next_timer = next_tmr;
                /* Take the next rcu event into account */
                next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
        }
Quoted 2 lines

Can you please figure out WHY the tick is requested to continue instead of blindly wreckaging the logic in that code?

Looks like the it hits in both cases during forced idle.
+ Josh
+ Paul

For the first case, it is always related to RCU. I found there are two CONFIG options to avoid this undesired tick in idle loop.
1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)

Quoted 6 lines

Either one works but my concern is that users may not realize the intricate CONFIG_ options and how they translate into energy savings. Consulted with Josh, it seems we could add a check here to recognize the forced idle state and relax rcu_needs_cpu() to return false even it has callbacks. Since we are blocking everybody for a short time (5 ticks default). It should not impact synchronize and kfree rcu.

Quoted 6 lines

Right; as long as you're blocking *everybody*, and RCU priority boosting doesn't come into play (meaning a real-time task is waiting on RCU callbacks), then I don't see any harm in blocking RCU callbacks for a while. You'd block completion of synchronize_rcu() and similar, as well as memory reclamation, but since you've blocked *every* CPU systemwide then that doesn't cause a problem.

True enough. But how does RCU distinguish between this being a normal idle cycle that might last indefinitely on the one hand and the five-jiffy system-wide throttling on the other? OK, maybe there is a global variable that says that the just-now-starting idle period is system-wide throttling. But then what about the CPU that just went idle 10 microseconds ago, and therefore left its timer tick running? Fine and well, we could IPI it to wake it up and let it see that we are now doing thermal throttling. But then we presumably also have to IPI it at the end of the thermal-throttling interval in order for it to re-evaluate whether or not it should have the tick going. :-/

On the one hand, I am sure that all of this can be made to work, but simply having systems using thermal throttling enable either CONFIG_RCU_NOCB_CPU_ALL or CONFIG_RCU_FAST_NO_HZ seems -way- simpler. CONFIG_RCU_FAST_NO_HZ is probably the better choice for generic workloads, but CONFIG_RCU_NOCB_CPU_ALL is the better choice for embedded workloads where it is less likely that RCU callbacks will be posted with continuous wild abandon.

Quoted 1 line

Or am I missing something subtle here?

I agree that it seems preferable to make this require an existing RCU solution rather than adding more complexity to the RCU idle path. One possible thing that may affect the choice of solution: this needs to idle *every* CPU, without leaving any CPU awake to handle callbacks or similar.

- Josh Triplett

#28 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 71 lines · Josh Triplett
Quoted 70 lines · Paul E. McKenney
Quoted 43 lines · Josh Triplett
Quoted 35 lines · Jacob Pan
Quoted 20 lines · On Mon, 16 Nov 2015 16:06:57 +0100 (CET) Thomas Gleixner <[email protected]>…
Quoted 4 lines
          <idle>-0     [000]    30.093474: bprint:

__tick_nohz_idle_enter: JPAN: tick_nohz_stop_sched_tick 609 delta 1000000 [JP] but sees delta is exactly 1 tick away. didn't stop tick.

If the delta is 1 tick then it is not supposed to stop it. Did you ever try to figure out WHY it is 1 tick?

There are two code pathes which can set it to basemono + TICK_NSEC:

        if (rcu_needs_cpu(basemono, &next_rcu) ||
            arch_needs_cpu() || irq_work_needs_cpu()) {
                next_tick = basemono + TICK_NSEC;
        } else {
                next_tmr = get_next_timer_interrupt(basejiff,
basemono); ts->next_timer = next_tmr;
                /* Take the next rcu event into account */
                next_tick = next_rcu < next_tmr ? next_rcu : next_tmr;
        }
Quoted 2 lines

Can you please figure out WHY the tick is requested to continue instead of blindly wreckaging the logic in that code?

Looks like the it hits in both cases during forced idle.
+ Josh
+ Paul

For the first case, it is always related to RCU. I found there are two CONFIG options to avoid this undesired tick in idle loop.
1. enable CONFIG_RCU_NOCB_CPU_ALL, offload to orcu kthreads
2. or enable CONFIG_RCU_FAST_NO_HZ (enter dytick idle w/ rcu callback)

Quoted 6 lines

Either one works but my concern is that users may not realize the intricate CONFIG_ options and how they translate into energy savings. Consulted with Josh, it seems we could add a check here to recognize the forced idle state and relax rcu_needs_cpu() to return false even it has callbacks. Since we are blocking everybody for a short time (5 ticks default). It should not impact synchronize and kfree rcu.

Quoted 6 lines

Right; as long as you're blocking *everybody*, and RCU priority boosting doesn't come into play (meaning a real-time task is waiting on RCU callbacks), then I don't see any harm in blocking RCU callbacks for a while. You'd block completion of synchronize_rcu() and similar, as well as memory reclamation, but since you've blocked *every* CPU systemwide then that doesn't cause a problem.

True enough. But how does RCU distinguish between this being a normal idle cycle that might last indefinitely on the one hand and the five-jiffy system-wide throttling on the other? OK, maybe there is a global variable that says that the just-now-starting idle period is system-wide throttling. But then what about the CPU that just went idle 10 microseconds ago, and therefore left its timer tick running? Fine and well, we could IPI it to wake it up and let it see that we are now doing thermal throttling. But then we presumably also have to IPI it at the end of the thermal-throttling interval in order for it to re-evaluate whether or not it should have the tick going. :-/

On the one hand, I am sure that all of this can be made to work, but simply having systems using thermal throttling enable either CONFIG_RCU_NOCB_CPU_ALL or CONFIG_RCU_FAST_NO_HZ seems -way- simpler. CONFIG_RCU_FAST_NO_HZ is probably the better choice for generic workloads, but CONFIG_RCU_NOCB_CPU_ALL is the better choice for embedded workloads where it is less likely that RCU callbacks will be posted with continuous wild abandon.

Quoted 7 lines
Quoted 1 line

Or am I missing something subtle here?

I agree that it seems preferable to make this require an existing RCU solution rather than adding more complexity to the RCU idle path. One possible thing that may affect the choice of solution: this needs to idle *every* CPU, without leaving any CPU awake to handle callbacks or similar.

Fair point. When in the five-jiffy throttling state, what can wake up a CPU? In an earlier version of this proposal, the answer was "nothing", but maybe that has changed.

Thanx, Paul

#29 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 3 lines · Paul E. McKenney

Fair point. When in the five-jiffy throttling state, what can wake up a CPU? In an earlier version of this proposal, the answer was "nothing", but maybe that has changed.

device interrupts are likely to wake the cpus.

#30 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 6 lines · Arjan van de Ven
Quoted 3 lines · Paul E. McKenney

Fair point. When in the five-jiffy throttling state, what can wake up a CPU? In an earlier version of this proposal, the answer was "nothing", but maybe that has changed.

device interrupts are likely to wake the cpus.

OK, that I cannot help you with. But presumably if the interrupt handler does a wakeup (or similar), that is deferred to the end of the throttling interval? Timers are also deferred, including hrtimers?

Thanx, Paul

maintainer
#31 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 11 lines · Paul E. McKenney
Quoted 6 lines · Arjan van de Ven
Quoted 3 lines · Paul E. McKenney

Fair point. When in the five-jiffy throttling state, what can wake up a CPU? In an earlier version of this proposal, the answer was "nothing", but maybe that has changed.

device interrupts are likely to wake the cpus.

OK, that I cannot help you with. But presumably if the interrupt handler does a wakeup (or similar), that is deferred to the end of the throttling interval? Timers are also deferred, including hrtimers?

This throttling thing only throttles 'normal' tasks, real-time tasks will still run.

author
#32 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 9 lines · On Tue, 17 Nov 2015 11:24:49 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 8 lines · Paul E. McKenney
Quoted 6 lines · Arjan van de Ven
Quoted 3 lines · Paul E. McKenney

Fair point. When in the five-jiffy throttling state, what can wake up a CPU? In an earlier version of this proposal, the answer was "nothing", but maybe that has changed.

device interrupts are likely to wake the cpus.

Quoted 7 lines
Quoted 4 lines

OK, that I cannot help you with. But presumably if the interrupt handler does a wakeup (or similar), that is deferred to the end of the throttling interval? Timers are also deferred, including hrtimers?

This throttling thing only throttles 'normal' tasks, real-time tasks will still run.

As an optimization or option, it might be useful to further defer the next timer interrupt if it falls within the idle injection period. But I guess we don't know if that timer belongs to a normal task or rt. Also we there could be more than one 'next' timer interrupts fall into that injection idle period.

#33 lore raw

Re: [PATCH 2/4] timer: relax tick stop in idle entry

Quoted 12 lines · Jacob Pan
Quoted 9 lines · On Tue, 17 Nov 2015 11:24:49 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 8 lines · Paul E. McKenney
Quoted 6 lines · Arjan van de Ven
Quoted 3 lines · Paul E. McKenney

Fair point. When in the five-jiffy throttling state, what can wake up a CPU? In an earlier version of this proposal, the answer was "nothing", but maybe that has changed.

device interrupts are likely to wake the cpus.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

OK, that I cannot help you with. But presumably if the interrupt handler does a wakeup (or similar), that is deferred to the end of the throttling interval? Timers are also deferred, including hrtimers?

This throttling thing only throttles 'normal' tasks, real-time tasks will still run.

Heh! Then RCU will be delayed or not based on the priority of the grace-period kthreads and softirqd. ;-)

In addition, this does sound like an excellent test for priority-inversion situations that might otherwise go unnoticed on overprovisioned systems. That said, I would expect many types of real-time systems to configure voltage, frequency, and cooling so as to avoid thermal throttling.

Quoted 5 lines

As an optimization or option, it might be useful to further defer the next timer interrupt if it falls within the idle injection period. But I guess we don't know if that timer belongs to a normal task or rt. Also we there could be more than one 'next' timer interrupts fall into that injection idle period.

Understood. This brings me back to my recommendation that throttling select RCU_FAST_NO_HZ unless RCU_NOCB_CPU_ALL is already set.

Thanx, Paul

author
#34 lore raw

PATCH 3/4 sched: introduce synchronized idle injection

In reply to Jacob Pan

With increasingly constrained power and thermal budget, it's often necessary to cap power via throttling. Throttling individual CPUs or devices at random times can help power capping but may not be optimal in terms of energy efficiency. Frequency scaling is also limited by certain range before losing energy efficiency.

In general, the optimal solution in terms of energy efficiency is to align idle periods such that more shared circuits can be power gated to enter lower power states. Combined with energy efficient frequency point, idle injection provides a way to scale power and performance efficiently.

This patch introduces a scheduler based idle injection method, it works by blocking CFS runqueue synchronously and periodically. The actions on all online CPUs are orchestrated by per CPU hrtimers.

Two sysctl knobs are given to the userspace for selecting the percentage of idle time as well as the forced idle duration for each idle period injected.

Since only CFS class is targeted, other high priority tasks are not affected, such as EDF and RT tasks as well as softirq and interrupts.

Hotpath in CFS pick_next_task is optimized by Peter Zijlstra, where a new runnable flag is introduced to combine forced idle and nr_running.

---
 include/linux/sched.h        |  11 ++
 include/linux/sched/sysctl.h |   5 +
 init/Kconfig                 |  10 ++
 kernel/sched/fair.c          | 353 ++++++++++++++++++++++++++++++++++++++++++-
 kernel/sched/sched.h         |  54 ++++++-
 kernel/sysctl.c              |  21 +++
 6 files changed, 449 insertions(+), 5 deletions(-)
include/linux/sched.h +11 -0
index b7b9501..ff551a3 100644
@@ -3180,4 +3180,15 @@ static inline unsigned long rlimit_max(unsigned int limit)
 	return task_rlimit_max(current, limit);
 }
 
+#ifdef CONFIG_CFS_IDLE_INJECT
+extern int proc_sched_cfs_idle_inject_pct_handler(struct ctl_table *table,
+						int write,
+						void __user *buffer,
+						size_t *length, loff_t *ppos);
+extern int proc_sched_cfs_idle_inject_duration_handler(struct ctl_table *table,
+						int write,
+						void __user *buffer,
+						size_t *length, loff_t *ppos);
+#endif
+
 #endif
include/linux/sched/sysctl.h +5 -0
index c9e4731..d32da45 100644
@@ -81,6 +81,11 @@ extern unsigned int sysctl_sched_cfs_bandwidth_slice;
 extern unsigned int sysctl_sched_autogroup_enabled;
 #endif
 
+#ifdef CONFIG_CFS_IDLE_INJECT
+extern unsigned int sysctl_sched_cfs_idle_inject_pct;
+extern unsigned int sysctl_sched_cfs_idle_inject_duration;
+#endif
+
 extern int sched_rr_timeslice;
 
 extern int sched_rr_handler(struct ctl_table *table, int write,
init/Kconfig +10 -0
index c24b6f7..4041c94 100644
@@ -1086,6 +1086,16 @@ menuconfig CGROUP_SCHED
 	  bandwidth allocation to such task groups. It uses cgroups to group
 	  tasks.
 
+config CFS_IDLE_INJECT
+	bool "Synchronized CFS idle injection"
+	depends on NO_HZ_IDLE && HIGH_RES_TIMERS
+	default n
+	help
+	  This feature let scheduler inject synchronized idle time across all online
+	  CPUs. Idle injection affects normal tasks only, yeilds to RT and interrupts.
+	  Effecitvely, CPUs can be duty cycled between running at the most power
+	  efficient performance state and deep idle states.
+
 if CGROUP_SCHED
 config FAIR_GROUP_SCHED
 	bool "Group scheduling for SCHED_OTHER"
kernel/sched/fair.c +349 -4

398 lines not on this page · raw message

kernel/sched/sched.h +53 -1

66 lines not on this page · raw message

kernel/sysctl.c +21 -0
index e69201d..e7d39f7 100644
@@ -124,6 +124,7 @@ static int __maybe_unused one = 1;
 static int __maybe_unused two = 2;
 static int __maybe_unused four = 4;
 static unsigned long one_ul = 1;
+static int fifty = 50;
 static int one_hundred = 100;
 #ifdef CONFIG_PRINTK
 static int ten_thousand = 10000;
@@ -433,6 +434,26 @@ static struct ctl_table kern_table[] = {
 		.extra1		= &one,
 	},
 #endif
+#ifdef CONFIG_CFS_IDLE_INJECT
+	{
+		.procname	= "sched_cfs_idle_inject_pct",
+		.data		= &sysctl_sched_cfs_idle_inject_pct,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_sched_cfs_idle_inject_pct_handler,
+		.extra1		= &zero,
+		.extra2		= &fifty,
+	},
+	{
+		.procname	= "sched_cfs_idle_inject_duration",
+		.data		= &sysctl_sched_cfs_idle_inject_duration,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_sched_cfs_idle_inject_duration_handler,
+		.extra1		= &four,
+		.extra2		= &one_hundred,
+	},
+#endif
 #ifdef CONFIG_PROVE_LOCKING
 	{
 		.procname	= "prove_locking",
Signature · 1 line
1.9.1
#35 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

reproduce:

Hi Jacob,

[auto build test WARNING on: tip/sched/core]
[also build test WARNING on: v4.3 next-20151113]

url:    https://github.com/0day-ci/linux/commits/Jacob-Pan/CFS-idle-injection/20151114-035732
config: i386-randconfig-s1-201545 (attached as .config)
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All warnings (new ones prefixed by >>):

>> kernel/sysctl.c:127:12: warning: 'fifty' defined but not used [-Wunused-variable]
    static int fifty = 50;
               ^

vim +/fifty +127 kernel/sysctl.c

111 #ifndef CONFIG_MMU
112 extern int sysctl_nr_trim_pages;
113 #endif
114
115 /* Constants used for minimum and maximum */
116 #ifdef CONFIG_LOCKUP_DETECTOR
117 static int sixty = 60;
118 #endif
119
120 static int __maybe_unused neg_one = -1;
121
122 static int zero;
123 static int __maybe_unused one = 1;
124 static int __maybe_unused two = 2;
125 static int __maybe_unused four = 4;
126 static unsigned long one_ul = 1;

 > 127	static int fifty = 50;

128 static int one_hundred = 100;
129 #ifdef CONFIG_PRINTK
130 static int ten_thousand = 10000;
131 #endif
132
133 /* this is needed for the proc_doulongvec_minmax of vm_dirty_bytes */
134 static unsigned long dirty_bytes_min = 2 * PAGE_SIZE;
135

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

maintainer
#36 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

In reply to Jacob Pan
Quoted 11 lines · Jacob Pan

With increasingly constrained power and thermal budget, it's often necessary to cap power via throttling. Throttling individual CPUs or devices at random times can help power capping but may not be optimal in terms of energy efficiency. Frequency scaling is also limited by certain range before losing energy efficiency.

In general, the optimal solution in terms of energy efficiency is to align idle periods such that more shared circuits can be power gated to enter lower power states. Combined with energy efficient frequency point, idle injection provides a way to scale power and performance efficiently.

Quoted 7 lines

This patch introduces a scheduler based idle injection method, it works by blocking CFS runqueue synchronously and periodically. The actions on all online CPUs are orchestrated by per CPU hrtimers.

Two sysctl knobs are given to the userspace for selecting the percentage of idle time as well as the forced idle duration for each idle period injected.

What's the purpose of these knobs? Just testing, or will some user-space daemon set them dynamically?

I.e. what mechanism will drive the throttling in the typical case?

Quoted 6 lines

Since only CFS class is targeted, other high priority tasks are not affected, such as EDF and RT tasks as well as softirq and interrupts.

Hotpath in CFS pick_next_task is optimized by Peter Zijlstra, where a new runnable flag is introduced to combine forced idle and nr_running.

init/Kconfig menuconfig CGROUP_SCHED · line 1097

Quoted 9 lines
init/Kconfig +9 -0
+config CFS_IDLE_INJECT
+	bool "Synchronized CFS idle injection"
+	depends on NO_HZ_IDLE && HIGH_RES_TIMERS
+	default n
+	help
+	  This feature let scheduler inject synchronized idle time across all online
+	  CPUs. Idle injection affects normal tasks only, yeilds to RT and interrupts.
+	  Effecitvely, CPUs can be duty cycled between running at the most power
+	  efficient performance state and deep idle states.

So there are 3 typos in this single paragraph alone ...

I also think that naming it 'idle injection' is pretty euphemistic: this is forced idling, right? So why not name it CFS_FORCED_IDLE?

What will such throttling do to latencies, as observed by user-space tasks? What's the typical expected frequency of the throttling frequency that you are targeting?

Thanks,

Ingo

maintainer
#37 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 2 lines · Ingo Molnar

What will such throttling do to latencies, as observed by user-space tasks? What's the typical expected frequency of the throttling frequency that you are targeting?

The default has 5ms (iirc) of forced idle, so depending on what you do, noticeable to outright painful.

#38 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Ingo Molnar

What will such throttling do to latencies, as observed by user-space tasks? What's the typical expected frequency of the throttling frequency that you are targeting?

The default has 5ms (iirc) of forced idle, so depending on what you do, noticeable to outright painful.

IIUC, it is 5 ticks, not ms.

Which raises the question, doesn't that mean that we get disturbed four times on each cpu during the forced idle period? So idle injection only makes sense if the platform has package states with a target residency less than a jiffy. Or, do we enter NOHZ idle? I haven't looked closely enough to figure out yet.

maintainer
#39 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 9 lines · Morten Rasmussen
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Ingo Molnar

What will such throttling do to latencies, as observed by user-space tasks? What's the typical expected frequency of the throttling frequency that you are targeting?

The default has 5ms (iirc) of forced idle, so depending on what you do, noticeable to outright painful.

IIUC, it is 5 ticks, not ms.

The code uses hrtimers (badly), this means there _should_ not be a tick dependency.

Quoted 5 lines

Which raises the question, doesn't that mean that we get disturbed four times on each cpu during the forced idle period? So idle injection only makes sense if the platform has package states with a target residency less than a jiffy. Or, do we enter NOHZ idle? I haven't looked closely enough to figure out yet.

The idea is to hit NOHZ.

#40 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Morten Rasmussen
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Ingo Molnar

What will such throttling do to latencies, as observed by user-space tasks? What's the typical expected frequency of the throttling frequency that you are targeting?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

The default has 5ms (iirc) of forced idle, so depending on what you do, noticeable to outright painful.

IIUC, it is 5 ticks, not ms.

The code uses hrtimers (badly), this means there _should_ not be a tick dependency.

Then I'm confused :-/

I see the hrtimers, but the actual idle duration appears to be in ticks rather than ms and then converted later.

+/* Duration of idle time in ticks of each injection period */ +unsigned int sysctl_sched_cfs_idle_inject_duration = 5UL;

...and futher down we have:

+ duration_msec = jiffies_to_msecs(sysctl_sched_cfs_idle_inject_duration);

I will go back and look harder.

Quoted 8 lines
Quoted 6 lines

Which raises the question, doesn't that mean that we get disturbed four times on each cpu during the forced idle period? So idle injection only makes sense if the platform has package states with a target residency less than a jiffy. Or, do we enter NOHZ idle? I haven't looked closely enough to figure out yet.

The idea is to hit NOHZ.

Nice!

author
#41 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 5 lines · On Wed, 18 Nov 2015 14:04:41 +0000 Morten Rasmussen <[email protected]>…

Then I'm confused :-/

I see the hrtimers, but the actual idle duration appears to be in ticks rather than ms and then converted later.

Quoted 7 lines

+/* Duration of idle time in ticks of each injection period */ +unsigned int sysctl_sched_cfs_idle_inject_duration = 5UL;

...and futher down we have:

+ duration_msec =
jiffies_to_msecs(sysctl_sched_cfs_idle_inject_duration);

I used hrtimers but still round around tick boundaries of ktime. It is more reliable than jiffies in terms of getting timely updates.

Jacob

#42 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 11 lines · Jacob Pan
Quoted 8 lines · On Wed, 18 Nov 2015 14:04:41 +0000 Morten Rasmussen <[email protected]>…

Then I'm confused :-/

I see the hrtimers, but the actual idle duration appears to be in ticks rather than ms and then converted later.

+/* Duration of idle time in ticks of each injection period */ +unsigned int sysctl_sched_cfs_idle_inject_duration = 5UL;

Quoted 6 lines
Quoted 4 lines

...and futher down we have:

+ duration_msec =
jiffies_to_msecs(sysctl_sched_cfs_idle_inject_duration);

I used hrtimers but still round around tick boundaries of ktime. It is more reliable than jiffies in terms of getting timely updates.

Okay. But it does mean that the defeault idle period is 5 ticks (50ms @ HZ=100) and not 5 ms.

author
#43 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 2 lines · On Wed, 18 Nov 2015 15:09:44 +0000 Morten Rasmussen <[email protected]>…

Okay. But it does mean that the defeault idle period is 5 ticks (50ms @ HZ=100) and not 5 ms

correct. my reason is to scale with various HZ values.

#44 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 6 lines · Jacob Pan
Quoted 2 lines · On Wed, 18 Nov 2015 15:09:44 +0000 Morten Rasmussen <[email protected]>…

Okay. But it does mean that the defeault idle period is 5 ticks (50ms @ HZ=100) and not 5 ms

correct. my reason is to scale with various HZ values.

So for smaller HZ values we get longer disruption. That's well thought out scaling.

Thanks,

tglx

author
#45 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 7 lines · On Wed, 18 Nov 2015 16:21:27 +0100 (CET) Thomas Gleixner <[email protected]>…
Quoted 4 lines
Quoted 3 lines

Okay. But it does mean that the defeault idle period is 5 ticks (50ms @ HZ=100) and not 5 ms

correct. my reason is to scale with various HZ values.

So for smaller HZ values we get longer disruption. That's well thought out scaling.

well it might be too long for embedded systems who uses 100HZ. Is there a better way to scale in sub tick level?

My original thought was for smaller HZ value, I assume they care less about latency, so the idle injection period is proportional to what they set for HZ.

#46 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

In reply to Jacob Pan
Quoted 6 lines · Jacob Pan
Quoted 2 lines · On Wed, 18 Nov 2015 15:09:44 +0000 Morten Rasmussen <[email protected]>…

Okay. But it does mean that the defeault idle period is 5 ticks (50ms @ HZ=100) and not 5 ms

correct. my reason is to scale with various HZ values.

50ms seems quite long for embedded applications. OTOH, it has to be long enough to enter a package idle state and stay there for a while. You would surely notice it when it kicks in on an Android device.

Do you intend to use it only as a last option? One might want to keep using inefficient frequencies (P-states) if those don't cause UI lag then.

maintainer
#47 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

In reply to Peter Zijlstra
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Ingo Molnar

What will such throttling do to latencies, as observed by user-space tasks? What's the typical expected frequency of the throttling frequency that you are targeting?

The default has 5ms (iirc) of forced idle, so depending on what you do, noticeable to outright painful.

That's absolutely painful!

Thanks,

Ingo

author
#48 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

In reply to Ingo Molnar
Quoted 5 lines · On Wed, 18 Nov 2015 09:36:22 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 5 lines

This patch introduces a scheduler based idle injection method, it works by blocking CFS runqueue synchronously and periodically. The actions on all online CPUs are orchestrated by per CPU hrtimers.

Quoted 7 lines
Quoted 3 lines

Two sysctl knobs are given to the userspace for selecting the percentage of idle time as well as the forced idle duration for each idle period injected.

What's the purpose of these knobs? Just testing, or will some user-space daemon set them dynamically?

yes, it is to be used by userspace daemon such as thermal daemon. Though there are interests from in kernel thermal governor but that is another story.

Quoted 2 lines

I also think that naming it 'idle injection' is pretty euphemistic:
this is forced idling, right? So why not name it CFS_FORCED_IDLE?

yes, it is forced idle. sounds good.

maintainer
#49 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 12 lines · Jacob Pan
Quoted 9 lines · On Wed, 18 Nov 2015 09:36:22 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 8 lines

This patch introduces a scheduler based idle injection method, it works by blocking CFS runqueue synchronously and periodically. The actions on all online CPUs are orchestrated by per CPU hrtimers.

Two sysctl knobs are given to the userspace for selecting the percentage of idle time as well as the forced idle duration for each idle period injected.

Quoted 6 lines
Quoted 3 lines

What's the purpose of these knobs? Just testing, or will some user-space daemon set them dynamically?

yes, it is to be used by userspace daemon such as thermal daemon. Though there are interests from in kernel thermal governor but that is another story.

Yeah, so let me make this very clear: for a kernel scheduling feature to be self-sufficient is not 'another story', but a must-have aspect for this feature to become upstream acceptable.

We don't add scheduler features that rely on pushing 'policy' to user-space. That's poor design with many disadvantages. This feature should offer a reasonable and automatic in-kernel default behavior with numbers that prove that it works.

Keeping an essential part of the feature in user-space earns a NAK from me.

Thanks,

Ingo

author
#50 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 11 lines · On Fri, 27 Nov 2015 10:17:02 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

Two sysctl knobs are given to the userspace for selecting the percentage of idle time as well as the forced idle duration for each idle period injected.

What's the purpose of these knobs? Just testing, or will some user-space daemon set them dynamically?

yes, it is to be used by userspace daemon such as thermal daemon. Though there are interests from in kernel thermal governor but that is another story.

Quoted 9 lines

Yeah, so let me make this very clear: for a kernel scheduling feature to be self-sufficient is not 'another story', but a must-have aspect for this feature to become upstream acceptable.

We don't add scheduler features that rely on pushing 'policy' to user-space. That's poor design with many disadvantages. This feature should offer a reasonable and automatic in-kernel default behavior with numbers that prove that it works.

Sorry about the late response, have been sick all this time.

So my intention here are two folds and two steps.

1. for system under thermal/power limit but still want to operate at optimal energy efficiency point

2. synchronize idle time for better energy efficiency (runtime identify suitable workload)

This patchset is intended for #1. Lay ground work for #2.

The knobs in #1 can be used by in kernel thermal-aware scheduling as some early discussion pointed out. Thus not pushing policy to userspace.

For #2, my plan is to have a PM QoS like knob with sensible default to specify scheduler latency tolerance w.r.t. idle synchronization.

Similar to timer slack (where default 5ms slack is used), we can give a default tolerance in the forms of two parameters:
- forced_idle_duration (default 5ms)
- forced_to_natural_idle_ratio (default 50%)

e.g. when playing a online game with 40% natural idle time, idle injection code will try to 20% synchronized idle time.

Quoted 2 lines

Keeping an essential part of the feature in user-space earns a NAK from me.

I agree this should be self contained. My intension is to have the essential part used in kernel by thermal aware scheduling or PM QoS with clear intuitive mode of operation.

#51 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

What will such throttling do to latencies, as observed by user-space tasks? What's the typical expected frequency of the throttling frequency that you are targeting?

for this to meaningfully reduce power consumption, deep system power states need to be reached, and for those to pay of, several milliseconds of idle are required.

for hard realtime stuff that is obviously insane, but I would assume that for those cases your system is thermally fine. This only kicks in at the end of a "we're in thermal problems" path, which can happen both on clients (thin devices) as well as servers (airconditioning issues). The objective for this is to kick in before the
hardware built-in protections kick in (which are power off/reboot depending on a bunch of things). The frequency of how often these 5 msec get injected depend on how deep the system is in trouble; and is zero if the system is not in trouble.

The idea is that for the user it is better to inject several 5 msec intervals than it is to inject one longer period.

You can compare this method to other ways of reducing thermal issues (like lowering cpu frequency), and in a typical setup, this is done after the benign of those methods are exhausted. Lowering frequency even lower is usually of a low efficiency (you need to lower the frequency a LOT to gain a little bit of power in the bottom parts of the frequency ranges), while this idle will not only put the CPU in low power, but will also put the system memory in low power and usually a big chunk of the rest of the SOC. In many client systems, memory power consumption is higher than CPU power consumption (and in big servers, it's also quite sizable), so there is a pretty hard limit of how much you can do on thermals if you're not also kicking some of the memory power savings. This means that to achieve a certain amount of reduction, the performance is impacted a lot less than the more drastic methods you would need on the cpu side, if possible at all.
(stepping your 2 Ghz cpu down to 50Mhz may sound less evil than injecting 5msec of idle time, but in reality that is impacting user tasks a heck of a lot more than 5msec of not being scheduled)

#52 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

In reply to Jacob Pan
Quoted 12 lines · Jacob Pan

With increasingly constrained power and thermal budget, it's often necessary to cap power via throttling. Throttling individual CPUs or devices at random times can help power capping but may not be optimal in terms of energy efficiency. Frequency scaling is also limited by certain range before losing energy efficiency.

In general, the optimal solution in terms of energy efficiency is to align idle periods such that more shared circuits can be power gated to enter lower power states. Combined with energy efficient frequency point, idle injection provides a way to scale power and performance efficiently.

Quoted 3 lines

This patch introduces a scheduler based idle injection method, it works by blocking CFS runqueue synchronously and periodically. The actions on all online CPUs are orchestrated by per CPU hrtimers.

I fully agree with the idea of synchronous duty cycling of cpus for thermal management to avoid those inefficient low frequencies where we loose a lot of energy to static power consumption and get very little work done. However, I would not necessarily want to punish all cpus system-wide if we have local overheating in one corner. If would rather have it apply to only the overheating socket in a multi-socket machine and only the big cores in a big.LITTLE system.

Several people have already brought up that we should look into integrating this with the thermal framework. It already has the concepts of thermal zones and cooling devices which could be used to control idle-injection separately for different sockets/clusters.

Also, letting user-space control idle-injection opens up for having two entities (user-space daemon and the in-kernel thermal governor) stepping on each others toes. Wouldn't the control of idle-injection naturally belong to thermal governors so we have everything controlled from a single place?

#53 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 4 lines · Morten Rasmussen

I would not necessarily want to punish all cpus
system-wide if we have local overheating in one corner. If would rather have it apply to only the overheating socket in a multi-socket machine and only the big cores in a big.LITTLE system.

most of the time thermal issues aren't inside the SOC, but on a system level due to cheap heat spreaders or outright lack of space due to thinness. But even if you have one part of the die too hot:

For core level idle injection, no need to synchronize that; the reason to synchronize is generally that when ALL cores are idle, additional power savings kick in (like memory going to self refresh, fabrics power gating etc); those additional power savings are what makes this more efficient than just voltage/frequency
scaling at the bottom of that range... not so much the fact that things are just idle.

#54 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 10 lines · Arjan van de Ven
Quoted 4 lines · Morten Rasmussen

I would not necessarily want to punish all cpus
system-wide if we have local overheating in one corner. If would rather have it apply to only the overheating socket in a multi-socket machine and only the big cores in a big.LITTLE system.

most of the time thermal issues aren't inside the SOC, but on a system level due to cheap heat spreaders or outright lack of space due to thinness. But even if you have one part of the die too hot:

Quoted 5 lines

For core level idle injection, no need to synchronize that; the reason to synchronize is generally that when ALL cores are idle, additional power savings kick in (like memory going to self refresh, fabrics power gating etc); those additional power savings are what makes this more efficient than just voltage/frequency
scaling at the bottom of that range... not so much the fact that things are just idle.

I could see this technique being useful within the SoC as well. Synchronized idle injection on all cpus in a cluster would allow us to reach cluster idle states where resources shared within the cluster can be gated or powered off as well. But yes, shutting down everything is more efficient if you are en serious trouble.

I was hoping this could be a good alternative to hotplugging cpus out for thermal management in non-critical situations, but it isn't that appealing if you have to throttle all the cpus. I would consider it an emergency-only mechanism (as in emergency brake) that isn't really suitable for normal thermal management. In which case: Does this sort of mechanism belong in the scheduler code?

maintainer
#55 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 4 lines · Morten Rasmussen

I would consider it an
emergency-only mechanism (as in emergency brake) that isn't really suitable for normal thermal management. In which case: Does this sort of mechanism belong in the scheduler code?

I would prefer it not to be, but Thomas is very much opposed to teaching the nohz code to 'work' on !idle threads.

#56 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Morten Rasmussen

I would consider it an
emergency-only mechanism (as in emergency brake) that isn't really suitable for normal thermal management. In which case: Does this sort of mechanism belong in the scheduler code?

I would prefer it not to be, but Thomas is very much opposed to teaching the nohz code to 'work' on !idle threads.

The whole concept of faking idle is simply crap.

If you want to avoid that stuff in the scheduler, then create a mechanism which just defers the next timer interrupt for X milliseconds and does not any fiddling with NOHZ state and such.

That might hurt RT tasks, but if someone really cares about real-time and deterministic behaviour, then running the machine on its thermal limits is simply stupid. In fact any sensible RT system will bring itself into a safe state way before the machine runs into that condition.

Thanks,

tglx

maintainer
#57 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 1 line · Thomas Gleixner

The whole concept of faking idle is simply crap.

:-)

Quoted 3 lines

If you want to avoid that stuff in the scheduler, then create a mechanism which just defers the next timer interrupt for X milliseconds and does not any fiddling with NOHZ state and such.

Not sure that will actually result in the machine going idle. Device interrupts will still wake tasks and get them ran.

#58 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Thomas Gleixner

The whole concept of faking idle is simply crap.

Quoted 8 lines

:-)

Quoted 3 lines

If you want to avoid that stuff in the scheduler, then create a mechanism which just defers the next timer interrupt for X milliseconds and does not any fiddling with NOHZ state and such.

Not sure that will actually result in the machine going idle. Device interrupts will still wake tasks and get them ran.

That's not what I meant. If you don't want to control all that from the scheduler than you are back to that thread which "runs" at RT priority and does

	 if (machine_on_fire) {
	        defer_timer_interrupt(5ms);
		end = now + 5ms:
	    	while (now < end)
		      mwait();
         }

That's what the existing code does, but the above does not longer claim it's idle and confuses the hell out of nohz and whatever. It's just a "runaway" RT task which "hogs" the CPU for 5ms and makes the next timer interrupt firing late.

Thanks,

tglx

maintainer
#59 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 11 lines · Thomas Gleixner

That's not what I meant. If you don't want to control all that from the scheduler than you are back to that thread which "runs" at RT priority and does

	 if (machine_on_fire) {
	        defer_timer_interrupt(5ms);
		end = now + 5ms:
	    	while (now < end)
		      mwait();
         }
Quoted 4 lines

That's what the existing code does, but the above does not longer claim it's idle and confuses the hell out of nohz and whatever. It's just a "runaway" RT task which "hogs" the CPU for 5ms and makes the next timer interrupt firing late.

Right; so the naive way of implementing that is by simply programing the timer hardware 5ms in the future and leaving it at that.

The problem with that would be a device interrupt happening and mucking with timers, this would result in the timer hardware being reprogrammed to a 'sane' value. I see two solutions for that:

- add another check in tick_program_event(); or,

- muck about with the evtdev pointer, such that we (temporarily) neuter its clock_event_device::set_next_*() methods.

The later is fugly, but avoids any runtime overhead.

This all makes the idle-injection muck hard depend on hres_active, but I think that's a sane constraint anyway.

#60 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 24 lines · Peter Zijlstra
Quoted 11 lines · Thomas Gleixner

That's not what I meant. If you don't want to control all that from the scheduler than you are back to that thread which "runs" at RT priority and does

	 if (machine_on_fire) {
	        defer_timer_interrupt(5ms);
		end = now + 5ms:
	    	while (now < end)
		      mwait();
         }
Quoted 4 lines

That's what the existing code does, but the above does not longer claim it's idle and confuses the hell out of nohz and whatever. It's just a "runaway" RT task which "hogs" the CPU for 5ms and makes the next timer interrupt firing late.

Right; so the naive way of implementing that is by simply programing the timer hardware 5ms in the future and leaving it at that.

The problem with that would be a device interrupt happening and mucking with timers, this would result in the timer hardware being reprogrammed to a 'sane' value. I see two solutions for that:

Quoted 6 lines

- add another check in tick_program_event(); or,

- muck about with the evtdev pointer, such that we (temporarily) neuter its clock_event_device::set_next_*() methods.

The later is fugly, but avoids any runtime overhead.

Yes, it's fugly, but it does not touch any of the states.

Quoted 2 lines

This all makes the idle-injection muck hard depend on hres_active, but I think that's a sane constraint anyway.

It makes it actually depend on NO_HZ || HRES. It comes with a few other restrictions as well: I'm not going to support that for TSCs which stop in C3. It's ugly enough already and I really don't want to muck with the broadcast device.

One other thing is that the caller has to ensure that the unblock() happens in a timely manner. I really would recommend to just disable preemption around that machine on fire wait loop and screw the RT tasks. We screw them anyway when they depend on a timer.

Untested patch below.

Thanks,

tglx

8<----------------
Subject: tick/nohz: Allow blocking the tick to prevent fire
From: Thomas Gleixner <[email protected]>
Date: Fri, 20 Nov 2015 14:28:17 +0100

---
 include/linux/tick.h     |    5 ++
 kernel/time/Kconfig      |    5 ++
 kernel/time/tick-sched.c |   99 +++++++++++++++++++++++++++++++++++++++++++++++
 kernel/time/tick-sched.h |    7 ++-
 4 files changed, 114 insertions(+), 2 deletions(-)
include/linux/tick.h +5 -0
Index: tip/include/linux/tick.h
===================================================================
@@ -203,4 +203,9 @@ static inline void tick_nohz_task_switch
 		__tick_nohz_task_switch();
 }
 
+#ifdef CONFIG_TICK_THROTTLING
+int tick_nohz_block_tick(u64 delay);
+void tick_nohz_unblock_tick(void);
+#endif
+
 #endif
kernel/time/Kconfig +5 -0
Index: tip/kernel/time/Kconfig
===================================================================
@@ -60,6 +60,11 @@ menu "Timers subsystem"
 config TICK_ONESHOT
 	bool
 
+# Special functions for tick throttling to avoid fire extinguishers
+config TICK_THROTTLING
+       bool
+       depends on TICK_ONESHOT
+
 config NO_HZ_COMMON
 	bool
 	depends on !ARCH_USES_GETTIMEOFFSET && GENERIC_CLOCKEVENTS
kernel/time/tick-sched.c +99 -0

105 lines not on this page · raw message

kernel/time/tick-sched.h +5 -2
Index: tip/kernel/time/tick-sched.h
===================================================================
@@ -9,8 +9,11 @@ enum tick_device_mode {
 };
 
 struct tick_device {
-	struct clock_event_device *evtdev;
-	enum tick_device_mode mode;
+	struct clock_event_device	*evtdev;
+	enum tick_device_mode		mode;
+#ifdef CONFIG_TICK_THROTTLING
+	struct clock_event_device	*real_evtdev;
+#endif
 };
 
 enum tick_nohz_mode {
author
#61 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 8 lines · On Fri, 20 Nov 2015 19:53:45 +0100 (CET) Thomas Gleixner <[email protected]>…
Quoted 2 lines

This all makes the idle-injection muck hard depend on hres_active, but I think that's a sane constraint anyway.

It makes it actually depend on NO_HZ || HRES. It comes with a few other restrictions as well: I'm not going to support that for TSCs which stop in C3. It's ugly enough already and I really don't want to muck with the broadcast device.

Quoted 7 lines

One other thing is that the caller has to ensure that the unblock() happens in a timely manner. I really would recommend to just disable preemption around that machine on fire wait loop and screw the RT tasks. We screw them anyway when they depend on a timer.

Untested patch below.

Sorry it took a while, but I tested the patch below with some small tweaks. Here are the testing results and issues.

1. Block tick works well to improve idle efficiency, e.g. on an Ivy Bridge 4 core 8 thread desktop system. Performance per watt is better than without blocking the ticks. It is also more consistent and predictable. For CPU bound floating point spin workload, compare idle percentage from 0-50%. The test below is limited to package C7. The improvement is likely better on newer CPUs or system tuned with deeper package C-states.

W/O patch
==========
IdlePct Perf RAPL WallPower PPW_RAPL
0 467.16 29.58 0.0 15.79
5 441.68 27.94 0.0 15.81
10 406.71 26.50 0.0 15.35
15 372.16 24.94 0.0 14.92
20 340.89 23.41 0.0 14.56
25 307.87 21.97 0.0 14.01
30 310.69 21.39 0.0 14.53
35 293.29 21.00 0.0 13.97
40 268.78 19.90 0.0 13.51
45 240.66 18.73 0.0 12.85
50 238.56 18.74 0.0 12.73

W/ patch
IdlePct Perf RAPL WallPower PPW_RAPL
0 467.19 29.71 0.0 15.73
5 445.07 28.53 0.0 15.6
10 420.34 27.43 0.0 15.32
15 398.47 26.25 0.0 15.18
20 376.47 25.05 0.0 15.03
25 354.13 23.81 0.0 14.87
30 329.50 22.57 0.0 14.6
35 304.38 21.27 0.0 14.31
40 290.42 20.21 0.0 14.37
45 254.82 16.70 0.0 15.26
50 236.10 17.75 0.0 13.3

Note: PPW_RAPL= performance per watt measured by RAPL package power meter.

If we just look at 20% forced idle, I can see significantly more deep package C-states(PC7) residency.

		Pkg%pc2 	Pkg%pc6 	Pkg%pc7 

w/o patch 3.17 4.39 13.21
w/ patch 1.15 1.43 17.24

2. run into warning

[  109.035699] WARNING: CPU: 0 PID: 2748 at
kernel/time/clockevents.c:325 clockevents_program_event+0x102/0x110() 
[  109.035700] Current state: 0

I think it is caused by tick_throttler in CLOCK_EVT_STATE_DETACHED state. perhaps fix it with below?

kernel/time/tick-sched.c +0 -0
@@ -1141,6 +1141,7 @@ static struct clock_event_device tick_throttler =

{
.features = CLOCK_EVT_FEAT_ONESHOT,
.event_handler = tick_throttling_handler, .set_next_event = tick_throttling_noop,
+ .state_use_accessors = CLOCK_EVT_STATE_ONESHOT,

        .mult           = 1,
        .shift          = 0,
        .irq            = -1,

3. typo? since we use ns as delay unit
@@ -1175,7 +1176,7 @@ int tick_nohz_block_tick(u64 delay)

        if (WARN_ON_ONCE(td->evtdev == &tick_throttler))
                goto out;

- if (delay > td->evtdev->max_delta_ticks) {
+ if (delay > td->evtdev->max_delta_ns)

4. I see random message about "sched: RT throttling activated", not sure accounting is being affected. I never got close to 95% RT.

5. Random soft lockup, stall, detected by NMI watchdog, or RCU preempt, still debugging but each idle duration is only 5 ticks. I also disabled preemption as suggested.

Thanks,

Jacob

Quoted 167 lines

Thanks,

tglx

8<----------------
Subject: tick/nohz: Allow blocking the tick to prevent fire
From: Thomas Gleixner <[email protected]>
Date: Fri, 20 Nov 2015 14:28:17 +0100

---
 include/linux/tick.h     |    5 ++
 kernel/time/Kconfig      |    5 ++
 kernel/time/tick-sched.c |   99

+++++++++++++++++++++++++++++++++++++++++++++++
kernel/time/tick-sched.h | 7 ++- 4 files changed, 114
insertions(+), 2 deletions(-)

Index: tip/include/linux/tick.h
===================================================================

include/linux/tick.h +5 -0
@@ -203,4 +203,9 @@ static inline void tick_nohz_task_switch
 		__tick_nohz_task_switch();
 }
 
+#ifdef CONFIG_TICK_THROTTLING
+int tick_nohz_block_tick(u64 delay);
+void tick_nohz_unblock_tick(void);
+#endif
+
 #endif

Index: tip/kernel/time/Kconfig
===================================================================

kernel/time/Kconfig +5 -0
@@ -60,6 +60,11 @@ menu "Timers subsystem"
 config TICK_ONESHOT
 	bool
 
+# Special functions for tick throttling to avoid fire extinguishers
+config TICK_THROTTLING
+       bool
+       depends on TICK_ONESHOT
+
 config NO_HZ_COMMON
 	bool
 	depends on !ARCH_USES_GETTIMEOFFSET && GENERIC_CLOCKEVENTS

Index: tip/kernel/time/tick-sched.c
===================================================================

kernel/time/tick-sched.c +4 -0
@@ -1119,6 +1119,105 @@ void tick_setup_sched_timer(void)
 }
 #endif /* HIGH_RES_TIMERS */
 
+#ifdef CONFIG_TICK_THROTTLING
+/*
+ * An interrupt might have been pending already, when we programmed
+ * the throttler time. Nothing to do here. The device is armed and

the
+ * interrupt will fire again. If this is the real wakeup event then
+ * the unblock call will retrigger it.
+ */

kernel/time/tick-sched.c +92 -0
+static void tick_throttling_handler(struct clock_event_device *dev)
+{
+}
+
+static int tick_throttling_noop(unsigned long evt,
+				struct clock_event_device *d)
+{
+	return 0;
+}
+
+static struct clock_event_device tick_throttler = {
+	.name		= "throttler",
+	.features	= CLOCK_EVT_FEAT_ONESHOT,
+	.event_handler	= tick_throttling_handler,
+	.set_next_event	= tick_throttling_noop,
+	.mult		= 1,
+	.shift		= 0,
+	.irq		= -1,
+};
+
+/**
+ * tick_nohz_block_tick - Force block the tick to prevent fire
+ * @delay:	Defer the tick for X nano seconds
+ *
+ * This is a special interface for thermal emergencies. Do not use
+ * otherwise!  Make sure to call tick_nohz_block_tick() right after
+ * the delay ends to undo the damage.
+ */
+int tick_nohz_block_tick(u64 delay)
+{
+	struct tick_device *td;
+	unsigned long flags;
+	int ret = -EBUSY;
+	ktime_t until;
+
+	if (!tick_nohz_active)
+		return -EBUSY;
+
+	local_irq_save(flags);
+	td = this_cpu_ptr(&tick_cpu_device);
+
+	/* No way to do that with broadcast */
+	if (td->evtdev->features & CLOCK_EVT_FEAT_C3STOP)
+		goto out;
+
+	/* Yes, I do not trust people! */
+	if (WARN_ON_ONCE(td->evtdev == &tick_throttler))
+		goto out;
+
+	if (delay > td->evtdev->max_delta_ticks) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	until = ktime_add_ns(ktime_get(), delay);
+	if (!tick_program_event(until, 0)) {
+		td->real_evtdev = td->evtdev;
+		td->evtdev = &tick_throttler;
+		ret = 0;
+	}
+out:
+	local_irq_restore(flags);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tick_nohz_block_tick);
+
+/**
+ * tick_nohz_force_unblock_tick - Undo the force blocking of the tick
+ *
+ * Pairs with tick_nohz_block_tick(). Can be called unconditionally
+ * even if the tick was not blocked by tick_nohz_block_tick().
+ */
+void tick_nohz_unblock_tick(void)
+{
+	struct tick_device *td;
+	unsigned long flags;
+
+	if (!tick_nohz_active)
+		return;
+
+	local_irq_save(flags);
+	td = this_cpu_ptr(&tick_cpu_device);
+	if (td->evtdev == &tick_throttler) {
+		td->evtdev = td->real_evtdev;
+		/* Force a timer interrupt now */
+		tick_program_event(ktime_get(), 1);
+	}
+	local_irq_restore(flags);
+}
+EXPORT_SYMBOL_GPL(tick_nohz_unblock_tick);
+#endif /* TICK_THROTTLING */
+
 #if defined CONFIG_NO_HZ_COMMON || defined CONFIG_HIGH_RES_TIMERS
 void tick_cancel_sched_timer(int cpu)
 {

Index: tip/kernel/time/tick-sched.h
===================================================================

kernel/time/tick-sched.h +0 -0
@@ -9,8 +9,11 @@ enum tick_device_mode {
 };
 
 struct tick_device {

kernel/time/tick-sched.h enum tick_device_mode { · line 19

Quoted 10 lines
kernel/time/tick-sched.h +5 -2
-	struct clock_event_device *evtdev;
-	enum tick_device_mode mode;
+	struct clock_event_device	*evtdev;
+	enum tick_device_mode		mode;
+#ifdef CONFIG_TICK_THROTTLING
+	struct clock_event_device	*real_evtdev;
+#endif
 };
 
 enum tick_nohz_mode {

[Jacob Pan]

maintainer
#62 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 8 lines · Jacob Pan

Note: PPW_RAPL= performance per watt measured by RAPL package power meter.

If we just look at 20% forced idle, I can see significantly more deep package C-states(PC7) residency.

		Pkg%pc2 	Pkg%pc6 	Pkg%pc7 

w/o patch 3.17 4.39 13.21
w/ patch 1.15 1.43 17.24

Can this be translated to: "this patch improved power efficiency (watts per second) by XYZ percent?", with XYZ filled in accordingly?

Thanks,

Ingo

author
#63 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 8 lines · On Tue, 19 Jan 2016 13:06:05 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 5 lines

If we just look at 20% forced idle, I can see significantly more deep package C-states(PC7) residency.

		Pkg%pc2 	Pkg%pc6 	Pkg%pc7 

w/o patch 3.17 4.39 13.21
w/ patch 1.15 1.43 17.24

Can this be translated to: "this patch improved power efficiency (watts per second) by XYZ percent?", with XYZ filled in accordingly?

With CPU bound workload and 20% idle injection ratio, this patch improves performance per watt by 3.2% on Ivy Bridge system with Package C7 as the deepest idle.

The above packages C-state residency explains why, i.e. more deeper PC7 when tick is stopped.

With newer CPUs, this improvements should be more visible in that low power package C-states are getting deeper/less power. I will test on Skylake client as soon as I have time.

Thanks,

Jacob

author
#64 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

In reply to Thomas Gleixner
Quoted 1 line · On Fri, 20 Nov 2015 11:58:12 +0100 (CET) Thomas Gleixner <[email protected]>…
	 if (machine_on_fire) {

Just to add data to show this is not just for emergency. I have an broadwell laptop, here is the test I did.
- clamp cpu freq to 900MHz (close to max efficiency)
- run gpu and cpu load (cut-the-rop, glxgears)
- turn fans off

run tmon (tools/thermal/tmon) to show temperatures and thermal trip points. (sorry about the long lines, quite a few thermal sensors) Also run turbostat to show CPU idle etc.

Observations:
without idle injection:
------------------------------------------------------------------
- package temp rise to 66C crossing the second active trip points (2nd fan speed should kick in if not disabled)
- package power 9.47w
- cpu c0 ~90%, 0% core c7
------------------------------------------------------------------
- with idle injection 20%
- cpu c0 ~67%, ~13% core c7, ~7% package c-states
- package power 4.53w
- package temp 52C

glxgears maintains the same frame rate. didnt measure cut-the-rope but still plays.

This is a large 14 inch laptop with much better cooling than a fanless tablet/convertible. So hitting thermal constraints can happen in _NORMAL_ condition (even at 900MHz), not just on fire case.

Sync up idle can save energy efficiency and i think scheduler is a better place to solve that problem, perhaps with PM QoS constraint will have less perceived intrusive behavior.

Thanks,

Jacob

│Trip Points:                   AAAAC       C                       AAPHCPP     APHCPP      PHCPP       AAPHCPP     AAAAPHCPP   APHCPP      AAAAPHC     PP                                                                                                    │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────── COOLING DEVICES ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ID  Cooling Dev   Cur    Max   Thermal Zone Binding                                                                                                                                                                                                          │
│00         TFN1     4     11   │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ ││││││││││││                                                                                          │
│01          Fan     0      1   │││*│││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ ││││││││││││                                                                                          │
│02          Fan     0      1   ││*││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ ││││││││││││                                                                                          │
│03          Fan     0      1   │*│││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ ││││││││││││                                                                                          │
│04          Fan     0      1   *││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ ││││││││││││                                                                                          │
│05         CHRG     4      4   │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ ││││││││││││                                                                                          │
│06 intel_powerc    -1     50   │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ │││││││││││ ││││││││││││                                                                                          │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                         10        20        30        40        50        60        70        80        90        100       110       120                                                                                                                   │
│acpitz 0:[ 66][→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→A55→→→→→→→A65       A75       A85                       C111          ]                                                                                                                │
│pch_wi 1:[ 59][→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→                                                       C115      ]                                                                                                                │
│INT340 2:[ 20][→→→→→→→→→→→→→→→→→→→→→                                                                                                        ]                                                                                                                │
│  SEN1 3:[ 47][P0→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→          A58  A63     P71                          H100103                  ]                                                                                                                │
│  SEN2 4:[ 51][P0→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→           A63     P71                          H100103                  ]                                                                                                                │
│  SEN3 5:[ 46][P0→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→                        P71                          H100103                  ]                                                                                                                │
│  SEN4 6:[ 45][P0→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→            A58  A63     P71                          H100103                  ]                                                                                                                │
│  SEN6 7:[ 46][P0→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→ A48  A53  A58  A63     P71                          H100103                  ]                                                                                                                │
│  SEN7 8:[ 27][P0→→→→→→→→→→→→→→→→→→→→→→→→→→                                   A63     P71                          H100103                  ]                                                                                                                │
│  B0D4 9:[ 66][→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→A55→→→→→→→A65       A75       A85              P102045                ]                                                                                                                │
│x86_pk10:[ 66][P0→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→→                                                          ]                                                                                                                │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
turbostat without idle injection
      1       3     776   86.37     898    2295       0   11.90
    Core     CPU Avg_MHz   %Busy Bzy_MHz TSC_MHz     SMI  CPU%c1  CPU%c3  CPU%c6  CPU%c7 CoreTmp  PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 Pkg%pc8 Pkg%pc9 Pk%pc10 PkgWatt CorWatt GFXWatt
       -       -     801   89.16     898    2295       0    9.77    0.55    0.07    0.44      66      70    0.00    0.00    0.00    0.00    0.00    0.00    0.00    9.47    4.31    1.17
       0       0     801   89.26     898    2294       0    9.76    0.57    0.07    0.34      64      70    0.00    0.00    0.00    0.00    0.00    0.00    0.00    9.47    4.31    1.17
       0       2     803   89.43     898    2294       0    9.58
       1       1     810   90.27     898    2295       0    8.59    0.52    0.07    0.55      66
       1       3     787   87.69     898    2295       0   11.17

turbostat with 20% idle injection

    Core     CPU Avg_MHz   %Busy Bzy_MHz TSC_MHz     SMI  CPU%c1  CPU%c3  CPU%c6  CPU%c7 CoreTmp  PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 Pkg%pc8 Pkg%pc9 Pk%pc10 PkgWatt CorWatt GFXWatt
       -       -     601   66.92     898    2295       0   13.48    4.65    1.09   13.86      52      52    3.50    0.60    0.46    2.30    0.00    0.00    0.00    4.53    1.99    0.17
       0       0     605   67.36     898    2294       0   12.75    5.95    1.51   12.42      51      52    3.50    0.60    0.46    2.30    0.00    0.00    0.00    4.53    1.99    0.17
       0       2     583   64.94     898    2295       0   15.18
       1       1     605   67.37     898    2295       0   13.31    3.34    0.67   15.30      52
       1       3     611   68.02     898    2295       0   12.66

~
~
~

Quoted 5 lines
	        defer_timer_interrupt(5ms);
		end = now + 5ms:
	    	while (now < end)
		      mwait();
         }

[Jacob Pan]

author
#65 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

In reply to Morten Rasmussen
Quoted 6 lines · On Thu, 19 Nov 2015 17:24:07 +0000 Morten Rasmussen <[email protected]>…

I could see this technique being useful within the SoC as well. Synchronized idle injection on all cpus in a cluster would allow us to reach cluster idle states where resources shared within the cluster can be gated or powered off as well. But yes, shutting down everything is more efficient if you are en serious trouble.

Quoted 6 lines

I was hoping this could be a good alternative to hotplugging cpus out for thermal management in non-critical situations, but it isn't that appealing if you have to throttle all the cpus. I would consider it an emergency-only mechanism (as in emergency brake) that isn't really suitable for normal thermal management. In which case: Does this sort of mechanism belong in the scheduler code?

I would think this is not an emergency-only mechanism. I know shutting down all CPUs at the same time sounds bad but here are advantage of having this in the scheduler in that we can still have certain QoS. e.g. we yeild softirq, RT. etc. There are other workload info readily available in the scheduler to better use this knob. e.g. sync with other input timers such as media.

For normal thermal conditions, Skylake for example, the max efficient frequency is ~900MHz. If thermal restriction happens (e.g. you want to do fanless), would you like to lose energy efficiency by further lowering frequency to 500Mhz? or do idle injection and take 5ms latency for normal tasks? Idle injection gives you near linear performance-power scaling.

Thanks,

Jacob

#66 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

In reply to Jacob Pan
Quoted 28 lines · Jacob Pan

With increasingly constrained power and thermal budget, it's often necessary to cap power via throttling. Throttling individual CPUs or devices at random times can help power capping but may not be optimal in terms of energy efficiency. Frequency scaling is also limited by certain range before losing energy efficiency.

In general, the optimal solution in terms of energy efficiency is to align idle periods such that more shared circuits can be power gated to enter lower power states. Combined with energy efficient frequency point, idle injection provides a way to scale power and performance efficiently.

This patch introduces a scheduler based idle injection method, it works by blocking CFS runqueue synchronously and periodically. The actions on all online CPUs are orchestrated by per CPU hrtimers.

Two sysctl knobs are given to the userspace for selecting the percentage of idle time as well as the forced idle duration for each idle period injected.

Since only CFS class is targeted, other high priority tasks are not affected, such as EDF and RT tasks as well as softirq and interrupts.

Hotpath in CFS pick_next_task is optimized by Peter Zijlstra, where a new runnable flag is introduced to combine forced idle and nr_running.

Quoted 8 lines
---
 include/linux/sched.h        |  11 ++
 include/linux/sched/sysctl.h |   5 +
 init/Kconfig                 |  10 ++
 kernel/sched/fair.c          | 353 ++++++++++++++++++++++++++++++++++++++++++-
 kernel/sched/sched.h         |  54 ++++++-
 kernel/sysctl.c              |  21 +++
 6 files changed, 449 insertions(+), 5 deletions(-)

I've tested this series on Juno (2xCortex-A57 4xCortex-A53). If you idle inject for 50% of the time, when I run 6 busy loops the scheduler sometimes keeps two of them in the same cpu while the another cpu is completely idle. Without idle injection the scheduler does the sensible thing: put one busy loop in each CPU. I'm running systemd and this only happens with CONFIG_SCHED_AUTOGROUP=y. If I unset CONFIG_SCHED_AUTOGROUP, the tasks are spread across all cpus as usual.

See below part of the trace that shows this problem. CPU3 has two 100% tasks: 1554 and 1549 but the scheduler never moves one of the tasks to CPU4, which has an empty runqueue. Both cpus are in the same domain. Juri helped me add two additional trace points to track the load of a task and cpu. This tracepoints are added at the end of update_load_avg().

          <idle>-0     [002]   164.739796: sched_cfs_idle_inject_timer: throttled=0
          <idle>-0     [000]   164.739797: sched_cfs_idle_inject_timer: throttled=0
          <idle>-0     [005]   164.739797: sched_cfs_idle_inject_timer: throttled=0
          <idle>-0     [001]   164.739797: sched_cfs_idle_inject_timer: throttled=0
          <idle>-0     [003]   164.739797: sched_cfs_idle_inject_timer: throttled=0
          <idle>-0     [004]   164.739798: sched_cfs_idle_inject_timer: throttled=0
          <idle>-0     [002]   164.739802: sched_load_avg_cpu:   cpu=2 load_avg=171 util_avg=406
          <idle>-0     [002]   164.739803: sched_load_avg_task:  comm=busy_loop pid=1552 cpu=2 load_avg=1006 util_avg=400 load_sum=48043453 util_sum=19130537 period_contrib=173
          <idle>-0     [001]   164.739803: sched_load_avg_cpu:   cpu=1 load_avg=170 util_avg=405
          <idle>-0     [002]   164.739804: sched_load_avg_cpu:   cpu=2 load_avg=1014 util_avg=403
          <idle>-0     [001]   164.739804: sched_load_avg_task:  comm=busy_loop pid=1551 cpu=1 load_avg=1008 util_avg=401 load_sum=48161276 util_sum=19177731 period_contrib=288
          <idle>-0     [005]   164.739804: sched_load_avg_cpu:   cpu=5 load_avg=169 util_avg=404
          <idle>-0     [002]   164.739805: sched_switch:         swapper/2:0 [120] R ==> busy_loop:1552 [120]
          <idle>-0     [001]   164.739805: sched_load_avg_cpu:   cpu=1 load_avg=1024 util_avg=407
          <idle>-0     [003]   164.739805: sched_load_avg_cpu:   cpu=3 load_avg=340 util_avg=405
          <idle>-0     [000]   164.739805: sched_load_avg_cpu:   cpu=0 load_avg=168 util_avg=400
          <idle>-0     [001]   164.739806: sched_switch:         swapper/1:0 [120] R ==> busy_loop:1551 [120]
          <idle>-0     [005]   164.739806: sched_load_avg_task:  comm=busy_loop pid=1550 cpu=5 load_avg=1010 util_avg=402 load_sum=48229881 util_sum=19205027 period_contrib=355
          <idle>-0     [003]   164.739807: sched_load_avg_task:  comm=busy_loop pid=1549 cpu=3 load_avg=1012 util_avg=193 load_sum=48316673 util_sum=9247244 period_contrib=441
          <idle>-0     [000]   164.739807: sched_load_avg_task:  comm=busy_loop pid=1553 cpu=0 load_avg=1005 util_avg=400 load_sum=48003551 util_sum=19119112 period_contrib=134
          <idle>-0     [005]   164.739808: sched_load_avg_cpu:   cpu=5 load_avg=1002 util_avg=399
          <idle>-0     [003]   164.739808: sched_load_avg_cpu:   cpu=3 load_avg=2045 util_avg=407
          <idle>-0     [000]   164.739809: sched_load_avg_cpu:   cpu=0 load_avg=1008 util_avg=401
          <idle>-0     [005]   164.739810: sched_switch:         swapper/5:0 [120] R ==> busy_loop:1550 [120]
          <idle>-0     [003]   164.739810: sched_switch:         swapper/3:0 [120] R ==> busy_loop:1549 [120]
          <idle>-0     [000]   164.739811: sched_switch:         swapper/0:0 [120] R ==> busy_loop:1553 [120]
       busy_loop-1552  [002]   164.743793: sched_stat_runtime:   comm=busy_loop pid=1552 runtime=3991560 [ns] vruntime=605432548 [ns]
       busy_loop-1549  [003]   164.743794: sched_stat_runtime:   comm=busy_loop pid=1549 runtime=3990040 [ns] vruntime=382380848 [ns]
       busy_loop-1552  [002]   164.743794: sched_load_avg_task:  comm=busy_loop pid=1552 cpu=2 load_avg=1024 util_avg=456 load_sum=48889883 util_sum=21796057 period_contrib=999
       busy_loop-1553  [000]   164.743794: sched_stat_runtime:   comm=busy_loop pid=1553 runtime=3990180 [ns] vruntime=590391894 [ns]
       busy_loop-1551  [001]   164.743794: sched_stat_runtime:   comm=busy_loop pid=1551 runtime=3992100 [ns] vruntime=272056341 [ns]
       busy_loop-1550  [005]   164.743794: sched_stat_runtime:   comm=busy_loop pid=1550 runtime=3990920 [ns] vruntime=198320034 [ns]
       busy_loop-1552  [002]   164.743795: sched_load_avg_cpu:   cpu=2 load_avg=1010 util_avg=450
       busy_loop-1551  [001]   164.743796: sched_load_avg_task:  comm=busy_loop pid=1551 cpu=1 load_avg=1004 util_avg=447 load_sum=47958941 util_sum=21380913 period_contrib=90
       busy_loop-1549  [003]   164.743796: sched_load_avg_task:  comm=busy_loop pid=1549 cpu=3 load_avg=1007 util_avg=257 load_sum=48112396 util_sum=12285572 period_contrib=241
       busy_loop-1552  [002]   164.743796: sched_load_avg_cpu:   cpu=2 load_avg=170 util_avg=453
       busy_loop-1553  [000]   164.743796: sched_load_avg_task:  comm=busy_loop pid=1553 cpu=0 load_avg=1023 util_avg=456 load_sum=48847931 util_sum=21780791 period_contrib=958
       busy_loop-1551  [001]   164.743796: sched_load_avg_cpu:   cpu=1 load_avg=1020 util_avg=454
       busy_loop-1550  [005]   164.743797: sched_load_avg_task:  comm=busy_loop pid=1550 cpu=5 load_avg=1005 util_avg=448 load_sum=48026522 util_sum=21410614 period_contrib=156
       busy_loop-1549  [003]   164.743797: sched_load_avg_cpu:   cpu=3 load_avg=2036 util_avg=454
       busy_loop-1553  [000]   164.743798: sched_load_avg_cpu:   cpu=0 load_avg=1004 util_avg=447
       busy_loop-1551  [001]   164.743798: sched_load_avg_cpu:   cpu=1 load_avg=169 util_avg=452
       busy_loop-1550  [005]   164.743798: sched_load_avg_cpu:   cpu=5 load_avg=1020 util_avg=455
       busy_loop-1553  [000]   164.743800: sched_load_avg_cpu:   cpu=0 load_avg=171 util_avg=456
       busy_loop-1549  [003]   164.743800: sched_load_avg_cpu:   cpu=3 load_avg=339 util_avg=452
       busy_loop-1550  [005]   164.743800: sched_load_avg_cpu:   cpu=5 load_avg=168 util_avg=450
       busy_loop-1552  [002]   164.747792: sched_stat_runtime:   comm=busy_loop pid=1552 runtime=3999320 [ns] vruntime=609431868 [ns]
       busy_loop-1553  [000]   164.747793: sched_stat_runtime:   comm=busy_loop pid=1553 runtime=3999380 [ns] vruntime=594391274 [ns]
       busy_loop-1549  [003]   164.747793: sched_stat_runtime:   comm=busy_loop pid=1549 runtime=3999540 [ns] vruntime=386380388 [ns]
       busy_loop-1552  [002]   164.747794: sched_load_avg_task:  comm=busy_loop pid=1552 cpu=2 load_avg=1019 util_avg=499 load_sum=48694671 util_sum=23849523 period_contrib=808
       busy_loop-1551  [001]   164.747794: sched_stat_runtime:   comm=busy_loop pid=1551 runtime=3999880 [ns] vruntime=276056221 [ns]
       busy_loop-1550  [005]   164.747795: sched_stat_runtime:   comm=busy_loop pid=1550 runtime=3999280 [ns] vruntime=202319314 [ns]
       busy_loop-1552  [002]   164.747795: sched_load_avg_cpu:   cpu=2 load_avg=1006 util_avg=492
       busy_loop-1551  [001]   164.747795: sched_load_avg_task:  comm=busy_loop pid=1551 cpu=1 load_avg=1022 util_avg=500 load_sum=48813533 util_sum=23907693 period_contrib=924
       busy_loop-1553  [000]   164.747795: sched_load_avg_task:  comm=busy_loop pid=1553 cpu=0 load_avg=1019 util_avg=499 load_sum=48652717 util_sum=23832040 period_contrib=767
       busy_loop-1549  [003]   164.747796: sched_load_avg_task:  comm=busy_loop pid=1549 cpu=3 load_avg=1003 util_avg=315 load_sum=47917292 util_sum=15063949 period_contrib=50
       busy_loop-1551  [001]   164.747796: sched_load_avg_cpu:   cpu=1 load_avg=1016 util_avg=497
       busy_loop-1552  [002]   164.747796: sched_load_avg_cpu:   cpu=2 load_avg=169 util_avg=496
       busy_loop-1550  [005]   164.747797: sched_load_avg_task:  comm=busy_loop pid=1550 cpu=5 load_avg=1023 util_avg=501 load_sum=48880090 util_sum=23938753 period_contrib=989
       busy_loop-1553  [000]   164.747797: sched_load_avg_cpu:   cpu=0 load_avg=1022 util_avg=500
       busy_loop-1549  [003]   164.747797: sched_load_avg_cpu:   cpu=3 load_avg=2028 util_avg=496
       busy_loop-1551  [001]   164.747797: sched_load_avg_cpu:   cpu=1 load_avg=169 util_avg=495
       busy_loop-1550  [005]   164.747798: sched_load_avg_cpu:   cpu=5 load_avg=1016 util_avg=497
       busy_loop-1553  [000]   164.747799: sched_load_avg_cpu:   cpu=0 load_avg=170 util_avg=499
       busy_loop-1549  [003]   164.747800: sched_load_avg_cpu:   cpu=3 load_avg=337 util_avg=494
       busy_loop-1550  [005]   164.747800: sched_load_avg_cpu:   cpu=5 load_avg=168 util_avg=492
       busy_loop-1552  [002]   164.751792: sched_stat_runtime:   comm=busy_loop pid=1552 runtime=4000260 [ns] vruntime=613432128 [ns]
       busy_loop-1549  [003]   164.751793: sched_stat_runtime:   comm=busy_loop pid=1549 runtime=3999760 [ns] vruntime=390380148 [ns]
       busy_loop-1553  [000]   164.751793: sched_stat_runtime:   comm=busy_loop pid=1553 runtime=3999920 [ns] vruntime=598391194 [ns]
       busy_loop-1552  [002]   164.751793: sched_load_avg_task:  comm=busy_loop pid=1552 cpu=2 load_avg=1015 util_avg=538 load_sum=48500452 util_sum=25717351 period_contrib=618
       busy_loop-1550  [005]   164.751793: sched_stat_runtime:   comm=busy_loop pid=1550 runtime=3999920 [ns] vruntime=206319234 [ns]
       busy_loop-1552  [002]   164.751794: sched_load_avg_cpu:   cpu=2 load_avg=1024 util_avg=542
       busy_loop-1551  [001]   164.751794: sched_stat_runtime:   comm=busy_loop pid=1551 runtime=4000120 [ns] vruntime=280056341 [ns]
       busy_loop-1549  [003]   164.751795: sched_load_avg_task:  comm=busy_loop pid=1549 cpu=3 load_avg=1021 util_avg=376 load_sum=48771927 util_sum=17985591 period_contrib=884
       busy_loop-1553  [000]   164.751795: sched_load_avg_task:  comm=busy_loop pid=1553 cpu=0 load_avg=1015 util_avg=538 load_sum=48458496 util_sum=25697835 period_contrib=577
       busy_loop-1551  [001]   164.751795: sched_load_avg_task:  comm=busy_loop pid=1551 cpu=1 load_avg=1018 util_avg=539 load_sum=48619308 util_sum=25780552 period_contrib=734
       busy_loop-1550  [005]   164.751795: sched_load_avg_task:  comm=busy_loop pid=1550 cpu=5 load_avg=1019 util_avg=540 load_sum=48685865 util_sum=25814558 period_contrib=799
       busy_loop-1552  [002]   164.751796: sched_load_avg_cpu:   cpu=2 load_avg=169 util_avg=535
       busy_loop-1551  [001]   164.751796: sched_load_avg_cpu:   cpu=1 load_avg=1011 util_avg=536
       busy_loop-1553  [000]   164.751797: sched_load_avg_cpu:   cpu=0 load_avg=1018 util_avg=539
       busy_loop-1549  [003]   164.751797: sched_load_avg_cpu:   cpu=3 load_avg=2020 util_avg=535
       busy_loop-1550  [005]   164.751797: sched_load_avg_cpu:   cpu=5 load_avg=1012 util_avg=536
       busy_loop-1551  [001]   164.751797: sched_load_avg_cpu:   cpu=1 load_avg=168 util_avg=533
       busy_loop-1553  [000]   164.751799: sched_load_avg_cpu:   cpu=0 load_avg=169 util_avg=538
       busy_loop-1549  [003]   164.751799: sched_load_avg_cpu:   cpu=3 load_avg=336 util_avg=533
       busy_loop-1550  [005]   164.751800: sched_load_avg_cpu:   cpu=5 load_avg=171 util_avg=543
       busy_loop-1549  [003]   164.751807: sched_stat_runtime:   comm=busy_loop pid=1549 runtime=13700 [ns] vruntime=390393848 [ns]
       busy_loop-1549  [003]   164.751809: sched_load_avg_task:  comm=busy_loop pid=1549 cpu=3 load_avg=1021 util_avg=376 load_sum=48785239 util_sum=17998903 period_contrib=897
       busy_loop-1549  [003]   164.751811: sched_load_avg_cpu:   cpu=3 load_avg=2020 util_avg=535
       busy_loop-1549  [003]   164.751812: sched_load_avg_task:  comm=busy_loop pid=1554 cpu=3 load_avg=1015 util_avg=163 load_sum=48472554 util_sum=7827475 period_contrib=593
       busy_loop-1549  [003]   164.751814: sched_load_avg_cpu:   cpu=3 load_avg=2020 util_avg=535
       busy_loop-1549  [003]   164.751816: sched_switch:         busy_loop:1549 [120] R ==> busy_loop:1554 [120]
       busy_loop-1552  [002]   164.755792: sched_stat_runtime:   comm=busy_loop pid=1552 runtime=3999800 [ns] vruntime=617431928 [ns]
       busy_loop-1553  [000]   164.755793: sched_stat_runtime:   comm=busy_loop pid=1553 runtime=3999880 [ns] vruntime=602391074 [ns]
       busy_loop-1552  [002]   164.755793: sched_load_avg_task:  comm=busy_loop pid=1552 cpu=2 load_avg=1011 util_avg=574 load_sum=48306205 util_sum=27414009 period_contrib=428
       busy_loop-1550  [005]   164.755793: sched_stat_runtime:   comm=busy_loop pid=1550 runtime=3999780 [ns] vruntime=210319014 [ns]
       busy_loop-1554  [003]   164.755793: sched_stat_runtime:   comm=busy_loop pid=1554 runtime=3986540 [ns] vruntime=382907621 [ns]
       busy_loop-1552  [002]   164.755794: sched_load_avg_cpu:   cpu=2 load_avg=1019 util_avg=578
       busy_loop-1551  [001]   164.755794: sched_stat_runtime:   comm=busy_loop pid=1551 runtime=3999860 [ns] vruntime=284056201 [ns]
       busy_loop-1553  [000]   164.755795: sched_load_avg_task:  comm=busy_loop pid=1553 cpu=0 load_avg=1010 util_avg=573 load_sum=48264247 util_sum=27392629 period_contrib=387
       busy_loop-1551  [001]   164.755795: sched_load_avg_task:  comm=busy_loop pid=1551 cpu=1 load_avg=1014 util_avg=575 load_sum=48425055 util_sum=27481823 period_contrib=544
       busy_loop-1552  [002]   164.755795: sched_load_avg_cpu:   cpu=2 load_avg=168 util_avg=570
       busy_loop-1550  [005]   164.755795: sched_load_avg_task:  comm=busy_loop pid=1550 cpu=5 load_avg=1015 util_avg=576 load_sum=48491612 util_sum=27518531 period_contrib=609
       busy_loop-1554  [003]   164.755796: sched_load_avg_task:  comm=busy_loop pid=1554 cpu=3 load_avg=1010 util_avg=230 load_sum=48265186 util_sum=10993484 period_contrib=390
       busy_loop-1551  [001]   164.755796: sched_load_avg_cpu:   cpu=1 load_avg=1007 util_avg=571
       busy_loop-1553  [000]   164.755796: sched_load_avg_cpu:   cpu=0 load_avg=1014 util_avg=575
       busy_loop-1550  [005]   164.755797: sched_load_avg_cpu:   cpu=5 load_avg=1008 util_avg=572
       busy_loop-1554  [003]   164.755797: sched_load_avg_cpu:   cpu=3 load_avg=2012 util_avg=571
       busy_loop-1551  [001]   164.755797: sched_load_avg_cpu:   cpu=1 load_avg=171 util_avg=581
       busy_loop-1553  [000]   164.755799: sched_load_avg_cpu:   cpu=0 load_avg=168 util_avg=574
       busy_loop-1550  [005]   164.755799: sched_load_avg_cpu:   cpu=5 load_avg=170 util_avg=579
       busy_loop-1554  [003]   164.755799: sched_load_avg_cpu:   cpu=3 load_avg=342 util_avg=581
       busy_loop-1552  [002]   164.759791: sched_cfs_idle_inject_timer: throttled=1
       busy_loop-1551  [001]   164.759791: sched_cfs_idle_inject_timer: throttled=1
       busy_loop-1550  [005]   164.759792: sched_cfs_idle_inject_timer: throttled=1
       busy_loop-1554  [003]   164.759792: sched_cfs_idle_inject_timer: throttled=1
       busy_loop-1553  [000]   164.759792: sched_cfs_idle_inject_timer: throttled=1

Cheers,
Javi

maintainer
#67 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 7 lines · Javi Merino

I've tested this series on Juno (2xCortex-A57 4xCortex-A53). If you idle inject for 50% of the time, when I run 6 busy loops the scheduler sometimes keeps two of them in the same cpu while the another cpu is completely idle. Without idle injection the scheduler does the sensible thing: put one busy loop in each CPU. I'm running systemd and this only happens with CONFIG_SCHED_AUTOGROUP=y. If I unset CONFIG_SCHED_AUTOGROUP, the tasks are spread across all cpus as usual.

That's not a plus for this patch though; but a bug report against AUTOGROUP/cgroup muck, right?

#68 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Javi Merino

I've tested this series on Juno (2xCortex-A57 4xCortex-A53). If you idle inject for 50% of the time, when I run 6 busy loops the scheduler sometimes keeps two of them in the same cpu while the another cpu is completely idle. Without idle injection the scheduler does the sensible thing: put one busy loop in each CPU. I'm running systemd and this only happens with CONFIG_SCHED_AUTOGROUP=y. If I unset CONFIG_SCHED_AUTOGROUP, the tasks are spread across all cpus as usual.

That's not a plus for this patch though; but a bug report against AUTOGROUP/cgroup muck, right?

The bug only happens when you apply this series and you set the system to idle inject 50% of the time. SCHED_AUTOGROUP is already part of the kernel and behaves properly with the kernel as is. I think that this patch should not introduce new bugs.

Cheers,
Javi

maintainer
#69 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 10 lines · Javi Merino
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Javi Merino

I've tested this series on Juno (2xCortex-A57 4xCortex-A53). If you idle inject for 50% of the time, when I run 6 busy loops the scheduler sometimes keeps two of them in the same cpu while the another cpu is completely idle. Without idle injection the scheduler does the sensible thing: put one busy loop in each CPU. I'm running systemd and this only happens with CONFIG_SCHED_AUTOGROUP=y. If I unset CONFIG_SCHED_AUTOGROUP, the tasks are spread across all cpus as usual.

Quoted 7 lines
Quoted 2 lines

That's not a plus for this patch though; but a bug report against AUTOGROUP/cgroup muck, right?

The bug only happens when you apply this series and you set the system to idle inject 50% of the time. SCHED_AUTOGROUP is already part of the kernel and behaves properly with the kernel as is. I think that this patch should not introduce new bugs.

Ah, then I misunderstood your email, agreed.

author
#70 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 15 lines · On Tue, 24 Nov 2015 11:09:56 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 14 lines · Javi Merino
Quoted 12 lines · Peter Zijlstra
Quoted 8 lines · Javi Merino

I've tested this series on Juno (2xCortex-A57 4xCortex-A53). If you idle inject for 50% of the time, when I run 6 busy loops the scheduler sometimes keeps two of them in the same cpu while the another cpu is completely idle. Without idle injection the scheduler does the sensible thing: put one busy loop in each CPU. I'm running systemd and this only happens with CONFIG_SCHED_AUTOGROUP=y. If I unset CONFIG_SCHED_AUTOGROUP, the tasks are spread across all cpus as usual.

That's not a plus for this patch though; but a bug report against AUTOGROUP/cgroup muck, right?

Quoted 6 lines
Quoted 4 lines

The bug only happens when you apply this series and you set the system to idle inject 50% of the time. SCHED_AUTOGROUP is already part of the kernel and behaves properly with the kernel as is. I think that this patch should not introduce new bugs.

Ah, then I misunderstood your email, agreed.

First of all, thanks for testing.
Just trying to reproduce this. So let me understand your set up.
- 8 cores in total?
- you first set 50% idle
- then launch 6 busy loops
How often you see this happen?

#71 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 23 lines · Jacob Pan
Quoted 20 lines · On Tue, 24 Nov 2015 11:09:56 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 18 lines · Javi Merino
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Javi Merino

I've tested this series on Juno (2xCortex-A57 4xCortex-A53). If you idle inject for 50% of the time, when I run 6 busy loops the scheduler sometimes keeps two of them in the same cpu while the another cpu is completely idle. Without idle injection the scheduler does the sensible thing: put one busy loop in each CPU. I'm running systemd and this only happens with CONFIG_SCHED_AUTOGROUP=y. If I unset CONFIG_SCHED_AUTOGROUP, the tasks are spread across all cpus as usual.

Quoted 2 lines

That's not a plus for this patch though; but a bug report against AUTOGROUP/cgroup muck, right?

The bug only happens when you apply this series and you set the system to idle inject 50% of the time. SCHED_AUTOGROUP is already part of the kernel and behaves properly with the kernel as is. I think that this patch should not introduce new bugs.

Quoted 4 lines
Quoted 1 line

Ah, then I misunderstood your email, agreed.

First of all, thanks for testing.
Just trying to reproduce this. So let me understand your set up.
- 8 cores in total?

- 6 cores in total

Quoted 2 lines

- you first set 50% idle
- then launch 6 busy loops

Correct

Quoted 1 line

How often you see this happen?

About 1 in 10. Sometimes, even though it ends up doing the right thing, it takes some time (>500ms) to move things around. This never happens with idle injection disabled (kernel.sched_cfs_idle_inject_pct = 0).

Admittedly, the situation is a lot better than with 4.3. When I first tried this series on top of v4.3-rc6, the scheduler consistently misbehaved with 50% idle injection, leaving up to 4 100% tasks on a cpu while three cpus remain completely idle

Cheers,
Javi

author
#72 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 4 lines · On Tue, 24 Nov 2015 12:00:07 +0000 Javi Merino <[email protected]> wrote:
Quoted 2 lines

- you first set 50% idle
- then launch 6 busy loops

Correct

Do you mind sharing the workload or test scripts? Just to reproduce here.

Thanks,

Jacob

#73 lore raw

Re: [PATCH 3/4] sched: introduce synchronized idle injection

Quoted 9 lines · Jacob Pan
Quoted 4 lines · On Tue, 24 Nov 2015 12:00:07 +0000 Javi Merino <[email protected]> wrote:
Quoted 2 lines

- you first set 50% idle
- then launch 6 busy loops

Correct

Do you mind sharing the workload or test scripts? Just to reproduce here.

I'm using rt-app[0] with the attached config. To run the test I use workload-automation[1], but I don't think you need workload automation.

[0] https://git.linaro.org/power/rt-app.git
[1] https://github.com/ARM-software/workload-automation/

author
#74 lore raw

PATCH 4/4 sched: add trace event for idle injection

In reply to Jacob Pan

Trace events for idle injection can be used to determine timer activities for checking synchronization. In addition they also helps to determine when the runqueue is throttled.

---
 include/linux/sched.h        |  5 +++++
 include/trace/events/sched.h | 25 +++++++++++++++++++++++++
 kernel/sched/fair.c          |  3 +++
 3 files changed, 33 insertions(+)
include/linux/sched.h +5 -0
index ff551a3..99c79bf 100644
@@ -3189,6 +3189,11 @@ extern int proc_sched_cfs_idle_inject_duration_handler(struct ctl_table *table,
 						int write,
 						void __user *buffer,
 						size_t *length, loff_t *ppos);
+enum cfs_idle_inject_action {
+	CFS_IDLE_INJECT_TIMER,    /* timer sync point */
+	CFS_IDLE_INJECT_FORCED,   /* idle forced in rq */
+	CFS_IDLE_INJECT_YIELD_SOFTIRQ   /* yield to pending softirq */
+};
 #endif
 
 #endif
include/trace/events/sched.h +25 -0
index 539d6bc..52c11c1 100644
@@ -566,6 +566,31 @@ TRACE_EVENT(sched_wake_idle_without_ipi,
 
 	TP_printk("cpu=%d", __entry->cpu)
 );
+
+#ifdef CONFIG_CFS_IDLE_INJECT
+/*
+ * Tracepoint for idle injection
+ */
+TRACE_EVENT(sched_cfs_idle_inject,
+
+	TP_PROTO(enum cfs_idle_inject_action action, int throttled),
+
+	TP_ARGS(action, throttled),
+
+	TP_STRUCT__entry(
+		__field(enum cfs_idle_inject_action, action)
+		__field(int, throttled)
+	),
+
+	TP_fast_assign(
+		__entry->action = action;
+		__entry->throttled = throttled;
+	),
+
+	TP_printk("action:%d throttled:%d", __entry->action, __entry->throttled)
+);
+#endif
+
 #endif /* _TRACE_SCHED_H */
 
 /* This part must be outside protection */
kernel/sched/fair.c +3 -0
index a0cd777..20027eb 100644
@@ -5248,8 +5248,10 @@ idle:
 	if (in_forced_idle(cfs_rq)) {
 		if (unlikely(local_softirq_pending())) {
 			__unthrottle_cfs_rq(cfs_rq);
+			trace_sched_cfs_idle_inject(CFS_IDLE_INJECT_YIELD_SOFTIRQ, 1);
 			goto again;
 		}
+		trace_sched_cfs_idle_inject(CFS_IDLE_INJECT_FORCED, 1);
 		return NULL;
 	}
 	/*
@@ -8432,6 +8434,7 @@ static enum hrtimer_restart idle_inject_timer_fn(struct hrtimer *hrtimer)
 		throttle_rq(cpu);
 	}
 	raw_cpu_write(idle_injected, !status);
+	trace_sched_cfs_idle_inject(CFS_IDLE_INJECT_TIMER, !status);
 
 	return HRTIMER_RESTART;
 }
Signature · 1 line
1.9.1
#75 lore raw

Re: [PATCH 4/4] sched: add trace event for idle injection

reproduce: wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree…

Hi Jacob,

[auto build test ERROR on: tip/sched/core]
[also build test ERROR on: v4.3 next-20151113]

url:    https://github.com/0day-ci/linux/commits/Jacob-Pan/CFS-idle-injection/20151114-035732
config: cris-etrax-100lx_v2_defconfig (attached as .config)

reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=cris

All errors (new ones prefixed by >>):

kernel/sched/fair.c: In function 'pick_next_task_fair':

>> kernel/sched/fair.c:5316:4: error: implicit declaration of function 'trace_sched_cfs_idle_inject' [-Werror=implicit-function-declaration]
>> kernel/sched/fair.c:5316:32: error: 'CFS_IDLE_INJECT_YIELD_SOFTIRQ' undeclared (first use in this function)
   kernel/sched/fair.c:5316:32: note: each undeclared identifier is reported only once for each function it appears in
>> kernel/sched/fair.c:5319:31: error: 'CFS_IDLE_INJECT_FORCED' undeclared (first use in this function)

cc1: some warnings being treated as errors

vim +/trace_sched_cfs_idle_inject +5316 kernel/sched/fair.c

5310 return p;
5311
5312 idle:
5313 if (in_forced_idle(cfs_rq)) {
5314 if (unlikely(local_softirq_pending())) {
5315 __unthrottle_cfs_rq(cfs_rq);

> 5316				trace_sched_cfs_idle_inject(CFS_IDLE_INJECT_YIELD_SOFTIRQ, 1);
  5317				goto again;
  5318			}
> 5319			trace_sched_cfs_idle_inject(CFS_IDLE_INJECT_FORCED, 1);

5320 return NULL;
5321 }
5322 /*

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#76 lore raw

Re: [PATCH 4/4] sched: add trace event for idle injection

In reply to Jacob Pan

[ CCing Steve Rostedt for the tracing bits ]

Quoted 50 lines · Jacob Pan

Trace events for idle injection can be used to determine timer activities for checking synchronization. In addition they also helps to determine when the runqueue is throttled.

---
 include/linux/sched.h        |  5 +++++
 include/trace/events/sched.h | 25 +++++++++++++++++++++++++
 kernel/sched/fair.c          |  3 +++
 3 files changed, 33 insertions(+)
include/linux/sched.h +5 -0
index ff551a3..99c79bf 100644
@@ -3189,6 +3189,11 @@ extern int proc_sched_cfs_idle_inject_duration_handler(struct ctl_table *table,
 						int write,
 						void __user *buffer,
 						size_t *length, loff_t *ppos);
+enum cfs_idle_inject_action {
+	CFS_IDLE_INJECT_TIMER,    /* timer sync point */
+	CFS_IDLE_INJECT_FORCED,   /* idle forced in rq */
+	CFS_IDLE_INJECT_YIELD_SOFTIRQ   /* yield to pending softirq */
+};
 #endif
 
 #endif
include/trace/events/sched.h +15 -0
index 539d6bc..52c11c1 100644
@@ -566,6 +566,31 @@ TRACE_EVENT(sched_wake_idle_without_ipi,
 
 	TP_printk("cpu=%d", __entry->cpu)
 );
+
+#ifdef CONFIG_CFS_IDLE_INJECT
+/*
+ * Tracepoint for idle injection
+ */
+TRACE_EVENT(sched_cfs_idle_inject,
+
+	TP_PROTO(enum cfs_idle_inject_action action, int throttled),
+
+	TP_ARGS(action, throttled),
+
+	TP_STRUCT__entry(
+		__field(enum cfs_idle_inject_action, action)
+		__field(int, throttled)
+	),

include/trace/events/sched.h TRACE_EVENT(sched_wake_idle_without_ipi, · line 593

Quoted 10 lines
include/trace/events/sched.h +10 -0
+
+	TP_fast_assign(
+		__entry->action = action;
+		__entry->throttled = throttled;
+	),
+
+	TP_printk("action:%d throttled:%d", __entry->action, __entry->throttled)
+);
+#endif
+

One minor nit: can you use key=value (i.e. "throttled=%d") instead for consistency with the rest of this file?

Other than that, I know that Peter suggested an enum for the action, but wouldn't it be better to create an EVENT_CLASS and subclass the three actions from it? Something like:

DECLARE_EVENT_CLASS(sched_cfs_idle_inject_template,

	TP_PROTO(int throttled),

	TP_ARGS(throttled),

	TP_STRUCT__entry(
		__field(int, throttled)
	),

	TP_fast_assign(
		__entry->throttled = throttled;
	),

	TP_printk("throttled=%d", __entry->throttled)

);

DEFINE_EVENT(sched_cfs_idle_inject_template, sched_cfs_idle_inject_timer,

	     TP_PROTO(int throttled),
	     TP_ARGS(throttled));

DEFINE_EVENT(sched_cfs_idle_inject_template, sched_cfs_idle_inject_forced,

	     TP_PROTO(int throttled),
	     TP_ARGS(throttled));

DEFINE_EVENT(sched_cfs_idle_inject_template,

	     sched_cfs_idle_inject_yield_softirq,
	     TP_PROTO(int throttled),
	     TP_ARGS(throttled));

kernel/sched/fair.c idle: · line 5251

Quoted 12 lines
include/trace/events/sched.h +0 -0
 #endif /* _TRACE_SCHED_H */
 
 /* This part must be outside protection */
kernel/sched/fair.c +1 -0
index a0cd777..20027eb 100644
@@ -5248,8 +5248,10 @@ idle:
 	if (in_forced_idle(cfs_rq)) {
 		if (unlikely(local_softirq_pending())) {
 			__unthrottle_cfs_rq(cfs_rq);
+			trace_sched_cfs_idle_inject(CFS_IDLE_INJECT_YIELD_SOFTIRQ, 1);

This becomes:

			trace_sched_cfs_idle_inject_yield_softirq(1);

kernel/sched/fair.c idle: · line 5254

Quoted 3 lines
kernel/sched/fair.c +1 -0
 			goto again;
 		}
+		trace_sched_cfs_idle_inject(CFS_IDLE_INJECT_FORCED, 1);

This becomes:

		trace_sched_cfs_idle_inject_forced(1);

kernel/sched/fair.c static enum hrtimer_restart idle_inject_timer_fn(struct hrtimer *hrtimer) · line 8437

Quoted 8 lines
kernel/sched/fair.c +1 -0
 		return NULL;
 	}
 	/*
@@ -8432,6 +8434,7 @@ static enum hrtimer_restart idle_inject_timer_fn(struct hrtimer *hrtimer)
 		throttle_rq(cpu);
 	}
 	raw_cpu_write(idle_injected, !status);
+	trace_sched_cfs_idle_inject(CFS_IDLE_INJECT_TIMER, !status);

and this would be:

	trace_sched_cfs_idle_inject_timer(!status);

kernel/sched/fair.c static enum hrtimer_restart idle_inject_timer_fn(struct hrtimer *hrtimer) · line 8440

Quoted 3 lines
kernel/sched/fair.c +0 -0
 	return HRTIMER_RESTART;
 }

Cheers,
Javi

author
#77 lore raw

Re: [PATCH 4/4] sched: add trace event for idle injection

Quoted 5 lines · On Thu, 19 Nov 2015 14:39:35 +0000 Javi Merino <[email protected]> wrote:
Quoted 1 line

+

One minor nit: can you use key=value (i.e. "throttled=%d") instead for consistency with the rest of this file?

will do.

Quoted 3 lines

Other than that, I know that Peter suggested an enum for the action, but wouldn't it be better to create an EVENT_CLASS and subclass the three actions from it? Something like:

I agree, it will be more readable in the trace.

thanks you