Skip to contentSkip to conversation

cpuidle: Always stop scheduler tick on adaptive-tick CPUs

PATCH In review Rafael J. Wysocki 3 participants 6 messages Jul 16, 2019 – Jul 18, 2019 Scheduler

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cpuidle: Always stop scheduler tick on adaptive-tick CPUs

From: Rafael J. Wysocki <[email protected]>

Running the scheduler tick on idle adaptive-tick CPUs is not useful and it may also be not expected by users (as reported by Thomas), so add a check to cpuidle_idle_call() to always stop the tick on them regardless of the idle duration predicted by the governor.

Fixes 554c8aa8ecad ("sched: idle: Select idle state before stopping the tick")
---
 kernel/sched/idle.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/sched/idle.c +2 -1
Index: linux-pm/kernel/sched/idle.c
===================================================================
@@ -191,7 +191,8 @@ static void cpuidle_idle_call(void)
 		 */
 		next_state = cpuidle_select(drv, dev, &stop_tick);
 
-		if (stop_tick || tick_nohz_tick_stopped())
+		if (stop_tick || tick_nohz_tick_stopped() ||
+		    !housekeeping_cpu(dev->cpu, HK_FLAG_TICK))
 			tick_nohz_idle_stop_tick();
 		else
 			tick_nohz_idle_retain_tick();
#1 lore raw
Quoted 3 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Running the scheduler tick on idle adaptive-tick CPUs is not useful

Judging by the below change, you mean full dynticks, right?

Quoted 17 lines

and it may also be not expected by users (as reported by Thomas), so add a check to cpuidle_idle_call() to always stop the tick on them regardless of the idle duration predicted by the governor.

Fixes 554c8aa8ecad ("sched: idle: Select idle state before stopping the tick")
---
 kernel/sched/idle.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-pm/kernel/sched/idle.c
===================================================================

kernel/sched/idle.c static void cpuidle_idle_call(void) · line 195

Quoted 6 lines
kernel/sched/idle.c +2 -1
@@ -191,7 +191,8 @@ static void cpuidle_idle_call(void)
 		 */
 		next_state = cpuidle_select(drv, dev, &stop_tick);
 
-		if (stop_tick || tick_nohz_tick_stopped())
+		if (stop_tick || tick_nohz_tick_stopped() ||
+		    !housekeeping_cpu(dev->cpu, HK_FLAG_TICK))

But tick_nohz_tick_stopped() also works on full dynticks CPUs. If the tick isn't stopped on a full dynticks CPU by the time we reach this path, it means that the conditions for the tick to be stopped are not met anyway (eg: more than one task and sched tick is needed, perf event requires the tick, posix CPU timer, etc...)

Or am I missing something else?

Thanks.

Quoted 6 lines
kernel/sched/idle.c +0 -0
 			tick_nohz_idle_stop_tick();
 		else
 			tick_nohz_idle_retain_tick();
#2 lore raw
Quoted 7 lines · Frederic Weisbecker
Quoted 3 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Running the scheduler tick on idle adaptive-tick CPUs is not useful

Judging by the below change, you mean full dynticks, right?

Right.

Quoted 20 lines
Quoted 20 lines

and it may also be not expected by users (as reported by Thomas), so add a check to cpuidle_idle_call() to always stop the tick on them regardless of the idle duration predicted by the governor.

Fixes 554c8aa8ecad ("sched: idle: Select idle state before stopping the tick")
---
 kernel/sched/idle.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-pm/kernel/sched/idle.c
===================================================================

kernel/sched/idle.c +0 -0
@@ -191,7 +191,8 @@ static void cpuidle_idle_call(void)
               */
              next_state = cpuidle_select(drv, dev, &stop_tick);
 
Quoted 9 lines

kernel/sched/idle.c static void cpuidle_idle_call(void) · line 195

Quoted 3 lines
kernel/sched/idle.c +2 -1
-             if (stop_tick || tick_nohz_tick_stopped())
+             if (stop_tick || tick_nohz_tick_stopped() ||
+                 !housekeeping_cpu(dev->cpu, HK_FLAG_TICK))

But tick_nohz_tick_stopped() also works on full dynticks CPUs. If the tick isn't stopped on a full dynticks CPU by the time we reach this path, it means that the conditions for the tick to be stopped are not met anyway (eg: more than one task and sched tick is needed, perf event requires the tick, posix CPU timer, etc...)

First of all, according to Thomas, the patch does make a difference, so evidently on his system(s) the full dynticks CPUs enter the idle loop with running tick.

This means that, indeed, the conditions for the tick to be stopped have not been met up to that point, but if the (full dynticks) CPU becomes idle, that's because it has been made idle on purpose (presumably by a user-space "orchestrator" or the sysadmin), so the kernel can assume that it will remain idle indefinitely. That, in turn, is when the tick would be stopped on it regardless of everything else (even if it wasn't a full dynticks CPU).

I guess I should add the above to the changelog.

Quoted 1 line

Or am I missing something else?

Well, if full dynticks CPUs are expected to always enter the idle loop with stopped tick, then something appears to be amiss, but I'm not sure if that expectation is entirely realistic.

Cheers!

#3 lore raw
Quoted 46 lines · Rafael J. Wysocki
Quoted 7 lines · Frederic Weisbecker
Quoted 3 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Running the scheduler tick on idle adaptive-tick CPUs is not useful

Judging by the below change, you mean full dynticks, right?

Right.

Quoted 20 lines
Quoted 20 lines

and it may also be not expected by users (as reported by Thomas), so add a check to cpuidle_idle_call() to always stop the tick on them regardless of the idle duration predicted by the governor.

Fixes 554c8aa8ecad ("sched: idle: Select idle state before stopping the tick")
---
 kernel/sched/idle.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-pm/kernel/sched/idle.c
===================================================================

kernel/sched/idle.c +0 -0
@@ -191,7 +191,8 @@ static void cpuidle_idle_call(void)
               */
              next_state = cpuidle_select(drv, dev, &stop_tick);
 
Quoted 9 lines

kernel/sched/idle.c static void cpuidle_idle_call(void) · line 195

Quoted 3 lines
kernel/sched/idle.c +2 -1
-             if (stop_tick || tick_nohz_tick_stopped())
+             if (stop_tick || tick_nohz_tick_stopped() ||
+                 !housekeeping_cpu(dev->cpu, HK_FLAG_TICK))

But tick_nohz_tick_stopped() also works on full dynticks CPUs. If the tick isn't stopped on a full dynticks CPU by the time we reach this path, it means that the conditions for the tick to be stopped are not met anyway (eg: more than one task and sched tick is needed, perf event requires the tick, posix CPU timer, etc...)

First of all, according to Thomas, the patch does make a difference, so evidently on his system(s) the full dynticks CPUs enter the idle loop with running tick.

Quoted 7 lines

This means that, indeed, the conditions for the tick to be stopped have not been met up to that point, but if the (full dynticks) CPU becomes idle, that's because it has been made idle on purpose (presumably by a user-space "orchestrator" or the sysadmin), so the kernel can assume that it will remain idle indefinitely. That, in turn, is when the tick would be stopped on it regardless of everything else (even if it wasn't a full dynticks CPU).

Well I think we disagree on that assumption that if a nohz_full CPU is put idle, it will remain there indefinitely. Nohz_full CPUs aren't really special in this regard, they can sleep on an IO, wait for a short event just like any other CPU.

The only difference with other CPUs is that they _might_ enter the idle loop with the tick already stopped. Ok that should be the case most of the time with regular full dynticks usecases, but there can be initialization work or stuff that make the CPU run with periodic tick for some time.

Thanks.

Quoted 10 lines

I guess I should add the above to the changelog.

Quoted 1 line

Or am I missing something else?

Well, if full dynticks CPUs are expected to always enter the idle loop with stopped tick, then something appears to be amiss, but I'm not sure if that expectation is entirely realistic.

Cheers!

#4 lore raw
Quoted 56 lines · Frederic Weisbecker
Quoted 46 lines · Rafael J. Wysocki
Quoted 7 lines · Frederic Weisbecker
Quoted 3 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Running the scheduler tick on idle adaptive-tick CPUs is not useful

Judging by the below change, you mean full dynticks, right?

Right.

Quoted 20 lines
Quoted 20 lines

and it may also be not expected by users (as reported by Thomas), so add a check to cpuidle_idle_call() to always stop the tick on them regardless of the idle duration predicted by the governor.

Fixes 554c8aa8ecad ("sched: idle: Select idle state before stopping the tick")
---
 kernel/sched/idle.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-pm/kernel/sched/idle.c
===================================================================

kernel/sched/idle.c +0 -0
@@ -191,7 +191,8 @@ static void cpuidle_idle_call(void)
               */
              next_state = cpuidle_select(drv, dev, &stop_tick);
 
Quoted 9 lines

kernel/sched/idle.c static void cpuidle_idle_call(void) · line 195

Quoted 3 lines
kernel/sched/idle.c +2 -1
-             if (stop_tick || tick_nohz_tick_stopped())
+             if (stop_tick || tick_nohz_tick_stopped() ||
+                 !housekeeping_cpu(dev->cpu, HK_FLAG_TICK))

But tick_nohz_tick_stopped() also works on full dynticks CPUs. If the tick isn't stopped on a full dynticks CPU by the time we reach this path, it means that the conditions for the tick to be stopped are not met anyway (eg: more than one task and sched tick is needed, perf event requires the tick, posix CPU timer, etc...)

First of all, according to Thomas, the patch does make a difference, so evidently on his system(s) the full dynticks CPUs enter the idle loop with running tick.

Quoted 7 lines

This means that, indeed, the conditions for the tick to be stopped have not been met up to that point, but if the (full dynticks) CPU becomes idle, that's because it has been made idle on purpose (presumably by a user-space "orchestrator" or the sysadmin), so the kernel can assume that it will remain idle indefinitely. That, in turn, is when the tick would be stopped on it regardless of everything else (even if it wasn't a full dynticks CPU).

Quoted 4 lines

Well I think we disagree on that assumption that if a nohz_full CPU is put idle, it will remain there indefinitely. Nohz_full CPUs aren't really special in this regard, they can sleep on an IO, wait for a short event just like any other CPU.

Fair enough.

This means that the governor (or rather governors) will need to be modified to address the issue reported by Thomas.

Fortunately, I have a patch going in that direction too. :-)

Cheers!

#5 lore raw
Quoted 8 lines · Rafael J. Wysocki
Quoted 6 lines · Frederic Weisbecker

On Wed, Jul 17, 2019 at 09:55:08AM +0200, Rafael J. Wysocki wrote:
Well I think we disagree on that assumption that if a nohz_full CPU is put idle, it will remain there indefinitely. Nohz_full CPUs aren't really special in this regard, they can sleep on an IO, wait for a short event just like any other CPU.

Quoted 6 lines

Fair enough.

This means that the governor (or rather governors) will need to be modified to address the issue reported by Thomas.

Fortunately, I have a patch going in that direction too. :-)

Good to hear, please also Cc me on that one, thanks!