sched/idle: Fix missing need_resched() checks after rcu_idle_enter()

PATCH In review Frederic Weisbecker 4 participants 11 messages Dec 22, 2020 – Jan 5, 2021

Subsystems (8)

Stable branch Arm (32-bit) Arm sub-architectures Arm i.MX and Layerscape +4 more subsystems

lore

Share & tools

lore

Frederic Weisbecker

author

2020-12-22 01:38 UTC #0 lore raw

cover sched/idle: Fix missing need_resched() checks after rcu_idle_enter()

With Paul, we've been thinking that the idle loop wasn't twisted enough yet to deserve 2020.

rcutorture, after some recent parameter changes, has been complaining about a hung task.

It appears that rcu_idle_enter() may wake up a NOCB kthread but this happens after the last generic need_resched() check. Some cpuidle drivers fix it by chance but many others don't.

Here is a proposed bunch of fixes. I will need to also fix the rcu_user_enter() case, likely using irq_work, since nohz_full requires irq work to support self IPI.

Also more generally, this raise the question of local task wake_up() under disabled interrupts. When a wake up occurs in a preempt disabled section, it gets handled by the outer preempt_enable() call. There is no similar mechanism when a wake up occurs with interrupts disabled. I guess it is assumed to be handled, at worst, in the next tick. But a local irq work would provide instant preemption once IRQs are re-enabled. Of course this would only make sense in CONFIG_PREEMPTION, and when the tick is disabled...

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
sched/idle

HEAD: f2fa6e4a070c1535b9edc9ee097167fd2b15d235

Thanks,
Frederic
---

Frederic Weisbecker (4):
sched/idle: Fix missing need_resched() check after rcu_idle_enter()
cpuidle: Fix missing need_resched() check after rcu_idle_enter()
ARM: imx6q: Fix missing need_resched() check after rcu_idle_enter()
ACPI: processor: Fix missing need_resched() check after rcu_idle_enter()

 arch/arm/mach-imx/cpuidle-imx6q.c |  7 ++++++-
 drivers/acpi/processor_idle.c     | 10 ++++++++--
 drivers/cpuidle/cpuidle.c         | 33 +++++++++++++++++++++++++--------
 kernel/sched/idle.c               | 18 ++++++++++++------
 4 files changed, 51 insertions(+), 17 deletions(-)

Frederic Weisbecker

author

2020-12-22 01:38 UTC #1 lore raw

PATCH 1/4 sched/idle: Fix missing need_resched() check after rcu_idle_enter()

Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced.

Usually a wake up happening while running the idle task is spotted in one of the need_resched() checks carefully placed within the idle loop that can break to the scheduler.

Unfortunately in default_idle_call(), the call to rcu_idle_enter() is already beyond the last need_resched() check and we may halt the CPU with a resched request unhandled, leaving the task hanging.

Fix this with performing a last minute need_resched() check after calling rcu_idle_enter().

Reported-by Paul E. McKenney <[email protected]>

Fixes 96d3fd0d315a (rcu: Break call_rcu() deadlock involving scheduler and perf)

Cc [email protected]

Cc Peter Zijlstra <[email protected]>

Cc Rafael J. Wysocki <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar<[email protected]>

Signed-off-by Frederic Weisbecker <[email protected]>

---
 kernel/sched/idle.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

kernel/sched/idle.c +12 -6

index 305727ea0677..1af60dc50beb 100644
@@ -109,15 +109,21 @@ void __cpuidle default_idle_call(void)
 		rcu_idle_enter();
 		lockdep_hardirqs_on(_THIS_IP_);
 
-		arch_cpu_idle();
+		/*
+		 * Last need_resched() check must come after rcu_idle_enter()
+		 * which may wake up RCU internal tasks.
+		 */
+		if (!need_resched()) {
+			arch_cpu_idle();
+			raw_local_irq_disable();
+		}
 
 		/*
-		 * OK, so IRQs are enabled here, but RCU needs them disabled to
-		 * turn itself back on.. funny thing is that disabling IRQs
-		 * will cause tracing, which needs RCU. Jump through hoops to
-		 * make it 'work'.
+		 * OK, so IRQs are enabled after arch_cpu_idle(), but RCU needs
+		 * them disabled to turn itself back on.. funny thing is that
+		 * disabling IRQs will cause tracing, which needs RCU. Jump through
+		 * hoops to make it 'work'.
 		 */
-		raw_local_irq_disable();
 		lockdep_hardirqs_off(_THIS_IP_);
 		rcu_idle_exit();
 		lockdep_hardirqs_on(_THIS_IP_);

Signature · 1 line

2.25.1

Paul E. McKenney

Tested-by on v1 1/4

2020-12-22 04:21 UTC #2 lore raw

Re: [PATCH 1/4] sched/idle: Fix missing need_resched() check after rcu_idle_enter()

Quoted 14 lines · Frederic Weisbecker

Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced.
Usually a wake up happening while running the idle task is spotted in one of the need_resched() checks carefully placed within the idle loop that can break to the scheduler.
Unfortunately in default_idle_call(), the call to rcu_idle_enter() is already beyond the last need_resched() check and we may halt the CPU with a resched request unhandled, leaving the task hanging.
Fix this with performing a last minute need_resched() check after calling rcu_idle_enter().

Quoted 8 lines

Reported-by Paul E. McKenney <[email protected]>
Fixes 96d3fd0d315a (rcu: Break call_rcu() deadlock involving scheduler and perf)
Cc [email protected]
Cc Peter Zijlstra <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar<[email protected]>
Signed-off-by Frederic Weisbecker <[email protected]>

Tested-by Paul E. McKenney <[email protected]>

Quoted 39 lines

---
 kernel/sched/idle.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

kernel/sched/idle.c +12 -6

index 305727ea0677..1af60dc50beb 100644
@@ -109,15 +109,21 @@ void __cpuidle default_idle_call(void)
 		rcu_idle_enter();
 		lockdep_hardirqs_on(_THIS_IP_);
 
-		arch_cpu_idle();
+		/*
+		 * Last need_resched() check must come after rcu_idle_enter()
+		 * which may wake up RCU internal tasks.
+		 */
+		if (!need_resched()) {
+			arch_cpu_idle();
+			raw_local_irq_disable();
+		}
 
 		/*
-		 * OK, so IRQs are enabled here, but RCU needs them disabled to
-		 * turn itself back on.. funny thing is that disabling IRQs
-		 * will cause tracing, which needs RCU. Jump through hoops to
-		 * make it 'work'.
+		 * OK, so IRQs are enabled after arch_cpu_idle(), but RCU needs
+		 * them disabled to turn itself back on.. funny thing is that
+		 * disabling IRQs will cause tracing, which needs RCU. Jump through
+		 * hoops to make it 'work'.
 		 */
-		raw_local_irq_disable();
 		lockdep_hardirqs_off(_THIS_IP_);
 		rcu_idle_exit();
 		lockdep_hardirqs_on(_THIS_IP_);

Signature · 1 line

2.25.1

Frederic Weisbecker

author

2020-12-22 01:38 UTC #3 lore raw

PATCH 2/4 cpuidle: Fix missing need_resched() check after rcu_idle_enter()

In reply to Frederic Weisbecker

Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced.

Usually a wake up happening while running the idle task is spotted in one of the need_resched() checks carefully placed within the idle loop that can break to the scheduler.

Unfortunately within cpuidle the call to rcu_idle_enter() is already beyond the last generic need_resched() check. Some drivers may perform their own checks like with mwait_idle_with_hints() but many others don't and we may halt the CPU with a resched request unhandled, leaving the task hanging.

Fix this with performing a last minute need_resched() check after calling rcu_idle_enter().

Reported-by Paul E. McKenney <[email protected]>

Fixes 1098582a0f6c (sched,idle,rcu: Push rcu_idle deeper into the idle path)

Cc [email protected]

Cc Daniel Lezcano <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Rafael J. Wysocki <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Signed-off-by Frederic Weisbecker <[email protected]>

---
 drivers/cpuidle/cpuidle.c | 33 +++++++++++++++++++++++++--------
 1 file changed, 25 insertions(+), 8 deletions(-)

drivers/cpuidle/cpuidle.c +25 -8

index ef2ea1b12cd8..4cc1ba49ce05 100644
@@ -134,8 +134,8 @@ int cpuidle_find_deepest_state(struct cpuidle_driver *drv,
 }
 
 #ifdef CONFIG_SUSPEND
-static void enter_s2idle_proper(struct cpuidle_driver *drv,
-				struct cpuidle_device *dev, int index)
+static int enter_s2idle_proper(struct cpuidle_driver *drv,
+			       struct cpuidle_device *dev, int index)
 {
 	ktime_t time_start, time_end;
 	struct cpuidle_state *target_state = &drv->states[index];

@@ -151,7 +151,14 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,
 	stop_critical_timings();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_enter();
-	target_state->enter_s2idle(dev, drv, index);
+	/*
+	 * Last need_resched() check must come after rcu_idle_enter()
+	 * which may wake up RCU internal tasks.
+	 */
+	if (!need_resched())
+		target_state->enter_s2idle(dev, drv, index);
+	else
+		index = -EBUSY;
 	if (WARN_ON_ONCE(!irqs_disabled()))
 		local_irq_disable();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))

@@ -159,10 +166,13 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,
 	tick_unfreeze();
 	start_critical_timings();
 
-	time_end = ns_to_ktime(local_clock());
+	if (index > 0) {
+		time_end = ns_to_ktime(local_clock());
+		dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start);
+		dev->states_usage[index].s2idle_usage++;
+	}
 
-	dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start);
-	dev->states_usage[index].s2idle_usage++;
+	return index;
 }
 
 /**

@@ -184,7 +194,7 @@ int cpuidle_enter_s2idle(struct cpuidle_driver *drv, struct cpuidle_device *dev)
 	 */
 	index = find_deepest_state(drv, dev, U64_MAX, 0, true);
 	if (index > 0) {
-		enter_s2idle_proper(drv, dev, index);
+		index = enter_s2idle_proper(drv, dev, index);
 		local_irq_enable();
 	}
 	return index;

@@ -234,7 +244,14 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,
 	stop_critical_timings();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_enter();
-	entered_state = target_state->enter(dev, drv, index);
+	/*
+	 * Last need_resched() check must come after rcu_idle_enter()
+	 * which may wake up RCU internal tasks.
+	 */
+	if (!need_resched())
+		entered_state = target_state->enter(dev, drv, index);
+	else
+		entered_state = -EBUSY;
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_exit();
 	start_critical_timings();

Signature · 1 line

2.25.1

Dmitry Osipenko

2021-01-05 17:26 UTC #4 lore raw

Re: [PATCH 2/4] cpuidle: Fix missing need_resched() check after rcu_idle_enter()

22.12.2020 04:37, Frederic Weisbecker пишет:

Quoted 61 lines

Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced.
Usually a wake up happening while running the idle task is spotted in one of the need_resched() checks carefully placed within the idle loop that can break to the scheduler.
Unfortunately within cpuidle the call to rcu_idle_enter() is already beyond the last generic need_resched() check. Some drivers may perform their own checks like with mwait_idle_with_hints() but many others don't and we may halt the CPU with a resched request unhandled, leaving the task hanging.
Fix this with performing a last minute need_resched() check after calling rcu_idle_enter().
Reported-by Paul E. McKenney <[email protected]>
Fixes 1098582a0f6c (sched,idle,rcu: Push rcu_idle deeper into the idle path)
Cc [email protected]
Cc Daniel Lezcano <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Signed-off-by Frederic Weisbecker <[email protected]>
---
 drivers/cpuidle/cpuidle.c | 33 +++++++++++++++++++++++++--------
 1 file changed, 25 insertions(+), 8 deletions(-)
drivers/cpuidle/cpuidle.c +10 -3
index ef2ea1b12cd8..4cc1ba49ce05 100644
@@ -134,8 +134,8 @@ int cpuidle_find_deepest_state(struct cpuidle_driver *drv,
 }
 
 #ifdef CONFIG_SUSPEND
-static void enter_s2idle_proper(struct cpuidle_driver *drv,
-				struct cpuidle_device *dev, int index)
+static int enter_s2idle_proper(struct cpuidle_driver *drv,
+			       struct cpuidle_device *dev, int index)
 {
 	ktime_t time_start, time_end;
 	struct cpuidle_state *target_state = &drv->states[index];
@@ -151,7 +151,14 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,
 	stop_critical_timings();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_enter();
-	target_state->enter_s2idle(dev, drv, index);
+	/*
+	 * Last need_resched() check must come after rcu_idle_enter()
+	 * which may wake up RCU internal tasks.
+	 */
+	if (!need_resched())
+		target_state->enter_s2idle(dev, drv, index);
+	else
+		index = -EBUSY;
 	if (WARN_ON_ONCE(!irqs_disabled()))
 		local_irq_disable();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))

drivers/cpuidle/cpuidle.c static void enter_s2idle_proper(struct cpuidle_driver *drv, · line 169

Quoted 5 lines

drivers/cpuidle/cpuidle.c +1 -1

@@ -159,10 +166,13 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,
 	tick_unfreeze();
 	start_critical_timings();
 
-	time_end = ns_to_ktime(local_clock());
+	if (index > 0) {

index=0 is valid too

Quoted 27 lines

drivers/cpuidle/cpuidle.c +7 -4

+		time_end = ns_to_ktime(local_clock());
+		dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start);
+		dev->states_usage[index].s2idle_usage++;
+	}
 
-	dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start);
-	dev->states_usage[index].s2idle_usage++;
+	return index;
 }
 
 /**

@@ -184,7 +194,7 @@ int cpuidle_enter_s2idle(struct cpuidle_driver *drv, struct cpuidle_device *dev)
 	 */
 	index = find_deepest_state(drv, dev, U64_MAX, 0, true);
 	if (index > 0) {
-		enter_s2idle_proper(drv, dev, index);
+		index = enter_s2idle_proper(drv, dev, index);
 		local_irq_enable();
 	}
 	return index;

@@ -234,7 +244,14 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,
 	stop_critical_timings();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_enter();
-	entered_state = target_state->enter(dev, drv, index);
+	/*

drivers/cpuidle/cpuidle.c int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv, · line 257

Quoted 10 lines

drivers/cpuidle/cpuidle.c +7 -0

+	 * Last need_resched() check must come after rcu_idle_enter()
+	 * which may wake up RCU internal tasks.
+	 */
+	if (!need_resched())
+		entered_state = target_state->enter(dev, drv, index);
+	else
+		entered_state = -EBUSY;
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_exit();
 	start_critical_timings();

This patch causes a hardlock on NVIDIA Tegra using today's linux-next. Disabling coupled idling state helps. Please fix thanks in advance.

Dmitry Osipenko

2021-01-05 18:11 UTC #5 lore raw

Re: [PATCH 2/4] cpuidle: Fix missing need_resched() check after rcu_idle_enter()

05.01.2021 20:25, Dmitry Osipenko пишет:

Quoted 102 lines

22.12.2020 04:37, Frederic Weisbecker пишет:
Quoted 61 lines
Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced.
Usually a wake up happening while running the idle task is spotted in one of the need_resched() checks carefully placed within the idle loop that can break to the scheduler.
Unfortunately within cpuidle the call to rcu_idle_enter() is already beyond the last generic need_resched() check. Some drivers may perform their own checks like with mwait_idle_with_hints() but many others don't and we may halt the CPU with a resched request unhandled, leaving the task hanging.
Fix this with performing a last minute need_resched() check after calling rcu_idle_enter().
Reported-by Paul E. McKenney <[email protected]>
Fixes 1098582a0f6c (sched,idle,rcu: Push rcu_idle deeper into the idle path)
Cc [email protected]
Cc Daniel Lezcano <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Signed-off-by Frederic Weisbecker <[email protected]>
---
 drivers/cpuidle/cpuidle.c | 33 +++++++++++++++++++++++++--------
 1 file changed, 25 insertions(+), 8 deletions(-)
drivers/cpuidle/cpuidle.c +10 -3
index ef2ea1b12cd8..4cc1ba49ce05 100644
@@ -134,8 +134,8 @@ int cpuidle_find_deepest_state(struct cpuidle_driver *drv,
 }
 
 #ifdef CONFIG_SUSPEND
-static void enter_s2idle_proper(struct cpuidle_driver *drv,
-				struct cpuidle_device *dev, int index)
+static int enter_s2idle_proper(struct cpuidle_driver *drv,
+			       struct cpuidle_device *dev, int index)
 {
 	ktime_t time_start, time_end;
 	struct cpuidle_state *target_state = &drv->states[index];
@@ -151,7 +151,14 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,
 	stop_critical_timings();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_enter();
-	target_state->enter_s2idle(dev, drv, index);
+	/*
+	 * Last need_resched() check must come after rcu_idle_enter()
+	 * which may wake up RCU internal tasks.
+	 */
+	if (!need_resched())
+		target_state->enter_s2idle(dev, drv, index);
+	else
+		index = -EBUSY;
 	if (WARN_ON_ONCE(!irqs_disabled()))
 		local_irq_disable();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
drivers/cpuidle/cpuidle.c static void enter_s2idle_proper(struct cpuidle_driver *drv, · line 169
Quoted 5 lines
drivers/cpuidle/cpuidle.c +1 -1
@@ -159,10 +166,13 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,
 	tick_unfreeze();
 	start_critical_timings();
 
-	time_end = ns_to_ktime(local_clock());
+	if (index > 0) {
index=0 is valid too
Quoted 32 lines
drivers/cpuidle/cpuidle.c +13 -4
+		time_end = ns_to_ktime(local_clock());
+		dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start);
+		dev->states_usage[index].s2idle_usage++;
+	}
 
-	dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start);
-	dev->states_usage[index].s2idle_usage++;
+	return index;
 }
 
 /**
@@ -184,7 +194,7 @@ int cpuidle_enter_s2idle(struct cpuidle_driver *drv, struct cpuidle_device *dev)
 	 */
 	index = find_deepest_state(drv, dev, U64_MAX, 0, true);
 	if (index > 0) {
-		enter_s2idle_proper(drv, dev, index);
+		index = enter_s2idle_proper(drv, dev, index);
 		local_irq_enable();
 	}
 	return index;
@@ -234,7 +244,14 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,
 	stop_critical_timings();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_enter();
-	entered_state = target_state->enter(dev, drv, index);
+	/*
+	 * Last need_resched() check must come after rcu_idle_enter()
+	 * which may wake up RCU internal tasks.
+	 */
+	if (!need_resched())
+		entered_state = target_state->enter(dev, drv, index);
+	else

Quoted 9 lines

drivers/cpuidle/cpuidle.c int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv, · line 257
Quoted 5 lines
drivers/cpuidle/cpuidle.c +1 -0
+		entered_state = -EBUSY;
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_exit();
 	start_critical_timings();
This patch causes a hardlock on NVIDIA Tegra using today's linux-next. Disabling coupled idling state helps. Please fix thanks in advance.

This isn't a proper fix, but it works:

(quoted diff) +0 -0

b/drivers/cpuidle/cpuidle-tegra.c
index 191966dc8d02..ecc5d9b31553 100644

drivers/cpuidle/cpuidle-tegra.c +1 -1

@@ -148,7 +148,7 @@ static int tegra_cpuidle_c7_enter(void)
 
 static int tegra_cpuidle_coupled_barrier(struct cpuidle_device *dev)
 {
-	if (tegra_pending_sgi()) {
+	if (tegra_pending_sgi() || need_resched()) {
 		/*
 		 * CPU got local interrupt that will be lost after GIC's
 		 * shutdown because GIC driver doesn't save/restore the

drivers/cpuidle/cpuidle.c +0 -0

index 4cc1ba49ce05..2bc52ccc339b 100644
@@ -248,7 +248,7 @@ int cpuidle_enter_state(struct cpuidle_device *dev,

struct cpuidle_driver *drv,
* Last need_resched() check must come after rcu_idle_enter()
* which may wake up RCU internal tasks.
*/
- if (!need_resched())
+ if ((target_state->flags & CPUIDLE_FLAG_COUPLED) || !need_resched())
entered_state = target_state->enter(dev, drv, index);
else
entered_state = -EBUSY;

Paul E. McKenney

2021-01-05 19:18 UTC #6 lore raw

Re: [PATCH 2/4] cpuidle: Fix missing need_resched() check after rcu_idle_enter()

Quoted 112 lines · Dmitry Osipenko

05.01.2021 20:25, Dmitry Osipenko пишет:
Quoted 102 lines
22.12.2020 04:37, Frederic Weisbecker пишет:
Quoted 61 lines
Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced.
Usually a wake up happening while running the idle task is spotted in one of the need_resched() checks carefully placed within the idle loop that can break to the scheduler.
Unfortunately within cpuidle the call to rcu_idle_enter() is already beyond the last generic need_resched() check. Some drivers may perform their own checks like with mwait_idle_with_hints() but many others don't and we may halt the CPU with a resched request unhandled, leaving the task hanging.
Fix this with performing a last minute need_resched() check after calling rcu_idle_enter().
Reported-by Paul E. McKenney <[email protected]>
Fixes 1098582a0f6c (sched,idle,rcu: Push rcu_idle deeper into the idle path)
Cc [email protected]
Cc Daniel Lezcano <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Signed-off-by Frederic Weisbecker <[email protected]>
---
 drivers/cpuidle/cpuidle.c | 33 +++++++++++++++++++++++++--------
 1 file changed, 25 insertions(+), 8 deletions(-)
drivers/cpuidle/cpuidle.c +10 -3
index ef2ea1b12cd8..4cc1ba49ce05 100644
@@ -134,8 +134,8 @@ int cpuidle_find_deepest_state(struct cpuidle_driver *drv,
 }
 
 #ifdef CONFIG_SUSPEND
-static void enter_s2idle_proper(struct cpuidle_driver *drv,
-				struct cpuidle_device *dev, int index)
+static int enter_s2idle_proper(struct cpuidle_driver *drv,
+			       struct cpuidle_device *dev, int index)
 {
 	ktime_t time_start, time_end;
 	struct cpuidle_state *target_state = &drv->states[index];
@@ -151,7 +151,14 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,
 	stop_critical_timings();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_enter();
-	target_state->enter_s2idle(dev, drv, index);
+	/*
+	 * Last need_resched() check must come after rcu_idle_enter()
+	 * which may wake up RCU internal tasks.
+	 */
+	if (!need_resched())
+		target_state->enter_s2idle(dev, drv, index);
+	else
+		index = -EBUSY;
 	if (WARN_ON_ONCE(!irqs_disabled()))
 		local_irq_disable();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
drivers/cpuidle/cpuidle.c static void enter_s2idle_proper(struct cpuidle_driver *drv, · line 169
Quoted 5 lines
drivers/cpuidle/cpuidle.c +1 -1
@@ -159,10 +166,13 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,
 	tick_unfreeze();
 	start_critical_timings();
 
-	time_end = ns_to_ktime(local_clock());
+	if (index > 0) {
index=0 is valid too
Quoted 32 lines
drivers/cpuidle/cpuidle.c +13 -4
+		time_end = ns_to_ktime(local_clock());
+		dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start);
+		dev->states_usage[index].s2idle_usage++;
+	}
 
-	dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start);
-	dev->states_usage[index].s2idle_usage++;
+	return index;
 }
 
 /**
@@ -184,7 +194,7 @@ int cpuidle_enter_s2idle(struct cpuidle_driver *drv, struct cpuidle_device *dev)
 	 */
 	index = find_deepest_state(drv, dev, U64_MAX, 0, true);
 	if (index > 0) {
-		enter_s2idle_proper(drv, dev, index);
+		index = enter_s2idle_proper(drv, dev, index);
 		local_irq_enable();
 	}
 	return index;
@@ -234,7 +244,14 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,
 	stop_critical_timings();
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_enter();
-	entered_state = target_state->enter(dev, drv, index);
+	/*
+	 * Last need_resched() check must come after rcu_idle_enter()
+	 * which may wake up RCU internal tasks.
+	 */
+	if (!need_resched())
+		entered_state = target_state->enter(dev, drv, index);
+	else
Quoted 8 lines
drivers/cpuidle/cpuidle.c int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv, · line 257
Quoted 5 lines
drivers/cpuidle/cpuidle.c +1 -0
+		entered_state = -EBUSY;
 	if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE))
 		rcu_idle_exit();
 	start_critical_timings();
This patch causes a hardlock on NVIDIA Tegra using today's linux-next. Disabling coupled idling state helps. Please fix thanks in advance.

Quoted 1 line

This isn't a proper fix, but it works:

There is some ongoing discussion about what an overall proper fix might look like, so in the meantime I am folding you changes below into Frederic's original. ;-)

Thanx, Paul

Quoted 29 lines

drivers/cpuidle/cpuidle-tegra.c +1 -1

index 191966dc8d02..ecc5d9b31553 100644
@@ -148,7 +148,7 @@ static int tegra_cpuidle_c7_enter(void)
 
 static int tegra_cpuidle_coupled_barrier(struct cpuidle_device *dev)
 {
-	if (tegra_pending_sgi()) {
+	if (tegra_pending_sgi() || need_resched()) {
 		/*
 		 * CPU got local interrupt that will be lost after GIC's
 		 * shutdown because GIC driver doesn't save/restore the

drivers/cpuidle/cpuidle.c +1 -1

index 4cc1ba49ce05..2bc52ccc339b 100644
@@ -248,7 +248,7 @@ int cpuidle_enter_state(struct cpuidle_device *dev,
 struct cpuidle_driver *drv,
 	 * Last need_resched() check must come after rcu_idle_enter()
 	 * which may wake up RCU internal tasks.
 	 */
-	if (!need_resched())
+	if ((target_state->flags & CPUIDLE_FLAG_COUPLED) || !need_resched())
 		entered_state = target_state->enter(dev, drv, index);
 	else

entered_state = -EBUSY;

Frederic Weisbecker

author

2020-12-22 01:38 UTC #7 lore raw

PATCH 3/4 ARM: imx6q: Fix missing need_resched() check after rcu_idle_enter()

In reply to Frederic Weisbecker

Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced.

Usually a wake up happening while running the idle task is spotted in one of the need_resched() checks carefully placed within the idle loop that can break to the scheduler.

Unfortunately imx6q_enter_wait() is beyond the last generic need_resched() check and it performs a wfi right away after the call to rcu_idle_enter(). We may halt the CPU with a resched request unhandled, leaving the task hanging.

Fix this with performing a last minute need_resched() check after calling rcu_idle_enter().

Reported-by Paul E. McKenney <[email protected]>

Fixes 1a67b9263e06 (ARM: imx6q: Fixup RCU usage for cpuidle)

Cc [email protected]

Cc Shawn Guo <[email protected]>

Cc Sascha Hauer <[email protected]>

Cc Pengutronix Kernel Team <[email protected]>

Cc Fabio Estevam <[email protected]>

Cc NXP Linux Team <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Rafael J. Wysocki <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Signed-off-by Frederic Weisbecker <[email protected]>

---
 arch/arm/mach-imx/cpuidle-imx6q.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

arch/arm/mach-imx/cpuidle-imx6q.c +6 -1

index 094337dc1bc7..31a60d257d3d 100644
@@ -25,7 +25,12 @@ static int imx6q_enter_wait(struct cpuidle_device *dev,
 	raw_spin_unlock(&cpuidle_lock);
 
 	rcu_idle_enter();
-	cpu_do_idle();
+	/*
+	 * Last need_resched() check must come after rcu_idle_enter()
+	 * which may wake up RCU internal tasks.
+	 */
+	if (!need_resched())
+		cpu_do_idle();
 	rcu_idle_exit();
 
 	raw_spin_lock(&cpuidle_lock);

Signature · 1 line

2.25.1

Frederic Weisbecker

author

2020-12-22 01:38 UTC #8 lore raw

PATCH 4/4 ACPI: processor: Fix missing need_resched() check after rcu_idle_enter()

In reply to Frederic Weisbecker

Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced.

Usually a wake up happening while running the idle task is spotted in one of the need_resched() checks carefully placed within the idle loop that can break to the scheduler.

Unfortunately within acpi_idle_enter_bm() the call to rcu_idle_enter() is already beyond the last generic need_resched() check. The cpu idle implementation happens to be ok because it ends up calling mwait_idle_with_hints() or acpi_safe_halt() which both perform their own need_resched() checks. But the suspend to idle implementation doesn't so it may suspend the CPU with a resched request unhandled, leaving the task hanging.

Fix this with performing a last minute need_resched() check after calling rcu_idle_enter().

Reported-by Paul E. McKenney <[email protected]>

Fixes 1fecfdbb7acc (ACPI: processor: Take over RCU-idle for C3-BM idle)

Cc [email protected]

Cc Len Brown <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Rafael J. Wysocki <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar<[email protected]>

Signed-off-by Frederic Weisbecker <[email protected]>

---
 drivers/acpi/processor_idle.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

drivers/acpi/processor_idle.c +8 -2

index d93e400940a3..c4939c49d972 100644
@@ -604,8 +604,14 @@ static int acpi_idle_enter_bm(struct cpuidle_driver *drv,
 	}
 
 	rcu_idle_enter();
-
-	acpi_idle_do_entry(cx);
+	/*
+	 * Last need_resched() check must come after rcu_idle_enter()
+	 * which may wake up RCU internal tasks. mwait_idle_with_hints()
+	 * and acpi_safe_halt() have their own checks but s2idle
+	 * implementation doesn't.
+	 */
+	if (!need_resched())
+		acpi_idle_do_entry(cx);
 
 	rcu_idle_exit();

Signature · 1 line

2.25.1

Rafael J. Wysocki

Reviewed-by on the v1 series

2020-12-22 16:21 UTC #9 lore raw

In reply to Frederic Weisbecker

Quoted 39 lines · Frederic Weisbecker

With Paul, we've been thinking that the idle loop wasn't twisted enough yet to deserve 2020.
rcutorture, after some recent parameter changes, has been complaining about a hung task.
It appears that rcu_idle_enter() may wake up a NOCB kthread but this happens after the last generic need_resched() check. Some cpuidle drivers fix it by chance but many others don't.
Here is a proposed bunch of fixes. I will need to also fix the rcu_user_enter() case, likely using irq_work, since nohz_full requires irq work to support self IPI.
Also more generally, this raise the question of local task wake_up() under disabled interrupts. When a wake up occurs in a preempt disabled section, it gets handled by the outer preempt_enable() call. There is no similar mechanism when a wake up occurs with interrupts disabled. I guess it is assumed to be handled, at worst, in the next tick. But a local irq work would provide instant preemption once IRQs are re-enabled. Of course this would only make sense in CONFIG_PREEMPTION, and when the tick is disabled...
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
sched/idle
HEAD: f2fa6e4a070c1535b9edc9ee097167fd2b15d235
Thanks,
Frederic
---
Frederic Weisbecker (4):
sched/idle: Fix missing need_resched() check after rcu_idle_enter()
cpuidle: Fix missing need_resched() check after rcu_idle_enter()
ARM: imx6q: Fix missing need_resched() check after rcu_idle_enter()
ACPI: processor: Fix missing need_resched() check after rcu_idle_enter()

Quoted 5 lines

  arch/arm/mach-imx/cpuidle-imx6q.c |  7 ++++++-
  drivers/acpi/processor_idle.c     | 10 ++++++++--
  drivers/cpuidle/cpuidle.c         | 33 +++++++++++++++++++++++++--------
  kernel/sched/idle.c               | 18 ++++++++++++------
  4 files changed, 51 insertions(+), 17 deletions(-)

Please feel free to add

Reviewed-by Rafael J. Wysocki <[email protected]>

to all patches in the series.

Thanks!

Paul E. McKenney

2021-01-01 16:06 UTC #10 lore raw

Quoted 46 lines · Rafael J. Wysocki

Quoted 39 lines · Frederic Weisbecker
With Paul, we've been thinking that the idle loop wasn't twisted enough yet to deserve 2020.
rcutorture, after some recent parameter changes, has been complaining about a hung task.
It appears that rcu_idle_enter() may wake up a NOCB kthread but this happens after the last generic need_resched() check. Some cpuidle drivers fix it by chance but many others don't.
Here is a proposed bunch of fixes. I will need to also fix the rcu_user_enter() case, likely using irq_work, since nohz_full requires irq work to support self IPI.
Also more generally, this raise the question of local task wake_up() under disabled interrupts. When a wake up occurs in a preempt disabled section, it gets handled by the outer preempt_enable() call. There is no similar mechanism when a wake up occurs with interrupts disabled. I guess it is assumed to be handled, at worst, in the next tick. But a local irq work would provide instant preemption once IRQs are re-enabled. Of course this would only make sense in CONFIG_PREEMPTION, and when the tick is disabled...
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
sched/idle
HEAD: f2fa6e4a070c1535b9edc9ee097167fd2b15d235
Thanks,
Frederic
---
Frederic Weisbecker (4):
sched/idle: Fix missing need_resched() check after rcu_idle_enter()
cpuidle: Fix missing need_resched() check after rcu_idle_enter()
ARM: imx6q: Fix missing need_resched() check after rcu_idle_enter()
ACPI: processor: Fix missing need_resched() check after rcu_idle_enter()
Quoted 5 lines
  arch/arm/mach-imx/cpuidle-imx6q.c |  7 ++++++-
  drivers/acpi/processor_idle.c     | 10 ++++++++--
  drivers/cpuidle/cpuidle.c         | 33 +++++++++++++++++++++++++--------
  kernel/sched/idle.c               | 18 ++++++++++++------
  4 files changed, 51 insertions(+), 17 deletions(-)

Quoted 5 lines

Please feel free to add
Reviewed-by Rafael J. Wysocki <[email protected]>
to all patches in the series.

I would guess that they will take some other path to mainline, but I have queued these to cut down on rcutorture's whining. ;-)

Thanx, Paul