Skip to contentSkip to conversation

timers/cpuidle: Fixes and cleanups

PATCH In review Frederic Weisbecker 4 participants 24 messages Aug 11, 2023 – Aug 29, 2023

Subsystems (6)
x86 ACPI Scheduler LoongArch +2 more subsystems
lore
Share & tools
lore
#0 lore raw

cover timers/cpuidle: Fixes and cleanups

Hi,

The first part of the series is cpuidle callback fixes against timers, some of which haven't been Signed by Peter yet.

Another part is removing the overhead of useless TIF_NR_POLLING clearing.

The rest is comments.

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
timers/idle

HEAD: b66dd12bb29cca558b9323f2b270a7dae8f56c48

Thanks,
Frederic
---

Frederic Weisbecker (8):
x86: Add a comment about the "magic" behind shadow sti before mwait
cpuidle: Report illegal tick stopped while polling
cpuidle: Comment about timers requirements VS idle handler
cpuidle: Remove unnecessary current_clr_polling_and_test() from haltpoll
cpuidle: Remove unnecessary current_clr_polling() on poll_idle()
x86: Remove __current_clr_polling() from mwait_idle()
x86: Remove the current_clr_polling() call upon mwait exit sched/timers: Explain why idle task schedules out on remote timer enqueue

Peter Zijlstra (2):
cpuidle: Fix CPUIDLE_FLAG_IRQ_ENABLE leaking timer reprogram
loongson: Fix idle VS timer enqueue

 arch/loongarch/kernel/genex.S      | 32 ++++++++++++++++++--------------
 arch/loongarch/kernel/idle.c       |  1 -
 arch/x86/include/asm/mwait.h       | 21 ++++++++++++++++++---
 arch/x86/kernel/process.c          |  1 -
 drivers/acpi/acpi_pad.c            |  1 +
 drivers/cpuidle/cpuidle-haltpoll.c |  5 +----
 drivers/cpuidle/poll_state.c       | 10 ++++++++--
 drivers/idle/intel_idle.c          | 19 +++++++------------
 kernel/sched/core.c                | 22 ++++++++++++++++++++++
 kernel/sched/idle.c                | 30 ++++++++++++++++++++++++++++++
 10 files changed, 105 insertions(+), 37 deletions(-)
#1 lore raw

PATCH 1/10 x86: Add a comment about the "magic" behind shadow sti before mwait

Add a note to make sure we never miss and break the requirements behind it.

---
 arch/x86/include/asm/mwait.h | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/x86/include/asm/mwait.h +9 -0
index 778df05f8539..341ee4f1d91e 100644
@@ -87,6 +87,15 @@ static __always_inline void __mwaitx(unsigned long eax, unsigned long ebx,
 		     :: "a" (eax), "b" (ebx), "c" (ecx));
 }
 
+/*
+ * Re-enable interrupts right upon calling mwait in such a way that
+ * no interrupt can fire _before_ the execution of mwait, ie: no
+ * instruction must be placed between "sti" and "mwait".
+ *
+ * This is necessary because if an interrupt queues a timer before
+ * executing mwait, it would otherwise go unnoticed and the next tick
+ * would not be reprogrammed accordingly before mwait ever wakes up.
+ */
 static __always_inline void __sti_mwait(unsigned long eax, unsigned long ecx)
 {
 	mds_idle_clear_cpu_buffers();
Signature · 1 line
2.34.1
maintainer Acked-by on v1 1/10
#2 lore raw

Re: [PATCH 01/10] x86: Add a comment about the "magic" behind shadow sti before mwait

Quoted 5 lines · Frederic Weisbecker

Add a note to make sure we never miss and break the requirements behind it.

Quoted 27 lines
---
 arch/x86/include/asm/mwait.h | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/x86/include/asm/mwait.h +9 -1
index 778df05f8539..341ee4f1d91e 100644
@@ -87,6 +87,15 @@ static __always_inline void __mwaitx(unsigned long eax, unsigned long ebx,
                     :: "a" (eax), "b" (ebx), "c" (ecx));
 }
 
+/*
+ * Re-enable interrupts right upon calling mwait in such a way that
+ * no interrupt can fire _before_ the execution of mwait, ie: no
+ * instruction must be placed between "sti" and "mwait".
+ *
+ * This is necessary because if an interrupt queues a timer before
+ * executing mwait, it would otherwise go unnoticed and the next tick
+ * would not be reprogrammed accordingly before mwait ever wakes up.
+ */
 static __always_inline void __sti_mwait(unsigned long eax, unsigned long ecx)
 {
        mds_idle_clear_cpu_buffers();
--

2.34.1

#3 lore raw

PATCH 2/10 cpuidle: Fix CPUIDLE_FLAG_IRQ_ENABLE leaking timer reprogram

In reply to Frederic Weisbecker

From: Peter Zijlstra <[email protected]>

intel_idle_irq() re-enables IRQs very early. As a result, an interrupt may fire before mwait() is eventually called. If such an interrupt queues a timer, it may go unnoticed until mwait returns and the idle loop handles the tick re-evaluation. And monitoring TIF_NEED_RESCHED doesn't help because a local timer enqueue doesn't set that flag.

The issue is mitigated by the fact that this idle handler is only invoked for shallow C-states when, presumably, the next tick is supposed to be close enough. There may still be rare cases though when the next tick is far away and the selected C-state is shallow, resulting in a timer getting ignored for a while.

Fix this with using sti_mwait() whose IRQ-reenablement only triggers upon calling mwait(), dealing with the race while keeping the interrupt latency within acceptable bounds.

Fixes c227233ad64c (intel_idle: enable interrupts before C1 on Xeons)
---
 arch/x86/include/asm/mwait.h | 11 +++++++++--
 drivers/idle/intel_idle.c    | 19 +++++++------------
 2 files changed, 16 insertions(+), 14 deletions(-)
arch/x86/include/asm/mwait.h +9 -2
index 341ee4f1d91e..920426d691ce 100644
@@ -124,8 +124,15 @@ static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned lo
 		}
 
 		__monitor((void *)&current_thread_info()->flags, 0, 0);
-		if (!need_resched())
-			__mwait(eax, ecx);
+
+		if (!need_resched()) {
+			if (ecx & 1) {
+				__mwait(eax, ecx);
+			} else {
+				__sti_mwait(eax, ecx);
+				raw_local_irq_disable();
+			}
+		}
 	}
 	current_clr_polling();
 }
drivers/idle/intel_idle.c +7 -12
index 256c2d42e350..d676d32741da 100644
@@ -131,11 +131,12 @@ static unsigned int mwait_substates __initdata;
 #define MWAIT2flg(eax) ((eax & 0xFF) << 24)
 
 static __always_inline int __intel_idle(struct cpuidle_device *dev,
-					struct cpuidle_driver *drv, int index)
+					struct cpuidle_driver *drv,
+					int index, bool irqoff)
 {
 	struct cpuidle_state *state = &drv->states[index];
 	unsigned long eax = flg2MWAIT(state->flags);
-	unsigned long ecx = 1; /* break on interrupt flag */
+	unsigned long ecx = 1*irqoff; /* break on interrupt flag */
 
 	mwait_idle_with_hints(eax, ecx);
 
@@ -159,19 +160,13 @@ static __always_inline int __intel_idle(struct cpuidle_device *dev,
 static __cpuidle int intel_idle(struct cpuidle_device *dev,
 				struct cpuidle_driver *drv, int index)
 {
-	return __intel_idle(dev, drv, index);
+	return __intel_idle(dev, drv, index, true);
 }
 
 static __cpuidle int intel_idle_irq(struct cpuidle_device *dev,
 				    struct cpuidle_driver *drv, int index)
 {
-	int ret;
-
-	raw_local_irq_enable();
-	ret = __intel_idle(dev, drv, index);
-	raw_local_irq_disable();
-
-	return ret;
+	return __intel_idle(dev, drv, index, false);
 }
 
 static __cpuidle int intel_idle_ibrs(struct cpuidle_device *dev,
@@ -184,7 +179,7 @@ static __cpuidle int intel_idle_ibrs(struct cpuidle_device *dev,
 	if (smt_active)
 		native_wrmsrl(MSR_IA32_SPEC_CTRL, 0);
 
-	ret = __intel_idle(dev, drv, index);
+	ret = __intel_idle(dev, drv, index, true);
 
 	if (smt_active)
 		native_wrmsrl(MSR_IA32_SPEC_CTRL, spec_ctrl);
@@ -196,7 +191,7 @@ static __cpuidle int intel_idle_xstate(struct cpuidle_device *dev,
 				       struct cpuidle_driver *drv, int index)
 {
 	fpu_idle_fpregs();
-	return __intel_idle(dev, drv, index);
+	return __intel_idle(dev, drv, index, true);
 }
 
 /**
Signature · 1 line
2.34.1
maintainer Acked-by on v1 2/10
#4 lore raw

Re: [PATCH 02/10] cpuidle: Fix CPUIDLE_FLAG_IRQ_ENABLE leaking timer reprogram

Quoted 15 lines · Frederic Weisbecker

From: Peter Zijlstra <[email protected]>

intel_idle_irq() re-enables IRQs very early. As a result, an interrupt may fire before mwait() is eventually called. If such an interrupt queues a timer, it may go unnoticed until mwait returns and the idle loop handles the tick re-evaluation. And monitoring TIF_NEED_RESCHED doesn't help because a local timer enqueue doesn't set that flag.

The issue is mitigated by the fact that this idle handler is only invoked for shallow C-states when, presumably, the next tick is supposed to be close enough. There may still be rare cases though when the next tick is far away and the selected C-state is shallow, resulting in a timer getting ignored for a while.

Quoted 7 lines

Fix this with using sti_mwait() whose IRQ-reenablement only triggers upon calling mwait(), dealing with the race while keeping the interrupt latency within acceptable bounds.

Fixes c227233ad64c (intel_idle: enable interrupts before C1 on Xeons)
Quoted 89 lines
---
 arch/x86/include/asm/mwait.h | 11 +++++++++--
 drivers/idle/intel_idle.c    | 19 +++++++------------
 2 files changed, 16 insertions(+), 14 deletions(-)
arch/x86/include/asm/mwait.h +9 -2
index 341ee4f1d91e..920426d691ce 100644
@@ -124,8 +124,15 @@ static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned lo
                }
 
                __monitor((void *)&current_thread_info()->flags, 0, 0);
-               if (!need_resched())
-                       __mwait(eax, ecx);
+
+               if (!need_resched()) {
+                       if (ecx & 1) {
+                               __mwait(eax, ecx);
+                       } else {
+                               __sti_mwait(eax, ecx);
+                               raw_local_irq_disable();
+                       }
+               }
        }
        current_clr_polling();
 }
drivers/idle/intel_idle.c +7 -13
index 256c2d42e350..d676d32741da 100644
@@ -131,11 +131,12 @@ static unsigned int mwait_substates __initdata;
 #define MWAIT2flg(eax) ((eax & 0xFF) << 24)
 
 static __always_inline int __intel_idle(struct cpuidle_device *dev,
-                                       struct cpuidle_driver *drv, int index)
+                                       struct cpuidle_driver *drv,
+                                       int index, bool irqoff)
 {
        struct cpuidle_state *state = &drv->states[index];
        unsigned long eax = flg2MWAIT(state->flags);
-       unsigned long ecx = 1; /* break on interrupt flag */
+       unsigned long ecx = 1*irqoff; /* break on interrupt flag */
 
        mwait_idle_with_hints(eax, ecx);
 
@@ -159,19 +160,13 @@ static __always_inline int __intel_idle(struct cpuidle_device *dev,
 static __cpuidle int intel_idle(struct cpuidle_device *dev,
                                struct cpuidle_driver *drv, int index)
 {
-       return __intel_idle(dev, drv, index);
+       return __intel_idle(dev, drv, index, true);
 }
 
 static __cpuidle int intel_idle_irq(struct cpuidle_device *dev,
                                    struct cpuidle_driver *drv, int index)
 {
-       int ret;
-
-       raw_local_irq_enable();
-       ret = __intel_idle(dev, drv, index);
-       raw_local_irq_disable();
-
-       return ret;
+       return __intel_idle(dev, drv, index, false);
 }
 
 static __cpuidle int intel_idle_ibrs(struct cpuidle_device *dev,
@@ -184,7 +179,7 @@ static __cpuidle int intel_idle_ibrs(struct cpuidle_device *dev,
        if (smt_active)
                native_wrmsrl(MSR_IA32_SPEC_CTRL, 0);
 
-       ret = __intel_idle(dev, drv, index);
+       ret = __intel_idle(dev, drv, index, true);
 
        if (smt_active)
                native_wrmsrl(MSR_IA32_SPEC_CTRL, spec_ctrl);
@@ -196,7 +191,7 @@ static __cpuidle int intel_idle_xstate(struct cpuidle_device *dev,
                                       struct cpuidle_driver *drv, int index)
 {
        fpu_idle_fpregs();
-       return __intel_idle(dev, drv, index);
+       return __intel_idle(dev, drv, index, true);
 }
 
 /**
--

2.34.1

#5 lore raw

PATCH 3/10 cpuidle: Report illegal tick stopped while polling

In reply to Frederic Weisbecker

poll_idle() can't be called while the tick is stopped because it enables interrupts and only polls on TIF_NEED_RESCHED, which doesn't tell if an interrupt queues a timer that would require a tick re-programming.

There is no point anyway to poll with the tick stopped so add a check to make sure it never happens.

---
 drivers/cpuidle/poll_state.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/cpuidle/poll_state.c +8 -0
index 9b6d90a72601..009f46f121ae 100644
@@ -7,6 +7,7 @@
 #include <linux/sched.h>
 #include <linux/sched/clock.h>
 #include <linux/sched/idle.h>
+#include <linux/tick.h>
 
 #define POLL_IDLE_RELAX_COUNT	200
 
@@ -19,6 +20,13 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,
 
 	dev->poll_time_limit = false;
 
+	/*
+	 * This re-enables IRQs and only polls on TIF_NEED_RESCHED.
+	 * A timer queued by an interrupt here may go unnoticed if
+	 * the tick is stopped.
+	 */
+	WARN_ON_ONCE(tick_nohz_tick_stopped());
+
 	raw_local_irq_enable();
 	if (!current_set_polling_and_test()) {
 		unsigned int loop_count = 0;
Signature · 1 line
2.34.1
#6 lore raw

Re: [PATCH 03/10] cpuidle: Report illegal tick stopped while polling

Quoted 7 lines · Frederic Weisbecker

poll_idle() can't be called while the tick is stopped because it enables interrupts and only polls on TIF_NEED_RESCHED, which doesn't tell if an interrupt queues a timer that would require a tick re-programming.

There is no point anyway to poll with the tick stopped so add a check to make sure it never happens.

I'd rather update governors so they never use polling states when the tick has been stopped and then add the WARN_ON().

Quoted 34 lines
---
 drivers/cpuidle/poll_state.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/cpuidle/poll_state.c +8 -1
index 9b6d90a72601..009f46f121ae 100644
@@ -7,6 +7,7 @@
 #include <linux/sched.h>
 #include <linux/sched/clock.h>
 #include <linux/sched/idle.h>
+#include <linux/tick.h>
 
 #define POLL_IDLE_RELAX_COUNT  200
 
@@ -19,6 +20,13 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,
 
        dev->poll_time_limit = false;
 
+       /*
+        * This re-enables IRQs and only polls on TIF_NEED_RESCHED.
+        * A timer queued by an interrupt here may go unnoticed if
+        * the tick is stopped.
+        */
+       WARN_ON_ONCE(tick_nohz_tick_stopped());
+
        raw_local_irq_enable();
        if (!current_set_polling_and_test()) {
                unsigned int loop_count = 0;
--

2.34.1

#7 lore raw

Re: [PATCH 03/10] cpuidle: Report illegal tick stopped while polling

Quoted 11 lines · Rafael J. Wysocki
Quoted 7 lines · Frederic Weisbecker

poll_idle() can't be called while the tick is stopped because it enables interrupts and only polls on TIF_NEED_RESCHED, which doesn't tell if an interrupt queues a timer that would require a tick re-programming.

There is no point anyway to poll with the tick stopped so add a check to make sure it never happens.

I'd rather update governors so they never use polling states when the tick has been stopped and then add the WARN_ON().

Sounds good, let's put this one aside.

In the meantime feel free to pick up those you acked and see fit in your queue.

Thanks!

#8 lore raw

PATCH 4/10 loongson: Fix idle VS timer enqueue

In reply to Frederic Weisbecker

From: Peter Zijlstra <[email protected]>

Loongson re-enables interrupts on its idle routine and performs a TIF_NEED_RESCHED check afterwards before putting the CPU to sleep.

The IRQs firing between the check and the idling instruction may set the TIF_NEED_RESCHED flag. In order to deal with the such a race, IRQs interrupting __arch_cpu_idle() rollback their return address to the beginning of __arch_cpu_idle() so that TIF_NEED_RESCHED is checked again before going back to sleep.

However idle IRQs can also queue timers that may require a tick reprogramming through a new generic idle loop iteration but those timers would go unnoticed here because __arch_cpu_idle() only checks TIF_NEED_RESCHED. It doesn't check for pending timers.

Fix this with fast-forwarding idle IRQs return value to the end of the idle routine instead of the beginning, so that the generic idle loop handles both TIF_NEED_RESCHED and pending timers.

Fixes 0603839b18f4 (LoongArch: Add exception/interrupt handling)
Cc WANG Xuerui <[email protected]>
---
 arch/loongarch/kernel/genex.S | 32 ++++++++++++++++++--------------
 arch/loongarch/kernel/idle.c  |  1 -
 2 files changed, 18 insertions(+), 15 deletions(-)
arch/loongarch/kernel/genex.S +18 -14
index 78f066384657..359d693f112e 100644
@@ -18,27 +18,31 @@
 
 	.align	5
 SYM_FUNC_START(__arch_cpu_idle)
-	/* start of rollback region */
-	LONG_L	t0, tp, TI_FLAGS
-	nop
-	andi	t0, t0, _TIF_NEED_RESCHED
-	bnez	t0, 1f
-	nop
-	nop
-	nop
+	/* start of idle interrupt region */
+	move	t0, CSR_CRMD_IE
+	csrxchg	t0, t0, LOONGARCH_CSR_CRMD
+	/*
+	 * If an interrupt lands here; between enabling interrupts above and
+	 * going idle on the next instruction, we must *NOT* go idle since the
+	 * interrupt could have set TIF_NEED_RESCHED or caused an timer to need
+	 * reprogramming. Fall through -- see handle_vint() below -- and have
+	 * the idle loop take care of things.
+	 */
 	idle	0
-	/* end of rollback region */
-1:	jr	ra
+	nop
+	/* end of idle interrupt region */
+SYM_INNER_LABEL(__arch_cpu_idle_exit, SYM_L_LOCAL)
+	jr	ra
 SYM_FUNC_END(__arch_cpu_idle)
 
 SYM_FUNC_START(handle_vint)
 	BACKUP_T0T1
 	SAVE_ALL
-	la_abs	t1, __arch_cpu_idle
+	la_abs	t1, __arch_cpu_idle_exit
 	LONG_L	t0, sp, PT_ERA
-	/* 32 byte rollback region */
-	ori	t0, t0, 0x1f
-	xori	t0, t0, 0x1f
+	/* 16 byte idle interrupt region */
+	ori	t0, t0, 0x0f
+	addi.d	t0, t0, 1
 	bne	t0, t1, 1f
 	LONG_S	t0, sp, PT_ERA
 1:	move	a0, sp
arch/loongarch/kernel/idle.c +0 -1
index 0b5dd2faeb90..5ba72d229920 100644
@@ -11,7 +11,6 @@
 
 void __cpuidle arch_cpu_idle(void)
 {
-	raw_local_irq_enable();
 	__arch_cpu_idle(); /* idle instruction needs irq enabled */
 	raw_local_irq_disable();
 }
Signature · 1 line
2.34.1
#9 lore raw

Re: [PATCH 04/10] loongson: Fix idle VS timer enqueue

+Add huacai

asm instruction move should be replaced with li.w, the other looks good to me.

arch/loongarch/kernel/genex.S +1 -1
index 359d693f112e..8a98023ecafd 100644
@@ -19,7 +19,7 @@
        .align  5
 SYM_FUNC_START(__arch_cpu_idle)
        /* start of idle interrupt region */
-       move    t0, CSR_CRMD_IE
+       li.w    t0, CSR_CRMD_IE
        csrxchg t0, t0, LOONGARCH_CSR_CRMD
        /*
         * If an interrupt lands here; between enabling interrupts above and

By the way __arch_cpu_idle is called by machine_halt/machine_power_off etc, irq will be enabled with new patch. Need we add something like this?

arch/loongarch/kernel/reset.c +14 -9
index 1ef8c6383535..9ecd42c0c804 100644
@@ -20,6 +20,11 @@
 void (*pm_power_off)(void);
 EXPORT_SYMBOL(pm_power_off);
 
+static __always_inline void native_halt(void)
+{
+       asm volatile("idle 0": : :"memory");
+}
+
 void machine_halt(void)
 {
 #ifdef CONFIG_SMP
@@ -32,9 +37,9 @@ void machine_halt(void)
        pr_notice("\n\n** You can safely turn off the power now **\n\n");
        console_flush_on_panic(CONSOLE_FLUSH_PENDING);
 
-       while (true) {
-               __arch_cpu_idle();
-       }
+       while (1) {
+               native_halt();
+       };
 }
 
 void machine_power_off(void)
@@ -52,9 +57,9 @@ void machine_power_off(void)
        efi.reset_system(EFI_RESET_SHUTDOWN, EFI_SUCCESS, 0, NULL);
 #endif
 
-       while (true) {
-               __arch_cpu_idle();
-       }
+       while (1) {
+               native_halt();
+       };
 }
 
 void machine_restart(char *command)
@@ -73,7 +78,7 @@ void machine_restart(char *command)
        if (!acpi_disabled)
                acpi_reboot();
 
-       while (true) {
-               __arch_cpu_idle();
-       }
+       while (1) {
+               native_halt();
+       };
 }

Regards
Bibo Mao

Quoted 92 lines · Frederic Weisbecker

From: Peter Zijlstra <[email protected]>

Loongson re-enables interrupts on its idle routine and performs a TIF_NEED_RESCHED check afterwards before putting the CPU to sleep.

The IRQs firing between the check and the idling instruction may set the TIF_NEED_RESCHED flag. In order to deal with the such a race, IRQs interrupting __arch_cpu_idle() rollback their return address to the beginning of __arch_cpu_idle() so that TIF_NEED_RESCHED is checked again before going back to sleep.

However idle IRQs can also queue timers that may require a tick reprogramming through a new generic idle loop iteration but those timers would go unnoticed here because __arch_cpu_idle() only checks TIF_NEED_RESCHED. It doesn't check for pending timers.

Fix this with fast-forwarding idle IRQs return value to the end of the idle routine instead of the beginning, so that the generic idle loop handles both TIF_NEED_RESCHED and pending timers.

Fixes 0603839b18f4 (LoongArch: Add exception/interrupt handling)
Cc WANG Xuerui <[email protected]>
---
 arch/loongarch/kernel/genex.S | 32 ++++++++++++++++++--------------
 arch/loongarch/kernel/idle.c  |  1 -
 2 files changed, 18 insertions(+), 15 deletions(-)
arch/loongarch/kernel/genex.S +18 -14
index 78f066384657..359d693f112e 100644
@@ -18,27 +18,31 @@
 
 	.align	5
 SYM_FUNC_START(__arch_cpu_idle)
-	/* start of rollback region */
-	LONG_L	t0, tp, TI_FLAGS
-	nop
-	andi	t0, t0, _TIF_NEED_RESCHED
-	bnez	t0, 1f
-	nop
-	nop
-	nop
+	/* start of idle interrupt region */
+	move	t0, CSR_CRMD_IE
+	csrxchg	t0, t0, LOONGARCH_CSR_CRMD
+	/*
+	 * If an interrupt lands here; between enabling interrupts above and
+	 * going idle on the next instruction, we must *NOT* go idle since the
+	 * interrupt could have set TIF_NEED_RESCHED or caused an timer to need
+	 * reprogramming. Fall through -- see handle_vint() below -- and have
+	 * the idle loop take care of things.
+	 */
 	idle	0
-	/* end of rollback region */
-1:	jr	ra
+	nop
+	/* end of idle interrupt region */
+SYM_INNER_LABEL(__arch_cpu_idle_exit, SYM_L_LOCAL)
+	jr	ra
 SYM_FUNC_END(__arch_cpu_idle)
 
 SYM_FUNC_START(handle_vint)
 	BACKUP_T0T1
 	SAVE_ALL
-	la_abs	t1, __arch_cpu_idle
+	la_abs	t1, __arch_cpu_idle_exit
 	LONG_L	t0, sp, PT_ERA
-	/* 32 byte rollback region */
-	ori	t0, t0, 0x1f
-	xori	t0, t0, 0x1f
+	/* 16 byte idle interrupt region */
+	ori	t0, t0, 0x0f
+	addi.d	t0, t0, 1
 	bne	t0, t1, 1f
 	LONG_S	t0, sp, PT_ERA
 1:	move	a0, sp
arch/loongarch/kernel/idle.c +0 -1
index 0b5dd2faeb90..5ba72d229920 100644
@@ -11,7 +11,6 @@
 
 void __cpuidle arch_cpu_idle(void)
 {
-	raw_local_irq_enable();
 	__arch_cpu_idle(); /* idle instruction needs irq enabled */
 	raw_local_irq_disable();
 }
#10 lore raw

PATCH 5/10 cpuidle: Comment about timers requirements VS idle handler

In reply to Frederic Weisbecker
---
 kernel/sched/idle.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
kernel/sched/idle.c +30 -0
index 342f58a329f5..d52f6e3e3854 100644
@@ -258,6 +258,36 @@ static void do_idle(void)
 	while (!need_resched()) {
 		rmb();
 
+		/*
+		 * Interrupts shouldn't be re-enabled from that point on until
+		 * the CPU sleeping instruction is reached. Otherwise an interrupt
+		 * may fire and queue a timer that would be ignored until the CPU
+		 * wakes from the sleeping instruction. And testing need_resched()
+		 * doesn't tell about pending needed timer reprogram.
+		 *
+		 * Several cases to consider:
+		 *
+		 * - SLEEP-UNTIL-PENDING-INTERRUPT based instructions such as
+		 *   "wfi" or "mwait" are fine because they can be entered with
+		 *   interrupt disabled.
+		 *
+		 * - sti;mwait() couple is fine because the interrupts are
+		 *   re-enabled only upon the execution of mwait, leaving no gap
+		 *   in-between.
+		 *
+		 * - ROLLBACK based idle handlers with the sleeping instruction
+		 *   called with interrupts enabled are NOT fine. In this scheme
+		 *   when the interrupt detects it has interrupted an idle handler,
+		 *   it rolls back to its beginning which performs the
+		 *   need_resched() check before re-executing the sleeping
+		 *   instruction. This can leak a pending needed timer reprogram.
+		 *   If such a scheme is really mandatory due to the lack of an
+		 *   appropriate CPU sleeping instruction, then a FAST-FORWARD
+		 *   must instead be applied: when the interrupt detects it has
+		 *   interrupted an idle handler, it must resume to the end of
+		 *   this idle handler so that the generic idle loop is iterated
+		 *   again to reprogram the tick.
+		 */
 		local_irq_disable();
 
 		if (cpu_is_offline(cpu)) {
Signature · 1 line
2.34.1
maintainer Acked-by on v1 5/10
#11 lore raw

Re: [PATCH 05/10] cpuidle: Comment about timers requirements VS idle handler

Quoted 2 lines · Frederic Weisbecker
Quoted 48 lines
---
 kernel/sched/idle.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
kernel/sched/idle.c +30 -1
index 342f58a329f5..d52f6e3e3854 100644
@@ -258,6 +258,36 @@ static void do_idle(void)
        while (!need_resched()) {
                rmb();
 
+               /*
+                * Interrupts shouldn't be re-enabled from that point on until
+                * the CPU sleeping instruction is reached. Otherwise an interrupt
+                * may fire and queue a timer that would be ignored until the CPU
+                * wakes from the sleeping instruction. And testing need_resched()
+                * doesn't tell about pending needed timer reprogram.
+                *
+                * Several cases to consider:
+                *
+                * - SLEEP-UNTIL-PENDING-INTERRUPT based instructions such as
+                *   "wfi" or "mwait" are fine because they can be entered with
+                *   interrupt disabled.
+                *
+                * - sti;mwait() couple is fine because the interrupts are
+                *   re-enabled only upon the execution of mwait, leaving no gap
+                *   in-between.
+                *
+                * - ROLLBACK based idle handlers with the sleeping instruction
+                *   called with interrupts enabled are NOT fine. In this scheme
+                *   when the interrupt detects it has interrupted an idle handler,
+                *   it rolls back to its beginning which performs the
+                *   need_resched() check before re-executing the sleeping
+                *   instruction. This can leak a pending needed timer reprogram.
+                *   If such a scheme is really mandatory due to the lack of an
+                *   appropriate CPU sleeping instruction, then a FAST-FORWARD
+                *   must instead be applied: when the interrupt detects it has
+                *   interrupted an idle handler, it must resume to the end of
+                *   this idle handler so that the generic idle loop is iterated
+                *   again to reprogram the tick.
+                */
                local_irq_disable();
 
                if (cpu_is_offline(cpu)) {
--

2.34.1

#12 lore raw

PATCH 6/10 cpuidle: Remove unnecessary current_clr_polling_and_test() from haltpoll

In reply to Frederic Weisbecker

When cpuidle drivers ->enter() callback are called, the TIF_NR_POLLING flag is cleared already and TIF_NEED_RESCHED checked by call_cpuidle().

Therefore calling current_clr_polling_and_test() is redundant here and further setting of TIF_NEED_RESCHED will result in an IPI and thus an idle loop exit. This call can be safely removed.

---
 drivers/cpuidle/cpuidle-haltpoll.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/cpuidle/cpuidle-haltpoll.c +1 -4
index e66df22f9695..b641bc535102 100644
@@ -28,11 +28,8 @@ static enum cpuhp_state haltpoll_hp_state;
 static int default_enter_idle(struct cpuidle_device *dev,
 			      struct cpuidle_driver *drv, int index)
 {
-	if (current_clr_polling_and_test()) {
-		local_irq_enable();
-		return index;
-	}
 	arch_cpu_idle();
+
 	return index;
 }
 
Signature · 1 line
2.34.1
maintainer Acked-by on v1 6/10
#13 lore raw

Re: [PATCH 06/10] cpuidle: Remove unnecessary current_clr_polling_and_test() from haltpoll

Quoted 10 lines · Frederic Weisbecker

When cpuidle drivers ->enter() callback are called, the TIF_NR_POLLING flag is cleared already and TIF_NEED_RESCHED checked by call_cpuidle().

Therefore calling current_clr_polling_and_test() is redundant here and further setting of TIF_NEED_RESCHED will result in an IPI and thus an idle loop exit. This call can be safely removed.

Quoted 24 lines
---
 drivers/cpuidle/cpuidle-haltpoll.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/cpuidle/cpuidle-haltpoll.c +1 -5
index e66df22f9695..b641bc535102 100644
@@ -28,11 +28,8 @@ static enum cpuhp_state haltpoll_hp_state;
 static int default_enter_idle(struct cpuidle_device *dev,
                              struct cpuidle_driver *drv, int index)
 {
-       if (current_clr_polling_and_test()) {
-               local_irq_enable();
-               return index;
-       }
        arch_cpu_idle();
+
        return index;
 }
 
--

2.34.1

#14 lore raw

PATCH 7/10 cpuidle: Remove unnecessary current_clr_polling() on poll_idle()

In reply to Frederic Weisbecker

There is no point in clearing TIF_NR_POLLING and folding TIF_NEED_RESCHED upon poll_idle() exit because cpuidle_idle_call() is going to set again TIF_NR_POLLING anyway. Also if TIF_NEED_RESCHED is set, it will be folded and TIF_NR_POLLING will be cleared at the end of do_idle().

---
 drivers/cpuidle/poll_state.c | 2 --
 1 file changed, 2 deletions(-)
drivers/cpuidle/poll_state.c +0 -2
index 009f46f121ae..44a656464d06 100644
@@ -48,8 +48,6 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,
 	}
 	raw_local_irq_disable();
 
-	current_clr_polling();
-
 	return index;
 }
 
Signature · 1 line
2.34.1
maintainer Acked-by on v1 7/10
#15 lore raw

Re: [PATCH 07/10] cpuidle: Remove unnecessary current_clr_polling() on poll_idle()

Quoted 7 lines · Frederic Weisbecker

There is no point in clearing TIF_NR_POLLING and folding TIF_NEED_RESCHED upon poll_idle() exit because cpuidle_idle_call() is going to set again TIF_NR_POLLING anyway. Also if TIF_NEED_RESCHED is set, it will be folded and TIF_NR_POLLING will be cleared at the end of do_idle().

Quoted 20 lines
---
 drivers/cpuidle/poll_state.c | 2 --
 1 file changed, 2 deletions(-)
drivers/cpuidle/poll_state.c +0 -3
index 009f46f121ae..44a656464d06 100644
@@ -48,8 +48,6 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,
        }
        raw_local_irq_disable();
 
-       current_clr_polling();
-
        return index;
 }
 
--

2.34.1

#16 lore raw

PATCH 8/10 x86: Remove __current_clr_polling() from mwait_idle()

In reply to Frederic Weisbecker

mwait_idle() is only ever called through by cpuidle, either from default_idle_call() or from cpuidle_enter(). In any case cpuidle_idle_call() sets again TIF_NR_POLLING after calling it so there is no point for this atomic operation upon idle exit.

---
 arch/x86/kernel/process.c | 1 -
 1 file changed, 1 deletion(-)
arch/x86/kernel/process.c +0 -1
index 72015dba72ab..f1eb769770b1 100644
@@ -918,7 +918,6 @@ static __cpuidle void mwait_idle(void)
 			raw_local_irq_disable();
 		}
 	}
-	__current_clr_polling();
 }
 
 void select_idle_routine(const struct cpuinfo_x86 *c)
Signature · 1 line
2.34.1
maintainer Acked-by on v1 8/10
#17 lore raw

Re: [PATCH 08/10] x86: Remove __current_clr_polling() from mwait_idle()

Quoted 7 lines · Frederic Weisbecker

mwait_idle() is only ever called through by cpuidle, either from default_idle_call() or from cpuidle_enter(). In any case cpuidle_idle_call() sets again TIF_NR_POLLING after calling it so there is no point for this atomic operation upon idle exit.

Quoted 19 lines
---
 arch/x86/kernel/process.c | 1 -
 1 file changed, 1 deletion(-)
arch/x86/kernel/process.c +0 -2
index 72015dba72ab..f1eb769770b1 100644
@@ -918,7 +918,6 @@ static __cpuidle void mwait_idle(void)
                        raw_local_irq_disable();
                }
        }
-       __current_clr_polling();
 }
 
 void select_idle_routine(const struct cpuinfo_x86 *c)
--

2.34.1

#18 lore raw

PATCH 9/10 x86: Remove the current_clr_polling() call upon mwait exit

In reply to Frederic Weisbecker

mwait_idle_with_hints() is mostly called from cpuidle which already sets back TIF_NR_POLLING and fold TIF_NEED_RESCHED if necessary.

The only non-cpuidle caller is power_saving_thread() which acts as an idle loop and is the only reason why mwait_idle_with_hints() carries a costly fully ordered atomic operation upon idle exit.

Make this overhead proper to power_saving_thread() instead which already duplicates quite some cpuidle code.

---
 arch/x86/include/asm/mwait.h | 1 -
 drivers/acpi/acpi_pad.c      | 1 +
 2 files changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/mwait.h +0 -1
index 920426d691ce..c1be3775b94a 100644
@@ -134,7 +134,6 @@ static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned lo
 			}
 		}
 	}
-	current_clr_polling();
 }
 
 /*
drivers/acpi/acpi_pad.c +1 -0
index 7a453c5ff303..5a44afafe185 100644
@@ -174,6 +174,7 @@ static int power_saving_thread(void *data)
 			stop_critical_timings();
 
 			mwait_idle_with_hints(power_saving_mwait_eax, 1);
+			current_clr_polling();
 
 			start_critical_timings();
 			tick_broadcast_exit();
Signature · 1 line
2.34.1
maintainer Acked-by on v1 9/10
#19 lore raw

Re: [PATCH 09/10] x86: Remove the current_clr_polling() call upon mwait exit

Quoted 4 lines · Frederic Weisbecker

mwait_idle_with_hints() is mostly called from cpuidle which already sets back TIF_NR_POLLING and fold TIF_NEED_RESCHED if necessary.

Quoted 8 lines

The only non-cpuidle caller is power_saving_thread() which acts as an idle loop and is the only reason why mwait_idle_with_hints() carries a costly fully ordered atomic operation upon idle exit.

Make this overhead proper to power_saving_thread() instead which already duplicates quite some cpuidle code.

Quoted 32 lines
---
 arch/x86/include/asm/mwait.h | 1 -
 drivers/acpi/acpi_pad.c      | 1 +
 2 files changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/mwait.h +0 -1
index 920426d691ce..c1be3775b94a 100644
@@ -134,7 +134,6 @@ static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned lo
                        }
                }
        }
-       current_clr_polling();
 }
 
 /*
drivers/acpi/acpi_pad.c +1 -1
index 7a453c5ff303..5a44afafe185 100644
@@ -174,6 +174,7 @@ static int power_saving_thread(void *data)
                        stop_critical_timings();
 
                        mwait_idle_with_hints(power_saving_mwait_eax, 1);
+                       current_clr_polling();
 
                        start_critical_timings();
                        tick_broadcast_exit();
--

2.34.1

#20 lore raw

PATCH 10/10 sched/timers: Explain why idle task schedules out on remote timer enqueue

In reply to Frederic Weisbecker

Trying to avoid that didn't bring much value after testing, add comment about this.

---
 kernel/sched/core.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
kernel/sched/core.c +22 -0
index c52c2eba7c73..e53b892167ad 100644
@@ -1135,6 +1135,28 @@ static void wake_up_idle_cpu(int cpu)
 	if (cpu == smp_processor_id())
 		return;
 
+	/*
+	 * Set TIF_NEED_RESCHED and send an IPI if in the non-polling
+	 * part of the idle loop. This forces an exit from the idle loop
+	 * and a round trip to schedule(). Now this could be optimized
+	 * because a simple new idle loop iteration is enough to
+	 * re-evaluate the next tick. Provided some re-ordering of tick
+	 * nohz functions that would need to follow TIF_NR_POLLING
+	 * clearing:
+	 *
+	 * - On most archs, a simple fetch_or on ti::flags with a
+	 *   "0" value would be enough to know if an IPI needs to be sent.
+	 *
+	 * - x86 needs to perform a last need_resched() check between
+	 *   monitor and mwait which doesn't take timers into account.
+	 *   There a dedicated TIF_TIMER flag would be required to
+	 *   fetch_or here and be checked along with TIF_NEED_RESCHED
+	 *   before mwait().
+	 *
+	 * However, remote timer enqueue is not such a frequent event
+	 * and testing of the above solutions didn't appear to report
+	 * much benefits.
+	 */
 	if (set_nr_and_not_polling(rq->idle))
 		smp_send_reschedule(cpu);
 	else
Signature · 1 line
2.34.1
maintainer Acked-by on v1 10/10
#21 lore raw

Re: [PATCH 10/10] sched/timers: Explain why idle task schedules out on remote timer enqueue

Quoted 5 lines · Frederic Weisbecker

Trying to avoid that didn't bring much value after testing, add comment about this.

Quoted 40 lines
---
 kernel/sched/core.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
kernel/sched/core.c +22 -1
index c52c2eba7c73..e53b892167ad 100644
@@ -1135,6 +1135,28 @@ static void wake_up_idle_cpu(int cpu)
        if (cpu == smp_processor_id())
                return;
 
+       /*
+        * Set TIF_NEED_RESCHED and send an IPI if in the non-polling
+        * part of the idle loop. This forces an exit from the idle loop
+        * and a round trip to schedule(). Now this could be optimized
+        * because a simple new idle loop iteration is enough to
+        * re-evaluate the next tick. Provided some re-ordering of tick
+        * nohz functions that would need to follow TIF_NR_POLLING
+        * clearing:
+        *
+        * - On most archs, a simple fetch_or on ti::flags with a
+        *   "0" value would be enough to know if an IPI needs to be sent.
+        *
+        * - x86 needs to perform a last need_resched() check between
+        *   monitor and mwait which doesn't take timers into account.
+        *   There a dedicated TIF_TIMER flag would be required to
+        *   fetch_or here and be checked along with TIF_NEED_RESCHED
+        *   before mwait().
+        *
+        * However, remote timer enqueue is not such a frequent event
+        * and testing of the above solutions didn't appear to report
+        * much benefits.
+        */
        if (set_nr_and_not_polling(rq->idle))
                smp_send_reschedule(cpu);
        else
--

2.34.1

maintainer
#22 lore raw
In reply to Frederic Weisbecker
Quoted 6 lines · Frederic Weisbecker

Hi,

The first part of the series is cpuidle callback fixes against timers, some of which haven't been Signed by Peter yet.

Another part is removing the overhead of useless TIF_NR_POLLING clearing.

So I've again forgotten why we don't simply set TIF_NEED_RESCHED if we need the timer re-programmed. That is by far the simplest fix.

I'm sure we talked about it, but this was a long time ago and I can't remember :/

Anyway, the patches look good, except I think there's a whole bunch of architectures that still need fixing. In particular since loongson 'borrowed' the whole lot from MIPS, they need an identical fix. But I'm sure there's more architectures affected.

#23 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker

Hi,

The first part of the series is cpuidle callback fixes against timers, some of which haven't been Signed by Peter yet.

Quoted 7 lines
Quoted 1 line

Another part is removing the overhead of useless TIF_NR_POLLING clearing.

So I've again forgotten why we don't simply set TIF_NEED_RESCHED if we need the timer re-programmed. That is by far the simplest fix.

I'm sure we talked about it, but this was a long time ago and I can't remember :/

I don't think we did but the rationale is that with forcing setting TIF_NEED_RESCHED, you force a needless timer restart (which is then going to be cancelled shortly after) and a round trip to the scheduler with the rq lock overhead, etc...

Just for the fun I just tried the following change:

kernel/sched/core.c +3 -1
index c52c2eba7c73..ec43d135cf65 100644
@@ -1132,8 +1132,10 @@ static void wake_up_idle_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
 
-	if (cpu == smp_processor_id())
+	if (cpu == smp_processor_id()) {
+		set_tsk_need_resched(current);
 		return;
+	}
 
 	if (set_nr_and_not_polling(rq->idle))
 		smp_send_reschedule(cpu);

Then I computed the average of 100 runs of "make clean -C tools/perf; make -C tools/perf/" before and after this patch.

I observed an average regression of 1.27% less time spent in C-states.

So this has a measurable impact.

Quoted 5 lines

Anyway, the patches look good, except I think there's a whole bunch of architectures that still need fixing. In particular since loongson 'borrowed' the whole lot from MIPS, they need an identical fix. But I'm sure there's more architectures affected.

MIPS at least yes, I only did a quick check and it seems that most archs use a "wfi" like instruction. I'll check for others.

Thanks.