Skip to contentSkip to conversation

sched: set TIF_NEED_RESCHED before calling __trace_set_need_resched()

PATCH Superseded Sechang Lim 5 participants 14 messages Jun 27 – Jul 16 Scheduler

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 sched: set TIF_NEED_RESCHED before calling __trace_set_need_resched()

set_tsk_need_resched() tests TIF_NEED_RESCHED, calls __trace_set_need_resched() if the flag is clear, then sets it via set_tsk_thread_flag(). A BPF raw_tp program attached to sched_set_need_resched executes synchronously inside __bpf_trace_run(). On return, __bpf_trace_run() drops the RCU lock with rcu_read_unlock_migrate(), which on the preempt-or-BH-disabled path calls set_need_resched_current() -> set_tsk_need_resched() again.

set_tsk_thread_flag() follows the tracepoint call, so every re-entrant frame sees TIF_NEED_RESCHED clear and calls __trace_set_need_resched() again:

  BUG: TASK stack guard page was hit at ffffc9001224ff98
  Oops: stack guard page: 0000 [#1] SMP KASAN PTI
  RIP: 0010:__bpf_trace_sched_set_need_resched_tp+0x1c/0x190
  Call Trace:
   trace_sched_set_need_resched_tp+0x110/0x130
   set_tsk_need_resched include/linux/sched.h:2076
   set_need_resched_current include/linux/sched.h:2094
   rcu_read_unlock_special+0x43a/0x440
   __rcu_read_unlock+0x9e/0x120
   rcu_read_unlock_migrate+0xa9/0x240
   __bpf_trace_run+0x131/0x180
   bpf_trace_run3+0x333/0x430
   __bpf_trace_sched_set_need_resched_tp+0x13a/0x190
   trace_sched_set_need_resched_tp+0x110/0x130
   set_tsk_need_resched include/linux/sched.h:2076
   ...

__resched_curr() has the same ordering, firing the tracepoint before setting the flag via set_ti_thread_flag() or set_nr_and_not_polling(). Fix it for consistency.

Replace the separate test_tsk_thread_flag() + set_tsk_thread_flag() pair in set_tsk_need_resched() with test_and_set_tsk_thread_flag(). In __resched_curr(), move the tracepoint call after the flag is set in each path.

Fixes adcc3bfa8806 ("sched: Adapt sched tracepoints for RV task model")

---
v2:
- also fix __resched_curr (Peter Zijlstra)

v1:
- https://lore.kernel.org/all/[email protected]/

 include/linux/sched.h | 5 ++---
 kernel/sched/core.c   | 7 +++++--
 2 files changed, 7 insertions(+), 5 deletions(-)
include/linux/sched.h +2 -3
index ee06cba5c6f5..c9efd08dae92 100644
@@ -2071,10 +2071,9 @@ static inline int test_tsk_thread_flag(struct task_struct *tsk, int flag)
 
 static inline void set_tsk_need_resched(struct task_struct *tsk)
 {
-	if (tracepoint_enabled(sched_set_need_resched_tp) &&
-	    !test_tsk_thread_flag(tsk, TIF_NEED_RESCHED))
+	if (!test_and_set_tsk_thread_flag(tsk, TIF_NEED_RESCHED) &&
+	    tracepoint_enabled(sched_set_need_resched_tp))
 		__trace_set_need_resched(tsk, TIF_NEED_RESCHED);
-	set_tsk_thread_flag(tsk,TIF_NEED_RESCHED);
 }
 
 static inline void clear_tsk_need_resched(struct task_struct *tsk)
kernel/sched/core.c +5 -2
index b8871449d3c6..b358fac315d0 100644
@@ -1172,6 +1172,7 @@ static void __resched_curr(struct rq *rq, int tif)
 	struct task_struct *curr = rq->curr;
 	struct thread_info *cti = task_thread_info(curr);
 	int cpu;
+	bool need_ipi;
 
 	lockdep_assert_rq_held(rq);
 
@@ -1187,15 +1188,17 @@ static void __resched_curr(struct rq *rq, int tif)
 
 	cpu = cpu_of(rq);
 
-	trace_sched_set_need_resched_tp(curr, cpu, tif);
 	if (cpu == smp_processor_id()) {
 		set_ti_thread_flag(cti, tif);
 		if (tif == TIF_NEED_RESCHED)
 			set_preempt_need_resched();
+		trace_sched_set_need_resched_tp(curr, cpu, tif);
 		return;
 	}
 
-	if (set_nr_and_not_polling(cti, tif)) {
+	need_ipi = set_nr_and_not_polling(cti, tif);
+	trace_sched_set_need_resched_tp(curr, cpu, tif);
+	if (need_ipi) {
 		if (tif == TIF_NEED_RESCHED)
 			smp_send_reschedule(cpu);
 	} else {
Signature · 1 line
2.43.0
#1 lore raw

Hello Sechang,

kernel/sched/core.c static void __resched_curr(struct rq *rq, int tif) · line 1175

Quoted 10 lines · Sechang Lim

static inline void clear_tsk_need_resched(struct task_struct *tsk)

kernel/sched/core.c +1 -0
index b8871449d3c6..b358fac315d0 100644
@@ -1172,6 +1172,7 @@ static void __resched_curr(struct rq *rq, int tif)
        struct task_struct *curr = rq->curr;
        struct thread_info *cti = task_thread_info(curr);
        int cpu;
+       bool need_ipi;

nit. This declaration can go before "int cpu;" to preserve the reverse x-mas arrangement for the independent variables.

Quoted 9 lines
kernel/sched/core.c +0 -1
        lockdep_assert_rq_held(rq);
 
@@ -1187,15 +1188,17 @@ static void __resched_curr(struct rq *rq, int tif)
 
        cpu = cpu_of(rq);
 
-       trace_sched_set_need_resched_tp(curr, cpu, tif);
        if (cpu == smp_processor_id()) {

kernel/sched/core.c static void __resched_curr(struct rq *rq, int tif) · line 1200

Quoted 10 lines
kernel/sched/core.c +3 -1
                set_ti_thread_flag(cti, tif);
                if (tif == TIF_NEED_RESCHED)
                        set_preempt_need_resched();
+               trace_sched_set_need_resched_tp(curr, cpu, tif);
                return;
        }
 
-       if (set_nr_and_not_polling(cti, tif)) {
+       need_ipi = set_nr_and_not_polling(cti, tif);
+       trace_sched_set_need_resched_tp(curr, cpu, tif);

Unrelated to changes here: This tracepoint seems to be missing from wake_up_idle_cpu() which modifies the ti->flags without taking the rq_lock.

A newly enqueued timer can race with the wakeup on idle CPU and you can see a wakeup + sched_switch without seeing a corresponding trace_sched_set_need_resched_tp() for the CPU.

Gabriele, is that a concern for RV?

Quoted 7 lines
kernel/sched/core.c +1 -1
+       if (need_ipi) {
                if (tif == TIF_NEED_RESCHED)
                        smp_send_reschedule(cpu);
        } else {
--

2.43.0

Signature · 2 lines
Thanks and Regards,
Prateek
#2 lore raw
Quoted 9 lines · K Prateek Nayak

Unrelated to changes here: This tracepoint seems to be missing from wake_up_idle_cpu() which modifies the ti->flags without taking the rq_lock.

A newly enqueued timer can race with the wakeup on idle CPU and you can see a wakeup + sched_switch without seeing a corresponding trace_sched_set_need_resched_tp() for the CPU.

Gabriele, is that a concern for RV?

Thanks Prateek for pointing this out. So as I'm understanding, we are missing the tracepoint in that path, but even after putting it, this race might cause a sched_switch to be visible concurrently or before the corresponding need_resched for that CPU, right?

That's potentially an issue.

Currently the only model really relying on the need_resched event is nrp, validating every kernel preemption has a corresponding need_resched set. The scenario passing through wake_up_idle_cpu() is setting the flag on a CPU that is idle (causing __schedule(SM_IDLE)), this doesn't count as a preemption for the model. Or am I missing something?

That said, we should indeed add the tracepoint to that path and probably adapt the monitor if that's making it fail indirectly.

Thanks,
Gabriele

Quoted 7 lines
Quoted 7 lines

+       if (need_ipi) {
                if (tif == TIF_NEED_RESCHED)
                        smp_send_reschedule(cpu);
        } else {
--
2.43.0

#3 lore raw

Hello Gabriele,

Quoted 9 lines · Gabriele Monaco
Quoted 8 lines · K Prateek Nayak

Unrelated to changes here: This tracepoint seems to be missing from wake_up_idle_cpu() which modifies the ti->flags without taking the rq_lock.

A newly enqueued timer can race with the wakeup on idle CPU and you can see a wakeup + sched_switch without seeing a corresponding trace_sched_set_need_resched_tp() for the CPU.

Quoted 6 lines
Quoted 1 line

Gabriele, is that a concern for RV?

Thanks Prateek for pointing this out. So as I'm understanding, we are missing the tracepoint in that path, but even after putting it, this race might cause a sched_switch to be visible concurrently or before the corresponding need_resched for that CPU, right?

Ack but if that comment in wake_up_idle_cpu() is accurate, it is a very infrequent operation and we might be able to get away with grabbing the rq_lock and synchronizing it too.

Quoted 8 lines

That's potentially an issue.

Currently the only model really relying on the need_resched event is nrp, validating every kernel preemption has a corresponding need_resched set. The scenario passing through wake_up_idle_cpu() is setting the flag on a CPU that is idle (causing __schedule(SM_IDLE)), this doesn't count as a preemption for the model. Or am I missing something?

I just saw the npr monitor and yes, the SM_IDLE path will not count as preemption and the state machine should just stay at "any_thread_running" from the "schedule_entry" edge.

Sorry for the scare!

Quoted 2 lines

That said, we should indeed add the tracepoint to that path and probably adapt the monitor if that's making it fail indirectly.

For the npr use-case, I think the current scheme is fine since only SM_PREEMPt counts as a "schedule_entry_preempt" transition and only that can transition the state machine out of the "any_thread_running" state.

Signature · 2 lines
Thanks and Regards,
Prateek
#4 lore raw
Quoted 7 lines · K Prateek Nayak
Quoted 2 lines

That said, we should indeed add the tracepoint to that path and probably adapt the monitor if that's making it fail indirectly.

For the npr use-case, I think the current scheme is fine since only SM_PREEMPt counts as a "schedule_entry_preempt" transition and only that can transition the state machine out of the "any_thread_running" state.

Right, the monitor can live without it, but I wonder if we need to put that tracepoint for correctness sake. After all, however unlikely, that's a need_resched too.

(then if the monitor really saw a need_resched after it's sched_entry, it would stay erroneously in rescheduling).

Anyway this all isn't related to the patch.

Thanks,
Gabriele

#5 lore raw

Hello Gabriele,

Quoted 9 lines · Gabriele Monaco
Quoted 7 lines · K Prateek Nayak
Quoted 2 lines

That said, we should indeed add the tracepoint to that path and probably adapt the monitor if that's making it fail indirectly.

For the npr use-case, I think the current scheme is fine since only SM_PREEMPt counts as a "schedule_entry_preempt" transition and only that can transition the state machine out of the "any_thread_running" state.

Quoted 8 lines

Right, the monitor can live without it, but I wonder if we need to put that tracepoint for correctness sake. After all, however unlikely, that's a need_resched too.

(then if the monitor really saw a need_resched after it's sched_entry, it would stay erroneously in rescheduling).

Anyway this all isn't related to the patch.

Ack! Would something like this work for completeness:

(Lightly tested; Based on current tip:sched/core)

kernel/sched/core.c +19 -7
index 96226707c2f6..934f540d0d3f 100644
@@ -1049,9 +1049,16 @@ static inline void hrtick_schedule_exit(struct rq *rq) { }
  * this avoids any races wrt polling state changes and thereby avoids
  * spurious IPIs.
  */
-static inline bool set_nr_and_not_polling(struct thread_info *ti, int tif)
+static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
 {
-	return !(fetch_or(&ti->flags, 1 << tif) & _TIF_POLLING_NRFLAG);
+	struct task_struct *curr = rq->curr;
+	struct thread_info *ti = task_thread_info(curr);
+	unsigned long old_flags = fetch_or(&ti->flags, 1 << tif);
+
+	if (trace_sched_set_need_resched_tp_enabled() && !(old_flags & (1 << tif)))
+		trace_call__sched_set_need_resched_tp(curr, cpu_of(rq), tif);
+
+	return !(old_flags & _TIF_POLLING_NRFLAG);
 }
 
 /*
@@ -1076,8 +1083,11 @@ static bool set_nr_if_polling(struct task_struct *p)
 }
 
 #else
-static inline bool set_nr_and_not_polling(struct thread_info *ti, int tif)
+static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
 {
+	struct task_struct *curr = rq->curr;
+	struct thread_info *ti = task_thread_info(curr);
+
 	set_ti_thread_flag(ti, tif);
 	return true;
 }
@@ -1202,15 +1212,17 @@ static void __resched_curr(struct rq *rq, int tif)
 
 	cpu = cpu_of(rq);
 
-	trace_sched_set_need_resched_tp(curr, cpu, tif);
 	if (cpu == smp_processor_id()) {
-		set_ti_thread_flag(cti, tif);
+		int set = test_and_set_ti_thread_flag(cti, tif);
+
+		if (trace_sched_set_need_resched_tp_enabled() && !set)
+			trace_call__sched_set_need_resched_tp(curr, cpu, tif);
 		if (tif == TIF_NEED_RESCHED)
 			set_preempt_need_resched();
 		return;
 	}
 
-	if (set_nr_and_not_polling(cti, tif)) {
+	if (set_nr_and_not_polling(rq, tif)) {
 		if (tif == TIF_NEED_RESCHED)
 			smp_send_reschedule(cpu);
 	} else {
@@ -1350,7 +1362,7 @@ static void wake_up_idle_cpu(int cpu)
 	 * and testing of the above solutions didn't appear to report
 	 * much benefits.
 	 */
-	if (set_nr_and_not_polling(task_thread_info(rq->idle), TIF_NEED_RESCHED))
+	if (set_nr_and_not_polling(rq, TIF_NEED_RESCHED))
 		smp_send_reschedule(cpu);
 	else
 		trace_sched_wake_idle_without_ipi(cpu);

---

Quoted 4 lines

Thanks,
Gabriele

Signature · 2 lines
Thanks and Regards,
Prateek
maintainer
#6 lore raw
Quoted 26 lines · K Prateek Nayak
kernel/sched/core.c +9 -2
index 96226707c2f6..934f540d0d3f 100644
@@ -1049,9 +1049,16 @@ static inline void hrtick_schedule_exit(struct rq *rq) { }
  * this avoids any races wrt polling state changes and thereby avoids
  * spurious IPIs.
  */
-static inline bool set_nr_and_not_polling(struct thread_info *ti, int tif)
+static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
 {
-	return !(fetch_or(&ti->flags, 1 << tif) & _TIF_POLLING_NRFLAG);
+	struct task_struct *curr = rq->curr;
+	struct thread_info *ti = task_thread_info(curr);
+	unsigned long old_flags = fetch_or(&ti->flags, 1 << tif);
+
+	if (trace_sched_set_need_resched_tp_enabled() && !(old_flags & (1 << tif)))
+		trace_call__sched_set_need_resched_tp(curr, cpu_of(rq), tif);
+
+	return !(old_flags & _TIF_POLLING_NRFLAG);
 }
 
 /*
@@ -1076,8 +1083,11 @@ static bool set_nr_if_polling(struct task_struct *p)
 }
 

kernel/sched/core.c static bool set_nr_if_polling(struct task_struct *p) · line 1093

Quoted 10 lines
kernel/sched/core.c +4 -1
 #else
-static inline bool set_nr_and_not_polling(struct thread_info *ti, int tif)
+static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
 {
+	struct task_struct *curr = rq->curr;
+	struct thread_info *ti = task_thread_info(curr);
+
 	set_ti_thread_flag(ti, tif);
 	return true;
 }

This !POLLING thing also needs tracing, no?

Quoted 41 lines
kernel/sched/core.c +6 -5
@@ -1202,15 +1212,17 @@ static void __resched_curr(struct rq *rq, int tif)
 
 	cpu = cpu_of(rq);
 
-	trace_sched_set_need_resched_tp(curr, cpu, tif);
 	if (cpu == smp_processor_id()) {
-		set_ti_thread_flag(cti, tif);
+		int set = test_and_set_ti_thread_flag(cti, tif);
+
+		if (trace_sched_set_need_resched_tp_enabled() && !set)
+			trace_call__sched_set_need_resched_tp(curr, cpu, tif);
 		if (tif == TIF_NEED_RESCHED)
 			set_preempt_need_resched();
 		return;
 	}
 
-	if (set_nr_and_not_polling(cti, tif)) {
+	if (set_nr_and_not_polling(rq, tif)) {
 		if (tif == TIF_NEED_RESCHED)
 			smp_send_reschedule(cpu);
 	} else {
@@ -1350,7 +1362,7 @@ static void wake_up_idle_cpu(int cpu)
 	 * and testing of the above solutions didn't appear to report
 	 * much benefits.
 	 */
-	if (set_nr_and_not_polling(task_thread_info(rq->idle), TIF_NEED_RESCHED))
+	if (set_nr_and_not_polling(rq, TIF_NEED_RESCHED))
 		smp_send_reschedule(cpu);
 	else
 		trace_sched_wake_idle_without_ipi(cpu);
---
Quoted 5 lines

Thanks,
Gabriele

Signature · 2 lines
Thanks and Regards,
Prateek
#7 lore raw

Hello Peter, Gabriele,

Quoted 11 lines · Peter Zijlstra

Review

Quoted 9 lines
(quoted diff) +4 -1
-static inline bool set_nr_and_not_polling(struct thread_info *ti, int tif)
+static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
 {
+	struct task_struct *curr = rq->curr;
+	struct thread_info *ti = task_thread_info(curr);
+
 	set_ti_thread_flag(ti, tif);
 	return true;
 }

This !POLLING thing also needs tracing, no?

Indeed! I'm stupid.

Here is the full diff with set_nr_if_polling() covered too on top of tip:sched/core + Sechang's v3 at
https://lore.kernel.org/lkml/[email protected]/
for completeness.

Let me know if you prefer folding it in or if this needs to be a separate patch.

(lightly testet)

kernel/sched/core.c +24 -11
index 7cbd541f656f..bd2f7fb87dc9 100644
@@ -1049,9 +1049,16 @@ static inline void hrtick_schedule_exit(struct rq *rq) { }
  * this avoids any races wrt polling state changes and thereby avoids
  * spurious IPIs.
  */
-static inline bool set_nr_and_not_polling(struct thread_info *ti, int tif)
+static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
 {
-	return !(fetch_or(&ti->flags, 1 << tif) & _TIF_POLLING_NRFLAG);
+	struct task_struct *curr = rq->curr;
+	struct thread_info *ti = task_thread_info(curr);
+	unsigned long old_flags = fetch_or(&ti->flags, 1 << tif);
+
+	if (trace_sched_set_need_resched_tp_enabled() && !(old_flags & (1 << tif)))
+		trace_call__sched_set_need_resched_tp(curr, cpu_of(rq), tif);
+
+	return !(old_flags & _TIF_POLLING_NRFLAG);
 }
 
 /*
@@ -1072,13 +1079,20 @@ static bool set_nr_if_polling(struct task_struct *p)
 			return true;
 	} while (!try_cmpxchg(&ti->flags, &val, val | _TIF_NEED_RESCHED));
 
+	trace_sched_set_need_resched_tp(p, task_cpu(p), TIF_NEED_RESCHED);
 	return true;
 }
 
 #else
-static inline bool set_nr_and_not_polling(struct thread_info *ti, int tif)
+static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
 {
-	set_ti_thread_flag(ti, tif);
+	struct task_struct *curr = rq->curr;
+	struct thread_info *ti = task_thread_info(curr);
+	int set = test_and_set_ti_thread_flag(ti, tif);
+
+	if (trace_sched_set_need_resched_tp_enabled() && !set)
+		trace_call__sched_set_need_resched_tp(curr, cpu_of(rq), tif);
+
 	return true;
 }
 
@@ -1186,7 +1200,6 @@ static void __resched_curr(struct rq *rq, int tif)
 {
 	struct task_struct *curr = rq->curr;
 	struct thread_info *cti = task_thread_info(curr);
-	bool need_ipi;
 	int cpu;
 
 	lockdep_assert_rq_held(rq);
@@ -1204,16 +1217,16 @@ static void __resched_curr(struct rq *rq, int tif)
 	cpu = cpu_of(rq);
 
 	if (cpu == smp_processor_id()) {
-		set_ti_thread_flag(cti, tif);
+		int set = test_and_set_ti_thread_flag(cti, tif);
+
+		if (trace_sched_set_need_resched_tp_enabled() && !set)
+			trace_call__sched_set_need_resched_tp(curr, cpu, tif);
 		if (tif == TIF_NEED_RESCHED)
 			set_preempt_need_resched();
-		trace_sched_set_need_resched_tp(curr, cpu, tif);
 		return;
 	}
 
-	need_ipi = set_nr_and_not_polling(cti, tif);
-	trace_sched_set_need_resched_tp(curr, cpu, tif);
-	if (need_ipi) {
+	if (set_nr_and_not_polling(rq, tif)) {
 		if (tif == TIF_NEED_RESCHED)
 			smp_send_reschedule(cpu);
 	} else {
@@ -1353,7 +1366,7 @@ static void wake_up_idle_cpu(int cpu)
 	 * and testing of the above solutions didn't appear to report
 	 * much benefits.
 	 */
-	if (set_nr_and_not_polling(task_thread_info(rq->idle), TIF_NEED_RESCHED))
+	if (set_nr_and_not_polling(rq, TIF_NEED_RESCHED))
 		smp_send_reschedule(cpu);
 	else
 		trace_sched_wake_idle_without_ipi(cpu);

---

Now, on a separate note to Gabriele, this trips up RV's sched:nrp monitor when running sched-messaging because of the following race:

CPU0 CPU1
==== ====

<Any thread running>
... /* Wakes up a task on CPU0 */

  <IRQ>                                        rq_lock(rq0)
                                               __resched_curr(rq0)
                                                 set = test_and_set_ti_thread_flag(curr, 0, TIF_NEED_RESCHED);
                                                 <... interrupted by something before tracepoint hits>
                                                 !!! rq0->curr has NEED_RESCHED set but trace_sched_set_need_resched_tp() is not called !!!

  </IRQ>
  raw_irqentry_exit_cond_resched()
    if (!preempt_count() /* Masks PREEMPT_NEED_RESCHED */) {
      if (need_resched() /* Only check TIF flags */) {
        preempt_schedule_irq()
          schedule(SM_PREEMPT)
            trace_sched_entry_tp(true /* SM_PREEMPT */)

!!! SPLAT: rv: monitor nrp does not allow event schedule_entry_preempt on state any_thread_running !!!

Easy way to solve this is by moving the trace_sched_entry_tp() within the rq_lock critical section. SM_PREEMPT has to happen on a !POLLING CPU and __resched_curr() is always under the rq_lock() so the __schedule() on a remote CPU cannot race with it before trace_sched_set_need_resched_tp() is executed.

(lightly tested again)

kernel/sched/core.c +3 -3
index bd2f7fb87dc93..961c325feca84 100644
@@ -7086,9 +7086,6 @@ static void __sched notrace __schedule(int sched_mode)
 	struct rq *rq;
 	int cpu;
 
-	/* Trace preemptions consistently with task switches */
-	trace_sched_entry_tp(sched_mode == SM_PREEMPT);
-
 	cpu = smp_processor_id();
 	rq = cpu_rq(cpu);
 	prev = rq->curr;
@@ -7121,6 +7118,9 @@ static void __sched notrace __schedule(int sched_mode)
 	rq_lock(rq, &rf);
 	smp_mb__after_spinlock();
 
+	/* Trace preemptions consistently with task switches */
+	trace_sched_entry_tp(sched_mode == SM_PREEMPT);
+
 	hrtick_schedule_enter(rq);
 
 	/* Promote REQ to ACT */

---

I'm not sure if this is a problem with current mainline but I did trip it in my testing and the above seems to solve it - your mileage may vary :-)

Signature · 2 lines
Thanks and Regards,
Prateek
maintainer
#8 lore raw
Quoted 58 lines · K Prateek Nayak
kernel/sched/core.c +18 -5
index 7cbd541f656f..bd2f7fb87dc9 100644
@@ -1049,9 +1049,16 @@ static inline void hrtick_schedule_exit(struct rq *rq) { }
  * this avoids any races wrt polling state changes and thereby avoids
  * spurious IPIs.
  */
-static inline bool set_nr_and_not_polling(struct thread_info *ti, int tif)
+static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
 {
-	return !(fetch_or(&ti->flags, 1 << tif) & _TIF_POLLING_NRFLAG);
+	struct task_struct *curr = rq->curr;
+	struct thread_info *ti = task_thread_info(curr);
+	unsigned long old_flags = fetch_or(&ti->flags, 1 << tif);
+
+	if (trace_sched_set_need_resched_tp_enabled() && !(old_flags & (1 << tif)))
+		trace_call__sched_set_need_resched_tp(curr, cpu_of(rq), tif);
+
+	return !(old_flags & _TIF_POLLING_NRFLAG);
 }
 
 /*
@@ -1072,13 +1079,20 @@ static bool set_nr_if_polling(struct task_struct *p)
 			return true;
 	} while (!try_cmpxchg(&ti->flags, &val, val | _TIF_NEED_RESCHED));
 
+	trace_sched_set_need_resched_tp(p, task_cpu(p), TIF_NEED_RESCHED);
 	return true;
 }
 
 #else
-static inline bool set_nr_and_not_polling(struct thread_info *ti, int tif)
+static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
 {
-	set_ti_thread_flag(ti, tif);
+	struct task_struct *curr = rq->curr;
+	struct thread_info *ti = task_thread_info(curr);
+	int set = test_and_set_ti_thread_flag(ti, tif);
+
+	if (trace_sched_set_need_resched_tp_enabled() && !set)
+		trace_call__sched_set_need_resched_tp(curr, cpu_of(rq), tif);
+
 	return true;
 }
 
@@ -1186,7 +1200,6 @@ static void __resched_curr(struct rq *rq, int tif)
 {
 	struct task_struct *curr = rq->curr;
 	struct thread_info *cti = task_thread_info(curr);
-	bool need_ipi;
 	int cpu;
 
 	lockdep_assert_rq_held(rq);
@@ -1204,16 +1217,16 @@ static void __resched_curr(struct rq *rq, int tif)
 	cpu = cpu_of(rq);
 
 	if (cpu == smp_processor_id()) {

kernel/sched/core.c static void __resched_curr(struct rq *rq, int tif) · line 1227

Quoted 10 lines
kernel/sched/core.c +4 -2
-		set_ti_thread_flag(cti, tif);
+		int set = test_and_set_ti_thread_flag(cti, tif);
+
+		if (trace_sched_set_need_resched_tp_enabled() && !set)
+			trace_call__sched_set_need_resched_tp(curr, cpu, tif);
 		if (tif == TIF_NEED_RESCHED)
 			set_preempt_need_resched();
-		trace_sched_set_need_resched_tp(curr, cpu, tif);
 		return;
 	}

I can't help but notice that the local and !POLLING cases show remarkable similarity. Just not sure extracting that isn't going to make a mess.

Anyway, yes this looks about right.

#9 lore raw

Hello Peter,

Quoted 19 lines · Peter Zijlstra
Quoted 4 lines
(quoted diff) +0 -0
@@ -1204,16 +1217,16 @@ static void __resched_curr(struct rq *rq, int tif)
 	cpu = cpu_of(rq);
 
 	if (cpu == smp_processor_id()) {

Review static void __resched_curr(struct rq *rq, int tif) · line 1227

Quoted 10 lines
(quoted diff) +4 -2
-		set_ti_thread_flag(cti, tif);
+		int set = test_and_set_ti_thread_flag(cti, tif);
+
+		if (trace_sched_set_need_resched_tp_enabled() && !set)
+			trace_call__sched_set_need_resched_tp(curr, cpu, tif);
 		if (tif == TIF_NEED_RESCHED)
 			set_preempt_need_resched();
-		trace_sched_set_need_resched_tp(curr, cpu, tif);
 		return;
 	}

I can't help but notice that the local and !POLLING cases show remarkable similarity. Just not sure extracting that isn't going to make a mess.

Quoted 1 line

Anyway, yes this looks about right.

If the fetch_or() based path is okay for !POLLING and local cases which uses a slightly (vastly?) worse instruction to set the ti->flags, we can instead do:

(Lightly tested after removing "HAVE_TIF_POLLING_NRFLAG" for x86)

kernel/sched/core.c +21 -32
index bd2f7fb87dc93..ea793e8a94a8f 100644
@@ -1029,6 +1029,15 @@ static inline void hrtick_schedule_enter(struct rq *rq) { }
 static inline void hrtick_schedule_exit(struct rq *rq) { }
 #endif /* !CONFIG_SCHED_HRTICK */
 
+#ifndef TIF_POLLING_NRFLAG
+/*
+ * If arch doesn't define _TIF_POLLING_NRFLAG, set it 0 to
+ * allow compilers to optimize (val & _TIF_POLLING_NRFLAG)
+ * based branches during build.
+ */
+#define _TIF_POLLING_NRFLAG	0U
+#endif
+
 /*
  * try_cmpxchg based fetch_or() macro so it works for different integer types:
  */
@@ -1043,7 +1052,6 @@ static inline void hrtick_schedule_exit(struct rq *rq) { }
 	_val;								\
 })
 
-#ifdef TIF_POLLING_NRFLAG
 /*
  * Atomically set TIF_NEED_RESCHED and test for TIF_POLLING_NRFLAG,
  * this avoids any races wrt polling state changes and thereby avoids
@@ -1083,25 +1091,6 @@ static bool set_nr_if_polling(struct task_struct *p)
 	return true;
 }
 
-#else
-static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
-{
-	struct task_struct *curr = rq->curr;
-	struct thread_info *ti = task_thread_info(curr);
-	int set = test_and_set_ti_thread_flag(ti, tif);
-
-	if (trace_sched_set_need_resched_tp_enabled() && !set)
-		trace_call__sched_set_need_resched_tp(curr, cpu_of(rq), tif);
-
-	return true;
-}
-
-static inline bool set_nr_if_polling(struct task_struct *p)
-{
-	return false;
-}
-#endif
-
 static bool __wake_q_add(struct wake_q_head *head, struct task_struct *task)
 {
 	struct wake_q_node *node = &task->wake_q;
@@ -1216,19 +1205,19 @@ static void __resched_curr(struct rq *rq, int tif)
 
 	cpu = cpu_of(rq);
 
-	if (cpu == smp_processor_id()) {
-		int set = test_and_set_ti_thread_flag(cti, tif);
-
-		if (trace_sched_set_need_resched_tp_enabled() && !set)
-			trace_call__sched_set_need_resched_tp(curr, cpu, tif);
-		if (tif == TIF_NEED_RESCHED)
-			set_preempt_need_resched();
-		return;
-	}
-
 	if (set_nr_and_not_polling(rq, tif)) {
-		if (tif == TIF_NEED_RESCHED)
-			smp_send_reschedule(cpu);
+		if (tif != TIF_NEED_RESCHED)
+			return;
+		/*
+		 * For local CPU, folding the NEED_RESCHED
+		 * into preempt_count() is sufficient.
+		 */
+		if (cpu == smp_processor_id()) {
+			set_preempt_need_resched();
+			return;
+		}
+		/* Use an IPI for remote CPUs. */
+		smp_send_reschedule(cpu);
 	} else {
 		trace_sched_wake_idle_without_ipi(cpu);
 	}

---

set_nr_and_not_polling() will always return true for !POLLING which will go down the above path in __resched_curr() that deals appropriately with local CPU case.

set_nr_if_polling() will always return false for !POLLING from the first condition in the do-while loop.

Thoughts?

Signature · 2 lines
Thanks and Regards,
Prateek
maintainer
#10 lore raw
Quoted 22 lines · K Prateek Nayak

Hello Peter,

Quoted 19 lines · Peter Zijlstra
Quoted 4 lines
(quoted diff) +0 -0
@@ -1204,16 +1217,16 @@ static void __resched_curr(struct rq *rq, int tif)
 	cpu = cpu_of(rq);
 
 	if (cpu == smp_processor_id()) {

Review static void __resched_curr(struct rq *rq, int tif) · line 1227

Quoted 10 lines
(quoted diff) +4 -2
-		set_ti_thread_flag(cti, tif);
+		int set = test_and_set_ti_thread_flag(cti, tif);
+
+		if (trace_sched_set_need_resched_tp_enabled() && !set)
+			trace_call__sched_set_need_resched_tp(curr, cpu, tif);
 		if (tif == TIF_NEED_RESCHED)
 			set_preempt_need_resched();
-		trace_sched_set_need_resched_tp(curr, cpu, tif);
 		return;
 	}

I can't help but notice that the local and !POLLING cases show remarkable similarity. Just not sure extracting that isn't going to make a mess.

Quoted 6 lines
Quoted 1 line

Anyway, yes this looks about right.

If the fetch_or() based path is okay for !POLLING and local cases which uses a slightly (vastly?) worse instruction to set the ti->flags, we can instead do:

Tempting, but at least arm64 would have to agree I think. That said, I wonder why ARM64 doesn't use LDXR+WFE for idle, just like we have MONITOR+MWAIT.

Probably too damn many idle flavours to deal with or somesuch.

Quoted 101 lines
kernel/sched/core.c +21 -33
index bd2f7fb87dc93..ea793e8a94a8f 100644
@@ -1029,6 +1029,15 @@ static inline void hrtick_schedule_enter(struct rq *rq) { }
 static inline void hrtick_schedule_exit(struct rq *rq) { }
 #endif /* !CONFIG_SCHED_HRTICK */
 
+#ifndef TIF_POLLING_NRFLAG
+/*
+ * If arch doesn't define _TIF_POLLING_NRFLAG, set it 0 to
+ * allow compilers to optimize (val & _TIF_POLLING_NRFLAG)
+ * based branches during build.
+ */
+#define _TIF_POLLING_NRFLAG	0U
+#endif
+
 /*
  * try_cmpxchg based fetch_or() macro so it works for different integer types:
  */
@@ -1043,7 +1052,6 @@ static inline void hrtick_schedule_exit(struct rq *rq) { }
 	_val;								\
 })
 
-#ifdef TIF_POLLING_NRFLAG
 /*
  * Atomically set TIF_NEED_RESCHED and test for TIF_POLLING_NRFLAG,
  * this avoids any races wrt polling state changes and thereby avoids
@@ -1083,25 +1091,6 @@ static bool set_nr_if_polling(struct task_struct *p)
 	return true;
 }
 
-#else
-static inline bool set_nr_and_not_polling(struct rq *rq, int tif)
-{
-	struct task_struct *curr = rq->curr;
-	struct thread_info *ti = task_thread_info(curr);
-	int set = test_and_set_ti_thread_flag(ti, tif);
-
-	if (trace_sched_set_need_resched_tp_enabled() && !set)
-		trace_call__sched_set_need_resched_tp(curr, cpu_of(rq), tif);
-
-	return true;
-}
-
-static inline bool set_nr_if_polling(struct task_struct *p)
-{
-	return false;
-}
-#endif
-
 static bool __wake_q_add(struct wake_q_head *head, struct task_struct *task)
 {
 	struct wake_q_node *node = &task->wake_q;
@@ -1216,19 +1205,19 @@ static void __resched_curr(struct rq *rq, int tif)
 
 	cpu = cpu_of(rq);
 
-	if (cpu == smp_processor_id()) {
-		int set = test_and_set_ti_thread_flag(cti, tif);
-
-		if (trace_sched_set_need_resched_tp_enabled() && !set)
-			trace_call__sched_set_need_resched_tp(curr, cpu, tif);
-		if (tif == TIF_NEED_RESCHED)
-			set_preempt_need_resched();
-		return;
-	}
-
 	if (set_nr_and_not_polling(rq, tif)) {
-		if (tif == TIF_NEED_RESCHED)
-			smp_send_reschedule(cpu);
+		if (tif != TIF_NEED_RESCHED)
+			return;
+		/*
+		 * For local CPU, folding the NEED_RESCHED
+		 * into preempt_count() is sufficient.
+		 */
+		if (cpu == smp_processor_id()) {
+			set_preempt_need_resched();
+			return;
+		}
+		/* Use an IPI for remote CPUs. */
+		smp_send_reschedule(cpu);
 	} else {
 		trace_sched_wake_idle_without_ipi(cpu);
 	}
---

set_nr_and_not_polling() will always return true for !POLLING which will go down the above path in __resched_curr() that deals appropriately with local CPU case.

set_nr_if_polling() will always return false for !POLLING from the first condition in the do-while loop.

Thoughts?

Signature · 2 lines
Thanks and Regards,
Prateek
#11 lore raw
Quoted 30 lines · Peter Zijlstra
Quoted 22 lines · K Prateek Nayak

Hello Peter,

Quoted 19 lines · Peter Zijlstra
Quoted 4 lines
(quoted diff) +0 -0
@@ -1204,16 +1217,16 @@ static void __resched_curr(struct rq *rq, int tif)
 	cpu = cpu_of(rq);
 
 	if (cpu == smp_processor_id()) {

Review static void __resched_curr(struct rq *rq, int tif) · line 1227

Quoted 10 lines
(quoted diff) +4 -2
-		set_ti_thread_flag(cti, tif);
+		int set = test_and_set_ti_thread_flag(cti, tif);
+
+		if (trace_sched_set_need_resched_tp_enabled() && !set)
+			trace_call__sched_set_need_resched_tp(curr, cpu, tif);
 		if (tif == TIF_NEED_RESCHED)
 			set_preempt_need_resched();
-		trace_sched_set_need_resched_tp(curr, cpu, tif);
 		return;
 	}

I can't help but notice that the local and !POLLING cases show remarkable similarity. Just not sure extracting that isn't going to make a mess.

Quoted 6 lines
Quoted 1 line

Anyway, yes this looks about right.

If the fetch_or() based path is okay for !POLLING and local cases which uses a slightly (vastly?) worse instruction to set the ti->flags, we can instead do:

Quoted 5 lines

Tempting, but at least arm64 would have to agree I think. That said, I wonder why ARM64 doesn't use LDXR+WFE for idle, just like we have MONITOR+MWAIT.

Probably too damn many idle flavours to deal with or somesuch.

I think the problem is a combination of:

1. You probably want to enter a deeper sleep state after polling for a bit, but you might just get stuck in the WFE if you don't get an update.
2. An IRQ doesn't wake WFE if IRQs are disabled (this is different to

     WFI).

The series to add timeout-based cond-waits fixes (1) for the cpuidle polling loop:

https://lore.kernel.org/r/[email protected]

Will

#12 lore raw

Hello Will,

Quoted 34 lines · Will Deacon
Quoted 33 lines · Peter Zijlstra
Quoted 22 lines · K Prateek Nayak

Hello Peter,

Quoted 19 lines · Peter Zijlstra
Quoted 4 lines
(quoted diff) +0 -0
@@ -1204,16 +1217,16 @@ static void __resched_curr(struct rq *rq, int tif)
 	cpu = cpu_of(rq);
 
 	if (cpu == smp_processor_id()) {

Review static void __resched_curr(struct rq *rq, int tif) · line 1227

Quoted 10 lines
(quoted diff) +4 -2
-		set_ti_thread_flag(cti, tif);
+		int set = test_and_set_ti_thread_flag(cti, tif);
+
+		if (trace_sched_set_need_resched_tp_enabled() && !set)
+			trace_call__sched_set_need_resched_tp(curr, cpu, tif);
 		if (tif == TIF_NEED_RESCHED)
 			set_preempt_need_resched();
-		trace_sched_set_need_resched_tp(curr, cpu, tif);
 		return;
 	}

I can't help but notice that the local and !POLLING cases show remarkable similarity. Just not sure extracting that isn't going to make a mess.

Quoted 6 lines
Quoted 1 line

Anyway, yes this looks about right.

If the fetch_or() based path is okay for !POLLING and local cases which uses a slightly (vastly?) worse instruction to set the ti->flags, we can instead do:

Tempting, but at least arm64 would have to agree I think. That said, I wonder why ARM64 doesn't use LDXR+WFE for idle, just like we have MONITOR+MWAIT.

Quoted 7 lines
Quoted 1 line

Probably too damn many idle flavours to deal with or somesuch.

I think the problem is a combination of:

1. You probably want to enter a deeper sleep state after polling for a bit, but you might just get stuck in the WFE if you don't get an update.

But you can idle with scheduler tick enabled with WFE and that should allow you to retrospect and go deeper at the next tick if needed, no?

Quoted 2 lines

2. An IRQ doesn't wake WFE if IRQs are disabled (this is different to

     WFI).

I'm out of my depths here but the on x86 side mwait_idle() does an MWAIT with interrupts enabled.

The polling flag being set is simply an optimization for the IPI sender to set RESCHED bit instead of sending a physical IPI - an external interrupt is still possible and will break out of idle.

Also, the other question on the thread was: Is try_cmpxchg() much worse than a test_and_set_ti_thread_flag() (which uses test_and_set_bit()) on ARM64?

The full diff in
https://lore.kernel.org/lkml/[email protected]/
replaces a test_and_set_ti_thread_flag() path with a try_cmpxchg() one and we were wondering if the code unification is worth the performance lost.

Signature · 2 lines
Thanks and Regards,
Prateek
#13 lore raw
In reply to K Prateek Nayak
Quoted 17 lines · K Prateek Nayak

Hello Sechang,

kernel/sched/core.c static void __resched_curr(struct rq *rq, int tif) · line 1175

Quoted 10 lines · Sechang Lim

static inline void clear_tsk_need_resched(struct task_struct *tsk)

kernel/sched/core.c +1 -0
index b8871449d3c6..b358fac315d0 100644
@@ -1172,6 +1172,7 @@ static void __resched_curr(struct rq *rq, int tif)
        struct task_struct *curr = rq->curr;
        struct thread_info *cti = task_thread_info(curr);
        int cpu;
+       bool need_ipi;

nit. This declaration can go before "int cpu;" to preserve the reverse x-mas arrangement for the independent variables.

Will fix in v3. Thanks!

Bests,
Sechang