Skip to contentSkip to conversation

sched: do not call __put_task_struct() on rt if pi_blocked_on is set

PATCH Superseded Luis Claudio R. Goncalves 3 participants 5 messages Apr 9, 2025 – May 12, 2025

Subsystems (2)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 sched: do not call __put_task_struct() on rt if pi_blocked_on is set

With PREEMPT_RT enabled, some of the calls to put_task_struct() coming from rt_mutex_adjust_prio_chain() could happen in preemptible context and with a mutex enqueued. That could lead to this sequence:

	rt_mutex_adjust_prio_chain()
	  put_task_struct()
	    __put_task_struct()
	      sched_ext_free()
	        spin_lock_irqsave()
	          rtlock_lock() --->  TRIGGERS
	                              lockdep_assert(!current->pi_blocked_on);

Adjust the check in put_task_struct() to also consider pi_blocked_on before calling __put_task_struct(), resorting to the deferred call in case it is set.

v2: Rostedt suggested removing the #ifdef from put_task_struct() and
creating tsk_is_pi_blocked_on() in sched.h to make the change cleaner.

---
 include/linux/sched.h      |   12 ++++++++++++
 include/linux/sched/task.h |   10 +++++++---
 2 files changed, 19 insertions(+), 3 deletions(-)
include/linux/sched.h +12 -0
index 5ec93e5ba53a9..9fbfa7f55a83d 100644
@@ -2148,6 +2148,18 @@ static inline bool task_is_runnable(struct task_struct *p)
 	return p->on_rq && !p->se.sched_delayed;
 }
 
+#ifdef CONFIG_RT_MUTEXES
+static inline bool tsk_is_pi_blocked_on(struct task_struct *tsk)
+{
+	return tsk->pi_blocked_on != NULL;
+}
+#else
+static inline bool tsk_is_pi_blocked_on(strut task_struct *tsk)
+{
+	return false;
+}
+#endif
+
 extern bool sched_task_on_rq(struct task_struct *p);
 extern unsigned long get_wchan(struct task_struct *p);
 extern struct task_struct *cpu_curr_snapshot(int cpu);
include/linux/sched/task.h +7 -3
index 0f2aeb37bbb04..1f17a3dd51774 100644
@@ -135,9 +135,11 @@ static inline void put_task_struct(struct task_struct *t)
 
 	/*
 	 * In !RT, it is always safe to call __put_task_struct().
-	 * Under RT, we can only call it in preemptible context.
+	 * Under RT, we can only call it in preemptible context,
+	 * when not blocked on a PI chain.
 	 */
-	if (!IS_ENABLED(CONFIG_PREEMPT_RT) || preemptible()) {
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT) ||
+	    (preemptible() || !tsk_is_pi_blocked_on(current))) {
 		static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP);
 
 		lock_map_acquire_try(&put_task_map);
@@ -149,7 +151,9 @@ static inline void put_task_struct(struct task_struct *t)
 	/*
 	 * under PREEMPT_RT, we can't call put_task_struct
 	 * in atomic context because it will indirectly
-	 * acquire sleeping locks.
+	 * acquire sleeping locks. The same is true if the
+	 * current process has a mutex enqueued (blocked on
+	 * a PI chain).
 	 *
 	 * call_rcu() will schedule delayed_put_task_struct_rcu()
 	 * to be called in process context.
Signature · 1 line
2.49.0
#1 lore raw

+ sched folks.

Quoted 12 lines · Luis Claudio R. Goncalves

With PREEMPT_RT enabled, some of the calls to put_task_struct() coming from rt_mutex_adjust_prio_chain() could happen in preemptible context and with a mutex enqueued. That could lead to this sequence:

	rt_mutex_adjust_prio_chain()
	  put_task_struct()
	    __put_task_struct()
	      sched_ext_free()
	        spin_lock_irqsave()
	          rtlock_lock() --->  TRIGGERS
	                              lockdep_assert(!current->pi_blocked_on);
Quoted 6 lines

Adjust the check in put_task_struct() to also consider pi_blocked_on before calling __put_task_struct(), resorting to the deferred call in case it is set.

v2: Rostedt suggested removing the #ifdef from put_task_struct() and
creating tsk_is_pi_blocked_on() in sched.h to make the change cleaner.

I complained about this special RT case in put_task_struct() when it was first got introduced. Couldn't we just just unconditionally do the RCU put?

Quoted 49 lines
---
 include/linux/sched.h      |   12 ++++++++++++
 include/linux/sched/task.h |   10 +++++++---
 2 files changed, 19 insertions(+), 3 deletions(-)
include/linux/sched.h +12 -0
index 5ec93e5ba53a9..9fbfa7f55a83d 100644
@@ -2148,6 +2148,18 @@ static inline bool task_is_runnable(struct task_struct *p)
 	return p->on_rq && !p->se.sched_delayed;
 }
 
+#ifdef CONFIG_RT_MUTEXES
+static inline bool tsk_is_pi_blocked_on(struct task_struct *tsk)
+{
+	return tsk->pi_blocked_on != NULL;
+}
+#else
+static inline bool tsk_is_pi_blocked_on(strut task_struct *tsk)
+{
+	return false;
+}
+#endif
+
 extern bool sched_task_on_rq(struct task_struct *p);
 extern unsigned long get_wchan(struct task_struct *p);
 extern struct task_struct *cpu_curr_snapshot(int cpu);
include/linux/sched/task.h +4 -2
index 0f2aeb37bbb04..1f17a3dd51774 100644
@@ -135,9 +135,11 @@ static inline void put_task_struct(struct task_struct *t)
 
 	/*
 	 * In !RT, it is always safe to call __put_task_struct().
-	 * Under RT, we can only call it in preemptible context.
+	 * Under RT, we can only call it in preemptible context,
+	 * when not blocked on a PI chain.
 	 */
-	if (!IS_ENABLED(CONFIG_PREEMPT_RT) || preemptible()) {
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT) ||
+	    (preemptible() || !tsk_is_pi_blocked_on(current))) {
 		static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP);
 
 		lock_map_acquire_try(&put_task_map);

include/linux/sched/task.h static inline void put_task_struct(struct task_struct *t) · line 159

Quoted 10 lines
include/linux/sched/task.h +3 -1
@@ -149,7 +151,9 @@ static inline void put_task_struct(struct task_struct *t)
 	/*
 	 * under PREEMPT_RT, we can't call put_task_struct
 	 * in atomic context because it will indirectly
-	 * acquire sleeping locks.
+	 * acquire sleeping locks. The same is true if the
+	 * current process has a mutex enqueued (blocked on
+	 * a PI chain).
 	 *
 	 * call_rcu() will schedule delayed_put_task_struct_rcu()
 	 * to be called in process context.

Sebastian

maintainer
#2 lore raw
Quoted 21 lines · Sebastian Andrzej Siewior

+ sched folks.

Quoted 18 lines · Luis Claudio R. Goncalves

With PREEMPT_RT enabled, some of the calls to put_task_struct() coming from rt_mutex_adjust_prio_chain() could happen in preemptible context and with a mutex enqueued. That could lead to this sequence:

	rt_mutex_adjust_prio_chain()
	  put_task_struct()
	    __put_task_struct()
	      sched_ext_free()
	        spin_lock_irqsave()
	          rtlock_lock() --->  TRIGGERS
	                              lockdep_assert(!current->pi_blocked_on);

Adjust the check in put_task_struct() to also consider pi_blocked_on before calling __put_task_struct(), resorting to the deferred call in case it is set.

v2: Rostedt suggested removing the #ifdef from put_task_struct() and
creating tsk_is_pi_blocked_on() in sched.h to make the change cleaner.

Oh gawd, this patch makes a sad situation worse.

Quoted 3 lines

I complained about this special RT case in put_task_struct() when it was first got introduced. Couldn't we just just unconditionally do the RCU put?

Yeah, please make it simpler, not more complex.

#3 lore raw
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines

I complained about this special RT case in put_task_struct() when it was first got introduced. Couldn't we just just unconditionally do the RCU put?

Yeah, please make it simpler, not more complex.

Just so we clear: simpler as in everyone does call_rcu() or RT does always call_rcu() and everyone else __put_task_struct()? I mean we would end up with one call chain I am just not sure how expensive it gets for !RT.

Sebastian

#4 lore raw
Quoted 11 lines · Sebastian Andrzej Siewior
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines

I complained about this special RT case in put_task_struct() when it was first got introduced. Couldn't we just just unconditionally do the RCU put?

Yeah, please make it simpler, not more complex.

Just so we clear: simpler as in everyone does call_rcu() or RT does always call_rcu() and everyone else __put_task_struct()? I mean we would end up with one call chain I am just not sure how expensive it gets for !RT.

Sebastian, I implemented the change where put_task_struct() unconditionally resorted to:

	call_rcu(&t->rcu, __put_task_struct_rcu_cb);

I submitted the kernels I built with that change and a pristine upstream kenrel to LTP and stress-ng and also ran 'perf bench all'. I built kernels with and without lockdep and extra debug. All kernels survived the tests without a scratch and I haven't observed differences in behaviors nor timings (for the tests that had that information).

What would be a good benchmark to compare the kernels with and without the put_task_struct() change? I would like to observe whether there is a penalty or added overhead with the change in place.

Best,
Luis

Quoted 2 lines

Sebastian

---end quoted text---