sched: do not call __put_task_struct() on rt if pi_blocked_on is set
PATCH 1/1 sched: do not call __put_task_struct() on rt if pi_blocked_on is set
With PREEMPT_RT enabled, some of the calls to put_task_struct() coming from rt_mutex_adjust_prio_chain() could happen in preemptible context and with a mutex enqueued. That could lead to this sequence:
rt_mutex_adjust_prio_chain() put_task_struct() __put_task_struct() sched_ext_free() spin_lock_irqsave() rtlock_lock() ---> TRIGGERS lockdep_assert(!current->pi_blocked_on);
Adjust the check in put_task_struct() to also consider pi_blocked_on before calling __put_task_struct(), resorting to the deferred call in case it is set.
--- include/linux/sched/task.h | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
include/linux/sched/task.h +13 -3
@@ -130,14 +130,22 @@ extern void __put_task_struct_rcu_cb(struct rcu_head *rhp);static inline void put_task_struct (struct task_struct *t) { +bool defer = false; +if (!refcount_dec_and_test (&t->usage))return ;/* * In !RT, it is always safe to call __put_task_struct(). -* Under RT, we can only call it in preemptible context. +* Under RT, we can only call it in preemptible context, +* when not blocked on a PI chain. */ -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { +#ifdef CONFIG_PREEMPT_RT +if (!preemptible () || current->pi_blocked_on) + defer = true; +#endif + +if (!defer) { static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP);lock_map_acquire_try (&put_task_map);
@@ -149,7 +156,9 @@ static inline void put_task_struct(struct task_struct *t)/* * under PREEMPT_RT, we can't call put_task_struct * in atomic context because it will indirectly -* acquire sleeping locks. +* acquire sleeping locks. The same is true if the +* current process has a mutex enqueued (blocked on +* a PI chain). * * call_rcu() will schedule delayed_put_task_struct_rcu() * to be called in process context.
Signature · 1 line
2.49.0
Steven Rostedt
maintainerQuoted 7 lines · On Tue, 8 Apr 2025 21:27:37 -0300 "Luis Claudio R. Goncalves"…
{
include/linux/sched/task.h +2
-0+bool defer = false; +if (!refcount_dec_and_test (&t->usage))return ;/*
include/linux/sched/task.h extern void __put_task_struct_rcu_cb(struct rcu_head *rhp); · line 146
Quoted 10 lines
include/linux/sched/task.h +6
-2* In !RT, it is always safe to call __put_task_struct(). -* Under RT, we can only call it in preemptible context. +* Under RT, we can only call it in preemptible context, +* when not blocked on a PI chain. */ -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { +#ifdef CONFIG_PREEMPT_RT +if (!preemptible () || current->pi_blocked_on) + defer = true; +#endif
Why add the ugly #ifdef back?
if (!IS_ENABLED(CONFIG_PREEMPT_RT) ||
(preemptible() && !current->pi_blocked_on)) {-- Steve
Quoted 5 lines
include/linux/sched/task.h +2
-0+ +if (!defer) { static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP);
Quoted 23 lines · Steven Rostedt
Quoted 7 lines · On Tue, 8 Apr 2025 21:27:37 -0300 "Luis Claudio R. Goncalves"…
{
include/linux/sched/task.h +2
-0+bool defer = false; +if (!refcount_dec_and_test (&t->usage))return ;/* include/linux/sched/task.h extern void __put_task_struct_rcu_cb(struct rcu_head *rhp); · line 146
Quoted 10 lines
include/linux/sched/task.h +6
-2* In !RT, it is always safe to call __put_task_struct(). -* Under RT, we can only call it in preemptible context. +* Under RT, we can only call it in preemptible context, +* when not blocked on a PI chain. */ -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { +#ifdef CONFIG_PREEMPT_RT +if (!preemptible () || current->pi_blocked_on) + defer = true; +#endif Why add the ugly #ifdef back?
Quoted 2 lines
if (!IS_ENABLED(CONFIG_PREEMPT_RT) ||
(preemptible() && !current->pi_blocked_on)) {I had to add the #ifdef to avoid the build failing if CONFIG_RT_MUTEXES is not set. I do know SMP, FUTEX, I2C, PSTORE and a few more things depend on CONFIG_RT_MUTEXES being enabled, but I opted for being thorough.
I would be more than glad getting rid of the #ifdef and simplifying the patch if that possible build failure is not a case to worry about, if RT_MUTEXES are always enabled nowadays.
Luis
Quoted 8 lines
-- Steve
include/linux/sched/task.h extern void __put_task_struct_rcu_cb(struct rcu_head *rhp); · line 149
Quoted 5 lines
include/linux/sched/task.h +2
-0+ +if (!defer) { static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP);
---end quoted text---
Steven Rostedt
maintainerQuoted 11 lines · On Wed, 9 Apr 2025 13:27:12 -0300 "Luis Claudio R. Goncalves"…
Quoted 10 lines
include/linux/sched/task.h extern void __put_task_struct_rcu_cb(struct rcu_head *rhp); · line 146
Quoted 5 lines
include/linux/sched/task.h +4
-1-if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { +#ifdef CONFIG_PREEMPT_RT +if (!preemptible () || current->pi_blocked_on) + defer = true; +#endif Why add the ugly #ifdef back?
if (!IS_ENABLED(CONFIG_PREEMPT_RT) || (preemptible() && !current->pi_blocked_on)) {
Quoted 8 lines
I had to add the #ifdef to avoid the build failing if
CONFIG_RT_MUTEXESis not set. I do know SMP, FUTEX, I2C, PSTORE and a few more things depend onCONFIG_RT_MUTEXESbeing enabled, but I opted for being thorough.I would be more than glad getting rid of the #ifdef and simplifying the patch if that possible build failure is not a case to worry about, if
RT_MUTEXESare always enabled nowadays.
Because pi_blocked_on is only defined when CONFIG_RT_MUTEXES is enabled?
OK, then perhaps we should add in sched.h:
#ifdef CONFIG_RT_MUTEXES
static inline bool tsk_is_pi_blocked_on(struct task_struct *tsk)
{
return tsk->pi_blocked_on != NULL;
}
#else
static inline bool tsk_is_pi_blocked_on(strut task_struct *tsk)
{
return false;
}#endif
??
-- Steve
Quoted 37 lines · Steven Rostedt
Quoted 11 lines · On Wed, 9 Apr 2025 13:27:12 -0300 "Luis Claudio R. Goncalves"…
Quoted 10 lines
include/linux/sched/task.h extern void __put_task_struct_rcu_cb(struct rcu_head *rhp); · line 146
Quoted 5 lines
include/linux/sched/task.h +4
-1-if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { +#ifdef CONFIG_PREEMPT_RT +if (!preemptible () || current->pi_blocked_on) + defer = true; +#endif Why add the ugly #ifdef back?
if (!IS_ENABLED(CONFIG_PREEMPT_RT) || (preemptible() && !current->pi_blocked_on)) {Quoted 8 lines
I had to add the #ifdef to avoid the build failing if
CONFIG_RT_MUTEXESis not set. I do know SMP, FUTEX, I2C, PSTORE and a few more things depend onCONFIG_RT_MUTEXESbeing enabled, but I opted for being thorough.I would be more than glad getting rid of the #ifdef and simplifying the patch if that possible build failure is not a case to worry about, if
RT_MUTEXESare always enabled nowadays.Because
pi_blocked_onis only defined whenCONFIG_RT_MUTEXESis enabled?OK, then perhaps we should add in
sched.h:#ifdef
CONFIG_RT_MUTEXES
static inline booltsk_is_pi_blocked_on(struct task_struct *tsk){ return tsk->pi_blocked_on != NULL; } #else static inline bool tsk_is_pi_blocked_on(strut task_struct *tsk) { return false; }
Quoted 3 lines
#endif
??
Thank you for the suggestion! I will rework the patch with that and send v2 after basic testing.
Luis
Quoted 2 lines
-- Steve
---end quoted text---