sched: do not call __put_task_struct() on rt if pi_blocked_on is set
PATCH v3 1/1 sched: do not call __put_task_struct() on rt if pi_blocked_on is set
With PREEMPT_RT enabled, some of the calls to put_task_struct() coming from rt_mutex_adjust_prio_chain() could happen in preemptible context and with a mutex enqueued. That could lead to this sequence:
rt_mutex_adjust_prio_chain() put_task_struct() __put_task_struct() sched_ext_free() spin_lock_irqsave() rtlock_lock() ---> TRIGGERS lockdep_assert(!current->pi_blocked_on);
Fix that by unconditionally resorting to the deferred call to __put_task_struct().
v2: (Rostedt) remove the #ifdef from put_task_struct() and create
tsk_is_pi_blocked_on() in sched.h to make the change cleaner.
v3: (Sebastian and PeterZ) always call the RCU deferred __put_task_struct().
--- include/linux/sched/task.h | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-)
include/linux/sched/task.h +5 -15
@@ -134,22 +134,12 @@ static inline void put_task_struct(struct task_struct *t)return ;/* -* In !RT, it is always safe to call __put_task_struct(). -* Under RT, we can only call it in preemptible context. -*/ -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { - static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP); - -lock_map_acquire_try (&put_task_map); -__put_task_struct (t); -lock_map_release (&put_task_map); -return ; - } - -/* -* under PREEMPT_RT, we can't call put_task_struct +* In !RT, it is always safe to call __put_task_struct(), +* but under PREEMPT_RT, we can't call put_task_struct * in atomic context because it will indirectly -* acquire sleeping locks. +* acquire sleeping locks. The same is true if the +* current process has a mutex enqueued (blocked on +* a PI chain). * * call_rcu() will schedule delayed_put_task_struct_rcu() * to be called in process context.
Signature · 1 line
2.49.0
Sebastian Andrzej Siewior
maintainerQuoted 21 lines · Luis Claudio R. Goncalves
include/linux/sched/task.h +0
-14@@ -134,22 +134,12 @@ static inline void put_task_struct(struct task_struct *t)return ;/* -* In !RT, it is always safe to call __put_task_struct(). -* Under RT, we can only call it in preemptible context. -*/ -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { - static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP); - -lock_map_acquire_try (&put_task_map); -__put_task_struct (t); -lock_map_release (&put_task_map); -return ; - } - -/* -* under PREEMPT_RT, we can't call put_task_struct
include/linux/sched/task.h static inline void put_task_struct(struct task_struct *t) · line 145
Quoted 10 lines
include/linux/sched/task.h +5
-1+* In !RT, it is always safe to call __put_task_struct(), +* but under PREEMPT_RT, we can't call put_task_struct * in atomic context because it will indirectly -* acquire sleeping locks. +* acquire sleeping locks. The same is true if the +* current process has a mutex enqueued (blocked on +* a PI chain). * * call_rcu() will schedule delayed_put_task_struct_rcu() * to be called in process context.
Did you test it with lockdep with and without PREEMPT_RT? It would be nice to throw some testing on it.
This comment here "call_rcu will schedule bla in process context" is wrong. It will schedule the callback in softirq context. Unless RCU is configured to run the callbacks in rcuc/ thread which is the default for PREEMPT_RT. Also delayed_put_task_struct_rcu() does not exist, imho never did.
Sebastian
Quoted 27 lines · Sebastian Andrzej Siewior
Quoted 26 lines · Luis Claudio R. Goncalves
include/linux/sched/task.h +3
-15@@ -134,22 +134,12 @@ static inline void put_task_struct(struct task_struct *t)return ;/* -* In !RT, it is always safe to call __put_task_struct(). -* Under RT, we can only call it in preemptible context. -*/ -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { - static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP); - -lock_map_acquire_try (&put_task_map); -__put_task_struct (t); -lock_map_release (&put_task_map); -return ; - } - -/* -* under PREEMPT_RT, we can't call put_task_struct +* In !RT, it is always safe to call __put_task_struct(), +* but under PREEMPT_RT, we can't call put_task_struct * in atomic context because it will indirectly -* acquire sleeping locks. +* acquire sleeping locks. The same is true if the
Quoted 8 lines
include/linux/sched/task.h static inline void put_task_struct(struct task_struct *t) · line 145
Quoted 5 lines
include/linux/sched/task.h +2
-0+* current process has a mutex enqueued (blocked on +* a PI chain). * * call_rcu() will schedule delayed_put_task_struct_rcu() * to be called in process context. Did you test it with lockdep with and without
PREEMPT_RT? It would be nice to throw some testing on it.
I will re-run the full set of tests on both kernels.
Quoted 5 lines
This comment here "
call_rcuwill schedule bla in process context" is wrong. It will schedule the callback in softirq context. Unless RCU is configured to run the callbacks in rcuc/ thread which is the default forPREEMPT_RT. Alsodelayed_put_task_struct_rcu()does not exist, imho never did.
I kept the original comment about the call_rcu in process context, but didn't realize that wouldn't hold true for !RT. Would you prefer I adjust the comments (for RT vs non-RT and other possibilities) or remove them entirely?
And I completely missed delayed_put_task_struct_rcu() vs __put_task_struct_rcu_cb() in the original comment.
Thank you again for the review!
Luis
Quoted 3 lines
Sebastian
---end quoted text---
Quoted 12 lines · Luis Claudio R. Goncalves
With
PREEMPT_RTenabled, some of the calls toput_task_struct()coming fromrt_mutex_adjust_prio_chain()could happen in preemptible context and with a mutex enqueued. That could lead to this sequence:rt_mutex_adjust_prio_chain() put_task_struct() __put_task_struct() sched_ext_free() spin_lock_irqsave() rtlock_lock() ---> TRIGGERS lockdep_assert(!current->pi_blocked_on);
Quoted 7 lines
Fix that by unconditionally resorting to the deferred call to
__put_task_struct().v2: (Rostedt) remove the #ifdef from
put_task_struct()and create
tsk_is_pi_blocked_on()insched.hto make the change cleaner.v3: (Sebastian and PeterZ) always call the RCU deferred
__put_task_struct().
Changelog goes below the --- line.
Quoted 15 lines
Suggested-by Crystal Wood <[email protected]>Signed-off-by Luis Claudio R. Goncalves <[email protected]>--- include/linux/sched/task.h | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-)include/linux/sched/task.h +0
-1@@ -134,22 +134,12 @@ static inline void put_task_struct(struct task_struct *t)return ;/* -* In !RT, it is always safe to call __put_task_struct().
include/linux/sched/task.h static inline void put_task_struct(struct task_struct *t) · line 147
Quoted 10 lines
include/linux/sched/task.h +0
-10-* Under RT, we can only call it in preemptible context. -*/ -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { - static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP); - -lock_map_acquire_try (&put_task_map); -__put_task_struct (t); -lock_map_release (&put_task_map); -return ; - }
I don't think you've substantiated why the !PREEMPT_RT case needs to go.
Quoted 16 lines
include/linux/sched/task.h +5
-4- -/* -* under PREEMPT_RT, we can't call put_task_struct +* In !RT, it is always safe to call __put_task_struct(), +* but under PREEMPT_RT, we can't call put_task_struct * in atomic context because it will indirectly -* acquire sleeping locks. +* acquire sleeping locks. The same is true if the +* current process has a mutex enqueued (blocked on +* a PI chain). * * call_rcu() will schedule delayed_put_task_struct_rcu() * to be called in process context. Signature · 1 line
2.49.0
Quoted 36 lines · Peter Zijlstra
Quoted 12 lines · Luis Claudio R. Goncalves
With
PREEMPT_RTenabled, some of the calls toput_task_struct()coming fromrt_mutex_adjust_prio_chain()could happen in preemptible context and with a mutex enqueued. That could lead to this sequence:rt_mutex_adjust_prio_chain() put_task_struct() __put_task_struct() sched_ext_free() spin_lock_irqsave() rtlock_lock() ---> TRIGGERS lockdep_assert(!current->pi_blocked_on);Quoted 7 lines
Fix that by unconditionally resorting to the deferred call to
__put_task_struct().v2: (Rostedt) remove the #ifdef from
put_task_struct()and create
tsk_is_pi_blocked_on()insched.hto make the change cleaner.v3: (Sebastian and PeterZ) always call the RCU deferred
__put_task_struct().Changelog goes below the --- line.
Quoted 13 lines
Suggested-by Crystal Wood <[email protected]>Signed-off-by Luis Claudio R. Goncalves <[email protected]>--- include/linux/sched/task.h | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-)include/linux/sched/task.h +0
-0@@ -134,22 +134,12 @@ static inline void put_task_struct(struct task_struct *t)return ;
Quoted 14 lines
include/linux/sched/task.h static inline void put_task_struct(struct task_struct *t) · line 147
Quoted 12 lines
include/linux/sched/task.h +0
-11/* -* In !RT, it is always safe to call __put_task_struct(). -* Under RT, we can only call it in preemptible context. -*/ -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { - static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP); - -lock_map_acquire_try (&put_task_map); -__put_task_struct (t); -lock_map_release (&put_task_map); -return ; - }I don't think you've substantiated why the !PREEMPT_RT case needs to go.
That was my misunderstanding of "unconditionally call the deferred function". I see I took it too far and made the patch wrong.
I am testing v4 (closer to the original code with fixed comments) that is basically:
if !RT __put_task_struct (original code) else call_rcu(__put_task_struct_rcu_cb)
With the corrected comments Sebastian pointed out.
As soon as the tests complete I will post v4.
Thanks,
Luis
Quoted 8 lines
Quoted 8 lines
include/linux/sched/task.h +3
-4- -/* -* under PREEMPT_RT, we can't call put_task_struct +* In !RT, it is always safe to call __put_task_struct(), +* but under PREEMPT_RT, we can't call put_task_struct * in atomic context because it will indirectly -* acquire sleeping locks. +* acquire sleeping locks. The same is true if the
Quoted 9 lines
Quoted 8 lines
include/linux/sched/task.h +2
-0+* current process has a mutex enqueued (blocked on +* a PI chain). * * call_rcu() will schedule delayed_put_task_struct_rcu() * to be called in process context. Signature · 1 line
2.49.0
---end quoted text---