sched: do not call __put_task_struct() on rt if pi_blocked_on is set
Subsystems (2)
PATCH v2 1/1 sched: do not call __put_task_struct() on rt if pi_blocked_on is set
With PREEMPT_RT enabled, some of the calls to put_task_struct() coming from rt_mutex_adjust_prio_chain() could happen in preemptible context and with a mutex enqueued. That could lead to this sequence:
rt_mutex_adjust_prio_chain() put_task_struct() __put_task_struct() sched_ext_free() spin_lock_irqsave() rtlock_lock() ---> TRIGGERS lockdep_assert(!current->pi_blocked_on);
Adjust the check in put_task_struct() to also consider pi_blocked_on before calling __put_task_struct(), resorting to the deferred call in case it is set.
v2: Rostedt suggested removing the #ifdef from put_task_struct() and
creating tsk_is_pi_blocked_on() in sched.h to make the change cleaner.
--- include/linux/sched.h | 12 ++++++++++++ include/linux/sched/task.h | 10 +++++++--- 2 files changed, 19 insertions(+), 3 deletions(-)
include/linux/sched.h +12 -0
@@ -2148,6 +2148,18 @@ static inline bool task_is_runnable(struct task_struct *p)return p->on_rq && !p->se.sched_delayed; } +#ifdef CONFIG_RT_MUTEXES +static inline bool tsk_is_pi_blocked_on (struct task_struct *tsk) +{ +return tsk->pi_blocked_on !=NULL ; +} +#else +static inline bool tsk_is_pi_blocked_on(strut task_struct *tsk) +{ +return false; +} +#endif +extern bool sched_task_on_rq (struct task_struct *p);extern unsigned long get_wchan (struct task_struct *p);extern struct task_struct *cpu_curr_snapshot (int cpu);
include/linux/sched/task.h +7 -3
@@ -135,9 +135,11 @@ static inline void put_task_struct(struct task_struct *t)/* * In !RT, it is always safe to call __put_task_struct(). -* Under RT, we can only call it in preemptible context. +* Under RT, we can only call it in preemptible context, +* when not blocked on a PI chain. */ -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { +if (!IS_ENABLED (CONFIG_PREEMPT_RT ) || + (preemptible () || !tsk_is_pi_blocked_on (current))) { static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP);lock_map_acquire_try (&put_task_map);
@@ -149,7 +151,9 @@ static inline void put_task_struct(struct task_struct *t)/* * under PREEMPT_RT, we can't call put_task_struct * in atomic context because it will indirectly -* acquire sleeping locks. +* acquire sleeping locks. The same is true if the +* current process has a mutex enqueued (blocked on +* a PI chain). * * call_rcu() will schedule delayed_put_task_struct_rcu() * to be called in process context.
Signature · 1 line
2.49.0
Sebastian Andrzej Siewior
maintainer+ sched folks.
Quoted 12 lines · Luis Claudio R. Goncalves
With
PREEMPT_RTenabled, some of the calls toput_task_struct()coming fromrt_mutex_adjust_prio_chain()could happen in preemptible context and with a mutex enqueued. That could lead to this sequence:rt_mutex_adjust_prio_chain() put_task_struct() __put_task_struct() sched_ext_free() spin_lock_irqsave() rtlock_lock() ---> TRIGGERS lockdep_assert(!current->pi_blocked_on);
Quoted 6 lines
Adjust the check in
put_task_struct()to also considerpi_blocked_onbefore calling__put_task_struct(), resorting to the deferred call in case it is set.v2: Rostedt suggested removing the #ifdef from
put_task_struct()and
creatingtsk_is_pi_blocked_on()insched.hto make the change cleaner.
I complained about this special RT case in put_task_struct() when it was first got introduced. Couldn't we just just unconditionally do the RCU put?
Quoted 49 lines
Suggested-by Crystal Wood <[email protected]>Signed-off-by Luis Claudio R. Goncalves <[email protected]>--- include/linux/sched.h | 12 ++++++++++++ include/linux/sched/task.h | 10 +++++++--- 2 files changed, 19 insertions(+), 3 deletions(-)include/linux/sched.h +12
-0@@ -2148,6 +2148,18 @@ static inline bool task_is_runnable(struct task_struct *p)return p->on_rq && !p->se.sched_delayed; } +#ifdef CONFIG_RT_MUTEXES +static inline bool tsk_is_pi_blocked_on (struct task_struct *tsk) +{ +return tsk->pi_blocked_on !=NULL ; +} +#else +static inline bool tsk_is_pi_blocked_on(strut task_struct *tsk) +{ +return false; +} +#endif +extern bool sched_task_on_rq (struct task_struct *p);extern unsigned long get_wchan (struct task_struct *p);extern struct task_struct *cpu_curr_snapshot (int cpu);include/linux/sched/task.h +4
-2@@ -135,9 +135,11 @@ static inline void put_task_struct(struct task_struct *t)/* * In !RT, it is always safe to call __put_task_struct(). -* Under RT, we can only call it in preemptible context. +* Under RT, we can only call it in preemptible context, +* when not blocked on a PI chain. */ -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { +if (!IS_ENABLED (CONFIG_PREEMPT_RT ) || + (preemptible () || !tsk_is_pi_blocked_on (current))) { static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP);lock_map_acquire_try (&put_task_map);
include/linux/sched/task.h static inline void put_task_struct(struct task_struct *t) · line 159
Quoted 10 lines
include/linux/sched/task.h +3
-1@@ -149,7 +151,9 @@ static inline void put_task_struct(struct task_struct *t)/* * under PREEMPT_RT, we can't call put_task_struct * in atomic context because it will indirectly -* acquire sleeping locks. +* acquire sleeping locks. The same is true if the +* current process has a mutex enqueued (blocked on +* a PI chain). * * call_rcu() will schedule delayed_put_task_struct_rcu() * to be called in process context.
Sebastian
Peter Zijlstra
maintainerQuoted 21 lines · Sebastian Andrzej Siewior
+ sched folks.
Quoted 18 lines · Luis Claudio R. Goncalves
With
PREEMPT_RTenabled, some of the calls toput_task_struct()coming fromrt_mutex_adjust_prio_chain()could happen in preemptible context and with a mutex enqueued. That could lead to this sequence:rt_mutex_adjust_prio_chain() put_task_struct() __put_task_struct() sched_ext_free() spin_lock_irqsave() rtlock_lock() ---> TRIGGERS lockdep_assert(!current->pi_blocked_on);Adjust the check in
put_task_struct()to also considerpi_blocked_onbefore calling__put_task_struct(), resorting to the deferred call in case it is set.v2: Rostedt suggested removing the #ifdef from
put_task_struct()and
creatingtsk_is_pi_blocked_on()insched.hto make the change cleaner.
Oh gawd, this patch makes a sad situation worse.
Quoted 3 lines
I complained about this special RT case in
put_task_struct()when it was first got introduced. Couldn't we just just unconditionally do the RCU put?
Yeah, please make it simpler, not more complex.
Sebastian Andrzej Siewior
maintainerQuoted 5 lines · Peter Zijlstra
Quoted 3 lines
I complained about this special RT case in
put_task_struct()when it was first got introduced. Couldn't we just just unconditionally do the RCU put?Yeah, please make it simpler, not more complex.
Just so we clear: simpler as in everyone does call_rcu() or RT does always call_rcu() and everyone else __put_task_struct()? I mean we would end up with one call chain I am just not sure how expensive it gets for !RT.
Sebastian
Quoted 11 lines · Sebastian Andrzej Siewior
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines
I complained about this special RT case in
put_task_struct()when it was first got introduced. Couldn't we just just unconditionally do the RCU put?Yeah, please make it simpler, not more complex.
Just so we clear: simpler as in everyone does
call_rcu()or RT does alwayscall_rcu()and everyone else__put_task_struct()? I mean we would end up with one call chain I am just not sure how expensive it gets for !RT.
Sebastian, I implemented the change where put_task_struct() unconditionally resorted to:
call_rcu(&t->rcu, __put_task_struct_rcu_cb);
I submitted the kernels I built with that change and a pristine upstream kenrel to LTP and stress-ng and also ran 'perf bench all'. I built kernels with and without lockdep and extra debug. All kernels survived the tests without a scratch and I haven't observed differences in behaviors nor timings (for the tests that had that information).
What would be a good benchmark to compare the kernels with and without the put_task_struct() change? I would like to observe whether there is a penalty or added overhead with the change in place.
Best,
Luis
Quoted 2 lines
Sebastian
---end quoted text---