Fix put_task_struct() calls under PREEMPT_RT
Subsystems (3)
v2 cover Fix put_task_struct() calls under PREEMPT_RT
put_task_struct() decrements a usage counter and calls __put_task_struct() if the counter reaches zero.
__put_task_struct() indirectly acquires a spinlock, which is a sleeping lock under PREEMPT_RT. Therefore, we can't call put_task_struct() in an atomic context in RT kernels.
This patch series introduces put_task_struct_atomic_safe(), which defers the call to __put_task_struct() to a process context when compiled with PREEMPT_RT.
It also fixes known problematic call sites.
Changelog:
==========
v2:
* Add the put_task_struct_atomic_safe() function that is responsible for handling the conditions to call put_task_struct().
* Replace put_task_struct() by put_task_struct_atomic_safe() in known atomic call sites.
Wander Lairson Costa (4):
sched/task: Add the put_task_struct_atomic_safe function sched/deadline: fix inactive_task_timer splat
sched/rt: use put_task_struct_atomic_safe() to avoid potential splat sched/core: use put_task_struct_atomic_safe() to avoid potential splat
include/linux/sched/task.h | 21 +++++++++++++++++++++ kernel/fork.c | 8 ++++++++ kernel/sched/core.c | 2 +- kernel/sched/deadline.c | 2 +- kernel/sched/rt.c | 4 ++-- 5 files changed, 33 insertions(+), 4 deletions(-)
Signature · 1 line
2.39.0
PATCH v2 1/4 sched/task: Add the put_task_struct_atomic_safe function
With PREEMPT_RT, it is unsafe to call put_task_struct() in atomic contexts because it indirectly acquires sleeping locks.
Introduce put_task_struct_atomic_safe(), which schedules __put_task_struct() through call_rcu() when the kernel is compiled with PREEMPT_RT.
A more natural approach would use a workqueue, but since in PREEMPT_RT we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put the work_struct instance in the task_struct and initialize it when we allocate a new task_struct.
--- include/linux/sched/task.h | 21 +++++++++++++++++++++ kernel/fork.c | 8 ++++++++ 2 files changed, 29 insertions(+)
include/linux/sched/task.h +21 -0
@@ -127,6 +127,27 @@ static inline void put_task_struct_many(struct task_struct *t, int nr)void put_task_struct_rcu_user (struct task_struct *task); +extern void __delayed_put_task_struct (struct rcu_head *rhp); + +static inline void put_task_struct_atomic_safe (struct task_struct *task) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT )) { +/* +* Decrement the refcount explicitly to avoid unnecessarily +* calling call_rcu. +*/ +if (refcount_dec_and_test (&task->usage)) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&task->rcu, __delayed_put_task_struct); + }else { +put_task_struct (task); + } +} +/* Free all architecture-specific resources held by a thread. */ void release_thread (struct task_struct *dead_task);
kernel/fork.c +8 -0
@@ -859,6 +859,14 @@ void __put_task_struct(struct task_struct *tsk) }EXPORT_SYMBOL_GPL (__put_task_struct); +void __delayed_put_task_struct (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +__put_task_struct (task); +} +EXPORT_SYMBOL_GPL (__delayed_put_task_struct); +void __init __weak arch_task_cache_init (void ) { }/*
Signature · 1 line
2.39.0
PATCH v2 2/4 sched/deadline: fix inactive_task_timer splat
In reply to Wander Lairson Costainactive_task_timer() executes in interrupt (atomic) context. It calls put_task_struct(), which indirectly acquires sleeping locks under PREEMPT_RT.
Below is an example of a splat that happened in a test environment:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5
Use put_task_struct_atomic_safe() instead.
--- kernel/sched/deadline.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/deadline.c +1 -1
@@ -1442,7 +1442,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) dl_se->dl_non_contending =0 ; unlock:task_rq_unlock (rq, p, &rf); -put_task_struct (p); +put_task_struct_atomic_safe (p);return HRTIMER_NORESTART ; }
Signature · 1 line
2.39.0
PATCH v2 3/4 sched/rt: use put_task_struct_atomic_safe() to avoid potential splat
In reply to Wander Lairson Costarto_push_irq_work_func() is called in hardirq context, and it calls push_rt_task(), which calls put_task_struct().
If the kernel is compiled with PREEMPT_RT and put_task_struct() reaches zero usage count, it triggers a splat because __put_task_struct() indirectly acquires sleeping locks.
The put_task_struct() call pairs with an earlier get_task_struct(), which makes the probability of the usage count reaches zero pretty low. In any case, let's play safe and use the atomic safe version.
--- kernel/sched/rt.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/rt.c +2 -2
@@ -2147,7 +2147,7 @@ static int push_rt_task(struct rq *rq, bool pull)/* * Something has shifted, try again. */ -put_task_struct (next_task); +put_task_struct_atomic_safe (next_task); next_task = task; goto retry; }
@@ -2160,7 +2160,7 @@ static int push_rt_task(struct rq *rq, bool pull)double_unlock_balance (rq, lowest_rq); out: -put_task_struct (next_task); +put_task_struct_atomic_safe (next_task);return ret; }
Signature · 1 line
2.39.0
PATCH v2 4/4 sched/core: use put_task_struct_atomic_safe() to avoid potential splat
In reply to Wander Lairson Costapush_cpu_stop() is called form a context in which it must not sleep. Since push_cpu_stop() calls put_task_struct(), it potentially can sleep under PREEMPT_RT if the usage count reaches zero.
Use put_task_struct_atomic_safe(0 instead to avoid this potential splat.
--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -2565,7 +2565,7 @@ int push_cpu_stop(void *arg)raw_spin_rq_unlock (rq);raw_spin_unlock_irq (&p->pi_lock); -put_task_struct (p); +put_task_struct_atomic_safe (p);return 0 ; }
Signature · 1 line
2.39.0
Valentin Schneider
reviewerQuoted 7 lines · Wander Lairson Costa
put_task_struct()decrements a usage counter and calls__put_task_struct()if the counter reaches zero.
__put_task_struct()indirectly acquires a spinlock, which is a sleeping lock underPREEMPT_RT. Therefore, we can't callput_task_struct()in an atomic context in RT kernels.
Quoted 6 lines
This patch series introduces
put_task_struct_atomic_safe(), which defers the call to__put_task_struct()to a process context when compiled withPREEMPT_RT.It also fixes known problematic call sites.
Browsing around put_task_struct() callsites gives me the impression there are more problematic call sites lurking around, which makes me wonder:
should we make the PREEMPT_RT put_task_struct() *always* be done via call_rcu()?
The task's stack is actually always freed that way in put_task_stack(), cf.
e540bf3162e8 ("fork: Only cache the VMAP stack in finish_task_switch()")
Quoted 24 lines
Changelog:
==========v2:
* Add theput_task_struct_atomic_safe()function that is responsible for handling the conditions to callput_task_struct().
* Replaceput_task_struct()byput_task_struct_atomic_safe()in known atomic call sites.Wander Lairson Costa (4):
sched/task: Add theput_task_struct_atomic_safefunction sched/deadline: fixinactive_task_timersplat
sched/rt: useput_task_struct_atomic_safe()to avoid potential splat sched/core: useput_task_struct_atomic_safe()to avoid potential splatinclude/linux/sched/task.h | 21 +++++++++++++++++++++ kernel/fork.c | 8 ++++++++ kernel/sched/core.c | 2 +- kernel/sched/deadline.c | 2 +- kernel/sched/rt.c | 4 ++-- 5 files changed, 33 insertions(+), 4 deletions(-)--
2.39.0
Quoted 13 lines · Valentin Schneider
Quoted 11 lines · Wander Lairson Costa
put_task_struct()decrements a usage counter and calls__put_task_struct()if the counter reaches zero.
__put_task_struct()indirectly acquires a spinlock, which is a sleeping lock underPREEMPT_RT. Therefore, we can't callput_task_struct()in an atomic context in RT kernels.This patch series introduces
put_task_struct_atomic_safe(), which defers the call to__put_task_struct()to a process context when compiled withPREEMPT_RT.
Quoted 8 lines
Quoted 2 lines
It also fixes known problematic call sites.
Browsing around
put_task_struct()callsites gives me the impression there are more problematic call sites lurking around, which makes me wonder:
should we make thePREEMPT_RTput_task_struct()*always* be done viacall_rcu()?
I thought about going on this route, but I was concerned about the performance side effects this approach could bring. Another idea I had was to check at runtime if we are in a preemptible context. Again, this would have a (minor?) performance penalty.
Quoted 29 lines
The task's stack is actually always freed that way in
put_task_stack(), cf.
e540bf3162e8("fork: Only cache the VMAP stack infinish_task_switch()")Quoted 24 lines
Changelog:
==========v2:
* Add theput_task_struct_atomic_safe()function that is responsible for handling the conditions to callput_task_struct().
* Replaceput_task_struct()byput_task_struct_atomic_safe()in known atomic call sites.Wander Lairson Costa (4):
sched/task: Add theput_task_struct_atomic_safefunction sched/deadline: fixinactive_task_timersplat
sched/rt: useput_task_struct_atomic_safe()to avoid potential splat sched/core: useput_task_struct_atomic_safe()to avoid potential splatinclude/linux/sched/task.h | 21 +++++++++++++++++++++ kernel/fork.c | 8 ++++++++ kernel/sched/core.c | 2 +- kernel/sched/deadline.c | 2 +- kernel/sched/rt.c | 4 ++-- 5 files changed, 33 insertions(+), 4 deletions(-)--
2.39.0
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
In reply to Wander Lairson CostaQuoted 5 lines · Wander Lairson Costa
include/linux/sched/task.h +4
-0+static inline void put_task_struct_atomic_safe (struct task_struct *task) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT )) { +/*
include/linux/sched/task.h static inline void put_task_struct_many(struct task_struct *t, int nr) · line 145
Quoted 10 lines
include/linux/sched/task.h +10
-0+* Decrement the refcount explicitly to avoid unnecessarily +* calling call_rcu. +*/ +if (refcount_dec_and_test (&task->usage)) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&task->rcu, __delayed_put_task_struct);
^^^^^^^^^
I am not sure the usage of task->rcu is safe...
Suppose that, before __delayed_put_task_struct() is called by RCU, this task does the last schedule and calls put_task_struct_rcu_user().
And, can't we simply turn put_task_struct() into something like
put_task_struct(struct task_struct *t)
{
if (refcount_dec_and_test(&t->usage)) {
if (IS_ENABLED(CONFIG_PREEMPT_RT)
&& (in_atomic() || irqs_disabled()))
call_rcu(...);
else
__put_task_struct(t);
}
}?
Oleg.
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
Quoted 18 lines · Oleg Nesterov
Quoted 5 lines · Wander Lairson Costa
include/linux/sched/task.h +4
-0+static inline void put_task_struct_atomic_safe (struct task_struct *task) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT )) { +/* include/linux/sched/task.h static inline void put_task_struct_many(struct task_struct *t, int nr) · line 145
Quoted 10 lines
include/linux/sched/task.h +10
-0+* Decrement the refcount explicitly to avoid unnecessarily +* calling call_rcu. +*/ +if (refcount_dec_and_test (&task->usage)) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&task->rcu, __delayed_put_task_struct);^^^^^^^^^
Quoted 4 lines
I am not sure the usage of
task->rcuis safe...Suppose that, before
__delayed_put_task_struct()is called by RCU, this task does the last schedule and callsput_task_struct_rcu_user().
Ah, sorry, please forget, rcu_users != 0 implies task->usage != 0.
Quoted 16 lines
And, can't we simply turn
put_task_struct()into something likeput_task_struct(struct task_struct *t) { if (refcount_dec_and_test(&t->usage)) { if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_atomic() || irqs_disabled())) call_rcu(...); else __put_task_struct(t); } }?
Oleg.
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
In reply to Oleg NesterovQuoted 36 lines · Oleg Nesterov
Quoted 5 lines · Wander Lairson Costa
include/linux/sched/task.h +4
-0+static inline void put_task_struct_atomic_safe (struct task_struct *task) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT )) { +/* include/linux/sched/task.h static inline void put_task_struct_many(struct task_struct *t, int nr) · line 145
Quoted 10 lines
include/linux/sched/task.h +10
-0+* Decrement the refcount explicitly to avoid unnecessarily +* calling call_rcu. +*/ +if (refcount_dec_and_test (&task->usage)) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&task->rcu, __delayed_put_task_struct);^^^^^^^^^I am not sure the usage of
task->rcuis safe...Suppose that, before
__delayed_put_task_struct()is called by RCU, this task does the last schedule and callsput_task_struct_rcu_user().And, can't we simply turn
put_task_struct()into something likeput_task_struct(struct task_struct *t) { if (refcount_dec_and_test(&t->usage)) { if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_atomic() || irqs_disabled())) call_rcu(...); else __put_task_struct(t); } }
Quoted 1 line
?
Yeah, that was one approach I thought about. I chose to use an explicit function because I assumed calling __put_task_struct() from a non-preemptable context should be the exception, not the rule. Therefore (if I am correct in my assumption), it would make sense for only some call sites to pay the overhead price for it. But this is just a guess, and I have no evidence to support my claim.
Re: [PATCH v2 3/4] sched/rt: use put_task_struct_atomic_safe() to avoid potential splat
In reply to Wander Lairson CostaQuoted 11 lines · On Fri, 20 Jan 2023 12:02:41 -0300 Wander Lairson Costa <[email protected]>…
rto_push_irq_work_func()is called in hardirq context, and it callspush_rt_task(), which callsput_task_struct().If the kernel is compiled with
PREEMPT_RTandput_task_struct()reaches zero usage count, it triggers a splat because__put_task_struct()indirectly acquires sleeping locks.The
put_task_struct()call pairs with an earlierget_task_struct(), which makes the probability of the usage count reaches zero pretty low. In any case, let's play safe and use the atomic safe version.
Quoted 6 lines
Signed-off-by Wander Lairson Costa <[email protected]>Cc Thomas Gleixner <[email protected]>--- kernel/sched/rt.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
For what it's worth:
-- Steve
Quoted 22 lines
kernel/sched/rt.c +2
-2@@ -2147,7 +2147,7 @@ static int push_rt_task(struct rq *rq, bool pull)/* * Something has shifted, try again. */ -put_task_struct (next_task); +put_task_struct_atomic_safe (next_task); next_task = task; goto retry; }@@ -2160,7 +2160,7 @@ static int push_rt_task(struct rq *rq, bool pull)double_unlock_balance (rq, lowest_rq); out: -put_task_struct (next_task); +put_task_struct_atomic_safe (next_task);return ret; }
Valentin Schneider
reviewerRe: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
In reply to Wander Lairson CostaQuoted 37 lines · Wander Lairson Costa
Quoted 36 lines · Oleg Nesterov
Quoted 5 lines · Wander Lairson Costa
include/linux/sched/task.h +4
-0+static inline void put_task_struct_atomic_safe (struct task_struct *task) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT )) { +/* include/linux/sched/task.h static inline void put_task_struct_many(struct task_struct *t, int nr) · line 145
Quoted 10 lines
include/linux/sched/task.h +10
-0+* Decrement the refcount explicitly to avoid unnecessarily +* calling call_rcu. +*/ +if (refcount_dec_and_test (&task->usage)) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&task->rcu, __delayed_put_task_struct);^^^^^^^^^I am not sure the usage of
task->rcuis safe...Suppose that, before
__delayed_put_task_struct()is called by RCU, this task does the last schedule and callsput_task_struct_rcu_user().And, can't we simply turn
put_task_struct()into something likeput_task_struct(struct task_struct *t) { if (refcount_dec_and_test(&t->usage)) { if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_atomic() || irqs_disabled())) call_rcu(...); else __put_task_struct(t); } }
Quoted 5 lines
Quoted 1 line
?
Yeah, that was one approach I thought about. I chose to use an explicit function because I assumed calling
__put_task_struct()from a non-preemptable context should be the exception, not the rule.
I'd tend to agree.
Quoted 3 lines
Therefore (if I am correct in my assumption), it would make sense for only some call sites to pay the overhead price for it. But this is just a guess, and I have no evidence to support my claim.
My worry here is that it's easy to miss problematic callgraphs, and it's potentially easy for new ones to creep in. Having a solution within put_task_struct() itself would prevent that.
Another thing, if you look at release_task_stack(), it either caches the outgoing stack for later use, or frees it via RCU (regardless of PREEMPT_RT). Perhaps we could follow that and just always punt the freeing of the task struct to RCU?
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
Quoted 47 lines · Valentin Schneider
Quoted 37 lines · Wander Lairson Costa
Quoted 36 lines · Oleg Nesterov
Quoted 5 lines · Wander Lairson Costa
include/linux/sched/task.h +4
-0+static inline void put_task_struct_atomic_safe (struct task_struct *task) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT )) { +/* include/linux/sched/task.h static inline void put_task_struct_many(struct task_struct *t, int nr) · line 145
Quoted 10 lines
include/linux/sched/task.h +10
-0+* Decrement the refcount explicitly to avoid unnecessarily +* calling call_rcu. +*/ +if (refcount_dec_and_test (&task->usage)) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&task->rcu, __delayed_put_task_struct);^^^^^^^^^I am not sure the usage of
task->rcuis safe...Suppose that, before
__delayed_put_task_struct()is called by RCU, this task does the last schedule and callsput_task_struct_rcu_user().And, can't we simply turn
put_task_struct()into something likeput_task_struct(struct task_struct *t) { if (refcount_dec_and_test(&t->usage)) { if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_atomic() || irqs_disabled())) call_rcu(...); else __put_task_struct(t); } }Quoted 5 lines
Quoted 1 line
?
Yeah, that was one approach I thought about. I chose to use an explicit function because I assumed calling
__put_task_struct()from a non-preemptable context should be the exception, not the rule.I'd tend to agree.
Quoted 8 lines
Quoted 3 lines
Therefore (if I am correct in my assumption), it would make sense for only some call sites to pay the overhead price for it. But this is just a guess, and I have no evidence to support my claim.
My worry here is that it's easy to miss problematic callgraphs, and it's potentially easy for new ones to creep in. Having a solution within
put_task_struct()itself would prevent that.
We could add a WARN_ON statement in put_task_struct() to detect such cases.
Quoted 5 lines
Another thing, if you look at
release_task_stack(), it either caches the outgoing stack for later use, or frees it via RCU (regardless ofPREEMPT_RT). Perhaps we could follow that and just always punt the freeing of the task struct to RCU?
That's a point. Do you mean doing that even for !PREEMPT_RT?
Valentin Schneider
reviewerRe: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
Quoted 7 lines · Wander Lairson Costa
Quoted 6 lines · Valentin Schneider
Quoted 3 lines · Wander Lairson Costa
Therefore (if I am correct in my assumption), it would make sense for only some call sites to pay the overhead price for it. But this is just a guess, and I have no evidence to support my claim.
Quoted 7 lines
Quoted 4 lines
My worry here is that it's easy to miss problematic callgraphs, and it's potentially easy for new ones to creep in. Having a solution within
put_task_struct()itself would prevent that.We could add a
WARN_ONstatement input_task_struct()to detect such cases.
Anyone running their kernel with DEBUG_ATOMIC_SLEEP should be able to detect misuse, but it doesn't change that some callgraphs will only materialize under certain hardware/configuration combos.
Quoted 7 lines
Quoted 5 lines
Another thing, if you look at
release_task_stack(), it either caches the outgoing stack for later use, or frees it via RCU (regardless ofPREEMPT_RT). Perhaps we could follow that and just always punt the freeing of the task struct to RCU?That's a point. Do you mean doing that even for !PREEMPT_RT?
Could be worth a try? I think because of the cache thing the task stack is a bit less aggressive wrt RCU callback processing, but at a quick glance I don't see any fundamental reason why the task_struct itself can't be given the same treatment.
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
Quoted 14 lines · Valentin Schneider
Quoted 12 lines · Wander Lairson Costa
Quoted 10 lines · Valentin Schneider
Quoted 3 lines · Wander Lairson Costa
Therefore (if I am correct in my assumption), it would make sense for only some call sites to pay the overhead price for it. But this is just a guess, and I have no evidence to support my claim.
My worry here is that it's easy to miss problematic callgraphs, and it's potentially easy for new ones to creep in. Having a solution within
put_task_struct()itself would prevent that.
Quoted 7 lines
Quoted 2 lines
We could add a
WARN_ONstatement input_task_struct()to detect such cases.Anyone running their kernel with
DEBUG_ATOMIC_SLEEPshould be able to detect misuse, but it doesn't change that some callgraphs will only materialize under certain hardware/configuration combos.
If we put a WARN_ON in put_task_struct(), we catch cases where the reference count didn't reach zero.
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Another thing, if you look at
release_task_stack(), it either caches the outgoing stack for later use, or frees it via RCU (regardless ofPREEMPT_RT). Perhaps we could follow that and just always punt the freeing of the task struct to RCU?That's a point. Do you mean doing that even for !PREEMPT_RT?
Could be worth a try?
Sure. But I would do it only for PREEMPT_RT.
Quoted 5 lines
I think because of the cache thing the task stack is
a bit less aggressive wrt RCU callback processing, but at a quick glance I don't see any fundamental reason why thetask_structitself can't be given the same treatment.
Any idea about tests to catch performance regressions?
I
Valentin Schneider
reviewerRe: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
Quoted 18 lines · Wander Lairson Costa
Quoted 17 lines · Valentin Schneider
Quoted 14 lines · Wander Lairson Costa
Quoted 10 lines · Valentin Schneider
Quoted 3 lines · Wander Lairson Costa
Therefore (if I am correct in my assumption), it would make sense for only some call sites to pay the overhead price for it. But this is just a guess, and I have no evidence to support my claim.
My worry here is that it's easy to miss problematic callgraphs, and it's potentially easy for new ones to creep in. Having a solution within
put_task_struct()itself would prevent that.We could add a
WARN_ONstatement input_task_struct()to detect such cases.
Quoted 8 lines
Quoted 4 lines
Anyone running their kernel with
DEBUG_ATOMIC_SLEEPshould be able to detect misuse, but it doesn't change that some callgraphs will only materialize under certain hardware/configuration combos.If we put a
WARN_ONinput_task_struct(), we catch cases where the reference count didn't reach zero.
True, that'd be an improvement.
Quoted 12 lines
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Another thing, if you look at
release_task_stack(), it either caches the outgoing stack for later use, or frees it via RCU (regardless ofPREEMPT_RT). Perhaps we could follow that and just always punt the freeing of the task struct to RCU?That's a point. Do you mean doing that even for !PREEMPT_RT?
Could be worth a try?
Sure. But I would do it only for
PREEMPT_RT.
Quoted 8 lines
Quoted 5 lines
I think because of the cache thing the task stack is
a bit less aggressive wrt RCU callback processing, but at a quick glance I don't see any fundamental reason why thetask_structitself can't be given the same treatment.Any idea about tests to catch performance regressions?
I would wager anything fork-heavy with short-lived tasks, say loops of short hackbench runs, I belive stress-ng also has a fork test case.
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
In reply to Wander Lairson CostaQuoted 7 lines · Wander Lairson Costa
Quoted 5 lines
Another thing, if you look at
release_task_stack(), it either caches the outgoing stack for later use, or frees it via RCU (regardless ofPREEMPT_RT). Perhaps we could follow that and just always punt the freeing of the task struct to RCU?That's a point. Do you mean doing that even for !PREEMPT_RT?
It seems that Luca Abeni (in Cc) is hitting the bug in the non-rt kernel with SCHED_DEADLINE...
adding him to the thread.
-- Daniel
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
Hi all,
Quoted 6 lines · On Fri, 17 Feb 2023 14:35:22 -0300 Daniel Bristot de Oliveira…
Quoted 5 lines · Wander Lairson Costa
Quoted 5 lines
Another thing, if you look at
release_task_stack(), it either caches the outgoing stack for later use, or frees it via RCU (regardless ofPREEMPT_RT). Perhaps we could follow that and just always punt the freeing of the task struct to RCU?
Quoted 7 lines
Quoted 2 lines
That's a point. Do you mean doing that even for !PREEMPT_RT?
It seems that Luca Abeni (in Cc) is hitting the bug in the non-rt kernel with
SCHED_DEADLINE...adding him to the thread.
Thanks Daniel for adding me.
I triggered this "BUG: Invalid wait context" with a 5.15.76 kernel, without PREEMPT_RT. I can reproduce it easily on a KVM-based VM; if more information or tests are needed, let me know.
Luca
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
Quoted 17 lines · luca abeni
Hi all,
Quoted 12 lines · On Fri, 17 Feb 2023 14:35:22 -0300 Daniel Bristot de Oliveira…
Quoted 7 lines · Wander Lairson Costa
Quoted 5 lines
Another thing, if you look at
release_task_stack(), it either caches the outgoing stack for later use, or frees it via RCU (regardless ofPREEMPT_RT). Perhaps we could follow that and just always punt the freeing of the task struct to RCU?That's a point. Do you mean doing that even for !PREEMPT_RT?
It seems that Luca Abeni (in Cc) is hitting the bug in the non-rt kernel with
SCHED_DEADLINE...
Quoted 8 lines
Quoted 1 line
adding him to the thread.
Thanks Daniel for adding me.
I triggered this "BUG: Invalid wait context" with a 5.15.76 kernel, without
PREEMPT_RT. I can reproduce it easily on a KVM-based VM; if more information or tests are needed, let me know.
Does it happen in linux-6.1 or linux-6.2?
Quoted 3 lines
Luca
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
On Wed, 22 Feb 2023 15:42:37 -0300
Wander Lairson Costa <[email protected]> wrote:
[...]
Quoted 6 lines
Quoted 4 lines
I triggered this "BUG: Invalid wait context" with a 5.15.76 kernel, without
PREEMPT_RT. I can reproduce it easily on a KVM-based VM; if more information or tests are needed, let me know.Does it happen in linux-6.1 or linux-6.2?
I only tried with 5.15.76... I am going to test 6.2 and I'll let you know ASAP.
Luca
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
Quoted 8 lines · On Wed, 22 Feb 2023 22:00:34 +0100 luca abeni <[email protected]>…
On Wed, 22 Feb 2023 15:42:37 -0300
Wander Lairson Costa <[email protected]> wrote:
[...]Quoted 5 lines
Quoted 4 lines
I triggered this "BUG: Invalid wait context" with a 5.15.76 kernel, without
PREEMPT_RT. I can reproduce it easily on a KVM-based VM; if more information or tests are needed, let me know.
Quoted 4 lines
Quoted 1 line
Does it happen in linux-6.1 or linux-6.2?
I only tried with 5.15.76... I am going to test 6.2 and I'll let you know ASAP.
For various reasons it took more time than expected, but I managed to reproduce the bug with 6.2... Here are the relevant kernel messages:
[ 1246.556100] =============================
[ 1246.559104] [ BUG: Invalid wait context ]
[ 1246.562270] 6.2.0 #4 Not tainted
[ 1246.564854] -----------------------------
[ 1246.567260] swapper/3/0 is trying to lock:
[ 1246.568665] ffff8c2c7ebb2c10 (&c->lock){..-.}-{3:3}, at: put_cpu_partial+0x24/0x1c0
[ 1246.571325] other info that might help us debug this:
[ 1246.573045] context-{2:2}
[ 1246.574166] no locks held by swapper/3/0.
[ 1246.575434] stack backtrace:
[ 1246.576207] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.2.0 #4
[ 1246.578184] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 1246.580815] Call Trace:
[ 1246.581723] <IRQ>
[ 1246.582570] dump_stack_lvl+0x49/0x61
[ 1246.583860] __lock_acquire.cold+0xc8/0x31c
[ 1246.584923] ? __lock_acquire+0x3be/0x1df0
[ 1246.585915] lock_acquire+0xce/0x2f0
[ 1246.586819] ? put_cpu_partial+0x24/0x1c0
[ 1246.588177] ? lock_is_held_type+0xdb/0x130
[ 1246.589519] put_cpu_partial+0x5b/0x1c0
[ 1246.590996] ? put_cpu_partial+0x24/0x1c0
[ 1246.592212] inactive_task_timer+0x263/0x4c0
[ 1246.593509] ? __pfx_inactive_task_timer+0x10/0x10
[ 1246.594953] __hrtimer_run_queues+0x1bf/0x470
[ 1246.596297] hrtimer_interrupt+0x117/0x250
[ 1246.597528] __sysvec_apic_timer_interrupt+0x99/0x270
[ 1246.599015] sysvec_apic_timer_interrupt+0x8d/0xc0
[ 1246.600416] </IRQ>
[ 1246.601170] <TASK>
[ 1246.601918] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 1246.603377] RIP: 0010:default_idle+0xf/0x20
[ 1246.604640] Code: f6 5d 41 5c e9 72 4a 6e ff cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 03 52 2a 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
[ 1246.609718] RSP: 0018:ffffa1a2c009bed0 EFLAGS: 00000202
[ 1246.611259] RAX: ffffffffa4961a60 RBX: ffff8c2c4126b000 RCX: 0000000000000000
[ 1246.613230] RDX: 0000000000000000 RSI: ffffffffa510271b RDI: ffffffffa50d5b15
[ 1246.615266] RBP: 0000000000000003 R08: 0000000000000001 R09: 0000000000000001
[ 1246.617275] R10: 0000000000000000 R11: ffff8c2c4126b000 R12: ffff8c2c4126b000
[ 1246.619318] R13: ffff8c2c4126b000 R14: 0000000000000000 R15: 0000000000000000
[ 1246.621293] ? __pfx_default_idle+0x10/0x10
[ 1246.622581] default_idle_call+0x71/0x220
[ 1246.623790] do_idle+0x210/0x290
[ 1246.624827] cpu_startup_entry+0x18/0x20
[ 1246.626016] start_secondary+0xf1/0x100
[ 1246.627200] secondary_startup_64_no_verify+0xe0/0xeb
[ 1246.628707] </TASK>Let me know if you need more information, or
I should run other tests/experiments.
Luca
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
Quoted 67 lines · luca abeni
Quoted 8 lines · On Wed, 22 Feb 2023 22:00:34 +0100 luca abeni <[email protected]>…
On Wed, 22 Feb 2023 15:42:37 -0300
Wander Lairson Costa <[email protected]> wrote:
[...]Quoted 5 lines
Quoted 4 lines
I triggered this "BUG: Invalid wait context" with a 5.15.76 kernel, without
PREEMPT_RT. I can reproduce it easily on a KVM-based VM; if more information or tests are needed, let me know.Quoted 4 lines
Quoted 1 line
Does it happen in linux-6.1 or linux-6.2?
I only tried with 5.15.76... I am going to test 6.2 and I'll let you know ASAP.
For various reasons it took more time than expected, but I managed to reproduce the bug with 6.2... Here are the relevant kernel messages:
[ 1246.556100] ============================= [ 1246.559104] [ BUG: Invalid wait context ] [ 1246.562270] 6.2.0 #4 Not tainted [ 1246.564854] ----------------------------- [ 1246.567260] swapper/3/0 is trying to lock: [ 1246.568665] ffff8c2c7ebb2c10 (&c->lock){..-.}-{3:3}, at: put_cpu_partial+0x24/0x1c0 [ 1246.571325] other info that might help us debug this: [ 1246.573045] context-{2:2} [ 1246.574166] no locks held by swapper/3/0. [ 1246.575434] stack backtrace: [ 1246.576207] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.2.0 #4 [ 1246.578184] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1.1 04/01/2014 [ 1246.580815] Call Trace: [ 1246.581723] <IRQ> [ 1246.582570] dump_stack_lvl+0x49/0x61 [ 1246.583860] __lock_acquire.cold+0xc8/0x31c [ 1246.584923] ? __lock_acquire+0x3be/0x1df0 [ 1246.585915] lock_acquire+0xce/0x2f0 [ 1246.586819] ? put_cpu_partial+0x24/0x1c0 [ 1246.588177] ? lock_is_held_type+0xdb/0x130 [ 1246.589519] put_cpu_partial+0x5b/0x1c0 [ 1246.590996] ? put_cpu_partial+0x24/0x1c0 [ 1246.592212] inactive_task_timer+0x263/0x4c0 [ 1246.593509] ? __pfx_inactive_task_timer+0x10/0x10 [ 1246.594953] __hrtimer_run_queues+0x1bf/0x470 [ 1246.596297] hrtimer_interrupt+0x117/0x250 [ 1246.597528] __sysvec_apic_timer_interrupt+0x99/0x270 [ 1246.599015] sysvec_apic_timer_interrupt+0x8d/0xc0 [ 1246.600416] </IRQ> [ 1246.601170] <TASK> [ 1246.601918] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 1246.603377] RIP: 0010:default_idle+0xf/0x20 [ 1246.604640] Code: f6 5d 41 5c e9 72 4a 6e ff cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 03 52 2a 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 [ 1246.609718] RSP: 0018:ffffa1a2c009bed0 EFLAGS: 00000202 [ 1246.611259] RAX: ffffffffa4961a60 RBX: ffff8c2c4126b000 RCX: 0000000000000000 [ 1246.613230] RDX: 0000000000000000 RSI: ffffffffa510271b RDI: ffffffffa50d5b15 [ 1246.615266] RBP: 0000000000000003 R08: 0000000000000001 R09: 0000000000000001 [ 1246.617275] R10: 0000000000000000 R11: ffff8c2c4126b000 R12: ffff8c2c4126b000 [ 1246.619318] R13: ffff8c2c4126b000 R14: 0000000000000000 R15: 0000000000000000 [ 1246.621293] ? __pfx_default_idle+0x10/0x10 [ 1246.622581] default_idle_call+0x71/0x220 [ 1246.623790] do_idle+0x210/0x290 [ 1246.624827] cpu_startup_entry+0x18/0x20 [ 1246.626016] start_secondary+0xf1/0x100 [ 1246.627200] secondary_startup_64_no_verify+0xe0/0xeb [ 1246.628707] </TASK>
Quoted 3 lines
Let me know if you need more information, or
I should run other tests/experiments.
This seems to be a different (maybe related?) issue. Would you mind sharing your .config and steps to reproduce it?
Quoted 3 lines
Luca
Re: [PATCH v2 1/4] sched/task: Add the put_task_struct_atomic_safe function
On Fri, 24 Feb 2023 10:02:40 -0300
Wander Lairson Costa <[email protected]> wrote:
[...]
Quoted 50 lines
Quoted 50 lines
[ 1246.556100] ============================= [ 1246.559104] [ BUG: Invalid wait context ] [ 1246.562270] 6.2.0 #4 Not tainted [ 1246.564854] ----------------------------- [ 1246.567260] swapper/3/0 is trying to lock: [ 1246.568665] ffff8c2c7ebb2c10 (&c->lock){..-.}-{3:3}, at: put_cpu_partial+0x24/0x1c0 [ 1246.571325] other info that might help us debug this: [ 1246.573045] context-{2:2} [ 1246.574166] no locks held by swapper/3/0. [ 1246.575434] stack backtrace: [ 1246.576207] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.2.0 #4 [ 1246.578184] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1.1 04/01/2014 [ 1246.580815] Call Trace: [ 1246.581723] <IRQ> [ 1246.582570] dump_stack_lvl+0x49/0x61 [ 1246.583860] __lock_acquire.cold+0xc8/0x31c [ 1246.584923] ? __lock_acquire+0x3be/0x1df0 [ 1246.585915] lock_acquire+0xce/0x2f0 [ 1246.586819] ? put_cpu_partial+0x24/0x1c0 [ 1246.588177] ? lock_is_held_type+0xdb/0x130 [ 1246.589519] put_cpu_partial+0x5b/0x1c0 [ 1246.590996] ? put_cpu_partial+0x24/0x1c0 [ 1246.592212] inactive_task_timer+0x263/0x4c0 [ 1246.593509] ? __pfx_inactive_task_timer+0x10/0x10 [ 1246.594953] __hrtimer_run_queues+0x1bf/0x470 [ 1246.596297] hrtimer_interrupt+0x117/0x250 [ 1246.597528] __sysvec_apic_timer_interrupt+0x99/0x270 [ 1246.599015] sysvec_apic_timer_interrupt+0x8d/0xc0 [ 1246.600416] </IRQ> [ 1246.601170] <TASK> [ 1246.601918] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 1246.603377] RIP: 0010:default_idle+0xf/0x20 [ 1246.604640] Code: f6 5d 41 5c e9 72 4a 6e ff cc cc 90 90 90 9090 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 03 52 2a 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 [ 1246.609718] RSP: 0018:ffffa1a2c009bed0 EFLAGS:
00000202 [ 1246.611259] RAX:ffffffffa4961a60RBX:ffff8c2c4126b000
RCX: 0000000000000000 [ 1246.613230] RDX: 0000000000000000 RSI:ffffffffa510271bRDI:ffffffffa50d5b15[ 1246.615266] RBP:
0000000000000003 R08: 0000000000000001 R09: 0000000000000001 [ 1246.617275] R10: 0000000000000000 R11:ffff8c2c4126b000R12:ffff8c2c4126b000[ 1246.619318] R13:ffff8c2c4126b000R14:
0000000000000000 R15: 0000000000000000 [ 1246.621293] ?__pfx_default_idle+0x10/0x10 [ 1246.622581] default_idle_call+0x71/0x220 [ 1246.623790] do_idle+0x210/0x290 [ 1246.624827] cpu_startup_entry+0x18/0x20 [ 1246.626016] start_secondary+0xf1/0x100 [ 1246.627200] secondary_startup_64_no_verify+0xe0/0xeb [ 1246.628707] </TASK>
Quoted 6 lines
Quoted 3 lines
Let me know if you need more information, or
I should run other tests/experiments.This seems to be a different (maybe related?) issue. Would you mind sharing your .config and steps to reproduce it?
Ah, sorry then... I probably misunderstood the kernel messages (in my understanding, this is lockdep complaining because put_task_struct() - which can take a sleeping lock - is invoked from a timer callback).
Anyway, I attach the config (it is basically a "make defconfig; make kvm_guest.config" with some debug options manually enabled - I think the relevant one is CONFIG_PROVE_RAW_LOCK_NESTING)
Luca