kernel/fork: beware of __put_task_struct calling context
Subsystems (3)
PATCH v4 1/1 kernel/fork: beware of __put_task_struct calling context
Under PREEMPT_RT, __put_task_struct() indirectly acquires sleeping locks. Therefore, it can't be called from an non-preemptible context.
One practical example is splat inside inactive_task_timer(), which is called in a interrupt context:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5
Instead of calling __put_task_struct() directly, we defer it using call_rcu(). A more natural approach would use a workqueue, but since in PREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put the work_struct instance in the task_struct and initialize it when we allocate a new task_struct.
Changelog
=========
v1:
* Initial implementation fixing the splat.
v2:
* Isolate the logic in its own function.
* Fix two more cases caught in review.
v3:
* Change __put_task_struct() to handle the issue internally.
v4:
* Explain why call_rcu() is safe to call from interrupt context.
--- kernel/fork.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)
kernel/fork.c +24 -1
@@ -840,7 +840,7 @@ static inline void put_signal_struct(struct signal_struct *sig)free_signal_struct (sig); } -void __put_task_struct (struct task_struct *tsk) +static void ___put_task_struct (struct task_struct *tsk) {WARN_ON (!tsk->exit_state);WARN_ON (refcount_read (&tsk->usage));
@@ -857,6 +857,29 @@ void __put_task_struct(struct task_struct *tsk)sched_core_free (tsk);free_task (tsk); } + +static void delayed_put_task_struct_rcu (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +___put_task_struct (task); +} + +void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ())) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +* +* call_rcu() will schedule delayed_put_task_struct_rcu() +* to be called in process context. +*/ +call_rcu (&tsk->rcu, delayed_put_task_struct_rcu); +else +___put_task_struct (tsk); +}EXPORT_SYMBOL_GPL (__put_task_struct);void __init __weak arch_task_cache_init (void ) { }
Signature · 1 line
2.39.1
Quoted 5 lines · Wander Lairson Costa
Under
PREEMPT_RT,__put_task_struct()indirectly acquires sleeping locks. Therefore, it can't be called from an non-preemptible context.One practical example is splat inside
inactive_task_timer(), which is called in a interrupt context:
Do you have more?
The inactive_task_timer() is marked as HRTIMER_MODE_REL_HARD which means in runs in hardirq-context. The author of commit
850377a875a48 ("sched/deadline: Ensure inactive_timer runs in hardirq context")
should have been aware of that.
We have on workaround of that put_task() in sched-switch. I wasn't aware of this shortcoming. So either we have more problems or potential problems or this is the only finding so far.
Quoted 5 lines
kernel/fork.c +0
-0@@ -857,6 +857,29 @@ void __put_task_struct(struct task_struct *tsk)
…
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 870
Quoted 3 lines
kernel/fork.c +3
-0+void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ()))
Is it safe to use the rcu member in any case? If so why not use it unconditionally?
Quoted 5 lines
kernel/fork.c +5
-0+/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 885
Quoted 10 lines
kernel/fork.c +7
-0+* call_rcu() will schedule delayed_put_task_struct_rcu() +* to be called in process context. +*/ +call_rcu (&tsk->rcu, delayed_put_task_struct_rcu); + else +___put_task_struct (tsk ); +}EXPORT_SYMBOL_GPL (__put_task_struct);void __init __weak arch_task_cache_init (void ) { }
Sebastian
Quoted 10 lines · Sebastian Andrzej Siewior
Quoted 1 line · Wander Lairson Costa
(quoted diff) +0
-0@@ -857,6 +857,29 @@ void __put_task_struct(struct task_struct *tsk)…
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 870
Quoted 3 lines
kernel/fork.c +3
-0+void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ()))Is it safe to use the rcu member in any case?
I thinks it is safe but deserves a comment. I guess Wander misunderstood me when I asked him to do this...
__put_task_struct() is called when refcount_dec_and_test(&t->usage) succeeds.
This means that it can't "conflict" with put_task_struct_rcu_user() which abuses ->rcu the same way; rcu_users has a reference so task->usage can't be zero after rcu_users 1 -> 0 transition.
Quoted 2 lines
If so why not use it
unconditionally?
performance ?
And... I still don't like the name of delayed_put_task_struct_rcu() to me ___put_task_struct_rcu() looks a bit less confusing, note that we already have delayed_put_task_struct(). But this is minor.
Oleg.
Quoted 12 lines · Oleg Nesterov
Quoted 10 lines · Sebastian Andrzej Siewior
Quoted 1 line · Wander Lairson Costa
(quoted diff) +0
-0@@ -857,6 +857,29 @@ void __put_task_struct(struct task_struct *tsk)…
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 870
Quoted 3 lines
kernel/fork.c +3
-0+void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ()))Is it safe to use the rcu member in any case?
Quoted 8 lines
I thinks it is safe but deserves a comment. I guess Wander misunderstood me when I asked him to do this...
__put_task_struct()is called whenrefcount_dec_and_test(&t->usage)succeeds.This means that it can't "conflict" with
put_task_struct_rcu_user()which abuses ->rcu the same way;rcu_usershas a reference sotask->usagecan't be zero afterrcu_users1 -> 0 transition.
Sounds good.
Quoted 4 lines
Quoted 2 lines
If so why not use it
unconditionally?performance ?
All the free() part is moved from the caller into rcu.
Quoted 4 lines
And... I still don't like the name of
delayed_put_task_struct_rcu()to me___put_task_struct_rcu()looks a bit less confusing, note that we already havedelayed_put_task_struct(). But this is minor.
So if we do it unconditionally then we could get rid of put_task_struct_rcu_user().
Otherwise we could use put_task_struct_rcu_user() in that timer callback because it will lead to lockdep warnings once printk is fixed.
Quoted 1 line
Oleg.
Sebastian
Quoted 9 lines · Sebastian Andrzej Siewior
Quoted 4 lines · Oleg Nesterov
Quoted 2 lines
If so why not use it
unconditionally?performance ?
All the
free()part is moved from the caller into rcu.
sorry, I don't understand,
Quoted 8 lines
Quoted 5 lines
And... I still don't like the name of
delayed_put_task_struct_rcu()to me___put_task_struct_rcu()looks a bit less confusing, note that we already havedelayed_put_task_struct(). But this is minor.So if we do it unconditionally then we could get rid of
put_task_struct_rcu_user().
Yes. But the whole purpose of rcu_users is that we want to avoid the unconditional rcu grace period before free_task() ?
Just in case... please note that delayed_put_task_struct() delays refcount_sub(t->usage), not free_task().
Why do we need this? Consider
rcu_read_lock(); task = find-task-in-rcu-protected-list; // Safe, task->usage can't be zero get_task_struct(task); rcu_read_unlock();
Quoted 2 lines
Otherwise we could use
put_task_struct_rcu_user()in that timer callback because it will lead to lockdep warnings once printk is fixed.
IIUC there are more in-atomic callers of put_task_struct(). But perhaps I misunderstood you...
Oleg.
Quoted 4 lines · Oleg Nesterov
Quoted 3 lines · Sebastian Andrzej Siewior
On 2023-02-06 16:27:12 [+0100], Oleg Nesterov wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
If so why not use it
unconditionally?performance ?
All the
free()part is moved from the caller into rcu.sorry, I don't understand,
That callback does mostly free() and it is batched with other free() invocations. This also is moved away from the caller which _might_ benefit.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
And... I still don't like the name of
delayed_put_task_struct_rcu()to me___put_task_struct_rcu()looks a bit less confusing, note that we already havedelayed_put_task_struct(). But this is minor.So if we do it unconditionally then we could get rid of
put_task_struct_rcu_user().Yes. But the whole purpose of
rcu_usersis that we want to avoid the unconditional rcu grace period beforefree_task()?
Oh, this is usage vs rcu_users. Okay, mixed that up.
Quoted 2 lines
Just in case... please note that
delayed_put_task_struct()delaysrefcount_sub(t->usage), notfree_task().
Just noticed ;)
Quoted 12 lines
Why do we need this? Consider
rcu_read_lock(); task = find-task-in-rcu-protected-list; // Safe, task->usage can't be zero get_task_struct(task); rcu_read_unlock();
Quoted 5 lines
Quoted 2 lines
Otherwise we could use
put_task_struct_rcu_user()in that timer callback because it will lead to lockdep warnings once printk is fixed.IIUC there are more in-atomic callers of
put_task_struct(). But perhaps I misunderstood you...
That is true. So you are saying that we don't what to use RCU for put_task_struct() unconditionally?
Quoted 1 line
Oleg.
Sebastian
Quoted 33 lines · Sebastian Andrzej Siewior
Quoted 12 lines · Oleg Nesterov
Quoted 10 lines · Sebastian Andrzej Siewior
Quoted 1 line · Wander Lairson Costa
(quoted diff) +0
-0@@ -857,6 +857,29 @@ void __put_task_struct(struct task_struct *tsk)…
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 870
Quoted 3 lines
kernel/fork.c +3
-0+void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ()))Is it safe to use the rcu member in any case?
Quoted 8 lines
I thinks it is safe but deserves a comment. I guess Wander misunderstood me when I asked him to do this...
__put_task_struct()is called whenrefcount_dec_and_test(&t->usage)succeeds.This means that it can't "conflict" with
put_task_struct_rcu_user()which abuses ->rcu the same way;rcu_usershas a reference sotask->usagecan't be zero afterrcu_users1 -> 0 transition.Sounds good.
Quoted 4 lines
Quoted 2 lines
If so why not use it
unconditionally?performance ?
All the
free()part is moved from the caller into rcu.Quoted 1 line
Quoted 8 lines
Quoted 3 lines
And... I still don't like the name of
delayed_put_task_struct_rcu()to me___put_task_struct_rcu()looks a bit less confusing, note that we already havedelayed_put_task_struct(). But this is minor.So if we do it unconditionally then we could get rid of
put_task_struct_rcu_user().
Otherwise we could useput_task_struct_rcu_user()in that timer callback because it will lead to lockdep warnings once printk is fixed.
put_task_struct_rcu_user() calls delayed_put_task_struct(), which does more than just call __put_task_struct(). I tried this approach at the beginning, but I got another splat (unfortunately, I don't remember where).
Quoted 7 lines · Oleg Nesterov
Quoted 5 lines · Sebastian Andrzej Siewior
Quoted 1 line · Wander Lairson Costa
(quoted diff) +0
-0@@ -857,6 +857,29 @@ void __put_task_struct(struct task_struct *tsk)…
Quoted 9 lines
Quoted 5 lines
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 870
Quoted 3 lines
kernel/fork.c +3
-0+void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ()))Is it safe to use the rcu member in any case?
I thinks it is safe but deserves a comment. I guess Wander misunderstood me when I asked him to do this...
Oops, sorry. Next version, I will include this description.
Quoted 9 lines
__put_task_struct()is called whenrefcount_dec_and_test(&t->usage)succeeds.This means that it can't "conflict" with
put_task_struct_rcu_user()which abuses ->rcu the same way;rcu_usershas a reference sotask->usagecan't be zero afterrcu_users1 -> 0 transition.Quoted 2 lines
If so why not use it
unconditionally?
Quoted 7 lines
performance ?
And... I still don't like the name of
delayed_put_task_struct_rcu()to me___put_task_struct_rcu()looks a bit less confusing, note that we already havedelayed_put_task_struct(). But this is minor.
Ok, I will change it.
Quoted 2 lines
Oleg.
Quoted 13 lines · Sebastian Andrzej Siewior
Quoted 5 lines · Wander Lairson Costa
Under
PREEMPT_RT,__put_task_struct()indirectly acquires sleeping locks. Therefore, it can't be called from an non-preemptible context.One practical example is splat inside
inactive_task_timer(), which is called in a interrupt context:Do you have more?
Theinactive_task_timer()is marked asHRTIMER_MODE_REL_HARDwhich means in runs in hardirq-context. The author of commit
850377a875a48("sched/deadline: Ensureinactive_timerruns in hardirq context")
Quoted 5 lines
should have been aware of that.
We have on workaround of thatput_task()in sched-switch. I wasn't aware of this shortcoming. So either we have more problems or potential problems or this is the only finding so far.
Valentin spotted two other potential issues, I fixed them in v2[1]. Also there is a discussion there that led to this implementation.
Quoted 6 lines
Quoted 5 lines
kernel/fork.c +0
-0@@ -857,6 +857,29 @@ void __put_task_struct(struct task_struct *tsk)…
Quoted 7 lines
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 870
Quoted 3 lines
kernel/fork.c +3
-0+void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ()))Is it safe to use the rcu member in any case? If so why not use it unconditionally?
I am unsure what would be the consequences of moving every call to RCU, so I thought it would be better to play safe and do it only when necessary.
Quoted 14 lines
Quoted 14 lines
kernel/fork.c +12
-0+/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +* +* call_rcu() will schedule delayed_put_task_struct_rcu() +* to be called in process context. +*/ +call_rcu (&tsk->rcu, delayed_put_task_struct_rcu); + else +___put_task_struct (tsk ); +}EXPORT_SYMBOL_GPL (__put_task_struct);
Quoted 4 lines
Quoted 1 line
void
__init__weakarch_task_cache_init(void){ }Sebastian
Andrew Morton
maintainerQuoted 2 lines · On Mon, 6 Feb 2023 10:04:47 -0300 Wander Lairson Costa <[email protected]>…
Under
PREEMPT_RT,__put_task_struct()indirectly acquires sleeping locks. Therefore, it can't be called from an non-preemptible context.
Well that's regrettable. Especially if non-preempt kernels don't do this.
Why does PREEMPT_RT do this and can it be fixed?
If it cannot be fixed then we should have a might_sleep() in __put_task_struct() for all kernel configurations, along with an apologetic comment explaining why.
Quoted 11 lines · Andrew Morton
Quoted 2 lines · On Mon, 6 Feb 2023 10:04:47 -0300 Wander Lairson Costa <[email protected]>…
Under
PREEMPT_RT,__put_task_struct()indirectly acquires sleeping locks. Therefore, it can't be cala from a non-preemptible context.Well that's regrettable. Especially if non-preempt kernels don't do this.
Why does
PREEMPT_RTdo this and can it be fixed?
Under PREEMPT_RT, spin_lock becomes a wrapper around rtmutex, which is a sleeping lock. This is necessary because of the deterministic scheduling requirements of the RT kernel. Most of the places that run in atomic context in the stock kernel, become process context in the RT kernel, so the change spin_lock -> rtmutex is safe. However, there are always exceptions.
In this particular case, __put_task_struct calls kcache_mem_free, which refill_obj_stock. refill_obj_stock acquires a local_lock, that is implemented using a spin_lock.
Quoted 4 lines
If it cannot be fixed then we should have a
might_sleep()in__put_task_struct()for all kernel configurations, along with an apologetic comment explaining why.
This patch is supposed to be the fix.
Quoted 7 lines · Andrew Morton
Quoted 2 lines · On Mon, 6 Feb 2023 10:04:47 -0300 Wander Lairson Costa <[email protected]>…
Under
PREEMPT_RT,__put_task_struct()indirectly acquires sleeping locks. Therefore, it can't be called from an non-preemptible context.Well that's regrettable. Especially if non-preempt kernels don't do this.
Non-preemptible context on PREEMPT_RT. Interrupts handler and timers don't count as non-preemptible because interrupt handler are threaded and hrtimers are invoked in softirq context (which is preemptible on PREEMPT_RT).
This here is different because the hrtimer in question was marked as HRTIMER_MODE_REL_HARD. In this case it is invoked in hardirq context as requested with all the problems that follow.
Quoted 1 line
Why does
PREEMPT_RTdo this and can it be fixed?
PREEMPT_RT tries to move as much as it can out of hardirq context into preemptible context. A spinlock_t is preemptible on PREEMPT_RT while it is not in other preemption models. The scheduler needs to use raw_spinlock_t in order to be able to schedule a task from hardirq-context without a deadlock.
For memory allocation only sleeping locks (spinlock_t) is used since there are no memory allocation/ deallocation on PREEMPT_RT in hardirq context. These two need to be separated.
Quoted 3 lines
If it cannot be fixed then we should have a
might_sleep()in__put_task_struct()for all kernel configurations, along with an apologetic comment explaining why.
__put_task_struct() should not be invoked in atomic context on PREEMPT_RT. It is fine however in a regular timer hrtimer. Adding might_sleep() will trigger a lot of false positives on a preemptible kernel and RT.
A might_lock() on a spinlock_t should do the trick from LOCKDEP perspective if CONFIG_PROVE_RAW_LOCK_NESTING is enabled. In this case it should be visible due to rq-lock or due to hrtimer.
Sebastian