kernel/fork: beware of __put_task_struct calling context
Subsystems (3)
v10 cover kernel/fork: beware of __put_task_struct calling context
Under PREEMPT_RT, __put_task_struct() indirectly acquires sleeping locks. Therefore, it can't be called from an non-preemptible context.
Instead of calling __put_task_struct() directly, we defer it using call_rcu(). A more natural approach would use a workqueue, but since in PREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put the work_struct instance in the task_struct and initialize it when we allocate a new task_struct.
Changelog
=========
v1:
* Initial implementation fixing the splat.
v2:
* Isolate the logic in its own function.
* Fix two more cases caught in review.
v3:
* Change __put_task_struct() to handle the issue internally.
v4:
* Explain why call_rcu() is safe to call from interrupt context.
v5:
* Explain why __put_task_struct() doesn't conflict with put_task_sruct_rcu_user.
v6:
* As per Sebastian's review, revert back the implementation of v2 with a distinct function.
* Add a check in put_task_struct() to warning when called from a non-sleepable context.
* Address more call sites.
v7:
* Fix typos.
* Add an explanation why the new function doesn't conflict with delayed_free_task().
v8:
* Bring back v5.
* Fix coding style.
v9:
* Reorganize to not need ___put_task_struct() by Oleg's suggestion.
v10:
* Add a patch preventing a splat when compile with CONFIG_PROVE_RAW_LOCK_NESTING.
Wander Lairson Costa (2):
kernel/fork: beware of __put_task_struct calling context
sched: avoid false lockdep splat in put_task_struct()
include/linux/sched/task.h | 38 +++++++++++++++++++++++++++++++++++++- kernel/fork.c | 8 ++++++++ 2 files changed, 45 insertions(+), 1 deletion(-)
Signature · 1 line
2.40.1
PATCH v10 1/2 kernel/fork: beware of __put_task_struct calling context
Under PREEMPT_RT, __put_task_struct() indirectly acquires sleeping locks. Therefore, it can't be called from an non-preemptible context.
One practical example is splat inside inactive_task_timer(), which is called in a interrupt context:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5
Instead of calling __put_task_struct() directly, we defer it using call_rcu(). A more natural approach would use a workqueue, but since in PREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put the work_struct instance in the task_struct and initialize it when we allocate a new task_struct.
The issue is reproducible with stress-ng:
while true; do
stress-ng --sched deadline --sched-period 1000000000 \
--sched-runtime 800000000 --sched-deadline \
1000000000 --mmapfork 23 -t 20done
--- include/linux/sched/task.h | 28 +++++++++++++++++++++++++++- kernel/fork.c | 8 ++++++++ 2 files changed, 35 insertions(+), 1 deletion(-)
include/linux/sched/task.h +27 -1
@@ -118,10 +118,36 @@ static inline struct task_struct *get_task_struct(struct task_struct *t) }extern void __put_task_struct (struct task_struct *t); +extern void __put_task_struct_rcu_cb (struct rcu_head *rhp);static inline void put_task_struct (struct task_struct *t) { -if (refcount_dec_and_test (&t->usage)) +if (!refcount_dec_and_test (&t->usage)) +return ; + +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +* +* call_rcu() will schedule delayed_put_task_struct_rcu() +* to be called in process context. +* +* __put_task_struct() is called when +* refcount_dec_and_test(&t->usage) succeeds. +* +* This means that it can't "conflict" with +* put_task_struct_rcu_user() which abuses ->rcu the same +* way; rcu_users has a reference so task->usage can't be +* zero after rcu_users 1 -> 0 transition. +* +* delayed_free_task() also uses ->rcu, but it is only called +* when it fails to fork a process. Therefore, there is no +* way it can conflict with put_task_struct(). +*/ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && !preemptible ()) +call_rcu (&t->rcu, __put_task_struct_rcu_cb); +else __put_task_struct (t); }
kernel/fork.c +8 -0
@@ -988,6 +988,14 @@ void __put_task_struct(struct task_struct *tsk) }EXPORT_SYMBOL_GPL (__put_task_struct); +void __put_task_struct_rcu_cb (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +__put_task_struct (task); +} +EXPORT_SYMBOL_GPL (__put_task_struct_rcu_cb); +void __init __weak arch_task_cache_init (void ) { }/*
Signature · 1 line
2.40.1
[tip: sched/core] kernel/fork: beware of __put_task_struct() calling context
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: d243b34459cea30cfe5f3a9b2feb44e7daff9938 Gitweb: https://git.kernel.org/tip/d243b34459cea30cfe5f3a9b2feb44e7daff9938 Author: Wander Lairson Costa <[email protected]> AuthorDate: Wed, 14 Jun 2023 09:23:21 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 13 Jul 2023 15:21:48 +02:00
kernel/fork: beware of __put_task_struct() calling context
Under PREEMPT_RT, __put_task_struct() indirectly acquires sleeping locks. Therefore, it can't be called from an non-preemptible context.
One practical example is splat inside inactive_task_timer(), which is called in a interrupt context:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 rt_spin_lock+0x27/0xe0 refill_obj_stock+0x3d/0x3a0 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5
Instead of calling __put_task_struct() directly, we defer it using call_rcu(). A more natural approach would use a workqueue, but since in PREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put the work_struct instance in the task_struct and initialize it when we allocate a new task_struct.
The issue is reproducible with stress-ng:
while true; do
stress-ng --sched deadline --sched-period 1000000000 \ --sched-runtime 800000000 --sched-deadline \ 1000000000 --mmapfork 23 -t 20
done
--- include/linux/sched/task.h | 28 +++++++++++++++++++++++++++- kernel/fork.c | 8 ++++++++ 2 files changed, 35 insertions(+), 1 deletion(-)
include/linux/sched/task.h +27 -1
@@ -118,10 +118,36 @@ static inline struct task_struct *get_task_struct(struct task_struct *t) }extern void __put_task_struct (struct task_struct *t); +extern void __put_task_struct_rcu_cb (struct rcu_head *rhp);static inline void put_task_struct (struct task_struct *t) { -if (refcount_dec_and_test (&t->usage)) +if (!refcount_dec_and_test (&t->usage)) +return ; + +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +* +* call_rcu() will schedule delayed_put_task_struct_rcu() +* to be called in process context. +* +* __put_task_struct() is called when +* refcount_dec_and_test(&t->usage) succeeds. +* +* This means that it can't "conflict" with +* put_task_struct_rcu_user() which abuses ->rcu the same +* way; rcu_users has a reference so task->usage can't be +* zero after rcu_users 1 -> 0 transition. +* +* delayed_free_task() also uses ->rcu, but it is only called +* when it fails to fork a process. Therefore, there is no +* way it can conflict with put_task_struct(). +*/ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && !preemptible ()) +call_rcu (&t->rcu, __put_task_struct_rcu_cb); +else __put_task_struct (t); }
kernel/fork.c +8 -0
@@ -985,6 +985,14 @@ void __put_task_struct(struct task_struct *tsk) }EXPORT_SYMBOL_GPL (__put_task_struct); +void __put_task_struct_rcu_cb (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +__put_task_struct (task); +} +EXPORT_SYMBOL_GPL (__put_task_struct_rcu_cb); +void __init __weak arch_task_cache_init (void ) { }/*
PATCH v10 2/2 sched: avoid false lockdep splat in put_task_struct()
In reply to Wander Lairson CostaIn put_task_struct(), a spin_lock is indirectly acquired under the kernel stock. When running the kernel in real-time (RT) configuration, the operation is dispatched to a preemptible context call to ensure guaranteed preemption. However, if PROVE_RAW_LOCK_NESTING is enabled and __put_task_struct() is called while holding a raw_spinlock, lockdep incorrectly reports an "Invalid lock context" in the stock kernel.
This false splat occurs because lockdep is unaware of the different route taken under RT. To address this issue, override the inner wait type to prevent the false lockdep splat.
--- include/linux/sched/task.h | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
include/linux/sched/task.h +14 -4
@@ -125,6 +125,19 @@ static inline void put_task_struct(struct task_struct *t)if (!refcount_dec_and_test (&t->usage))return ; +/* +* In !RT, it is always safe to call __put_task_struct(). +* Under RT, we can only call it in preemptible context. +*/ +if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { + static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP); + +lock_map_acquire_try (&put_task_map); +__put_task_struct (t); +lock_map_release (&put_task_map); +return ; + } +/* * under PREEMPT_RT, we can't call put_task_struct * in atomic context because it will indirectly
@@ -145,10 +158,7 @@ static inline void put_task_struct(struct task_struct *t)* when it fails to fork a process. Therefore, there is no * way it can conflict with put_task_struct(). */ -if (IS_ENABLED (CONFIG_PREEMPT_RT ) && !preemptible ()) -call_rcu (&t->rcu, __put_task_struct_rcu_cb); -else -__put_task_struct (t); +call_rcu (&t->rcu, __put_task_struct_rcu_cb); }static inline void put_task_struct_many (struct task_struct *t,int nr)
Signature · 1 line
2.40.1
[tip: sched/core] sched: avoid false lockdep splat in put_task_struct()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 893cdaaa3977be6afb3a7f756fbfd7be83f68d8c Gitweb: https://git.kernel.org/tip/893cdaaa3977be6afb3a7f756fbfd7be83f68d8c Author: Wander Lairson Costa <[email protected]> AuthorDate: Wed, 14 Jun 2023 09:23:22 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 13 Jul 2023 15:21:48 +02:00
sched: avoid false lockdep splat in put_task_struct()
In put_task_struct(), a spin_lock is indirectly acquired under the kernel stock. When running the kernel in real-time (RT) configuration, the operation is dispatched to a preemptible context call to ensure guaranteed preemption. However, if PROVE_RAW_LOCK_NESTING is enabled and __put_task_struct() is called while holding a raw_spinlock, lockdep incorrectly reports an "Invalid lock context" in the stock kernel.
This false splat occurs because lockdep is unaware of the different route taken under RT. To address this issue, override the inner wait type to prevent the false lockdep splat.
--- include/linux/sched/task.h | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
include/linux/sched/task.h +14 -4
@@ -126,6 +126,19 @@ static inline void put_task_struct(struct task_struct *t)return ;/* +* In !RT, it is always safe to call __put_task_struct(). +* Under RT, we can only call it in preemptible context. +*/ +if (!IS_ENABLED (CONFIG_PREEMPT_RT ) ||preemptible ()) { + static DEFINE_WAIT_OVERRIDE_MAP(put_task_map, LD_WAIT_SLEEP); + +lock_map_acquire_try (&put_task_map); +__put_task_struct (t); +lock_map_release (&put_task_map); +return ; + } + +/* * under PREEMPT_RT, we can't call put_task_struct * in atomic context because it will indirectly * acquire sleeping locks.
@@ -145,10 +158,7 @@ static inline void put_task_struct(struct task_struct *t)* when it fails to fork a process. Therefore, there is no * way it can conflict with put_task_struct(). */ -if (IS_ENABLED (CONFIG_PREEMPT_RT ) && !preemptible ()) -call_rcu (&t->rcu, __put_task_struct_rcu_cb); -else -__put_task_struct (t); +call_rcu (&t->rcu, __put_task_struct_rcu_cb); } DEFINE_FREE(put_task, struct task_struct *, if (_T) put_task_struct(_T))
Peter Zijlstra
maintainerQuoted 11 lines · Wander Lairson Costa
Under
PREEMPT_RT,__put_task_struct()indirectly acquires sleeping locks. Therefore, it can't be called from an non-preemptible context.Instead of calling
__put_task_struct()directly, we defer it usingcall_rcu(). A more natural approach would use a workqueue, but since inPREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put thework_structinstance in thetask_structand initialize it when we allocate a newtask_struct.
Quoted 3 lines
Wander Lairson Costa (2):
kernel/fork: beware of__put_task_structcalling context
sched: avoid false lockdep splat input_task_struct()
Thanks!