kernel/fork: beware of __put_task_struct calling context
Subsystems (3)
PATCH v3 1/1 kernel/fork: beware of __put_task_struct calling context
Under PREEMPT_RT, __put_task_struct() indirectly acquires sleeping locks. Therefore, it can't be called from an non-preemptible context.
One practical example is splat inside inactive_task_timer(), which is called in a interrupt context:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5
Instead of calling __put_task_struct() directly, we defer it using call_rcu(). A more natural approach would use a workqueue, but since in PREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put the work_struct instance in the task_struct and initialize it when we allocate a new task_struct.
--- kernel/fork.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
kernel/fork.c +21 -1
@@ -840,7 +840,7 @@ static inline void put_signal_struct(struct signal_struct *sig)free_signal_struct (sig); } -void __put_task_struct (struct task_struct *tsk) +static void ___put_task_struct (struct task_struct *tsk) {WARN_ON (!tsk->exit_state);WARN_ON (refcount_read (&tsk->usage));
@@ -857,6 +857,26 @@ void __put_task_struct(struct task_struct *tsk)sched_core_free (tsk);free_task (tsk); } + +static void __delayed_put_task_struct (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +___put_task_struct (task); +} + +void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ())) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&tsk->rcu, __delayed_put_task_struct); +else +___put_task_struct (tsk); +}EXPORT_SYMBOL_GPL (__put_task_struct);void __init __weak arch_task_cache_init (void ) { }
Signature · 1 line
2.39.1
Quoted 7 lines · Wander Lairson Costa
Instead of calling
__put_task_struct()directly, we defer it usingcall_rcu(). A more natural approach would use a workqueue, but since inPREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put thework_structinstance in thetask_structand initialize it when we allocate a newtask_struct.
I don't think I can ack the changes in PREEMPT_RT but this version LGTM.
just a couple of purely cosmetic nits, feel free to ignore...
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 866
Quoted 6 lines
kernel/fork.c +6
-0+static void __delayed_put_task_struct (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +___put_task_struct (task); +}
We already have delayed_put_task_struct() which differs very much. Perhaps something like ___put_task_struct() will look less confusing.
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 876
Quoted 9 lines
kernel/fork.c +9
-0+void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ())) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&tsk->rcu, __delayed_put_task_struct);
Perhaps this deserves additional note to explain why is it safe to use tsk->rcu union. May be this is obvious, but I was confused when I looked at the previous version ;)
but again, feel free to ignore.
Oleg.
Quoted 17 lines · Oleg Nesterov
Quoted 7 lines · Wander Lairson Costa
Instead of calling
__put_task_struct()directly, we defer it usingcall_rcu(). A more natural approach would use a workqueue, but since inPREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put thework_structinstance in thetask_structand initialize it when we allocate a newtask_struct.I don't think I can ack the changes in
PREEMPT_RTbut this version LGTM.just a couple of purely cosmetic nits, feel free to ignore...
Quoted 10 lines
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 866
Quoted 6 lines
kernel/fork.c +6
-0+static void __delayed_put_task_struct (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +___put_task_struct (task); +}We already have
delayed_put_task_struct()which differs very much. Perhaps something like___put_task_struct()will look less confusing.
___put_task_struct()? I already added a function with this name below.
Quoted 5 lines
Quoted 5 lines
kernel/fork.c +5
-0+void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ())) +/* +* under PREEMPT_RT, we can't call put_task_struct
Quoted 9 lines
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 876
Quoted 4 lines
kernel/fork.c +4
-0+* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&tsk->rcu, __delayed_put_task_struct);Perhaps this deserves additional note to explain why is it safe to use
tsk->rcuunion. May be this is obvious, but I was confused when I looked at the previous version ;)
Makes sense, I will add it in the next version.
Quoted 4 lines
but again, feel free to ignore.
Oleg.
Quoted 6 lines · Wander Lairson Costa
Quoted 4 lines
We already have
delayed_put_task_struct()which differs very much. Perhaps something like___put_task_struct()will look less confusing.
___put_task_struct()? I already added a function with this name below.
Ah, I meant ___put_task_struct_rcu() or something like this. Bug again this is just cosmetic nit, please ignore
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
kernel/fork.c +5
-0+void __put_task_struct (struct task_struct *tsk) +{ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (!preemptible () || !in_task ())) +/* +* under PREEMPT_RT, we can't call put_task_struct Quoted 9 lines
kernel/fork.c void __put_task_struct(struct task_struct *tsk) · line 876
Quoted 4 lines
kernel/fork.c +4
-0+* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&tsk->rcu, __delayed_put_task_struct);Perhaps this deserves additional note to explain why is it safe to use
tsk->rcuunion. May be this is obvious, but I was confused when I looked at the previous version ;)
Quoted 1 line
Makes sense, I will add it in the next version.
Thanks ;)
Oleg.