sched/deadline: fix inactive_task_timer splat with CONFIG_PREEMPT_RT
PATCH 1/1 sched/deadline: fix inactive_task_timer splat with CONFIG_PREEMPT_RT
inactive_task_timer() executes in interrupt (atomic) context. It calls put_task_struct(), which indirectly acquires sleeping locks under PREEMPT_RT.
Below is an example of a splat that happened in a test environment:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5
Instead of calling put_task_struct() directly, we defer it using call_rcu(). A more natural approach would use a workqueue, but since in PREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put the work_struct instance in the task_struct and initialize it when we allocate a new task_struct.
--- kernel/sched/build_policy.c | 1 + kernel/sched/deadline.c | 24 +++++++++++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-)
kernel/sched/build_policy.c +1 -0
@@ -28,6 +28,7 @@#include <linux/suspend.h> #include <linux/tsacct_kern.h> #include <linux/vtime.h> +#include <linux/rcupdate.h> #include <uapi/linux/sched/types.h>
kernel/sched/deadline.c +23 -1
@@ -1405,6 +1405,13 @@ static void update_curr_dl(struct rq *rq) } } +static void delayed_put_task_struct (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +__put_task_struct (task); +} +static enum hrtimer_restart inactive_task_timer (struct hrtimer *timer) { struct sched_dl_entity *dl_se = container_of(timer,
@@ -1442,7 +1449,22 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) dl_se->dl_non_contending =0 ; unlock:task_rq_unlock (rq, p, &rf); -put_task_struct (p); + +if (IS_ENABLED (CONFIG_PREEMPT_RT )) { +/* +* Decrement the refcount explicitly to avoid unnecessarily +* calling call_rcu. +*/ +if (refcount_dec_and_test (&p->usage)) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&p->rcu, delayed_put_task_struct); + }else { +put_task_struct (p); + }return HRTIMER_NORESTART ; }
Signature · 1 line
2.39.0
Quoted 70 lines · Wander Lairson Costa
inactive_task_timer()executes in interrupt (atomic) context. It callsput_task_struct(), which indirectly acquires sleeping locks underPREEMPT_RT.Below is an example of a splat that happened in a test environment:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5Instead of calling
put_task_struct()directly, we defer it usingcall_rcu(). A more natural approach would use a workqueue, but since inPREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put thework_structinstance in thetask_structand initialize it when we allocate a newtask_struct.Signed-off-by Wander Lairson Costa <[email protected]>Cc Paul McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>--- kernel/sched/build_policy.c | 1 + kernel/sched/deadline.c | 24 +++++++++++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-)kernel/sched/build_policy.c +1
-0@@ -28,6 +28,7 @@#include <linux/suspend.h> #include <linux/tsacct_kern.h> #include <linux/vtime.h> +#include <linux/rcupdate.h> #include <uapi/linux/sched/types.h> kernel/sched/deadline.c +0
-0
kernel/sched/deadline.c static void update_curr_dl(struct rq *rq) · line 1412
Quoted 8 lines
kernel/sched/deadline.c +5
-0@@ -1405,6 +1405,13 @@ static void update_curr_dl(struct rq *rq) } } +static void delayed_put_task_struct (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +__put_task_struct (task);
Please note that BH is disabled here. Don't you therefore need to schedule a workqueue handler? Perhaps directly from inactive_task_timer(), or maybe from this point. If the latter, one way to skip the extra step is to use queue_rcu_work().
Thanx, Paul
Quoted 32 lines
kernel/sched/deadline.c +18
-1+} +static enum hrtimer_restart inactive_task_timer (struct hrtimer *timer) { struct sched_dl_entity *dl_se = container_of(timer,@@ -1442,7 +1449,22 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) dl_se->dl_non_contending =0 ; unlock:task_rq_unlock (rq, p, &rf); -put_task_struct (p); + +if (IS_ENABLED (CONFIG_PREEMPT_RT )) { +/* +* Decrement the refcount explicitly to avoid unnecessarily +* calling call_rcu. +*/ +if (refcount_dec_and_test (&p->usage)) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&p->rcu, delayed_put_task_struct); + }else { +put_task_struct (p); + }return HRTIMER_NORESTART ; }Signature · 1 line
2.39.0
Quoted 75 lines · Paul E. McKenney
Quoted 73 lines · Wander Lairson Costa
inactive_task_timer()executes in interrupt (atomic) context. It callsput_task_struct(), which indirectly acquires sleeping locks underPREEMPT_RT.Below is an example of a splat that happened in a test environment:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5Instead of calling
put_task_struct()directly, we defer it usingcall_rcu(). A more natural approach would use a workqueue, but since inPREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put thework_structinstance in thetask_structand initialize it when we allocate a newtask_struct.Signed-off-by Wander Lairson Costa <[email protected]>Cc Paul McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>--- kernel/sched/build_policy.c | 1 + kernel/sched/deadline.c | 24 +++++++++++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-)kernel/sched/build_policy.c +1
-0@@ -28,6 +28,7 @@#include <linux/suspend.h> #include <linux/tsacct_kern.h> #include <linux/vtime.h> +#include <linux/rcupdate.h> #include <uapi/linux/sched/types.h> kernel/sched/deadline.c +0
-0@@ -1405,6 +1405,13 @@ static void update_curr_dl(struct rq *rq) } }
Quoted 11 lines
kernel/sched/deadline.c static void update_curr_dl(struct rq *rq) · line 1412
Quoted 5 lines
kernel/sched/deadline.c +5
-0+static void delayed_put_task_struct (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +__put_task_struct (task);Please note that BH is disabled here. Don't you therefore need to schedule a workqueue handler? Perhaps directly from
inactive_task_timer(), or maybe from this point. If the latter, one way to skip the extra step is to usequeue_rcu_work().
My initial work was using a workqueue [1,2]. However, I realized I could reach a much simpler code with call_rcu().
I am afraid my ignorance doesn't allow me to get your point. Does disabling softirq imply atomic context?
[1] https://gitlab.com/walac/kernel-ark/-/commit/ec8addbe38d5c318f1789b4c0fa480a9d2afdb65
[2] https://gitlab.com/walac/kernel-ark/-/commit/0bde233235ffed233a7466a36a4866bc48064f54
Quoted 35 lines
Thanx, Paul
Quoted 32 lines
kernel/sched/deadline.c +18
-2+} +static enum hrtimer_restart inactive_task_timer (struct hrtimer *timer) { struct sched_dl_entity *dl_se = container_of(timer,@@ -1442,7 +1449,22 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) dl_se->dl_non_contending =0 ; unlock:task_rq_unlock (rq, p, &rf); -put_task_struct (p); + +if (IS_ENABLED (CONFIG_PREEMPT_RT )) { +/* +* Decrement the refcount explicitly to avoid unnecessarily +* calling call_rcu. +*/ +if (refcount_dec_and_test (&p->usage)) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&p->rcu, delayed_put_task_struct); + }else { +put_task_struct (p); + }return HRTIMER_NORESTART ; } --2.39.0
Quoted 88 lines · Wander Lairson Costa
Quoted 75 lines · Paul E. McKenney
Quoted 73 lines · Wander Lairson Costa
inactive_task_timer()executes in interrupt (atomic) context. It callsput_task_struct(), which indirectly acquires sleeping locks underPREEMPT_RT.Below is an example of a splat that happened in a test environment:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5Instead of calling
put_task_struct()directly, we defer it usingcall_rcu(). A more natural approach would use a workqueue, but since inPREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put thework_structinstance in thetask_structand initialize it when we allocate a newtask_struct.Signed-off-by Wander Lairson Costa <[email protected]>Cc Paul McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>--- kernel/sched/build_policy.c | 1 + kernel/sched/deadline.c | 24 +++++++++++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-)kernel/sched/build_policy.c +1
-0@@ -28,6 +28,7 @@#include <linux/suspend.h> #include <linux/tsacct_kern.h> #include <linux/vtime.h> +#include <linux/rcupdate.h> #include <uapi/linux/sched/types.h> kernel/sched/deadline.c +0
-0@@ -1405,6 +1405,13 @@ static void update_curr_dl(struct rq *rq) } }Quoted 11 lines
kernel/sched/deadline.c static void update_curr_dl(struct rq *rq) · line 1412
Quoted 5 lines
kernel/sched/deadline.c +5
-0+static void delayed_put_task_struct (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +__put_task_struct (task);Please note that BH is disabled here. Don't you therefore need to schedule a workqueue handler? Perhaps directly from
inactive_task_timer(), or maybe from this point. If the latter, one way to skip the extra step is to usequeue_rcu_work().
Quoted 4 lines
My initial work was using a workqueue [1,2]. However, I realized I could reach a much simpler code with
call_rcu().
I am afraid my ignorance doesn't allow me to get your point. Does disabling softirq imply atomic context?
Given that this problem occurred in PREEMPT_RT, I am assuming that the appropriate definition of "atomic context" is "cannot call schedule()". And you are in fact not permitted to call schedule() from a bh-disabled region.
This also means that you cannot acquire a non-raw spinlock in a bh-disabled region of code in a PREEMPT_RT kernel, because doing so can invoke schedule.
Of course, using a workqueue does incur needless overhead in non-PREEMPT_RT kernels. So one alternative approach is to use the workqueue only in PREEMPT_RT kernels and to just invoke __put_task_struct() directly (without call_rcu() along the way) otherwise.
Does that help, or am I missing your point?
Thanx, Paul
Quoted 40 lines
[1] https://gitlab.com/walac/kernel-ark/-/commit/ec8addbe38d5c318f1789b4c0fa480a9d2afdb65
[2] https://gitlab.com/walac/kernel-ark/-/commit/0bde233235ffed233a7466a36a4866bc48064f54Quoted 35 lines
Thanx, Paul
Quoted 32 lines
kernel/sched/deadline.c +18
-2+} +static enum hrtimer_restart inactive_task_timer (struct hrtimer *timer) { struct sched_dl_entity *dl_se = container_of(timer,@@ -1442,7 +1449,22 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) dl_se->dl_non_contending =0 ; unlock:task_rq_unlock (rq, p, &rf); -put_task_struct (p); + +if (IS_ENABLED (CONFIG_PREEMPT_RT )) { +/* +* Decrement the refcount explicitly to avoid unnecessarily +* calling call_rcu. +*/ +if (refcount_dec_and_test (&p->usage)) +/* +* under PREEMPT_RT, we can't call put_task_struct +* in atomic context because it will indirectly +* acquire sleeping locks. +*/ +call_rcu (&p->rcu, delayed_put_task_struct); + }else { +put_task_struct (p); + }return HRTIMER_NORESTART ; } --2.39.0
Valentin Schneider
reviewerQuoted 94 lines · Paul E. McKenney
Quoted 88 lines · Wander Lairson Costa
Quoted 75 lines · Paul E. McKenney
Quoted 73 lines · Wander Lairson Costa
inactive_task_timer()executes in interrupt (atomic) context. It callsput_task_struct(), which indirectly acquires sleeping locks underPREEMPT_RT.Below is an example of a splat that happened in a test environment:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5Instead of calling
put_task_struct()directly, we defer it usingcall_rcu(). A more natural approach would use a workqueue, but since inPREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put thework_structinstance in thetask_structand initialize it when we allocate a newtask_struct.Signed-off-by Wander Lairson Costa <[email protected]>Cc Paul McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>--- kernel/sched/build_policy.c | 1 + kernel/sched/deadline.c | 24 +++++++++++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-)kernel/sched/build_policy.c +1
-0@@ -28,6 +28,7 @@#include <linux/suspend.h> #include <linux/tsacct_kern.h> #include <linux/vtime.h> +#include <linux/rcupdate.h> #include <uapi/linux/sched/types.h> kernel/sched/deadline.c +0
-0@@ -1405,6 +1405,13 @@ static void update_curr_dl(struct rq *rq) } }Quoted 11 lines
kernel/sched/deadline.c static void update_curr_dl(struct rq *rq) · line 1412
Quoted 5 lines
kernel/sched/deadline.c +5
-0+static void delayed_put_task_struct (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +__put_task_struct (task);Please note that BH is disabled here. Don't you therefore need to schedule a workqueue handler? Perhaps directly from
inactive_task_timer(), or maybe from this point. If the latter, one way to skip the extra step is to usequeue_rcu_work().Quoted 4 lines
My initial work was using a workqueue [1,2]. However, I realized I could reach a much simpler code with
call_rcu().
I am afraid my ignorance doesn't allow me to get your point. Does disabling softirq imply atomic context?
Quoted 9 lines
Given that this problem occurred in
PREEMPT_RT, I am assuming that the appropriate definition of "atomic context" is "cannot callschedule()". And you are in fact not permitted to callschedule()from a bh-disabled region.This also means that you cannot acquire a non-raw spinlock in a bh-disabled region of code in a
PREEMPT_RTkernel, because doing so can invoke schedule.
But per the PREEMPT_RT lock "replacement", non-raw spinlocks end up invoking schedule_rtlock(), which should be safe vs BH disabled (local_lock() + rcu_read_lock()):
6991436c2b5d ("sched/core: Provide a scheduling point for RT locks")
Unless I'm missing something else?
Quoted 104 lines · Valentin Schneider
Quoted 94 lines · Paul E. McKenney
Quoted 88 lines · Wander Lairson Costa
Quoted 75 lines · Paul E. McKenney
Quoted 73 lines · Wander Lairson Costa
inactive_task_timer()executes in interrupt (atomic) context. It callsput_task_struct(), which indirectly acquires sleeping locks underPREEMPT_RT.Below is an example of a splat that happened in a test environment:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5Instead of calling
put_task_struct()directly, we defer it usingcall_rcu(). A more natural approach would use a workqueue, but since inPREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put thework_structinstance in thetask_structand initialize it when we allocate a newtask_struct.Signed-off-by Wander Lairson Costa <[email protected]>Cc Paul McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>--- kernel/sched/build_policy.c | 1 + kernel/sched/deadline.c | 24 +++++++++++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-)kernel/sched/build_policy.c +1
-0@@ -28,6 +28,7 @@#include <linux/suspend.h> #include <linux/tsacct_kern.h> #include <linux/vtime.h> +#include <linux/rcupdate.h> #include <uapi/linux/sched/types.h> kernel/sched/deadline.c +0
-0@@ -1405,6 +1405,13 @@ static void update_curr_dl(struct rq *rq) } }Quoted 11 lines
kernel/sched/deadline.c static void update_curr_dl(struct rq *rq) · line 1412
Quoted 5 lines
kernel/sched/deadline.c +5
-0+static void delayed_put_task_struct (struct rcu_head *rhp) +{ + struct task_struct *task = container_of(rhp, struct task_struct, rcu); + +__put_task_struct (task);Please note that BH is disabled here. Don't you therefore need to schedule a workqueue handler? Perhaps directly from
inactive_task_timer(), or maybe from this point. If the latter, one way to skip the extra step is to usequeue_rcu_work().Quoted 4 lines
My initial work was using a workqueue [1,2]. However, I realized I could reach a much simpler code with
call_rcu().
I am afraid my ignorance doesn't allow me to get your point. Does disabling softirq imply atomic context?Quoted 8 lines
Given that this problem occurred in
PREEMPT_RT, I am assuming that the appropriate definition of "atomic context" is "cannot callschedule()". And you are in fact not permitted to callschedule()from a bh-disabled region.This also means that you cannot acquire a non-raw spinlock in a bh-disabled region of code in a
PREEMPT_RTkernel, because doing so can invoke schedule.
Quoted 7 lines
But per the
PREEMPT_RTlock "replacement", non-raw spinlocks end up invokingschedule_rtlock(), which should be safe vs BH disabled (local_lock()+rcu_read_lock()):
6991436c2b5d("sched/core: Provide a scheduling point for RT locks")Unless I'm missing something else?
No, you miss nothing. Apologies for my confusion!
(I could have sworn that someone else corrected me on this earlier, but I don't see it right off hand.)
Thanx, Paul
Quoted 16 lines · Paul E. McKenney
Quoted 15 lines · Valentin Schneider
Quoted 8 lines
Given that this problem occurred in
PREEMPT_RT, I am assuming that the appropriate definition of "atomic context" is "cannot callschedule()". And you are in fact not permitted to callschedule()from a bh-disabled region.This also means that you cannot acquire a non-raw spinlock in a bh-disabled region of code in a
PREEMPT_RTkernel, because doing so can invoke schedule.But per the
PREEMPT_RTlock "replacement", non-raw spinlocks end up invokingschedule_rtlock(), which should be safe vs BH disabled (local_lock()+rcu_read_lock()):
6991436c2b5d("sched/core: Provide a scheduling point for RT locks")
Quoted 8 lines
Quoted 1 line
Unless I'm missing something else?
No, you miss nothing. Apologies for my confusion!
(I could have sworn that someone else corrected me on this earlier, but I don't see it right off hand.)
Thanx, Paul
Heh, I had a smidge of doubt myself, but since we've cleared this up:
Valentin Schneider
reviewerQuoted 38 lines · Wander Lairson Costa
inactive_task_timer()executes in interrupt (atomic) context. It callsput_task_struct(), which indirectly acquires sleeping locks underPREEMPT_RT.Below is an example of a splat that happened in a test environment:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5
Quoted 7 lines
Instead of calling
put_task_struct()directly, we defer it usingcall_rcu(). A more natural approach would use a workqueue, but since inPREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put thework_structinstance in thetask_structand initialize it when we allocate a newtask_struct.
Sorry to come back on this; Juri reminded me offline that put_task_struct() is invoked in other non-sleepable contexts, not just inactive_task_timer().
e.g.
rto_push_irq_work_func() // hard irq work so hardirq context
`\
push_rt_task()
`\
put_task_struct()Or
cpu_stopper_thread() // stopper callbacks must not sleep
`\
push_cpu_stop()
`\
put_task_struct()... But then again I'm not aware of any splats happening in these paths. Is there something special about inactive_task_timer(), or could it be the issue is there for those other paths but we just haven't had them reported yet?
Quoted 48 lines · Valentin Schneider
Quoted 38 lines · Wander Lairson Costa
inactive_task_timer()executes in interrupt (atomic) context. It callsput_task_struct(), which indirectly acquires sleeping locks underPREEMPT_RT.Below is an example of a splat that happened in a test environment:
CPU: 1 PID: 2848 Comm: life Kdump: loaded Tainted: G W --------- Hardware name: HP ProLiant DL388p Gen8, BIOS P70 07/15/2012 Call Trace: dump_stack_lvl+0x57/0x7d mark_lock_irq.cold+0x33/0xba ? stack_trace_save+0x4b/0x70 ? save_trace+0x55/0x150 mark_lock+0x1e7/0x400 mark_usage+0x11d/0x140 __lock_acquire+0x30d/0x930 lock_acquire.part.0+0x9c/0x210 ? refill_obj_stock+0x3d/0x3a0 ? rcu_read_lock_sched_held+0x3f/0x70 ? trace_lock_acquire+0x38/0x140 ? lock_acquire+0x30/0x80 ? refill_obj_stock+0x3d/0x3a0 rt_spin_lock+0x27/0xe0 ? refill_obj_stock+0x3d/0x3a0 refill_obj_stock+0x3d/0x3a0 ? inactive_task_timer+0x1ad/0x340 kmem_cache_free+0x357/0x560 inactive_task_timer+0x1ad/0x340 ? switched_from_dl+0x2d0/0x2d0 __run_hrtimer+0x8a/0x1a0 __hrtimer_run_queues+0x91/0x130 hrtimer_interrupt+0x10f/0x220 __sysvec_apic_timer_interrupt+0x7b/0xd0 sysvec_apic_timer_interrupt+0x4f/0xd0 ? asm_sysvec_apic_timer_interrupt+0xa/0x20 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0033:0x7fff196bf6f5Quoted 7 lines
Instead of calling
put_task_struct()directly, we defer it usingcall_rcu(). A more natural approach would use a workqueue, but since inPREEMPT_RT, we can't allocate dynamic memory from atomic context, the code would become more complex because we would need to put thework_structinstance in thetask_structand initialize it when we allocate a newtask_struct.
Quoted 3 lines
Sorry to come back on this; Juri reminded me offline that
put_task_struct()is invoked in other non-sleepable contexts, not justinactive_task_timer().
I guess there is no splat because the usage count doesn't reach zero in those code paths.
Quoted 8 lines
e.g.
rto_push_irq_work_func() // hard irq work so hardirq context `\ push_rt_task() `\ put_task_struct()
This is paired with a get_task_struct() a few lines above inside in the same function.
Quoted 8 lines
Or
cpu_stopper_thread() // stopper callbacks must not sleep `\ push_cpu_stop() `\ put_task_struct()
This is paired with a get_task_struct() from get_push_task()
Quoted 5 lines
... But then again I'm not aware of any splats happening in these paths. Is there something special about
inactive_task_timer(), or could it be the issue is there for those other paths but we just haven't had them reported yet?
Given that those calls have corresponding get_task_struct() calls that are close in time, there is a low probability of the usage count reaching zero and triggering the splat. In any case, I will work in a v2 that also addresses those call sites.