WARN ON at kernel/sched/deadline.c task_non_contending
Hi.
When looking to test SCHED_DEADLINE syzkaller report an warn in task_non_contending(). I tested the mainline kernel with the C program and captured the same call trace.
[The previous message contains some strings in other formats,
making the mail less readable. So I resend it. SORRY.]
[ 948.126369] WARNING: CPU: 4 PID: 17089 at kernel/sched/deadline.c:255 task_non_contending+0xae0/0x1950
[ 948.130198] Kernel panic - not syncing: panic_on_warn set ...
[ 948.130198]
[ 948.134221] CPU: 4 PID: 17089 Comm: syz-executor.1 Not tainted 4.19.27 #2
[ 948.139072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 948.141603] Call Trace:
[ 948.142277] dump_stack+0xca/0x13e
[ 948.164636] panic+0x1f7/0x543
[ 948.168704] ? refcount_error_report+0x29d/0x29d
[ 948.172438] ? __warn+0x1d1/0x210
[ 948.183359] ? task_non_contending+0xae0/0x1950
[ 948.191747] __warn+0x1ec/0x210
[ 948.196276] ? task_non_contending+0xae0/0x1950
[ 948.202476] report_bug+0x1ee/0x2b0
[ 948.204622] fixup_bug.part.7+0x37/0x80
[ 948.206879] do_error_trap+0x22c/0x290
[ 948.211340] ? math_error+0x2f0/0x2f0
[ 948.217033] ? trace_hardirqs_off_caller+0x40/0x190
[ 948.222477] ? trace_hardirqs_off_thunk+0x1a/0x1c
[ 948.229877] invalid_op+0x14/0x20
[ 948.238317] RIP: 0010:task_non_contending+0xae0/0x1950
[ 948.253825] Code: 6d 29 83 48 89 4c 24 20 48 89 54 24 10 c6 05 d0 89 5a 03 01 e8 11 ea ee ff 0f 0b 48 8b 4c 24 20 48 8b 54 24 10 e9 bb f7 ff ff <0f> 0b e9 1d f6 ff ff e8 d4 a7 09 00 85 c0 0f 85 74 f8 ff ff 48 c7 [ 948.272329] RSP: 0018:ffff8883d443f8c0 EFLAGS: 00010002
[ 948.293045] RAX: 0000000000000001 RBX: ffff8883d3572468 RCX: ffffffff813a6571
[ 948.300323] RDX: 00000000000008ab RSI: ffffc900030e4000 RDI: ffff8883e2fe6278
[ 948.305278] RBP: ffff8883e2f00000 R08: ffffed1078ea3ab2 R09: ffffed1078ea3ab2
[ 948.316441] R10: 0000000000000001 R11: ffffed1078ea3ab1 R12: 000000000002c680
[ 948.320257] R13: ffff8883d357217c R14: 0000000000000001 R15: ffff8883d3572140
[ 948.324500] ? hrtimer_active+0x171/0x1f0
[ 948.327421] ? dequeue_task_dl+0x38/0x970
[ 948.330572] __schedule+0x94b/0x1a80
[ 948.333578] ? __sched_text_start+0x8/0x8
[ 948.336141] ? lock_downgrade+0x5e0/0x5e0
[ 948.338111] ? plist_add+0x23e/0x480
[ 948.339706] schedule+0x7c/0x1a0
[ 948.341395] futex_wait_queue_me+0x319/0x600
[ 948.343329] ? get_futex_key_refs+0xd0/0xd0
[ 948.345037] ? lock_downgrade+0x5e0/0x5e0
[ 948.347206] ? get_futex_key_refs+0xa4/0xd0
[ 948.353007] futex_wait+0x1e7/0x590
[ 948.355328] ? futex_wait_setup+0x2b0/0x2b0
[ 948.360578] ? __lock_acquire+0x60c/0x3b70
[ 948.369186] ? __save_stack_trace+0x92/0x100
[ 948.374344] ? hash_futex+0x15/0x210
[ 948.376832] ? drop_futex_key_refs+0x3c/0xd0
[ 948.378591] ? futex_wake+0x14e/0x450
[ 948.381609] do_futex+0x5c9/0x15e0
[ 948.384567] ? perf_syscall_enter+0xb1/0xc80
[ 948.390307] ? exit_robust_list+0x240/0x240
[ 948.393566] ? ftrace_syscall_exit+0x5c0/0x5c0
[ 948.396369] ? lock_downgrade+0x5e0/0x5e0
[ 948.401748] ? __might_fault+0x17c/0x1c0
[ 948.404171] __x64_sys_futex+0x296/0x380
[ 948.406472] ? __ia32_sys_futex+0x370/0x370
[ 948.440630] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 948.441774] ? trace_hardirqs_off_caller+0x40/0x190
[ 948.442770] ? do_syscall_64+0x3b/0x580
[ 948.486728] do_syscall_64+0xc8/0x580
[ 948.489138] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 948.492072] RIP: 0033:0x462eb9
[ 948.492788] Code: f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 [ 948.532016] RSP: 002b:00007f7ac8a67cd8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[ 948.536811] RAX: ffffffffffffffda RBX: 000000000073bf08 RCX: 0000000000462eb9
[ 948.542138] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000000073bf08
[ 948.548077] RBP: 000000000073bf00 R08: 0000000000000000 R09: 0000000000000000
[ 948.562535] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000073bf0c
[ 948.569184] R13: 0000000000000000 R14: 000000000073bf00 R15: 00007fff106d8c10
WARNING at :
```cpp
static void task_non_contending(struct task_struct *p){
// ......
WARN_ON(hrtimer_active(&dl_se->inactive_timer));
WARN_ON(dl_se->dl_non_contending);
}
```I have debug for it and found that hrtimer_try_to_cancel FAILED(return -1) in migrate_task_rq_dl() because the timer handler inactive_task_timer() is running at that time. so when the task blocks later, theinactive_timer is still active indequeue_task_dl().
```cpp
static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused){
/*
* If the timer handler is currently running and the
* timer cannot be cancelled, inactive_task_timer()
* will see that dl_not_contending is not set, and
* will not touch the rq's active utilization,
* so we are still safe.
*/
if (hrtimer_try_to_cancel(&p->dl.inactive_timer) == 1)
put_task_struct(p);
}
```I also read the comment and i think this is a Non-issue phenomenon. If you delete WARN_ON, the kernel can still work very well. correct?
However the task_struct's refcount is still held, so the code looks like this :
kernel/sched/deadline.c +0 -0
@@ -252,7 +252,6 @@ static void task_non_contending(struct task_struct *p)
if (dl_entity_is_special(dl_se))
return;
- WARN_ON(hrtimer_active(&dl_se->inactive_timer)); WARN_ON(dl_se->dl_non_contending);
zerolag_time = dl_se->deadline -
@@ -287,7 +286,9 @@ static void task_non_contending(struct task_struct *p) }
dl_se->dl_non_contending = 1;
- get_task_struct(p);
+
+ if (!hrtimer_active(&dl_se->inactive_timer));
+ get_task_struct(p);
hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL); }
Did I miss something ?
I saw it directly remove the hrtimer in hrtime_start() if hrtime is queued, it may be unsafe here when the timer handler is running.
Help ?
I put the syzkaller log and C demo in attachments.
Thanks.
Hi all,
Quoted 5 lines · On Tue, 12 Mar 2019 10:03:12 +0800 "chengjian (D)" <[email protected]>…
Hi.
When looking to test
SCHED_DEADLINEsyzkaller report an warn intask_non_contending(). I tested the mainline kernel with the C program and captured the same call trace.
[...]
Quoted 13 lines
kernel/sched/deadline.c +0
-1@@ -252,7 +252,6 @@ static void task_non_contending(struct task_struct *p) if (dl_entity_is_special(dl_se)) return; - WARN_ON(hrtimer_active(&dl_se->inactive_timer)); WARN_ON(dl_se->dl_non_contending); zerolag_time = dl_se->deadline -
kernel/sched/deadline.c static void task_non_contending(struct · line 293
Quoted 9 lines
kernel/sched/deadline.c +3
-1@@ -287,7 +286,9 @@ static void task_non_contending(struct task_struct *p) } dl_se->dl_non_contending = 1; - get_task_struct(p); + + if (!hrtimer_active(&dl_se->inactive_timer)); + get_task_struct(p); hrtimer_start(timer, ns_to_ktime(zerolag_time),HRTIMER_MODE_REL ); }
At a first glance, I think the patch is OK, but I need some more time to look at the details.
I'll run some experiments with the reproducer, and I'll let you know my conclusions.
Quoted 4 lines
Did I miss something ?
I saw it directly remove the hrtimer in
hrtime_start()if hrtime is queued, it may be unsafe here when the timer handler is running.
This is probably why I added that WARN_ON()... I'll look at a possible solution.
Thanks,
Luca
Quoted 9 lines
Help ?
I put the syzkaller log and C demo in attachments.
Thanks.
Hi,
(I added Juri in cc)
On Tue, 12 Mar 2019 10:03:12 +0800
"chengjian (D)" <[email protected]> wrote:
[...]
Quoted 13 lines
kernel/sched/deadline.c +0
-1@@ -252,7 +252,6 @@ static void task_non_contending(struct task_struct *p) if (dl_entity_is_special(dl_se)) return; - WARN_ON(hrtimer_active(&dl_se->inactive_timer)); WARN_ON(dl_se->dl_non_contending); zerolag_time = dl_se->deadline -
kernel/sched/deadline.c static void task_non_contending(struct · line 293
Quoted 9 lines
kernel/sched/deadline.c +3
-1@@ -287,7 +286,9 @@ static void task_non_contending(struct task_struct *p) } dl_se->dl_non_contending = 1; - get_task_struct(p); + + if (!hrtimer_active(&dl_se->inactive_timer)); + get_task_struct(p); hrtimer_start(timer, ns_to_ktime(zerolag_time),HRTIMER_MODE_REL ); }
After looking at the patch a little bit more and running some tests, I suspect this solution might be racy:
when the timer is already active, (and hrtimer_start() fails), it relies on its handler to decrease the running bw (by setting dl_non_contending to 1)... But inactive_task_timer() might have already checked dl_non_contending, finding it equal to 0 (so, it ends up doing nothing and the running bw is not decreased).
So, I would prefer a different solution. I think this patch should work:
kernel/sched/deadline.c +1 -2
@@ -252,7 +252,6 @@ static void task_non_contending(struct task_struct *p)if (dl_entity_is_special (dl_se))return ; -WARN_ON (hrtimer_active (&dl_se->inactive_timer));WARN_ON (dl_se->dl_non_contending); zerolag_time = dl_se->deadline -
@@ -269,7 +268,7 @@ static void task_non_contending(struct task_struct *p)* If the "0-lag time" already passed, decrease the active * utilization now, instead of starting a timer */ -if (zerolag_time <0 ) { +if ((zerolag_time <0 ) ||hrtimer_active (&dl_se->inactive_timer)) {if (dl_task (p))sub_running_bw (dl_se, dl_rq);if (!dl_task (p) || p->state ==TASK_DEAD ) {
The idea is that if the timer is active, we leave dl_non_contending set to 0 (so that the timer handler does nothing), and we immediately decrease the running bw.
I think this is OK, because this situation can happen only if the task blocks, wakes up while the timer handler is running, and then immediately blocks again - while the timer handler is still running. So, the "zero lag time" cannot be too much in the future.
Thanks,
Luca
Quoted 45 lines · luca abeni
Hi,
After looking at the patch a little bit more and running some tests, I suspect this solution might be racy:
when the timer is already active, (andhrtimer_start()fails), it relies on its handler to decrease the running bw (by settingdl_non_contendingto 1)... Butinactive_task_timer()might have already checkeddl_non_contending, finding it equal to 0 (so, it ends up doing nothing and the running bw is not decreased).So, I would prefer a different solution. I think this patch should work:
kernel/sched/deadline.c +1
-2@@ -252,7 +252,6 @@ static void task_non_contending(struct task_struct *p)if (dl_entity_is_special (dl_se))return ; -WARN_ON (hrtimer_active (&dl_se->inactive_timer));WARN_ON (dl_se->dl_non_contending); zerolag_time = dl_se->deadline -@@ -269,7 +268,7 @@ static void task_non_contending(struct task_struct *p)* If the "0-lag time" already passed, decrease the active * utilization now, instead of starting a timer */ -if (zerolag_time <0 ) { +if ((zerolag_time <0 ) ||hrtimer_active (&dl_se->inactive_timer)) {if (dl_task (p))sub_running_bw (dl_se, dl_rq);if (!dl_task (p) || p->state ==TASK_DEAD ) {The idea is that if the timer is active, we leave
dl_non_contendingset to 0 (so that the timer handler does nothing), and we immediately decrease the running bw.
I think this is OK, because this situation can happen only if the task blocks, wakes up while the timer handler is running, and then immediately blocks again - while the timer handler is still running. So, the "zero lag time" cannot be too much in the future.
Quoted 4 lines
Thanks,
Luca.
Yeah, it looks good.
I can do some experiments with it ,
Do you have some testcases to help me with the test ?
Thanks,
Cheng Jian.
Hi,
On Fri, 15 Mar 2019 08:43:00 +0800
"chengjian (D)" <[email protected]> wrote:
[...]
Quoted 36 lines
Quoted 36 lines
kernel/sched/deadline.c +1
-2@@ -252,7 +252,6 @@ static void task_non_contending(struct task_struct *p) if (dl_entity_is_special(dl_se))return ; -WARN_ON (hrtimer_active (&dl_se->inactive_timer));WARN_ON (dl_se->dl_non_contending); zerolag_time = dl_se->deadline -@@ -269,7 +268,7 @@ static void task_non_contending(structtask_struct *p) * If the "0-lag time" already passed, decrease the active * utilization now, instead of starting a timer */ -if (zerolag_time <0 ) { +if ((zerolag_time <0 ) ||hrtimer_active (&dl_se->inactive_timer)) {if (dl_task (p))sub_running_bw (dl_se, dl_rq);if (!dl_task(p) || p->state == TASK_DEAD) {The idea is that if the timer is active, we leave
dl_non_contendingset to 0 (so that the timer handler does nothing), and we immediately decrease the running bw.
I think this is OK, because this situation can happen only if the task blocks, wakes up while the timer handler is running, and then immediately blocks again - while the timer handler is still running. So, the "zero lag time" cannot be too much in the future.Thanks,
Luca
Quoted 8 lines
Quoted 1 line
.
Yeah, it looks good.
I can do some experiments with it ,
Do you have some testcases to help me with the test ?
I just tried the test you provided... I also have some other SCHED_DEADLINE tests at https://github.com/lucabe72/ReclaimingTests but I did not try them with this patch yet.
Claudio Scordino also had some SCHED_DEADLINE tests here:
https://github.com/evidence/test-sched-dl
Luca
Juri Lelli
maintainerHi,
Quoted 64 lines · luca abeni
Hi,
(I added Juri in cc)
On Tue, 12 Mar 2019 10:03:12 +0800
"chengjian (D)" <[email protected]> wrote:
[...]Quoted 13 lines
kernel/sched/deadline.c +0
-1@@ -252,7 +252,6 @@ static void task_non_contending(struct task_struct *p) if (dl_entity_is_special(dl_se)) return; - WARN_ON(hrtimer_active(&dl_se->inactive_timer)); WARN_ON(dl_se->dl_non_contending); zerolag_time = dl_se->deadline -kernel/sched/deadline.c static void task_non_contending(struct · line 293
Quoted 9 lines
kernel/sched/deadline.c +3
-1@@ -287,7 +286,9 @@ static void task_non_contending(struct task_struct *p) } dl_se->dl_non_contending = 1; - get_task_struct(p); + + if (!hrtimer_active(&dl_se->inactive_timer)); + get_task_struct(p); hrtimer_start(timer, ns_to_ktime(zerolag_time),HRTIMER_MODE_REL ); }After looking at the patch a little bit more and running some tests, I suspect this solution might be racy:
when the timer is already active, (andhrtimer_start()fails), it relies on its handler to decrease the running bw (by settingdl_non_contendingto 1)... Butinactive_task_timer()might have already checkeddl_non_contending, finding it equal to 0 (so, it ends up doing nothing and the running bw is not decreased).So, I would prefer a different solution. I think this patch should work:
kernel/sched/deadline.c +1
-2@@ -252,7 +252,6 @@ static void task_non_contending(struct task_struct *p)if (dl_entity_is_special (dl_se))return ; -WARN_ON (hrtimer_active (&dl_se->inactive_timer));WARN_ON (dl_se->dl_non_contending); zerolag_time = dl_se->deadline -@@ -269,7 +268,7 @@ static void task_non_contending(struct task_struct *p)* If the "0-lag time" already passed, decrease the active * utilization now, instead of starting a timer */ -if (zerolag_time <0 ) { +if ((zerolag_time <0 ) ||hrtimer_active (&dl_se->inactive_timer)) {if (dl_task (p))sub_running_bw (dl_se, dl_rq);if (!dl_task (p) || p->state ==TASK_DEAD ) {
Quoted 7 lines
The idea is that if the timer is active, we leave
dl_non_contendingset to 0 (so that the timer handler does nothing), and we immediately decrease the running bw.
I think this is OK, because this situation can happen only if the task blocks, wakes up while the timer handler is running, and then immediately blocks again - while the timer handler is still running. So, the "zero lag time" cannot be too much in the future.
And if we get here and the handler is running it means that the handler is spinning on rq->lock waiting the dequeue to release it. So, this looks safe to me as well.
BTW, I could reproduce with Steve's deadline_test [1], and this seems to fix it.
Would you mind sending out a proper patch Luca?
Thanks!
- Juri
Hi Juri,
On Fri, 22 Mar 2019 15:32:32 +0100 Juri Lelli <[email protected]> wrote:
[...]
Quoted 36 lines
Quoted 24 lines
kernel/sched/deadline.c +1
-2@@ -252,7 +252,6 @@ static void task_non_contending(struct task_struct *p) if (dl_entity_is_special(dl_se))return ; -WARN_ON (hrtimer_active (&dl_se->inactive_timer));WARN_ON (dl_se->dl_non_contending); zerolag_time = dl_se->deadline -@@ -269,7 +268,7 @@ static void task_non_contending(structtask_struct *p) * If the "0-lag time" already passed, decrease the active * utilization now, instead of starting a timer */ -if (zerolag_time <0 ) { +if ((zerolag_time <0 ) ||hrtimer_active (&dl_se->inactive_timer)) {if (dl_task (p))sub_running_bw (dl_se, dl_rq);if (!dl_task(p) ||
p->state==TASK_DEAD) {Quoted 7 lines
The idea is that if the timer is active, we leave
dl_non_contendingset to 0 (so that the timer handler does nothing), and we immediately decrease the running bw.
I think this is OK, because this situation can happen only if the task blocks, wakes up while the timer handler is running, and then immediately blocks again - while the timer handler is still running. So, the "zero lag time" cannot be too much in the future.And if we get here and the handler is running it means that the handler is spinning on
rq->lockwaiting the dequeue to release it. So, this looks safe to me as well.
Quoted 4 lines
BTW, I could reproduce with Steve's
deadline_test[1], and this seems to fix it.Would you mind sending out a proper patch Luca?
Thanks for looking at this. I'll try to prepare and send a patch in next week.
Thanks,
Luca