Skip to contentSkip to conversation

io_uring: fix task hung in io_uring_setup

PATCH Jens Axboe 3 participants 7 messages Sep 3, 2020 – Sep 9, 2020 3 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Jens Axboe

author maintainer
#1 lore raw
Quoted 24 lines · Hillf Danton

The smart syzbot found the following issue:

INFO: task syz-executor047:6853 blocked for more than 143 seconds.
      Not tainted 5.9.0-rc3-next-20200902-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor047 state:D stack:28104 pid: 6853 ppid:  6847 flags:0x00004000
Call Trace:
 context_switch kernel/sched/core.c:3777 [inline]
 __schedule+0xea9/0x2230 kernel/sched/core.c:4526
 schedule+0xd0/0x2a0 kernel/sched/core.c:4601
 schedule_timeout+0x1d8/0x250 kernel/time/timer.c:1855
 do_wait_for_common kernel/sched/completion.c:85 [inline]
 __wait_for_common kernel/sched/completion.c:106 [inline]
 wait_for_common kernel/sched/completion.c:117 [inline]
 wait_for_completion+0x163/0x260 kernel/sched/completion.c:138
 io_sq_thread_stop fs/io_uring.c:6906 [inline]
 io_finish_async fs/io_uring.c:6920 [inline]
 io_sq_offload_create fs/io_uring.c:7595 [inline]
 io_uring_create fs/io_uring.c:8671 [inline]
 io_uring_setup+0x1495/0x29a0 fs/io_uring.c:8744
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 3 lines

because the sqo_thread kthread is created in io_sq_offload_create() without being waked up. Then in the error branch of that function we will wait for the sqo kthread that never runs. It's fixed by waking it up before waiting.

Looks good - applied, thanks.

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 25 lines · Jens Axboe
Quoted 24 lines · Hillf Danton

The smart syzbot found the following issue:

INFO: task syz-executor047:6853 blocked for more than 143 seconds.
      Not tainted 5.9.0-rc3-next-20200902-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor047 state:D stack:28104 pid: 6853 ppid:  6847 flags:0x00004000
Call Trace:
 context_switch kernel/sched/core.c:3777 [inline]
 __schedule+0xea9/0x2230 kernel/sched/core.c:4526
 schedule+0xd0/0x2a0 kernel/sched/core.c:4601
 schedule_timeout+0x1d8/0x250 kernel/time/timer.c:1855
 do_wait_for_common kernel/sched/completion.c:85 [inline]
 __wait_for_common kernel/sched/completion.c:106 [inline]
 wait_for_common kernel/sched/completion.c:117 [inline]
 wait_for_completion+0x163/0x260 kernel/sched/completion.c:138
 io_sq_thread_stop fs/io_uring.c:6906 [inline]
 io_finish_async fs/io_uring.c:6920 [inline]
 io_sq_offload_create fs/io_uring.c:7595 [inline]
 io_uring_create fs/io_uring.c:8671 [inline]
 io_uring_setup+0x1495/0x29a0 fs/io_uring.c:8744
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 5 lines
Quoted 3 lines

because the sqo_thread kthread is created in io_sq_offload_create() without being waked up. Then in the error branch of that function we will wait for the sqo kthread that never runs. It's fixed by waking it up before waiting.

Looks good - applied, thanks.

BTW, I don't see the patch itself, and it's neither in io_uring, block nor fs mailing lists. Hillf, could you please CC proper lists next time?

Signature · 1 line
Pavel Begunkov
author maintainer
#3 lore raw
Quoted 26 lines · Pavel Begunkov
Quoted 25 lines · Jens Axboe
Quoted 24 lines · Hillf Danton

The smart syzbot found the following issue:

INFO: task syz-executor047:6853 blocked for more than 143 seconds.
      Not tainted 5.9.0-rc3-next-20200902-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor047 state:D stack:28104 pid: 6853 ppid:  6847 flags:0x00004000
Call Trace:
 context_switch kernel/sched/core.c:3777 [inline]
 __schedule+0xea9/0x2230 kernel/sched/core.c:4526
 schedule+0xd0/0x2a0 kernel/sched/core.c:4601
 schedule_timeout+0x1d8/0x250 kernel/time/timer.c:1855
 do_wait_for_common kernel/sched/completion.c:85 [inline]
 __wait_for_common kernel/sched/completion.c:106 [inline]
 wait_for_common kernel/sched/completion.c:117 [inline]
 wait_for_completion+0x163/0x260 kernel/sched/completion.c:138
 io_sq_thread_stop fs/io_uring.c:6906 [inline]
 io_finish_async fs/io_uring.c:6920 [inline]
 io_sq_offload_create fs/io_uring.c:7595 [inline]
 io_uring_create fs/io_uring.c:8671 [inline]
 io_uring_setup+0x1495/0x29a0 fs/io_uring.c:8744
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

because the sqo_thread kthread is created in io_sq_offload_create() without being waked up. Then in the error branch of that function we will wait for the sqo kthread that never runs. It's fixed by waking it up before waiting.

Looks good - applied, thanks.

BTW, I don't see the patch itself, and it's neither in io_uring, block nor fs mailing lists. Hillf, could you please CC proper lists next time?

He did, but I'm guessing that vger didn't like the email for whatever reason. Hillf, did you get an error back from vger when sending the patch?

Signature · 1 line
Jens Axboe

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 8 lines · Hillf Danton
Quoted 4 lines · On Mon, 7 Sep 2020 06:55:04 Jens Axboe wrote:
Quoted 3 lines · Pavel Begunkov

BTW, I don't see the patch itself, and it's neither in io_uring, block nor fs mailing lists. Hillf, could you please CC proper lists next time?

Yes, I can. So will I send io_uring patches with Pavel Cced.

Thanks

Quoted 5 lines
Quoted 3 lines

He did, but I'm guessing that vger didn't like the email for whatever reason. Hillf, did you get an error back from vger when sending the pat

My inbox, a simple free mail, is perhaps blocked as spam at the vger end.

Signature · 1 line
Pavel Begunkov
author maintainer
#6 lore raw
In reply to a message not in the archive
Quoted 8 lines · Hillf Danton
Quoted 4 lines · On Mon, 7 Sep 2020 06:55:04 Jens Axboe wrote:
Quoted 3 lines · Pavel Begunkov

BTW, I don't see the patch itself, and it's neither in io_uring, block nor fs mailing lists. Hillf, could you please CC proper lists next time?

Yes, I can. So will I send io_uring patches with Pavel Cced.

While that is nice, it should not be necessary. We need to ensure that your emails reach the list, that's more important than needing to CC a specific person, because it still means that everyone else doesn't see it.

Do you get an error from vger, or does it simply not show up?

Signature · 1 line
Jens Axboe

A message that is not in the archive (sent off-list or not yet received): [email protected]

author maintainer
#8 lore raw
Quoted 17 lines · Hillf Danton
Quoted 7 lines · On Tue, 8 Sep 2020 17:34:26 -0600 Jens Axboe wrote:
Quoted 6 lines · Hillf Danton
Quoted 4 lines · On Mon, 7 Sep 2020 06:55:04 Jens Axboe wrote:
Quoted 3 lines · Pavel Begunkov

BTW, I don't see the patch itself, and it's neither in io_uring, block nor fs mailing lists. Hillf, could you please CC proper lists next time?

Quoted 7 lines
Quoted 1 line

Yes, I can. So will I send io_uring patches with Pavel Cced.

While that is nice, it should not be necessary. We need to ensure that your emails reach the list, that's more important than needing to CC a specific person, because it still means that everyone else doesn't see it.

Do you get an error from vger, or does it simply not show up?

Quoted 9 lines

After tapping the send button for this message, I will receive a message from the sina mail server saying it failed to deliver it to one of the targets ([email protected]), which has been happing over the past a couple of years. One of the redhat guys, I can't remmenber his name, once tryied to help me solve the problem, by sending somebody@vger a message explaining what was going on, but failed. AFAIC there's a glitch in exchanging info between the sina server and the server at the vger end, and it seems it would take more time than thought to figure it out. So let it be for now.

Might be worthwhile to just have a gmail account for sending patches and replying to list emails?

Signature · 1 line
Jens Axboe
Reviewed-by
#9 lore raw
In reply to a message not in the archive
Quoted 43 lines · Hillf Danton

The smart syzbot found the following issue:

INFO: task syz-executor047:6853 blocked for more than 143 seconds.
      Not tainted 5.9.0-rc3-next-20200902-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor047 state:D stack:28104 pid: 6853 ppid:  6847 flags:0x00004000
Call Trace:
 context_switch kernel/sched/core.c:3777 [inline]
 __schedule+0xea9/0x2230 kernel/sched/core.c:4526
 schedule+0xd0/0x2a0 kernel/sched/core.c:4601
 schedule_timeout+0x1d8/0x250 kernel/time/timer.c:1855
 do_wait_for_common kernel/sched/completion.c:85 [inline]
 __wait_for_common kernel/sched/completion.c:106 [inline]
 wait_for_common kernel/sched/completion.c:117 [inline]
 wait_for_completion+0x163/0x260 kernel/sched/completion.c:138
 io_sq_thread_stop fs/io_uring.c:6906 [inline]
 io_finish_async fs/io_uring.c:6920 [inline]
 io_sq_offload_create fs/io_uring.c:7595 [inline]
 io_uring_create fs/io_uring.c:8671 [inline]
 io_uring_setup+0x1495/0x29a0 fs/io_uring.c:8744
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9

because the sqo_thread kthread is created in io_sq_offload_create() without being waked up. Then in the error branch of that function we will wait for the sqo kthread that never runs. It's fixed by waking it up before waiting.

Reported-by [email protected]
Fixes dfe127799f8e ("io_uring: allow disabling rings during the creation")
Cc Stefano Garzarella <[email protected]>
Cc Kees Cook <[email protected]>

---

fs/io_uring.c +1 -0
@@ -6903,6 +6903,13 @@ static int io_sqe_files_unregister(struc
 static void io_sq_thread_stop(struct io_ring_ctx *ctx)
 {
 	if (ctx->sqo_thread) {
+		/*

fs/io_uring.c static int io_sqe_files_unregister(struc

Quoted 10 lines
fs/io_uring.c +6 -1
+		 * We may arrive here from the error branch in
+		 * io_sq_offload_create() where the kthread is created without
+		 * being waked up, thus wake it up now to make sure the wait will
+		 * complete.
+		 */
+		wake_up_process(ctx->sqo_thread);
 		wait_for_completion(&ctx->sq_thread_comp);
 		/*
 		 * The park is a bit of a work-around, without it we get
--

Thanks for fixing this issue!
Jens already queued this, but just for recording:

Thanks,
Stefano