test: Add kworker-hang test
PATCH 1/1 test: Add kworker-hang test
Add kworker-hang test to reproduce this:
[11133.980033] INFO: task kworker/u8:3:16315 blocked for more than 10 seconds.
[11133.980041] Tainted: G OE 5.15.0-rc4-for-5.16-io-uring-00060-g4922ab639eb6 #4
[11133.980047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[11133.980052] task:kworker/u8:3 state:D stack: 0 pid:16315 ppid: 2 flags:0x00004000
[11133.980061] Workqueue: events_unbound io_ring_exit_work
[11133.980068] Call Trace:
[11133.980076] __schedule+0x453/0x1850
[11133.980097] ? usleep_range+0x90/0x90
[11133.980104] schedule+0x59/0xc0
[11133.980111] schedule_timeout+0x1aa/0x1f0
[11133.980119] ? mark_held_locks+0x49/0x70
[11133.980128] ? lockdep_hardirqs_on_prepare+0xff/0x180
[11133.980134] ? _raw_spin_unlock_irq+0x24/0x40
[11133.980143] __wait_for_common+0xc2/0x170
[11133.980158] io_ring_exit_work+0x42c/0x44a
[11133.980166] ? io_uring_del_tctx_node+0xad/0xad
[11133.980178] ? verify_cpu+0xf0/0x100
[11133.980193] process_one_work+0x23b/0x550
[11133.980209] worker_thread+0x55/0x3c0
[11133.980214] ? process_one_work+0x550/0x550
[11133.980223] kthread+0x140/0x160
[11133.980229] ? set_kthread_struct+0x40/0x40
[11133.980237] ret_from_fork+0x1f/0x30
[11133.980278]
Showing all locks held in the system:
[11133.980282] 2 locks held by kworker/u8:0/8:
[11133.980286] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[11133.980304] #1: ffffc90000187e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[11133.980322] 1 lock held by khungtaskd/39:
[11133.980325] #0: ffffffff82977740 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x15/0x174
[11133.980348] 2 locks held by kworker/u8:9/760:
[11133.980351] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[11133.980366] #1: ffffc9000176fe70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[11133.980385] 1 lock held by in:imklog/926:
[11133.980388] #0: ffff88813208c2f0 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x4a/0x60
[11133.980434] 1 lock held by htop/13609:
[11133.980438] 2 locks held by kworker/u8:3/16315:
[11133.980441] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[11133.980456] #1: ffffc90008d43e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[11133.980475] 1 lock held by dmesg/18421:
[11133.980478] #0: ffff88824be300d0 (&user->lock){+.+.}-{3:3}, at: devkmsg_read+0x4b/0x230[11133.980498] =============================================
--- test/Makefile | 2 + test/kworker-hang.c | 289 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 291 insertions(+) create mode 100644 test/kworker-hang.c
test/Makefile +2 -0
@@ -75,6 +75,7 @@ test_targets += \ io_uring_register \ io_uring_setup \ iopoll \ + kworker-hang \ lfs-openat \ lfs-openat-write \ link \
@@ -227,6 +228,7 @@ test_srcs := \ io_uring_register.c \ io_uring_setup.c \ iopoll.c \ + kworker-hang.c \ lfs-openat-write.c \ lfs-openat.c \ link-timeout.c \
test/kworker-hang.c new file +289 -0
289 lines not on this page · raw message
Signature · 1 line
2.30.2
PATCH v2 1/1 test: Add kworker-hang test
Add kworker-hang test to reproduce this:
[28335.037622] INFO: task kworker/u8:3:77596 blocked for more than 10 seconds.
[28335.037629] Tainted: G W OE 5.15.0-rc4-for-5.16-io-uring-00060-g4922ab639eb6 #4
[28335.037637] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[28335.037642] task:kworker/u8:3 state:D stack: 0 pid:77596 ppid: 2 flags:0x00004000
[28335.037650] Workqueue: events_unbound io_ring_exit_work
[28335.037658] Call Trace:
[28335.037667] __schedule+0x453/0x1850
[28335.037681] ? lock_acquire+0xc8/0x2d0
[28335.037688] ? io_ring_exit_work+0x98/0x44a
[28335.037700] ? usleep_range+0x90/0x90
[28335.037707] schedule+0x59/0xc0
[28335.037715] schedule_timeout+0x1aa/0x1f0
[28335.037723] ? mark_held_locks+0x49/0x70
[28335.037733] ? lockdep_hardirqs_on_prepare+0xff/0x180
[28335.037740] ? _raw_spin_unlock_irq+0x24/0x40
[28335.037750] __wait_for_common+0xc2/0x170
[28335.037767] io_ring_exit_work+0x42c/0x44a
[28335.037776] ? io_uring_del_tctx_node+0xad/0xad
[28335.037790] ? verify_cpu+0xf0/0x100
[28335.037806] process_one_work+0x23b/0x550
[28335.037824] worker_thread+0x55/0x3c0
[28335.037830] ? process_one_work+0x550/0x550
[28335.037840] kthread+0x140/0x160
[28335.037846] ? set_kthread_struct+0x40/0x40
[28335.037856] ret_from_fork+0x1f/0x30
[28335.037887] INFO: task kworker/u8:4:78057 blocked for more than 10 seconds.
[28335.037895] Tainted: G W OE 5.15.0-rc4-for-5.16-io-uring-00060-g4922ab639eb6 #4
[28335.037901] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[28335.037907] task:kworker/u8:4 state:D stack: 0 pid:78057 ppid: 2 flags:0x00004000
[28335.037915] Workqueue: events_unbound io_ring_exit_work
[28335.037922] Call Trace:
[28335.037931] __schedule+0x453/0x1850
[28335.037946] ? lock_acquire+0xc8/0x2d0
[28335.037953] ? io_ring_exit_work+0x98/0x44a
[28335.037965] ? usleep_range+0x90/0x90
[28335.037973] schedule+0x59/0xc0
[28335.037980] schedule_timeout+0x1aa/0x1f0
[28335.037989] ? mark_held_locks+0x49/0x70
[28335.037999] ? lockdep_hardirqs_on_prepare+0xff/0x180
[28335.038005] ? _raw_spin_unlock_irq+0x24/0x40
[28335.038016] __wait_for_common+0xc2/0x170
[28335.038032] io_ring_exit_work+0x42c/0x44a
[28335.038041] ? io_uring_del_tctx_node+0xad/0xad
[28335.038055] ? verify_cpu+0xf0/0x100
[28335.038070] process_one_work+0x23b/0x550
[28335.038089] worker_thread+0x55/0x3c0
[28335.038095] ? process_one_work+0x550/0x550
[28335.038105] kthread+0x140/0x160
[28335.038111] ? set_kthread_struct+0x40/0x40
[28335.038120] ret_from_fork+0x1f/0x30
[28335.038148]
Showing all locks held in the system:
[28335.038155] 1 lock held by khungtaskd/39:
[28335.038159] #0: ffffffff82977740 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x15/0x174
[28335.038189] 1 lock held by in:imklog/926:
[28335.038193] #0: ffff88813208c2f0 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x4a/0x60
[28335.038253] 2 locks held by kworker/u8:1/68219:
[28335.038257] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[28335.038274] #1: ffffc90003f7fe70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[28335.038291] 2 locks held by kworker/u8:0/76320:
[28335.038295] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[28335.038311] #1: ffffc90003ca7e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[28335.038328] 2 locks held by kworker/u8:2/76681:
[28335.038331] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[28335.038347] #1: ffffc9000837be70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[28335.038365] 2 locks held by kworker/u8:3/77596:
[28335.038368] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[28335.038384] #1: ffffc900083e3e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[28335.038401] 1 lock held by dmesg/78014:
[28335.038405] #0: ffff888217c180d0 (&user->lock){+.+.}-{3:3}, at: devkmsg_read+0x4b/0x230
[28335.038423] 2 locks held by kworker/u8:4/78057:
[28335.038427] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[28335.038443] #1: ffffc900088b7e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550[28335.038463] =============================================
[28335.038468] NMI backtrace for cpu 2 [28335.038471] CPU: 2 PID: 39 Comm: khungtaskd Tainted: G W OE 5.15.0-rc4-for-5.16-io-uring-00060-g4922ab639eb6 #4 8b24b2500a34cedea2e69c8d84eb4c855e713e61 [28335.038478] Hardware name: Acer Aspire ES1-421/OLVIA_BE, BIOS V1.05 07/02/2015 [28335.038481] Call Trace: [28335.038486] dump_stack_lvl+0x57/0x72 [28335.038495] nmi_cpu_backtrace.cold+0x32/0x7f [28335.038500] ? lapic_can_unplug_cpu+0x80/0x80 [28335.038510] nmi_trigger_cpumask_backtrace+0xd1/0xe0 [28335.038520] watchdog+0x5b3/0x670 [28335.038528] ? _raw_spin_unlock_irqrestore+0x37/0x40 [28335.038535] ? hungtask_pm_notify+0x40/0x40 [28335.038544] kthread+0x140/0x160 [28335.038549] ? set_kthread_struct+0x40/0x40 [28335.038558] ret_from_fork+0x1f/0x30 [28335.038586] Sending NMI from CPU 2 to CPUs 0-1,3: [28335.038601] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0xb/0x10 [28335.038609] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0xb/0x10 [28335.038615] NMI backtrace for cpu 3 [28335.038618] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G W OE 5.15.0-rc4-for-5.16-io-uring-00060-g4922ab639eb6 #4 8b24b2500a34cedea2e69c8d84eb4c855e713e61 [28335.038622] Hardware name: Acer Aspire ES1-421/OLVIA_BE, BIOS V1.05 07/02/2015 [28335.038624] RIP: 0010:acpi_idle_enter+0x92/0x100 [28335.038631] Code: 00 41 bc 01 00 00 00 48 8b 2c 02 0f b6 45 01 3c 03 75 07 0f 09 0f 1f 44 00 00 0f b6 45 08 3c 01 74 3a 3c 02 74 45 8b 55 04 ec <48> 8b 05 07 fd 51 01 a9 00 00 00 80 75 08 48 8b 15 3d 74 9d 02 ed [28335.038634] RSP: 0018:ffffc900001d7e90 EFLAGS: 00000093 [28335.038637] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000040 [28335.038639] RDX: 0000000000000414 RSI: ffff88810135b000 RDI: ffff888103161000 [28335.038641] RBP: ffff88810135b098 R08: ffffffff82ae8f00 R09: 0000000000000018 [28335.038643] R10: 0000000000000170 R11: 0000000000000af8 R12: 0000000000000002 [28335.038644] R13: ffffffff82ae8fe8 R14: 0000000000000002 R15: 0000000000000000 [28335.038646] FS: 0000000000000000(0000) GS:ffff888313d80000(0000) knlGS:0000000000000000 [28335.038649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28335.038651] CR2: 00001e5478e1a518 CR3: 0000000199650000 CR4: 00000000000406e0 [28335.038653] Call Trace: [28335.038656] cpuidle_enter_state+0x9b/0x460 [28335.038665] cpuidle_enter+0x29/0x40 [28335.038670] do_idle+0x1fb/0x2a0 [28335.038676] cpu_startup_entry+0x19/0x20 [28335.038680] secondary_startup_64_no_verify+0xc2/0xcb
---
v2:
- Add child exit code check.
- closedir() the opendir().
- Add test/kworker-hang to .gitignore.
- Small trival cleanups.
.gitignore | 1 + test/Makefile | 2 + test/kworker-hang.c | 309 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 312 insertions(+) create mode 100644 test/kworker-hang.c
.gitignore +1 -0
@@ -65,6 +65,7 @@ /test/io_uring_register /test/io_uring_setup /test/iopoll +/test/kworker-hang /test/lfs-openat /test/lfs-openat-write /test/link
test/Makefile +2 -0
@@ -75,6 +75,7 @@ test_targets += \ io_uring_register \ io_uring_setup \ iopoll \ + kworker-hang \ lfs-openat \ lfs-openat-write \ link \
@@ -227,6 +228,7 @@ test_srcs := \ io_uring_register.c \ io_uring_setup.c \ iopoll.c \ + kworker-hang.c \ lfs-openat-write.c \ lfs-openat.c \ link-timeout.c \
test/kworker-hang.c new file +309 -0
309 lines not on this page · raw message
Signature · 1 line
2.30.2
Quoted 8 lines · Ammar Faizi
Add kworker-hang test to reproduce this:
[28335.037622] INFO: task kworker/u8:3:77596 blocked for more than 10 seconds. [28335.037629] Tainted: G W OE 5.15.0-rc4-for-5.16-io-uring-00060-g4922ab639eb6 #4 [28335.037637] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [28335.037642] task:kworker/u8:3 state:D stack: 0 pid:77596 ppid: 2 flags:0x00004000 [28335.037650] Workqueue: events_unbound io_ring_exit_work
Hi Jens, will you take this patch? If yes, I will spin the v3 because this one no longer applies since the test/Makefile changed.
This kworker hang bug still lives in Linux 5.15 stable anyway.
Signature · 1 line
Ammar Faizi
Jens Axboe
maintainerQuoted 5 lines · Ammar Faizi
Quoted 3 lines · Ammar Faizi
Add kworker-hang test to reproduce this:
Quoted 8 lines
Quoted 5 lines
[28335.037622] INFO: task kworker/u8:3:77596 blocked for more than 10 seconds. [28335.037629] Tainted: G W OE 5.15.0-rc4-for-5.16-io-uring-00060-g4922ab639eb6 #4 [28335.037637] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [28335.037642] task:kworker/u8:3 state:D stack: 0 pid:77596 ppid: 2 flags:0x00004000 [28335.037650] Workqueue: events_unbound io_ring_exit_workHi Jens, will you take this patch? If yes, I will spin the v3 because this one no longer applies since the test/Makefile changed.
Yes I will, sorry it dropped off my radar. Please just respin it.
Quoted 1 line
This kworker hang bug still lives in Linux 5.15 stable anyway.
We just need to backport the fix, then it should be fixed there too :-)
Signature · 1 line
Jens Axboe
PATCH v3 1/1 test: Add kworker-hang test
This is the reproducer for the kworker hang bug.
Reproduction Steps:
1) A user task calls io_uring_queue_exit().
2) Suspend the task with SIGSTOP / SIGTRAP before the ring exit is finished (do it as soon as step (1) is done).
3) Wait for /proc/sys/kernel/hung_task_timeout_secs seconds elapsed.
4) We get a complaint from the khungtaskd because the kworker is stuck in an uninterruptible state (D).
The kworkers waiting on ring exit are not progressing as the task cannot proceed. When the user task is continued (e.g. get SIGCONT after SIGSTOP, or continue after SIGTRAP breakpoint), the kworker then can finish the ring exit.
So here we need a special handling for this case to avoid khungtaskd complaint. Currently we don't have the fix for this.
The dmesg says:
[247390.432294] INFO: task kworker/u8:2:358488 blocked for more than 10 seconds.
[247390.432314] Tainted: G OE 5.15.0-stable #5
[247390.432322] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[247390.432329] task:kworker/u8:2 state:D stack: 0 pid:358488 ppid: 2 flags:0x00004000
[247390.432341] Workqueue: events_unbound io_ring_exit_work
[247390.432354] Call Trace:
[247390.432368] __schedule+0x453/0x1850
[247390.432388] ? lock_acquire+0xc8/0x2d0
[247390.432404] ? usleep_range+0x90/0x90
[247390.432412] schedule+0x59/0xc0
[247390.432420] schedule_timeout+0x1aa/0x1f0
[247390.432429] ? mark_held_locks+0x49/0x70
[247390.432439] ? lockdep_hardirqs_on_prepare+0xff/0x180
[247390.432445] ? _raw_spin_unlock_irq+0x24/0x40
[247390.432456] __wait_for_common+0xc2/0x170
[247390.432473] io_ring_exit_work+0x1d9/0x750
[247390.432486] ? io_uring_del_tctx_node+0xe0/0xe0
[247390.432502] ? verify_cpu+0xf0/0x100
[247390.432520] process_one_work+0x23b/0x550
[247390.432540] worker_thread+0x55/0x3c0
[247390.432546] ? process_one_work+0x550/0x550
[247390.432556] kthread+0x140/0x160
[247390.432564] ? set_kthread_struct+0x40/0x40
[247390.432574] ret_from_fork+0x1f/0x30
[247390.432605] INFO: task kworker/u8:0:359615 blocked for more than 10 seconds.
[247390.432613] Tainted: G OE 5.15.0-stable #5
[247390.432620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[247390.432626] task:kworker/u8:0 state:D stack: 0 pid:359615 ppid: 2 flags:0x00004000
[247390.432635] Workqueue: events_unbound io_ring_exit_work
[247390.432643] Call Trace:
[247390.432653] __schedule+0x453/0x1850
[247390.432676] ? usleep_range+0x90/0x90
[247390.432684] schedule+0x59/0xc0
[247390.432691] schedule_timeout+0x1aa/0x1f0
[247390.432700] ? mark_held_locks+0x49/0x70
[247390.432710] ? lockdep_hardirqs_on_prepare+0xff/0x180
[247390.432717] ? _raw_spin_unlock_irq+0x24/0x40
[247390.432728] __wait_for_common+0xc2/0x170
[247390.432744] io_ring_exit_work+0x1d9/0x750
[247390.432758] ? io_uring_del_tctx_node+0xe0/0xe0
[247390.432772] ? verify_cpu+0xf0/0x100
[247390.432788] process_one_work+0x23b/0x550
[247390.432807] worker_thread+0x55/0x3c0
[247390.432813] ? process_one_work+0x550/0x550
[247390.432824] kthread+0x140/0x160
[247390.432830] ? set_kthread_struct+0x40/0x40
[247390.432839] ret_from_fork+0x1f/0x30
[247390.432870]
Showing all locks held in the system:
[247390.432877] 1 lock held by khungtaskd/40:
[247390.432880] #0: ffffffff82976700 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x15/0x174
[247390.432911] 1 lock held by in:imklog/922:
[247390.432915] #0: ffff8881041cfcf0 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x4a/0x60
[247390.432977] 2 locks held by pager/318088:
[247390.432981] #0: ffff8881208d4898 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x24/0x50
[247390.433001] #1: ffffc900010fd2e8 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x49e/0x660
[247390.433024] 1 lock held by htop/341462:
[247390.433032] 2 locks held by kworker/u8:2/358488:
[247390.433035] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433053] #1: ffffc90003797e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433071] 2 locks held by kworker/u8:0/359615:
[247390.433075] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433092] #1: ffffc90003597e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433110] 1 lock held by dmesg/361178:
[247390.433113] #0: ffff88810b5300d0 (&user->lock){+.+.}-{3:3}, at: devkmsg_read+0x4b/0x230[247390.433134] =============================================
--- .gitignore | 1 + test/Makefile | 1 + test/kworker-hang.c | 309 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 311 insertions(+) create mode 100644 test/kworker-hang.c
.gitignore +1 -0
@@ -65,6 +65,7 @@ /test/io_uring_register /test/io_uring_setup /test/iopoll +/test/kworker-hang /test/lfs-openat /test/lfs-openat-write /test/link
test/Makefile +1 -0
@@ -83,6 +83,7 @@ test_srcs := \ io_uring_enter.c \ io_uring_register.c \ io_uring_setup.c \ + kworker-hang.c \ lfs-openat.c \ lfs-openat-write.c \ link.c \
test/kworker-hang.c new file +309 -0
309 lines not on this page · raw message
Signature · 1 line
2.30.2
PATCH v4 1/1 test: Add kworker-hang test
This is the reproducer for the kworker hang bug.
Reproduction Steps:
1) A user task calls io_uring_queue_exit().
2) Suspend the task with SIGSTOP / SIGTRAP before the ring exit is finished (do it as soon as step (1) is done).
3) Wait for /proc/sys/kernel/hung_task_timeout_secs seconds elapsed.
4) We get a complaint from the khungtaskd because the kworker is stuck in an uninterruptible state (D).
The kworkers waiting on ring exit are not progressing as the task cannot proceed. When the user task is continued (e.g. get SIGCONT after SIGSTOP, or continue after SIGTRAP breakpoint), the kworkers then can finish the ring exit.
We need a special handling for this case to avoid khungtaskd complaint. Currently we don't have the fix for this.
The dmesg says:
[247390.432294] INFO: task kworker/u8:2:358488 blocked for more than 10 seconds.
[247390.432314] Tainted: G OE 5.15.0-stable #5
[247390.432322] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[247390.432329] task:kworker/u8:2 state:D stack: 0 pid:358488 ppid: 2 flags:0x00004000
[247390.432341] Workqueue: events_unbound io_ring_exit_work
[247390.432354] Call Trace:
[247390.432368] __schedule+0x453/0x1850
[247390.432388] ? lock_acquire+0xc8/0x2d0
[247390.432404] ? usleep_range+0x90/0x90
[247390.432412] schedule+0x59/0xc0
[247390.432420] schedule_timeout+0x1aa/0x1f0
[247390.432429] ? mark_held_locks+0x49/0x70
[247390.432439] ? lockdep_hardirqs_on_prepare+0xff/0x180
[247390.432445] ? _raw_spin_unlock_irq+0x24/0x40
[247390.432456] __wait_for_common+0xc2/0x170
[247390.432473] io_ring_exit_work+0x1d9/0x750
[247390.432486] ? io_uring_del_tctx_node+0xe0/0xe0
[247390.432502] ? verify_cpu+0xf0/0x100
[247390.432520] process_one_work+0x23b/0x550
[247390.432540] worker_thread+0x55/0x3c0
[247390.432546] ? process_one_work+0x550/0x550
[247390.432556] kthread+0x140/0x160
[247390.432564] ? set_kthread_struct+0x40/0x40
[247390.432574] ret_from_fork+0x1f/0x30
[247390.432605] INFO: task kworker/u8:0:359615 blocked for more than 10 seconds.
[247390.432613] Tainted: G OE 5.15.0-stable #5
[247390.432620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[247390.432626] task:kworker/u8:0 state:D stack: 0 pid:359615 ppid: 2 flags:0x00004000
[247390.432635] Workqueue: events_unbound io_ring_exit_work
[247390.432643] Call Trace:
[247390.432653] __schedule+0x453/0x1850
[247390.432676] ? usleep_range+0x90/0x90
[247390.432684] schedule+0x59/0xc0
[247390.432691] schedule_timeout+0x1aa/0x1f0
[247390.432700] ? mark_held_locks+0x49/0x70
[247390.432710] ? lockdep_hardirqs_on_prepare+0xff/0x180
[247390.432717] ? _raw_spin_unlock_irq+0x24/0x40
[247390.432728] __wait_for_common+0xc2/0x170
[247390.432744] io_ring_exit_work+0x1d9/0x750
[247390.432758] ? io_uring_del_tctx_node+0xe0/0xe0
[247390.432772] ? verify_cpu+0xf0/0x100
[247390.432788] process_one_work+0x23b/0x550
[247390.432807] worker_thread+0x55/0x3c0
[247390.432813] ? process_one_work+0x550/0x550
[247390.432824] kthread+0x140/0x160
[247390.432830] ? set_kthread_struct+0x40/0x40
[247390.432839] ret_from_fork+0x1f/0x30
[247390.432870]
Showing all locks held in the system:
[247390.432877] 1 lock held by khungtaskd/40:
[247390.432880] #0: ffffffff82976700 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x15/0x174
[247390.432911] 1 lock held by in:imklog/922:
[247390.432915] #0: ffff8881041cfcf0 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x4a/0x60
[247390.432977] 2 locks held by pager/318088:
[247390.432981] #0: ffff8881208d4898 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x24/0x50
[247390.433001] #1: ffffc900010fd2e8 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x49e/0x660
[247390.433024] 1 lock held by htop/341462:
[247390.433032] 2 locks held by kworker/u8:2/358488:
[247390.433035] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433053] #1: ffffc90003797e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433071] 2 locks held by kworker/u8:0/359615:
[247390.433075] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433092] #1: ffffc90003597e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433110] 1 lock held by dmesg/361178:
[247390.433113] #0: ffff88810b5300d0 (&user->lock){+.+.}-{3:3}, at: devkmsg_read+0x4b/0x230[247390.433134] =============================================
---
v4:
- Update comment and function name.
.gitignore | 1 + test/Makefile | 1 + test/kworker-hang.c | 320 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 322 insertions(+) create mode 100644 test/kworker-hang.c
.gitignore +1 -0
@@ -65,6 +65,7 @@ /test/io_uring_register /test/io_uring_setup /test/iopoll +/test/kworker-hang /test/lfs-openat /test/lfs-openat-write /test/link
test/Makefile +1 -0
@@ -83,6 +83,7 @@ test_srcs := \ io_uring_enter.c \ io_uring_register.c \ io_uring_setup.c \ + kworker-hang.c \ lfs-openat.c \ lfs-openat-write.c \ link.c \
test/kworker-hang.c new file +320 -0
320 lines not on this page · raw message
Signature · 1 line
2.30.2
PATCH v5 1/1 test: Add kworker-hang test
This is the reproducer for the kworker hang bug.
Reproduction Steps:
1) A user task calls io_uring_queue_exit().
2) Suspend the task with SIGSTOP / SIGTRAP before the ring exit is finished (do it as soon as step (1) is done).
3) Wait for /proc/sys/kernel/hung_task_timeout_secs seconds elapsed.
4) We get a complaint from the khungtaskd because the kworker is stuck in an uninterruptible state (D).
The kworkers waiting on ring exit are not progressing as the task cannot proceed. When the user task is continued (e.g. get SIGCONT after SIGSTOP, or continue after SIGTRAP breakpoint), the kworkers then can finish the ring exit.
We need a special handling for this case to avoid khungtaskd complaint. Currently we don't have the fix for this.
The dmesg says:
[247390.432294] INFO: task kworker/u8:2:358488 blocked for more than 10 seconds.
[247390.432314] Tainted: G OE 5.15.0-stable #5
[247390.432322] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[247390.432329] task:kworker/u8:2 state:D stack: 0 pid:358488 ppid: 2 flags:0x00004000
[247390.432341] Workqueue: events_unbound io_ring_exit_work
[247390.432354] Call Trace:
[247390.432368] __schedule+0x453/0x1850
[247390.432388] ? lock_acquire+0xc8/0x2d0
[247390.432404] ? usleep_range+0x90/0x90
[247390.432412] schedule+0x59/0xc0
[247390.432420] schedule_timeout+0x1aa/0x1f0
[247390.432429] ? mark_held_locks+0x49/0x70
[247390.432439] ? lockdep_hardirqs_on_prepare+0xff/0x180
[247390.432445] ? _raw_spin_unlock_irq+0x24/0x40
[247390.432456] __wait_for_common+0xc2/0x170
[247390.432473] io_ring_exit_work+0x1d9/0x750
[247390.432486] ? io_uring_del_tctx_node+0xe0/0xe0
[247390.432502] ? verify_cpu+0xf0/0x100
[247390.432520] process_one_work+0x23b/0x550
[247390.432540] worker_thread+0x55/0x3c0
[247390.432546] ? process_one_work+0x550/0x550
[247390.432556] kthread+0x140/0x160
[247390.432564] ? set_kthread_struct+0x40/0x40
[247390.432574] ret_from_fork+0x1f/0x30
[247390.432605] INFO: task kworker/u8:0:359615 blocked for more than 10 seconds.
[247390.432613] Tainted: G OE 5.15.0-stable #5
[247390.432620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[247390.432626] task:kworker/u8:0 state:D stack: 0 pid:359615 ppid: 2 flags:0x00004000
[247390.432635] Workqueue: events_unbound io_ring_exit_work
[247390.432643] Call Trace:
[247390.432653] __schedule+0x453/0x1850
[247390.432676] ? usleep_range+0x90/0x90
[247390.432684] schedule+0x59/0xc0
[247390.432691] schedule_timeout+0x1aa/0x1f0
[247390.432700] ? mark_held_locks+0x49/0x70
[247390.432710] ? lockdep_hardirqs_on_prepare+0xff/0x180
[247390.432717] ? _raw_spin_unlock_irq+0x24/0x40
[247390.432728] __wait_for_common+0xc2/0x170
[247390.432744] io_ring_exit_work+0x1d9/0x750
[247390.432758] ? io_uring_del_tctx_node+0xe0/0xe0
[247390.432772] ? verify_cpu+0xf0/0x100
[247390.432788] process_one_work+0x23b/0x550
[247390.432807] worker_thread+0x55/0x3c0
[247390.432813] ? process_one_work+0x550/0x550
[247390.432824] kthread+0x140/0x160
[247390.432830] ? set_kthread_struct+0x40/0x40
[247390.432839] ret_from_fork+0x1f/0x30
[247390.432870]
Showing all locks held in the system:
[247390.432877] 1 lock held by khungtaskd/40:
[247390.432880] #0: ffffffff82976700 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x15/0x174
[247390.432911] 1 lock held by in:imklog/922:
[247390.432915] #0: ffff8881041cfcf0 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x4a/0x60
[247390.432977] 2 locks held by pager/318088:
[247390.432981] #0: ffff8881208d4898 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x24/0x50
[247390.433001] #1: ffffc900010fd2e8 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x49e/0x660
[247390.433024] 1 lock held by htop/341462:
[247390.433032] 2 locks held by kworker/u8:2/358488:
[247390.433035] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433053] #1: ffffc90003797e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433071] 2 locks held by kworker/u8:0/359615:
[247390.433075] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433092] #1: ffffc90003597e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433110] 1 lock held by dmesg/361178:
[247390.433113] #0: ffff88810b5300d0 (&user->lock){+.+.}-{3:3}, at: devkmsg_read+0x4b/0x230[247390.433134] =============================================
---
v5:
- Use stderr for printing errors.
.gitignore | 1 + test/Makefile | 1 + test/kworker-hang.c | 322 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 324 insertions(+) create mode 100644 test/kworker-hang.c
.gitignore +1 -0
@@ -65,6 +65,7 @@ /test/io_uring_register /test/io_uring_setup /test/iopoll +/test/kworker-hang /test/lfs-openat /test/lfs-openat-write /test/link
test/Makefile +1 -0
@@ -83,6 +83,7 @@ test_srcs := \ io_uring_enter.c \ io_uring_register.c \ io_uring_setup.c \ + kworker-hang.c \ lfs-openat.c \ lfs-openat-write.c \ link.c \
test/kworker-hang.c new file +322 -0
322 lines not on this page · raw message
Signature · 1 line
2.30.2
PATCH v6 1/1 test: Add kworker-hang test
This is the reproducer for the kworker hang bug.
Reproduction Steps:
1) A user task calls io_uring_queue_exit().
2) Suspend the task with SIGSTOP / SIGTRAP before the ring exit is finished (do it as soon as step (1) is done).
3) Wait for /proc/sys/kernel/hung_task_timeout_secs seconds elapsed.
4) We get a complaint from the khungtaskd because the kworker is stuck in an uninterruptible state (D).
The kworkers waiting on ring exit are not progressing as the task cannot proceed. When the user task is continued (e.g. get SIGCONT after SIGSTOP, or continue after SIGTRAP breakpoint), the kworkers then can finish the ring exit.
We need a special handling for this case to avoid khungtaskd complaint. Currently we don't have the fix for this.
The dmesg says:
[247390.432294] INFO: task kworker/u8:2:358488 blocked for more than 10 seconds.
[247390.432314] Tainted: G OE 5.15.0-stable #5
[247390.432322] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[247390.432329] task:kworker/u8:2 state:D stack: 0 pid:358488 ppid: 2 flags:0x00004000
[247390.432341] Workqueue: events_unbound io_ring_exit_work
[247390.432354] Call Trace:
[247390.432368] __schedule+0x453/0x1850
[247390.432388] ? lock_acquire+0xc8/0x2d0
[247390.432404] ? usleep_range+0x90/0x90
[247390.432412] schedule+0x59/0xc0
[247390.432420] schedule_timeout+0x1aa/0x1f0
[247390.432429] ? mark_held_locks+0x49/0x70
[247390.432439] ? lockdep_hardirqs_on_prepare+0xff/0x180
[247390.432445] ? _raw_spin_unlock_irq+0x24/0x40
[247390.432456] __wait_for_common+0xc2/0x170
[247390.432473] io_ring_exit_work+0x1d9/0x750
[247390.432486] ? io_uring_del_tctx_node+0xe0/0xe0
[247390.432502] ? verify_cpu+0xf0/0x100
[247390.432520] process_one_work+0x23b/0x550
[247390.432540] worker_thread+0x55/0x3c0
[247390.432546] ? process_one_work+0x550/0x550
[247390.432556] kthread+0x140/0x160
[247390.432564] ? set_kthread_struct+0x40/0x40
[247390.432574] ret_from_fork+0x1f/0x30
[247390.432605] INFO: task kworker/u8:0:359615 blocked for more than 10 seconds.
[247390.432613] Tainted: G OE 5.15.0-stable #5
[247390.432620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[247390.432626] task:kworker/u8:0 state:D stack: 0 pid:359615 ppid: 2 flags:0x00004000
[247390.432635] Workqueue: events_unbound io_ring_exit_work
[247390.432643] Call Trace:
[247390.432653] __schedule+0x453/0x1850
[247390.432676] ? usleep_range+0x90/0x90
[247390.432684] schedule+0x59/0xc0
[247390.432691] schedule_timeout+0x1aa/0x1f0
[247390.432700] ? mark_held_locks+0x49/0x70
[247390.432710] ? lockdep_hardirqs_on_prepare+0xff/0x180
[247390.432717] ? _raw_spin_unlock_irq+0x24/0x40
[247390.432728] __wait_for_common+0xc2/0x170
[247390.432744] io_ring_exit_work+0x1d9/0x750
[247390.432758] ? io_uring_del_tctx_node+0xe0/0xe0
[247390.432772] ? verify_cpu+0xf0/0x100
[247390.432788] process_one_work+0x23b/0x550
[247390.432807] worker_thread+0x55/0x3c0
[247390.432813] ? process_one_work+0x550/0x550
[247390.432824] kthread+0x140/0x160
[247390.432830] ? set_kthread_struct+0x40/0x40
[247390.432839] ret_from_fork+0x1f/0x30
[247390.432870]
Showing all locks held in the system:
[247390.432877] 1 lock held by khungtaskd/40:
[247390.432880] #0: ffffffff82976700 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x15/0x174
[247390.432911] 1 lock held by in:imklog/922:
[247390.432915] #0: ffff8881041cfcf0 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x4a/0x60
[247390.432977] 2 locks held by pager/318088:
[247390.432981] #0: ffff8881208d4898 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x24/0x50
[247390.433001] #1: ffffc900010fd2e8 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x49e/0x660
[247390.433024] 1 lock held by htop/341462:
[247390.433032] 2 locks held by kworker/u8:2/358488:
[247390.433035] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433053] #1: ffffc90003797e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433071] 2 locks held by kworker/u8:0/359615:
[247390.433075] #0: ffff888100106938 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433092] #1: ffffc90003597e70 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x1c1/0x550
[247390.433110] 1 lock held by dmesg/361178:
[247390.433113] #0: ffff88810b5300d0 (&user->lock){+.+.}-{3:3}, at: devkmsg_read+0x4b/0x230[247390.433134] =============================================
---
v6:
- Fix missing call to restore_hung_entries() when fork() fails.
.gitignore | 1 + test/Makefile | 1 + test/kworker-hang.c | 323 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 325 insertions(+) create mode 100644 test/kworker-hang.c
.gitignore +1 -0
@@ -65,6 +65,7 @@ /test/io_uring_register /test/io_uring_setup /test/iopoll +/test/kworker-hang /test/lfs-openat /test/lfs-openat-write /test/link
test/Makefile +1 -0
@@ -83,6 +83,7 @@ test_srcs := \ io_uring_enter.c \ io_uring_register.c \ io_uring_setup.c \ + kworker-hang.c \ lfs-openat.c \ lfs-openat-write.c \ link.c \
test/kworker-hang.c new file +323 -0
323 lines not on this page · raw message
Signature · 1 line
2.30.2
Quoted 22 lines · Ammar Faizi
This is the reproducer for the kworker hang bug.
Reproduction Steps:
1) A user task callsio_uring_queue_exit().2) Suspend the task with SIGSTOP / SIGTRAP before the ring exit is finished (do it as soon as step (1) is done).
3) Wait for
/proc/sys/kernel/hung_task_timeout_secsseconds elapsed.4) We get a complaint from the khungtaskd because the kworker is stuck in an uninterruptible state (D).
The kworkers waiting on ring exit are not progressing as the task cannot proceed. When the user task is continued (e.g. get SIGCONT after SIGSTOP, or continue after SIGTRAP breakpoint), the kworkers then can finish the ring exit.
We need a special handling for this case to avoid khungtaskd complaint. Currently we don't have the fix for this.
[...]
Quoted 5 lines
Cc Pavel Begunkov <[email protected]>Signed-off-by Ammar Faizi <[email protected]>---
Quoted 8 lines
v6:
- Fix missing call torestore_hung_entries()whenfork()fails..gitignore | 1 + test/Makefile | 1 + test/kworker-hang.c | 323 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 325 insertions(+) create mode 100644 test/kworker-hang.c
It's ready for review.
Signature · 1 line
Ammar Faizi
Jens Axboe
maintainerQuoted 32 lines · Ammar Faizi
Quoted 22 lines · Ammar Faizi
This is the reproducer for the kworker hang bug.
Reproduction Steps:
1) A user task callsio_uring_queue_exit().2) Suspend the task with SIGSTOP / SIGTRAP before the ring exit is finished (do it as soon as step (1) is done).
3) Wait for
/proc/sys/kernel/hung_task_timeout_secsseconds elapsed.4) We get a complaint from the khungtaskd because the kworker is stuck in an uninterruptible state (D).
The kworkers waiting on ring exit are not progressing as the task cannot proceed. When the user task is continued (e.g. get SIGCONT after SIGSTOP, or continue after SIGTRAP breakpoint), the kworkers then can finish the ring exit.
We need a special handling for this case to avoid khungtaskd complaint. Currently we don't have the fix for this.
[...]
Quoted 8 lines
Cc Pavel Begunkov <[email protected]>Signed-off-by Ammar Faizi <[email protected]>---
v6:
- Fix missing call torestore_hung_entries()whenfork()fails.
Quoted 7 lines
Quoted 5 lines
.gitignore | 1 + test/Makefile | 1 + test/kworker-hang.c | 323 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 325 insertions(+) create mode 100644 test/kworker-hang.cIt's ready for review.
This one is still triggering in the current tree, I'd prefer waiting with queueing it up until it's fixed. I can park it in another branch until that happens.
Signature · 1 line
Jens Axboe
Quoted 39 lines · Jens Axboe
Quoted 38 lines · Ammar Faizi
Quoted 22 lines · Ammar Faizi
This is the reproducer for the kworker hang bug.
Reproduction Steps:
1) A user task callsio_uring_queue_exit().2) Suspend the task with SIGSTOP / SIGTRAP before the ring exit is finished (do it as soon as step (1) is done).
3) Wait for
/proc/sys/kernel/hung_task_timeout_secsseconds elapsed.4) We get a complaint from the khungtaskd because the kworker is stuck in an uninterruptible state (D).
The kworkers waiting on ring exit are not progressing as the task cannot proceed. When the user task is continued (e.g. get SIGCONT after SIGSTOP, or continue after SIGTRAP breakpoint), the kworkers then can finish the ring exit.
We need a special handling for this case to avoid khungtaskd complaint. Currently we don't have the fix for this.
[...]
Quoted 13 lines
Cc Pavel Begunkov <[email protected]>Signed-off-by Ammar Faizi <[email protected]>---
v6:
- Fix missing call torestore_hung_entries()whenfork()fails..gitignore | 1 + test/Makefile | 1 + test/kworker-hang.c | 323 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 325 insertions(+) create mode 100644 test/kworker-hang.c
Quoted 6 lines
Quoted 1 line
It's ready for review.
This one is still triggering in the current tree, I'd prefer waiting with queueing it up until it's fixed. I can park it in another branch until that happens.
Understand, thanks!
Signature · 1 line
Ammar Faizi