test/defer: fix deadlock when io_uring_submit fail
PATCH 1/1 test/defer: fix deadlock when io_uring_submit fail
While performing fault injection testing, a bug report was triggered:
FAULT_INJECTION: forcing a failure.
name fail_usercopy, interval 1, probability 0, space 0, times 0
CPU: 12 UID: 0 PID: 18795 Comm: defer.t Tainted: G O 6.13.0-rc6-gf2a0a37b174b #17 Tainted: [O]=OOT_MODULE Hardware name: linux,dummy-virt (DT) Call trace: show_stack+0x20/0x38 (C) dump_stack_lvl+0x78/0x90 dump_stack+0x1c/0x28 should_fail_ex+0x544/0x648 should_fail+0x14/0x20 should_fail_usercopy+0x1c/0x28 get_timespec64+0x7c/0x258 __io_timeout_prep+0x31c/0x798 io_link_timeout_prep+0x1c/0x30 io_submit_sqes+0x59c/0x1d50 __arm64_sys_io_uring_enter+0x8dc/0xfa0 invoke_syscall+0x74/0x270 el0_svc_common.constprop.0+0xb4/0x240 do_el0_svc+0x48/0x68 el0_svc+0x38/0x78 el0t_64_sync_handler+0xc8/0xd0 el0t_64_sync+0x198/0x1a0
The deadlock stack is as follows:
io_cqring_wait+0xa64/0x1060 __arm64_sys_io_uring_enter+0x46c/0xfa0 invoke_syscall+0x74/0x270 el0_svc_common.constprop.0+0xb4/0x240 do_el0_svc+0x48/0x68 el0_svc+0x38/0x78 el0t_64_sync_handler+0xc8/0xd0 el0t_64_sync+0x198/0x1a0
This is because after the submission fails, the defer.t testcase is still waiting to submit the failed request, resulting in an eventual deadlock. Solve the problem by telling wait_cqes the number of requests to wait for.
--- test/defer.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
test/defer.c +6 -6
@@ -69,12 +69,12 @@ err:return 1 ; } -static int wait_cqes (struct test_context *ctx) +static int wait_cqes (struct test_context *ctx,int num) {int ret, i;struct io_uring_cqe *cqe; -for (i =0 ; i < ctx->nr; i++) { +for (i =0 ; i < num; i++) { ret =io_uring_wait_cqe (ctx->ring, &cqe);if (ret <0 ) {
@@ -105,7 +105,7 @@ static int test_canceled_userdata(struct io_uring *ring) goto err; } -if (wait_cqes (&ctx)) +if (wait_cqes (&ctx, ret)) goto err;for (i =0 ; i < nr; i++) {
@@ -139,7 +139,7 @@ static int test_thread_link_cancel(struct io_uring *ring) goto err; } -if (wait_cqes (&ctx)) +if (wait_cqes (&ctx, ret)) goto err;for (i =0 ; i < nr; i++) {
@@ -185,7 +185,7 @@ static int test_drain_with_linked_timeout(struct io_uring *ring) goto err; } -if (wait_cqes (&ctx)) +if (wait_cqes (&ctx, ret)) goto err;free_context (&ctx);
@@ -212,7 +212,7 @@ static int run_drained(struct io_uring *ring, int nr) goto err; } -if (wait_cqes (&ctx)) +if (wait_cqes (&ctx, ret)) goto err;free_context (&ctx);
--
2.33.0
Jens Axboe
maintainerQuoted 37 lines · lizetao
While performing fault injection testing, a bug report was triggered:
FAULT_INJECTION: forcing a failure.
namefail_usercopy, interval 1, probability 0, space 0, times 0CPU: 12 UID: 0 PID: 18795 Comm: defer.t Tainted: G O 6.13.0-rc6-gf2a0a37b174b #17 Tainted: [O]=OOT_MODULE Hardware name: linux,dummy-virt (DT) Call trace: show_stack+0x20/0x38 (C) dump_stack_lvl+0x78/0x90 dump_stack+0x1c/0x28 should_fail_ex+0x544/0x648 should_fail+0x14/0x20 should_fail_usercopy+0x1c/0x28 get_timespec64+0x7c/0x258 __io_timeout_prep+0x31c/0x798 io_link_timeout_prep+0x1c/0x30 io_submit_sqes+0x59c/0x1d50 __arm64_sys_io_uring_enter+0x8dc/0xfa0 invoke_syscall+0x74/0x270 el0_svc_common.constprop.0+0xb4/0x240 do_el0_svc+0x48/0x68 el0_svc+0x38/0x78 el0t_64_sync_handler+0xc8/0xd0 el0t_64_sync+0x198/0x1a0The deadlock stack is as follows:
io_cqring_wait+0xa64/0x1060 __arm64_sys_io_uring_enter+0x46c/0xfa0 invoke_syscall+0x74/0x270 el0_svc_common.constprop.0+0xb4/0x240 do_el0_svc+0x48/0x68 el0_svc+0x38/0x78 el0t_64_sync_handler+0xc8/0xd0 el0t_64_sync+0x198/0x1a0
Quoted 2 lines
This is because after the submission fails, the defer.t testcase is still waiting to submit the failed request, resulting in an eventual deadlock. Solve the problem by telling
wait_cqesthe number of requests to wait for.
I suspect this would be fixed by setting IORING_SETUP_SUBMIT_ALL for ring init, something probably all/most tests should set.
Signature · 1 line
Jens Axboe
Hi,
Quoted 46 lines
-----Original Message-----
From: Jens Axboe <[email protected]>
Sent: Thursday, January 16, 2025 10:51 PM
To: lizetao <[email protected]>; Pavel Begunkov <[email protected]>
Cc: [email protected]
Subject: Re: [PATCH] test/defer: fix deadlock whenio_uring_submitfailQuoted 5 lines · lizetao
While performing fault injection testing, a bug report was triggered:
FAULT_INJECTION: forcing a failure.
namefail_usercopy, interval 1, probability 0, space 0, times 0CPU: 12 UID: 0 PID: 18795 Comm: defer.t Tainted: G O6.13.0-rc6-gf2a0a37b174b #17
Quoted 32 lines
Tainted: [O]=OOT_MODULE Hardware name: linux,dummy-virt (DT) Call trace: show_stack+0x20/0x38 (C) dump_stack_lvl+0x78/0x90 dump_stack+0x1c/0x28 should_fail_ex+0x544/0x648 should_fail+0x14/0x20 should_fail_usercopy+0x1c/0x28 get_timespec64+0x7c/0x258 __io_timeout_prep+0x31c/0x798 io_link_timeout_prep+0x1c/0x30 io_submit_sqes+0x59c/0x1d50 __arm64_sys_io_uring_enter+0x8dc/0xfa0 invoke_syscall+0x74/0x270 el0_svc_common.constprop.0+0xb4/0x240 do_el0_svc+0x48/0x68 el0_svc+0x38/0x78 el0t_64_sync_handler+0xc8/0xd0 el0t_64_sync+0x198/0x1a0The deadlock stack is as follows:
io_cqring_wait+0xa64/0x1060 __arm64_sys_io_uring_enter+0x46c/0xfa0 invoke_syscall+0x74/0x270 el0_svc_common.constprop.0+0xb4/0x240 do_el0_svc+0x48/0x68 el0_svc+0x38/0x78 el0t_64_sync_handler+0xc8/0xd0 el0t_64_sync+0x198/0x1a0
Quoted 6 lines
Quoted 1 line
This is because after the submission fails, the defer.t testcase is still waiting to
submit the failed request, resulting in an eventual deadlock.
Quoted 1 line
Solve the problem by telling
wait_cqesthe number of requests to wait for.I suspect this would be fixed by setting
IORING_SETUP_SUBMIT_ALLfor ring init, something probably all/most tests should set.
I tested it and found that IORING_SETUP_SUBMIT_ALL can indeed solve this problem. Should I just modify this problem or add IORING_SETUP_SUBMIT_ALL to the general path to solve most possible problems?
Quoted 3 lines
--
Jens Axboe
---
Li Zetao
Jens Axboe
maintainerQuoted 52 lines · lizetao
Hi,
Quoted 50 lines
-----Original Message-----
From: Jens Axboe <[email protected]>
Sent: Thursday, January 16, 2025 10:51 PM
To: lizetao <[email protected]>; Pavel Begunkov <[email protected]>
Cc: [email protected]
Subject: Re: [PATCH] test/defer: fix deadlock whenio_uring_submitfailQuoted 5 lines · lizetao
While performing fault injection testing, a bug report was triggered:
FAULT_INJECTION: forcing a failure.
namefail_usercopy, interval 1, probability 0, space 0, times 0CPU: 12 UID: 0 PID: 18795 Comm: defer.t Tainted: G O6.13.0-rc6-gf2a0a37b174b #17
Quoted 32 lines
Tainted: [O]=OOT_MODULE Hardware name: linux,dummy-virt (DT) Call trace: show_stack+0x20/0x38 (C) dump_stack_lvl+0x78/0x90 dump_stack+0x1c/0x28 should_fail_ex+0x544/0x648 should_fail+0x14/0x20 should_fail_usercopy+0x1c/0x28 get_timespec64+0x7c/0x258 __io_timeout_prep+0x31c/0x798 io_link_timeout_prep+0x1c/0x30 io_submit_sqes+0x59c/0x1d50 __arm64_sys_io_uring_enter+0x8dc/0xfa0 invoke_syscall+0x74/0x270 el0_svc_common.constprop.0+0xb4/0x240 do_el0_svc+0x48/0x68 el0_svc+0x38/0x78 el0t_64_sync_handler+0xc8/0xd0 el0t_64_sync+0x198/0x1a0The deadlock stack is as follows:
io_cqring_wait+0xa64/0x1060 __arm64_sys_io_uring_enter+0x46c/0xfa0 invoke_syscall+0x74/0x270 el0_svc_common.constprop.0+0xb4/0x240 do_el0_svc+0x48/0x68 el0_svc+0x38/0x78 el0t_64_sync_handler+0xc8/0xd0 el0t_64_sync+0x198/0x1a0Quoted 1 line
This is because after the submission fails, the defer.t testcase is still waiting to
submit the failed request, resulting in an eventual deadlock.
Quoted 1 line
Solve the problem by telling
wait_cqesthe number of requests to wait for.
Quoted 8 lines
Quoted 2 lines
I suspect this would be fixed by setting
IORING_SETUP_SUBMIT_ALLfor ring init, something probably all/most tests should set.I tested it and found that
IORING_SETUP_SUBMIT_ALLcan indeed solve this problem. Should I just modify this problem or addIORING_SETUP_SUBMIT_ALLto the general path to solve most possible problems?
I think just fix up this one. We really should have all the tests use t_create_ring*() first, and those helpers should just set SUBMIT_ALL. But that's a separate change.
Signature · 1 line
Jens Axboe