block/nbd.c: Fixed IO request coroutine not being wakeup when kill NBD server.
PATCH 1/1 block/nbd.c: Fixed IO request coroutine not being wakeup when kill NBD server.
During the stress test, the IO request coroutine has a probability that it can't be awakened when the NBD server is killed.
The GDB statck is as follows:
(gdb) bt
0 0x00007f2ff990cbf6 in __ppoll (fds=0x55575de85000, nfds=1, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44
1 0x000055575c302e7c in qemu_poll_ns (fds=0x55575de85000, nfds=1, timeout=599999603140) at ../util/qemu-timer.c:348
2 0x000055575c2d3c34 in fdmon_poll_wait (ctx=0x55575dc480f0, ready_list=0x7ffd9dd1dae0, timeout=599999603140) at ../util/fdmon-poll.c:80
3 0x000055575c2d350d in aio_poll (ctx=0x55575dc480f0, blocking=true) at ../util/aio-posix.c:655
4 0x000055575c16eabd in bdrv_do_drained_begin (bs=0x55575dee7fe0, recursive=false, parent=0x0, ignore_bds_parents=false, poll=true) at ../block/io.c:474
5 0x000055575c16eba6 in bdrv_drained_begin (bs=0x55575dee7fe0) at ../block/io.c:480
6 0x000055575c1aff33 in quorum_del_child (bs=0x55575dee7fe0, child=0x55575dcea690, errp=0x7ffd9dd1dd08) at ../block/quorum.c:1130
7 0x000055575c14239b in bdrv_del_child (parent_bs=0x55575dee7fe0, child=0x55575dcea690, errp=0x7ffd9dd1dd08) at ../block.c:7705
8 0x000055575c12da28 in qmp_x_blockdev_change
(parent=0x55575df404c0 "colo-disk0", has_child=true, child=0x55575de867f0 "children.1", has_node=false, node=0x0, errp=0x7ffd9dd1dd08)
at ../blockdev.c:36769 0x000055575c258435 in qmp_marshal_x_blockdev_change (args=0x7f2fec008190, ret=0x7f2ff7b0bd98, errp=0x7f2ff7b0bd90) at qapi/qapi-commands-block-core.c:1675 10 0x000055575c2c6201 in do_qmp_dispatch_bh (opaque=0x7f2ff7b0be30) at ../qapi/qmp-dispatch.c:129
11 0x000055575c2ebb1c in aio_bh_call (bh=0x55575dc429c0) at ../util/async.c:141
12 0x000055575c2ebc2a in aio_bh_poll (ctx=0x55575dc480f0) at ../util/async.c:169
13 0x000055575c2d2d96 in aio_dispatch (ctx=0x55575dc480f0) at ../util/aio-posix.c:415
14 0x000055575c2ec07f in aio_ctx_dispatch (source=0x55575dc480f0, callback=0x0, user_data=0x0) at ../util/async.c:311
15 0x00007f2ff9e7cfbd in g_main_context_dispatch () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
16 0x000055575c2fd581 in glib_pollfds_poll () at ../util/main-loop.c:232
17 0x000055575c2fd5ff in os_host_main_loop_wait (timeout=0) at ../util/main-loop.c:255
18 0x000055575c2fd710 in main_loop_wait (nonblocking=0) at ../util/main-loop.c:531
19 0x000055575bfa7588 in qemu_main_loop () at ../softmmu/runstate.c:726
20 0x000055575bbee57a in main (argc=60, argv=0x7ffd9dd1e0e8, envp=0x7ffd9dd1e2d0) at ../softmmu/main.c:50
(gdb) qemu coroutine 0x55575e16aac0
0 0x000055575c2ee7dc in qemu_coroutine_switch (from_=0x55575e16aac0, to_=0x7f2ff830fba0, action=COROUTINE_YIELD) at ../util/coroutine-ucontext.c:302
1 0x000055575c2fe2a9 in qemu_coroutine_yield () at ../util/qemu-coroutine.c:195
2 0x000055575c2fe93c in qemu_co_queue_wait_impl (queue=0x55575dc46170, lock=0x7f2b32ad9850) at ../util/qemu-coroutine-lock.c:56
3 0x000055575c17ddfb in nbd_co_send_request (bs=0x55575ebfaf20, request=0x7f2b32ad9920, qiov=0x55575dfc15d8) at ../block/nbd.c:478
4 0x000055575c17f931 in nbd_co_request (bs=0x55575ebfaf20, request=0x7f2b32ad9920, write_qiov=0x55575dfc15d8) at ../block/nbd.c:1182
5 0x000055575c17fe14 in nbd_client_co_pwritev (bs=0x55575ebfaf20, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at ../block/nbd.c:1284
6 0x000055575c170d25 in bdrv_driver_pwritev (bs=0x55575ebfaf20, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0)
at ../block/io.c:1264
7 0x000055575c1733b4 in bdrv_aligned_pwritev
(child=0x55575dff6890, req=0x7f2b32ad9ad0, offset=403487858688, bytes=4538368, align=1, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at ../block/io.c:2126
8 0x000055575c173c67 in bdrv_co_pwritev_part (child=0x55575dff6890, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0)
at ../block/io.c:2314
9 0x000055575c17391b in bdrv_co_pwritev (child=0x55575dff6890, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at ../block/io.c:2233
10 0x000055575c1ee506 in replication_co_writev (bs=0x55575e9824f0, sector_num=788062224, remaining_sectors=8864, qiov=0x55575dfc15d8, flags=0)
at ../block/replication.c:270
11 0x000055575c170eed in bdrv_driver_pwritev (bs=0x55575e9824f0, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0)
at ../block/io.c:1297
12 0x000055575c1733b4 in bdrv_aligned_pwritev
(child=0x55575dcea690, req=0x7f2b32ad9e00, offset=403487858688, bytes=4538368, align=512, qiov=0x55575dfc15d8, qiov_offset=0, flags=0)
at ../block/io.c:2126
13 0x000055575c173c67 in bdrv_co_pwritev_part (child=0x55575dcea690, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0)
at ../block/io.c:231414 0x000055575c17391b in bdrv_co_pwritev (child=0x55575dcea690, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at ../block/io.c:2233 15 0x000055575c1aeffa in write_quorum_entry (opaque=0x7f2fddaf8c50) at ../block/quorum.c:699
16 0x000055575c2ee4db in coroutine_trampoline (i0=1578543808, i1=21847) at ../util/coroutine-ucontext.c:173
17 0x00007f2ff9855660 in __start_context () at ../sysdeps/unix/sysv/linux/x86_64/__start_context.S:91
When we do failover in COLO mode, QEMU will hang while it is waiting for the in flight IO. From the call trace, we can see the IO request coroutine which is waiting for send_mutex has yield in nbd_co_send_request(). When we kill nbd server, it will never be wake up. So, it is necessary to wake up the coroutine in nbd_channel_error().
--- block/nbd.c | 1 + 1 file changed, 1 insertion(+)
block/nbd.c +1 -0
@@ -167,6 +167,7 @@ static void nbd_channel_error(BDRVNBDState *s, int ret) s->state =NBD_CLIENT_QUIT ; } +qemu_co_queue_restart_all (&s->free_sema);nbd_recv_coroutines_wake (s, true); }
Signature · 1 line
2.32.0
Quoted 52 lines · Rao Lei
During the stress test, the IO request coroutine has a probability that it can't be awakened when the NBD server is killed.
The GDB statck is as follows:
(gdb) bt0 0x00007f2ff990cbf6 in __ppoll (fds=0x55575de85000, nfds=1, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44 1 0x000055575c302e7c in qemu_poll_ns (fds=0x55575de85000, nfds=1, timeout=599999603140) at ../util/qemu-timer.c:348 2 0x000055575c2d3c34 in fdmon_poll_wait (ctx=0x55575dc480f0, ready_list=0x7ffd9dd1dae0, timeout=599999603140) at ../util/fdmon-poll.c:80 3 0x000055575c2d350d in aio_poll (ctx=0x55575dc480f0, blocking=true) at ../util/aio-posix.c:655 4 0x000055575c16eabd in bdrv_do_drained_begin (bs=0x55575dee7fe0, recursive=false, parent=0x0, ignore_bds_parents=false, poll=true) at ../block/io.c:474 5 0x000055575c16eba6 in bdrv_drained_begin (bs=0x55575dee7fe0) at ../block/io.c:480 6 0x000055575c1aff33 in quorum_del_child (bs=0x55575dee7fe0, child=0x55575dcea690, errp=0x7ffd9dd1dd08) at ../block/quorum.c:1130 7 0x000055575c14239b in bdrv_del_child (parent_bs=0x55575dee7fe0, child=0x55575dcea690, errp=0x7ffd9dd1dd08) at ../block.c:7705 8 0x000055575c12da28 in qmp_x_blockdev_change (parent=0x55575df404c0 "colo-disk0", has_child=true, child=0x55575de867f0 "children.1", has_node=false, node=0x0, errp=0x7ffd9dd1dd08) at ../blockdev.c:36769
0x000055575c258435inqmp_marshal_x_blockdev_change(args=0x7f2fec008190, ret=0x7f2ff7b0bd98, errp=0x7f2ff7b0bd90) atqapi/qapi-commands-block-core.c:1675100x000055575c2c6201indo_qmp_dispatch_bh(opaque=0x7f2ff7b0be30) at../qapi/qmp-dispatch.c:129
110x000055575c2ebb1cinaio_bh_call(bh=0x55575dc429c0) at../util/async.c:141
120x000055575c2ebc2ainaio_bh_poll(ctx=0x55575dc480f0) at../util/async.c:169
130x000055575c2d2d96inaio_dispatch(ctx=0x55575dc480f0) at../util/aio-posix.c:415
140x000055575c2ec07finaio_ctx_dispatch(source=0x55575dc480f0, callback=0x0, user_data=0x0) at../util/async.c:311
150x00007f2ff9e7cfbding_main_context_dispatch() at/lib/x86_64-linux-gnu/libglib-2.0.so.0
160x000055575c2fd581inglib_pollfds_poll() at../util/main-loop.c:232
170x000055575c2fd5ffinos_host_main_loop_wait(timeout=0) at../util/main-loop.c:255
180x000055575c2fd710inmain_loop_wait(nonblocking=0) at../util/main-loop.c:531
190x000055575bfa7588inqemu_main_loop() at../softmmu/runstate.c:726
200x000055575bbee57ain main (argc=60, argv=0x7ffd9dd1e0e8, envp=0x7ffd9dd1e2d0) at../softmmu/main.c:50(gdb) qemu coroutine
0x55575e16aac00 0x000055575c2ee7dc in qemu_coroutine_switch (from_=0x55575e16aac0, to_=0x7f2ff830fba0, action=COROUTINE_YIELD) at ../util/coroutine-ucontext.c:302 1 0x000055575c2fe2a9 in qemu_coroutine_yield () at ../util/qemu-coroutine.c:195 2 0x000055575c2fe93c in qemu_co_queue_wait_impl (queue=0x55575dc46170, lock=0x7f2b32ad9850) at ../util/qemu-coroutine-lock.c:56 3 0x000055575c17ddfb in nbd_co_send_request (bs=0x55575ebfaf20, request=0x7f2b32ad9920, qiov=0x55575dfc15d8) at ../block/nbd.c:478 4 0x000055575c17f931 in nbd_co_request (bs=0x55575ebfaf20, request=0x7f2b32ad9920, write_qiov=0x55575dfc15d8) at ../block/nbd.c:1182 5 0x000055575c17fe14 in nbd_client_co_pwritev (bs=0x55575ebfaf20, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at ../block/nbd.c:1284 6 0x000055575c170d25 in bdrv_driver_pwritev (bs=0x55575ebfaf20, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at ../block/io.c:1264 7 0x000055575c1733b4 in bdrv_aligned_pwritev (child=0x55575dff6890, req=0x7f2b32ad9ad0, offset=403487858688, bytes=4538368, align=1, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at ../block/io.c:2126 8 0x000055575c173c67 in bdrv_co_pwritev_part (child=0x55575dff6890, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at ../block/io.c:2314 9 0x000055575c17391b in bdrv_co_pwritev (child=0x55575dff6890, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at ../block/io.c:2233 10 0x000055575c1ee506 in replication_co_writev (bs=0x55575e9824f0, sector_num=788062224, remaining_sectors=8864, qiov=0x55575dfc15d8, flags=0) at ../block/replication.c:270 11 0x000055575c170eed in bdrv_driver_pwritev (bs=0x55575e9824f0, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at ../block/io.c:1297 12 0x000055575c1733b4 in bdrv_aligned_pwritev (child=0x55575dcea690, req=0x7f2b32ad9e00, offset=403487858688, bytes=4538368, align=512, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at ../block/io.c:2126 13 0x000055575c173c67 in bdrv_co_pwritev_part (child=0x55575dcea690, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at ../block/io.c:2314
Quoted 7 lines
14
0x000055575c17391binbdrv_co_pwritev(child=0x55575dcea690, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at../block/io.c:2233150x000055575c1aeffainwrite_quorum_entry(opaque=0x7f2fddaf8c50) at../block/quorum.c:699
160x000055575c2ee4dbincoroutine_trampoline(i0=1578543808, i1=21847) at../util/coroutine-ucontext.c:173
170x00007f2ff9855660in__start_context() at../sysdeps/unix/sysv/linux/x86_64/__start_context.S:91When we do failover in COLO mode, QEMU will hang while it is waiting for the in flight IO.
From the call trace, we can see the IO request coroutine which is waiting forsend_mutex
Hmm, in call-trace above, it's not waiting for send_mutex, but sitting in yield() inside qemu_co_queue_wait()..
Quoted 13 lines
has yield in
nbd_co_send_request(). When we kill nbd server, it will never be wake up. So, it is necessary to wake up the coroutine innbd_channel_error().Signed-off-by Rao Lei <[email protected]>--- block/nbd.c | 1 + 1 file changed, 1 insertion(+)block/nbd.c +0
-0
block/nbd.c static void nbd_channel_error(BDRVNBDState *s, int ret) · line 173
Quoted 7 lines
block/nbd.c +1
-0@@ -167,6 +167,7 @@ static void nbd_channel_error(BDRVNBDState *s, int ret) s->state =NBD_CLIENT_QUIT ; } +qemu_co_queue_restart_all (&s->free_sema);nbd_recv_coroutines_wake (s, true); }
Hmm. I think, I understand the problem.
Actually, when request finishes, it calls qemu_co_queue_next(&s->free_sema). But I assume the problem is that we have a lot of requests in free_sema queue. So, when error occured, current MAX_NBD_REQUESTS finishes with error and they wake up at most MAX_NBD_REQUESTS from the queue.. But what if we still have a lot more requests in free_sema queue? Seems right, nobody will wake them.
I think better is simply move one line:
block/nbd.c +1 -1
@@ -529,8 +529,8 @@ err:if (i !=-1 ) { s->requests[i].coroutine =NULL ; s->in_flight--; -qemu_co_queue_next (&s->free_sema); } +qemu_co_queue_next (&s->free_sema); }qemu_co_mutex_unlock (&s->send_mutex);return rc;
Could you check, does it help you?
This way, every coroutine, that goes out from
while (s->in_flight == MAX_NBD_REQUESTS ||
(!nbd_client_connected(s) && s->in_flight > 0))
{
qemu_co_queue_wait(&s->free_sema, &s->send_mutex);
}will wake next coroutine, and thus we will not hang.
In other words, I think, that the real problem is that in
case
if (!nbd_client_connected(s)) {
rc = -EIO;
goto err;
}of nbd_co_send_request(), we don't wake next coroutine in free_sema. So, let's fix it.
Signature · 2 lines
Best regards, Vladimir
Quoted 119 lines · Vladimir Sementsov-Ogievskiy
Quoted 57 lines · Rao Lei
During the stress test, the IO request coroutine has a probability that it can't be awakened when the NBD server is killed.
The GDB statck is as follows:
(gdb) bt
00x00007f2ff990cbf6in__ppoll(fds=0x55575de85000, nfds=1, timeout=<optimized out>, sigmask=0x0) at../sysdeps/unix/sysv/linux/ppoll.c:4410x000055575c302e7cinqemu_poll_ns(fds=0x55575de85000, nfds=1, timeout=599999603140) at../util/qemu-timer.c:348
20x000055575c2d3c34infdmon_poll_wait(ctx=0x55575dc480f0, ready_list=0x7ffd9dd1dae0, timeout=599999603140) at../util/fdmon-poll.c:8030x000055575c2d350dinaio_poll(ctx=0x55575dc480f0, blocking=true) at../util/aio-posix.c:655
40x000055575c16eabdinbdrv_do_drained_begin(bs=0x55575dee7fe0, recursive=false, parent=0x0, ignore_bds_parents=false, poll=true) at../block/io.c:47450x000055575c16eba6inbdrv_drained_begin(bs=0x55575dee7fe0) at../block/io.c:480
60x000055575c1aff33inquorum_del_child(bs=0x55575dee7fe0, child=0x55575dcea690, errp=0x7ffd9dd1dd08) at../block/quorum.c:113070x000055575c14239binbdrv_del_child(parent_bs=0x55575dee7fe0, child=0x55575dcea690, errp=0x7ffd9dd1dd08) at../block.c:7705
80x000055575c12da28inqmp_x_blockdev_change
(parent=0x55575df404c0 "colo-disk0", has_child=true, child=0x55575de867f0 "children.1", has_node=false, node=0x0, errp=0x7ffd9dd1dd08) at../blockdev.c:3676
90x000055575c258435inqmp_marshal_x_blockdev_change(args=0x7f2fec008190, ret=0x7f2ff7b0bd98, errp=0x7f2ff7b0bd90) atqapi/qapi-commands-block-core.c:1675100x000055575c2c6201indo_qmp_dispatch_bh(opaque=0x7f2ff7b0be30) at../qapi/qmp-dispatch.c:129
110x000055575c2ebb1cinaio_bh_call(bh=0x55575dc429c0) at../util/async.c:141
120x000055575c2ebc2ainaio_bh_poll(ctx=0x55575dc480f0) at../util/async.c:169
130x000055575c2d2d96inaio_dispatch(ctx=0x55575dc480f0) at../util/aio-posix.c:415
140x000055575c2ec07finaio_ctx_dispatch(source=0x55575dc480f0, callback=0x0, user_data=0x0) at../util/async.c:311
150x00007f2ff9e7cfbding_main_context_dispatch() at/lib/x86_64-linux-gnu/libglib-2.0.so.0
160x000055575c2fd581inglib_pollfds_poll() at../util/main-loop.c:232
170x000055575c2fd5ffinos_host_main_loop_wait(timeout=0) at../util/main-loop.c:255
180x000055575c2fd710inmain_loop_wait(nonblocking=0) at../util/main-loop.c:531
190x000055575bfa7588inqemu_main_loop() at../softmmu/runstate.c:726
200x000055575bbee57ain main (argc=60, argv=0x7ffd9dd1e0e8, envp=0x7ffd9dd1e2d0) at../softmmu/main.c:50(gdb) qemu coroutine
0x55575e16aac0
00x000055575c2ee7dcinqemu_coroutine_switch(from_=0x55575e16aac0, to_=0x7f2ff830fba0, action=COROUTINE_YIELD) at../util/coroutine-ucontext.c:30210x000055575c2fe2a9inqemu_coroutine_yield() at../util/qemu-coroutine.c:195
20x000055575c2fe93cinqemu_co_queue_wait_impl(queue=0x55575dc46170, lock=0x7f2b32ad9850) at../util/qemu-coroutine-lock.c:56
30x000055575c17ddfbinnbd_co_send_request(bs=0x55575ebfaf20, request=0x7f2b32ad9920, qiov=0x55575dfc15d8) at../block/nbd.c:478
40x000055575c17f931innbd_co_request(bs=0x55575ebfaf20, request=0x7f2b32ad9920, write_qiov=0x55575dfc15d8) at../block/nbd.c:118250x000055575c17fe14innbd_client_co_pwritev(bs=0x55575ebfaf20, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at../block/nbd.c:128460x000055575c170d25inbdrv_driver_pwritev(bs=0x55575ebfaf20, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:1264
70x000055575c1733b4inbdrv_aligned_pwritev
(child=0x55575dff6890, req=0x7f2b32ad9ad0, offset=403487858688, bytes=4538368, align=1, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:212680x000055575c173c67inbdrv_co_pwritev_part(child=0x55575dff6890, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:2314
90x000055575c17391binbdrv_co_pwritev(child=0x55575dff6890, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at../block/io.c:2233100x000055575c1ee506inreplication_co_writev(bs=0x55575e9824f0, sector_num=788062224, remaining_sectors=8864, qiov=0x55575dfc15d8, flags=0) at../block/replication.c:270
110x000055575c170eedinbdrv_driver_pwritev(bs=0x55575e9824f0, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:1297
120x000055575c1733b4inbdrv_aligned_pwritev
(child=0x55575dcea690, req=0x7f2b32ad9e00, offset=403487858688, bytes=4538368, align=512, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:2126
130x000055575c173c67inbdrv_co_pwritev_part(child=0x55575dcea690, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:2314
140x000055575c17391binbdrv_co_pwritev(child=0x55575dcea690, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at../block/io.c:2233150x000055575c1aeffainwrite_quorum_entry(opaque=0x7f2fddaf8c50) at../block/quorum.c:699
160x000055575c2ee4dbincoroutine_trampoline(i0=1578543808, i1=21847) at../util/coroutine-ucontext.c:173
170x00007f2ff9855660in__start_context() at../sysdeps/unix/sysv/linux/x86_64/__start_context.S:91Quoted 2 lines
When we do failover in COLO mode, QEMU will hang while it is waiting for the in flight IO. From the call trace, we can see the IO request coroutine which is waiting for
send_mutexHmm, in call-trace above, it's not waiting for
send_mutex, but sitting inyield()insideqemu_co_queue_wait()..Quoted 13 lines
has yield in
nbd_co_send_request(). When we kill nbd server, it will never be wake up. So, it is necessary to wake up the coroutine innbd_channel_error().Signed-off-by Rao Lei <[email protected]>---
block/nbd.c| 1 +1 file changed, 1 insertion(+)block/nbd.c +0
-0block/nbd.c static void nbd_channel_error(BDRVNBDState *s, int ret) · line 171
Quoted 5 lines
block/nbd.c +1
-0@@ -167,6 +167,7 @@ static void nbd_channel_error(BDRVNBDState *s, int ret) s->state = NBD_CLIENT_QUIT; } + qemu_co_queue_restart_all(&s->free_sema); nbd_recv_coroutines_wake(s, true); }Hmm. I think, I understand the problem.
Actually, when request finishes, it calls
qemu_co_queue_next(&s->free_sema). But I assume the problem is that we have a lot of requests infree_semaqueue. So, when error occured, currentMAX_NBD_REQUESTSfinishes with error and they wake up at mostMAX_NBD_REQUESTSfrom the queue.. But what if we still have a lot more requests infree_semaqueue? Seems right, nobody will wake them.I think better is simply move one line:
block/nbd.c +1
-1@@ -529,8 +529,8 @@ err: if (i != -1) { s->requests[i].coroutine = NULL; s->in_flight--; - qemu_co_queue_next(&s->free_sema); } + qemu_co_queue_next(&s->free_sema); } qemu_co_mutex_unlock(&s->send_mutex); return rc;Could you check, does it help you?
This way, every coroutine, that goes out from
while (s->in_flight==MAX_NBD_REQUESTS|| (!nbd_client_connected(s) &&s->in_flight> 0)) {
qemu_co_queue_wait(&s->free_sema, &s->send_mutex); }will wake next coroutine, and thus we will not hang.
In other words, I think, that the real problem is that in
Quoted 8 lines
case
if (!nbd_client_connected(s)) {
rc =-EIO;
goto err;
}of
nbd_co_send_request(), we don't wake next coroutine infree_sema. So, let's fix it.
I think you are right and will rerun the stress test.
Thanks
Lei
Quoted 1 line
Quoted 91 lines · Lei
Quoted 88 lines · Vladimir Sementsov-Ogievskiy
Quoted 57 lines · Rao Lei
During the stress test, the IO request coroutine has a probability that it can't be awakened when the NBD server is killed.
The GDB statck is as follows:
(gdb) bt
00x00007f2ff990cbf6in__ppoll(fds=0x55575de85000, nfds=1, timeout=<optimized out>, sigmask=0x0) at../sysdeps/unix/sysv/linux/ppoll.c:4410x000055575c302e7cinqemu_poll_ns(fds=0x55575de85000, nfds=1, timeout=599999603140) at../util/qemu-timer.c:348
20x000055575c2d3c34infdmon_poll_wait(ctx=0x55575dc480f0, ready_list=0x7ffd9dd1dae0, timeout=599999603140) at../util/fdmon-poll.c:8030x000055575c2d350dinaio_poll(ctx=0x55575dc480f0, blocking=true) at../util/aio-posix.c:655
40x000055575c16eabdinbdrv_do_drained_begin(bs=0x55575dee7fe0, recursive=false, parent=0x0, ignore_bds_parents=false, poll=true) at../block/io.c:47450x000055575c16eba6inbdrv_drained_begin(bs=0x55575dee7fe0) at../block/io.c:480
60x000055575c1aff33inquorum_del_child(bs=0x55575dee7fe0, child=0x55575dcea690, errp=0x7ffd9dd1dd08) at../block/quorum.c:113070x000055575c14239binbdrv_del_child(parent_bs=0x55575dee7fe0, child=0x55575dcea690, errp=0x7ffd9dd1dd08) at../block.c:7705
80x000055575c12da28inqmp_x_blockdev_change
(parent=0x55575df404c0 "colo-disk0", has_child=true, child=0x55575de867f0 "children.1", has_node=false, node=0x0, errp=0x7ffd9dd1dd08) at../blockdev.c:3676
90x000055575c258435inqmp_marshal_x_blockdev_change(args=0x7f2fec008190, ret=0x7f2ff7b0bd98, errp=0x7f2ff7b0bd90) atqapi/qapi-commands-block-core.c:1675100x000055575c2c6201indo_qmp_dispatch_bh(opaque=0x7f2ff7b0be30) at../qapi/qmp-dispatch.c:129
110x000055575c2ebb1cinaio_bh_call(bh=0x55575dc429c0) at../util/async.c:141
120x000055575c2ebc2ainaio_bh_poll(ctx=0x55575dc480f0) at../util/async.c:169
130x000055575c2d2d96inaio_dispatch(ctx=0x55575dc480f0) at../util/aio-posix.c:415
140x000055575c2ec07finaio_ctx_dispatch(source=0x55575dc480f0, callback=0x0, user_data=0x0) at../util/async.c:311
150x00007f2ff9e7cfbding_main_context_dispatch() at/lib/x86_64-linux-gnu/libglib-2.0.so.0
160x000055575c2fd581inglib_pollfds_poll() at../util/main-loop.c:232
170x000055575c2fd5ffinos_host_main_loop_wait(timeout=0) at../util/main-loop.c:255
180x000055575c2fd710inmain_loop_wait(nonblocking=0) at../util/main-loop.c:531
190x000055575bfa7588inqemu_main_loop() at../softmmu/runstate.c:726
200x000055575bbee57ain main (argc=60, argv=0x7ffd9dd1e0e8, envp=0x7ffd9dd1e2d0) at../softmmu/main.c:50(gdb) qemu coroutine
0x55575e16aac0
00x000055575c2ee7dcinqemu_coroutine_switch(from_=0x55575e16aac0, to_=0x7f2ff830fba0, action=COROUTINE_YIELD) at../util/coroutine-ucontext.c:30210x000055575c2fe2a9inqemu_coroutine_yield() at../util/qemu-coroutine.c:195
20x000055575c2fe93cinqemu_co_queue_wait_impl(queue=0x55575dc46170, lock=0x7f2b32ad9850) at../util/qemu-coroutine-lock.c:56
30x000055575c17ddfbinnbd_co_send_request(bs=0x55575ebfaf20, request=0x7f2b32ad9920, qiov=0x55575dfc15d8) at../block/nbd.c:478
40x000055575c17f931innbd_co_request(bs=0x55575ebfaf20, request=0x7f2b32ad9920, write_qiov=0x55575dfc15d8) at../block/nbd.c:118250x000055575c17fe14innbd_client_co_pwritev(bs=0x55575ebfaf20, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at../block/nbd.c:128460x000055575c170d25inbdrv_driver_pwritev(bs=0x55575ebfaf20, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:1264
70x000055575c1733b4inbdrv_aligned_pwritev
(child=0x55575dff6890, req=0x7f2b32ad9ad0, offset=403487858688, bytes=4538368, align=1, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:212680x000055575c173c67inbdrv_co_pwritev_part(child=0x55575dff6890, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:2314
90x000055575c17391binbdrv_co_pwritev(child=0x55575dff6890, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at../block/io.c:2233100x000055575c1ee506inreplication_co_writev(bs=0x55575e9824f0, sector_num=788062224, remaining_sectors=8864, qiov=0x55575dfc15d8, flags=0) at../block/replication.c:270
110x000055575c170eedinbdrv_driver_pwritev(bs=0x55575e9824f0, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:1297
120x000055575c1733b4inbdrv_aligned_pwritev
(child=0x55575dcea690, req=0x7f2b32ad9e00, offset=403487858688, bytes=4538368, align=512, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:2126
130x000055575c173c67inbdrv_co_pwritev_part(child=0x55575dcea690, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, qiov_offset=0, flags=0) at../block/io.c:2314
140x000055575c17391binbdrv_co_pwritev(child=0x55575dcea690, offset=403487858688, bytes=4538368, qiov=0x55575dfc15d8, flags=0) at../block/io.c:2233150x000055575c1aeffainwrite_quorum_entry(opaque=0x7f2fddaf8c50) at../block/quorum.c:699
160x000055575c2ee4dbincoroutine_trampoline(i0=1578543808, i1=21847) at../util/coroutine-ucontext.c:173
170x00007f2ff9855660in__start_context() at../sysdeps/unix/sysv/linux/x86_64/__start_context.S:91Quoted 2 lines
When we do failover in COLO mode, QEMU will hang while it is waiting for the in flight IO. From the call trace, we can see the IO request coroutine which is waiting for
send_mutexHmm, in call-trace above, it's not waiting for
send_mutex, but sitting inyield()insideqemu_co_queue_wait()..Quoted 13 lines
has yield in
nbd_co_send_request(). When we kill nbd server, it will never be wake up. So, it is necessary to wake up the coroutine innbd_channel_error().Signed-off-by Rao Lei <[email protected]>---
block/nbd.c| 1 +1 file changed, 1 insertion(+)block/nbd.c +0
-0block/nbd.c static void nbd_channel_error(BDRVNBDState *s, int ret) · line 171
Quoted 5 lines
block/nbd.c +1
-0@@ -167,6 +167,7 @@ static void nbd_channel_error(BDRVNBDState *s, int ret) s->state = NBD_CLIENT_QUIT; } + qemu_co_queue_restart_all(&s->free_sema); nbd_recv_coroutines_wake(s, true); }Hmm. I think, I understand the problem.
Actually, when request finishes, it calls
qemu_co_queue_next(&s->free_sema). But I assume the problem is that we have a lot of requests infree_semaqueue. So, when error occured, currentMAX_NBD_REQUESTSfinishes with error and they wake up at mostMAX_NBD_REQUESTSfrom the queue.. But what if we still have a lot more requests infree_semaqueue? Seems right, nobody will wake them.I think better is simply move one line:
Quoted 15 lines
Quoted 6 lines
block/nbd.c +0
-0block/nbd.c err: · line 536
Quoted 9 lines
block/nbd.c +1
-1@@ -529,8 +529,8 @@ err: if (i != -1) { s->requests[i].coroutine = NULL; s->in_flight--; - qemu_co_queue_next(&s->free_sema); } + qemu_co_queue_next(&s->free_sema); } qemu_co_mutex_unlock(&s->send_mutex); return rc;
In our 50 rounds IO stress test, it worked well and QEMU didn't hang. I will send the V2.
Lei
Quoted 31 lines
Quoted 16 lines
Could you check, does it help you?
This way, every coroutine, that goes out from while (
s->in_flight==MAX_NBD_REQUESTS|| (!nbd_client_connected(s) &&s->in_flight> 0)) {
qemu_co_queue_wait(&s->free_sema, &s->send_mutex); }will wake next coroutine, and thus we will not hang.
In other words, I think, that the real problem is that in
Quoted 8 lines
case
if (!nbd_client_connected(s)) {
rc =-EIO;
goto err;
}of
nbd_co_send_request(), we don't wake next coroutine infree_sema. So, let's fix it.I think you are right and will rerun the stress test > Thanks
LeiQuoted 1 line