block/nbd: rework client connection
Subsystems (2)
v3 cover block/nbd: rework client connection
The series substitutes "[PATCH v2 00/10] block/nbd: move connection code to separate file"
Supersedes: <[email protected]>
so it's called v3
block/nbd.c is overcomplicated. These series is a big refactoring, which finally drops all the complications around drained sections and context switching, including abuse of bs->in_flight counter.
Also, at the end of the series we don't cancel reconnect on drained sections (and don't cancel requests waiting for reconnect on drained section begin), which fixes a problem reported by Roman.
The series is also available at tag up-nbd-client-connection-v3 in git https://src.openvz.org/scm/~vsementsov/qemu.git
v3:
Changes in first part of the series (main thing is not using refcnt, but instead (modified) Roman's patch):
01-04: new
05: add Roman's r-b
06: new
07: now, new aio_co_schedule(NULL, thr->wait_co) is used 08: reworked, we now need also bool detached, as we don't have refcnt 09,10: add Roman's r-b
11: rebased, don't modify nbd_free_connect_thread() name at this point 12: add Roman's r-b
13: new
14: rebased
Other patches are new.
Roman Kagan (2):
block/nbd: fix channel object leak block/nbd: ensure ->connection_thread is always valid
Vladimir Sementsov-Ogievskiy (31):
block/nbd: fix how state is cleared on nbd_open() failure paths block/nbd: nbd_client_handshake(): fix leak of s->ioc block/nbd: BDRVNBDState: drop unused connect_err and connect_status util/async: aio_co_schedule(): support reschedule in same ctx block/nbd: simplify waking of nbd_co_establish_connection() block/nbd: drop thr->state
block/nbd: bs-independent interface for nbd_co_establish_connection() block/nbd: make nbd_co_establish_connection_cancel() bs-independent block/nbd: rename NBDConnectThread to NBDClientConnection block/nbd: introduce nbd_client_connection_new() block/nbd: introduce nbd_client_connection_release()
nbd: move connection code from block/nbd to nbd/client-connection nbd/client-connection: use QEMU_LOCK_GUARD nbd/client-connection: add possibility of negotiation nbd/client-connection: implement connection retry nbd/client-connection: shutdown connection on release block/nbd: split nbd_handle_updated_info out of nbd_client_handshake() block/nbd: use negotiation of NBDClientConnection
qemu-socket: pass monitor link to socket_get_fd directly block/nbd: pass monitor directly to connection thread block/nbd: nbd_teardown_connection() don't touch s->sioc block/nbd: drop BDRVNBDState::sioc
nbd/client-connection: return only one io channel
block-coroutine-wrapper: allow non bdrv_ prefix
block/nbd: split nbd_co_do_establish_connection out of
nbd_reconnect_attempt
nbd/client-connection: do qio_channel_set_delay(false) nbd/client-connection: add option for non-blocking connection attempt block/nbd: reuse nbd_co_do_establish_connection() in nbd_open() block/nbd: add nbd_clinent_connected() helper
block/nbd: safer transition to receiving request block/nbd: drop connection_co
block/coroutines.h | 6 + include/block/aio.h | 2 +- include/block/nbd.h | 19 + include/io/channel-socket.h | 20 + include/qemu/sockets.h | 2 +- block/nbd.c | 908 +++++++---------------------- io/channel-socket.c | 17 +- nbd/client-connection.c | 364 ++++++++++++ nbd/client.c | 2 - tests/unit/test-util-sockets.c | 16 +- util/async.c | 8 + util/qemu-sockets.c | 10 +- nbd/meson.build | 1 + scripts/block-coroutine-wrapper.py | 7 +- 14 files changed, 666 insertions(+), 716 deletions(-) create mode 100644 nbd/client-connection.c
Signature · 1 line
2.29.2
PATCH v3 1/33 block/nbd: fix channel object leak
From: Roman Kagan <[email protected]>
nbd_free_connect_thread leaks the channel object if it hasn't been stolen.
Unref it and fix the leak.
--- block/nbd.c | 1 + 1 file changed, 1 insertion(+)
block/nbd.c +1 -0
@@ -385,6 +385,7 @@ static void nbd_free_connect_thread(NBDConnectThread *thr) {if (thr->sioc) {qio_channel_close (QIO_CHANNEL (thr->sioc),NULL ); +object_unref (OBJECT (thr->sioc)); }error_free (thr->err);qapi_free_SocketAddress (thr->saddr);
Signature · 1 line
2.29.2
Re: [PATCH v3 01/33] block/nbd: fix channel object leak
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
From: Roman Kagan <[email protected]>
nbd_free_connect_threadleaks the channel object if it hasn't been stolen.Unref it and fix the leak.
Signed-off-by Roman Kagan <[email protected]>--- block/nbd.c | 1 + 1 file changed, 1 insertion(+)
Does nbd_yank() have a similar problem?
At any rate, this makes sense to me.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 01/33] block/nbd: fix channel object leak
Quoted 6 lines · Eric Blake
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
From: Roman Kagan <[email protected]>
nbd_free_connect_threadleaks the channel object if it hasn't been stolen.
Quoted 8 lines
Quoted 6 lines
Unref it and fix the leak.
Signed-off-by Roman Kagan <[email protected]>--- block/nbd.c | 1 + 1 file changed, 1 insertion(+)Does
nbd_yank()have a similar problem?
No, I think not. nbd_yank() just shutdown the socket to cancel in-flight requests. I doesn't release the state.
Quoted 4 lines
At any rate, this makes sense to me.
Reviewed-by Eric Blake <[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH v3 2/33 block/nbd: fix how state is cleared on nbd_open() failure paths
In reply to Vladimir Sementsov-OgievskiyWe have two "return error" paths in nbd_open() after nbd_process_options(). Actually we should call nbd_clear_bdrvstate() on these paths. Interesting that nbd_process_options() calls nbd_clear_bdrvstate() by itself.
Let's fix leaks and refactor things to be more obvious:
- intialize yank at top of nbd_open()
- move yank cleanup to nbd_clear_bdrvstate()
- refactor nbd_open() so that all failure paths except for yank-register goes through nbd_clear_bdrvstate()
--- block/nbd.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)
block/nbd.c +18 -18
@@ -152,8 +152,12 @@ static void nbd_co_establish_connection_cancel(BlockDriverState *bs,static int nbd_client_handshake (BlockDriverState *bs,Error **errp);static void nbd_yank (void *opaque); -static void nbd_clear_bdrvstate (BDRVNBDState *s) +static void nbd_clear_bdrvstate (BlockDriverState *bs) { +BDRVNBDState *s = (BDRVNBDState *)bs->opaque; + +yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); +object_unref (OBJECT (s->tlscreds));qapi_free_SocketAddress (s->saddr); s->saddr =NULL ;
@@ -2279,9 +2283,6 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options, ret =0 ; error: -if (ret <0 ) { -nbd_clear_bdrvstate (s); - }qemu_opts_del (opts);return ret; }
@@ -2292,11 +2293,6 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,int ret;BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - ret =nbd_process_options (bs, options, errp); -if (ret <0 ) { -return ret; - } - s->bs = bs;qemu_co_mutex_init (&s->send_mutex);qemu_co_queue_init (&s->free_sema);
@@ -2305,20 +2301,23 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,return -EEXIST ; } + ret =nbd_process_options (bs, options, errp); +if (ret <0 ) { + goto fail; + } +/* * establish TCP connection, return error if it fails * TODO: Configurable retry-until-timeout behaviour. */ if (nbd_establish_connection (bs, s->saddr, errp) <0 ) { -yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); -return -ECONNREFUSED ; + ret = -ECONNREFUSED ; + goto fail; } ret =nbd_client_handshake (bs, errp);if (ret <0 ) { -yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); -nbd_clear_bdrvstate (s); -return ret; + goto fail; }/* successfully connected */ s->state =NBD_CLIENT_CONNECTED ;
@@ -2330,6 +2329,10 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,aio_co_schedule (bdrv_get_aio_context (bs), s->connection_co);return 0 ; + +fail: +nbd_clear_bdrvstate (bs); +return ret; }static int nbd_co_flush (BlockDriverState *bs)
@@ -2373,11 +2376,8 @@ static void nbd_refresh_limits(BlockDriverState *bs, Error **errp)static void nbd_close (BlockDriverState *bs) { -BDRVNBDState *s = bs->opaque; -nbd_client_close (bs); -yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); -nbd_clear_bdrvstate (s); +nbd_clear_bdrvstate (bs); }/*
Signature · 1 line
2.29.2
Re: [PATCH v3 02/33] block/nbd: fix how state is cleared on nbd_open() failure paths
Quoted 68 lines · Vladimir Sementsov-Ogievskiy
We have two "return error" paths in
nbd_open()afternbd_process_options(). Actually we should callnbd_clear_bdrvstate()on these paths. Interesting thatnbd_process_options()callsnbd_clear_bdrvstate()by itself.Let's fix leaks and refactor things to be more obvious:
- intialize yank at top of
nbd_open()
- move yank cleanup tonbd_clear_bdrvstate()
- refactornbd_open()so that all failure paths except for yank-register goes throughnbd_clear_bdrvstate()Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)block/nbd.c +10
-9@@ -152,8 +152,12 @@ static void nbd_co_establish_connection_cancel(BlockDriverState *bs,static int nbd_client_handshake (BlockDriverState *bs,Error **errp);static void nbd_yank (void *opaque); -static void nbd_clear_bdrvstate (BDRVNBDState *s) +static void nbd_clear_bdrvstate (BlockDriverState *bs) { +BDRVNBDState *s = (BDRVNBDState *)bs->opaque; + +yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); +object_unref (OBJECT (s->tlscreds));qapi_free_SocketAddress (s->saddr); s->saddr =NULL ;@@ -2279,9 +2283,6 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options, ret =0 ; error: -if (ret <0 ) { -nbd_clear_bdrvstate (s); - }qemu_opts_del (opts);return ret; }@@ -2292,11 +2293,6 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,int ret;BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - ret =nbd_process_options (bs, options, errp); -if (ret <0 ) { -return ret; - } - s->bs = bs;qemu_co_mutex_init (&s->send_mutex);qemu_co_queue_init (&s->free_sema);@@ -2305,20 +2301,23 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,return -EEXIST ; } + ret =nbd_process_options (bs, options, errp); +if (ret <0 ) { + goto fail; + } +/* * establish TCP connection, return error if it fails
block/nbd.c static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 2318
Quoted 10 lines
block/nbd.c +2
-2* TODO: Configurable retry-until-timeout behaviour. */ if (nbd_establish_connection (bs, s->saddr, errp) <0 ) { -yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); -return -ECONNREFUSED ; + ret = -ECONNREFUSED ; + goto fail; } ret =nbd_client_handshake (bs, errp);
Not that this was introduced by this patch, but once you're at it:
AFAICT nbd_client_handshake() calls yank_unregister_instance() on some error path(s); I assume this needs to go too, otherwise it's called twice (and asserts).
Roman.
Quoted 32 lines
block/nbd.c +6
-7if (ret <0 ) { -yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); -nbd_clear_bdrvstate (s); -return ret; + goto fail; }/* successfully connected */ s->state =NBD_CLIENT_CONNECTED ;@@ -2330,6 +2329,10 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,aio_co_schedule (bdrv_get_aio_context (bs), s->connection_co);return 0 ; + +fail: +nbd_clear_bdrvstate (bs); +return ret; }static int nbd_co_flush (BlockDriverState *bs)@@ -2373,11 +2376,8 @@ static void nbd_refresh_limits(BlockDriverState *bs, Error **errp)static void nbd_close (BlockDriverState *bs) { -BDRVNBDState *s = bs->opaque; -nbd_client_close (bs); -yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); -nbd_clear_bdrvstate (s); +nbd_clear_bdrvstate (bs); }/*
Re: [PATCH v3 02/33] block/nbd: fix how state is cleared on nbd_open() failure paths
Quoted 78 lines · Roman Kagan
Quoted 77 lines · Vladimir Sementsov-Ogievskiy
We have two "return error" paths in
nbd_open()afternbd_process_options(). Actually we should callnbd_clear_bdrvstate()on these paths. Interesting thatnbd_process_options()callsnbd_clear_bdrvstate()by itself.Let's fix leaks and refactor things to be more obvious:
- intialize yank at top of
nbd_open()
- move yank cleanup tonbd_clear_bdrvstate()
- refactornbd_open()so that all failure paths except for yank-register goes throughnbd_clear_bdrvstate()Signed-off-by Vladimir Sementsov-Ogievskiy<[email protected]>--- block/nbd.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)block/nbd.c +12
-11@@ -152,8 +152,12 @@ static void nbd_co_establish_connection_cancel(BlockDriverState *bs,static int nbd_client_handshake (BlockDriverState *bs,Error **errp);static void nbd_yank (void *opaque); -static void nbd_clear_bdrvstate (BDRVNBDState *s) +static void nbd_clear_bdrvstate (BlockDriverState *bs) { +BDRVNBDState *s = (BDRVNBDState *)bs->opaque; + +yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); +object_unref (OBJECT (s->tlscreds));qapi_free_SocketAddress (s->saddr); s->saddr =NULL ;@@ -2279,9 +2283,6 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options, ret =0 ; error: -if (ret <0 ) { -nbd_clear_bdrvstate (s); - }qemu_opts_del (opts);return ret; }@@ -2292,11 +2293,6 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,int ret;BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - ret =nbd_process_options (bs, options, errp); -if (ret <0 ) { -return ret; - } - s->bs = bs;qemu_co_mutex_init (&s->send_mutex);qemu_co_queue_init (&s->free_sema);@@ -2305,20 +2301,23 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,return -EEXIST ; } + ret =nbd_process_options (bs, options, errp); +if (ret <0 ) { + goto fail; + } +/* * establish TCP connection, return error if it fails * TODO: Configurable retry-until-timeout behaviour. */ if (nbd_establish_connection (bs, s->saddr, errp) <0 ) { -yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); -return -ECONNREFUSED ; + ret = -ECONNREFUSED ; + goto fail; }
Quoted 8 lines
block/nbd.c static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 2318
Quoted 1 line
block/nbd.c +0
-0ret =nbd_client_handshake (bs, errp);Not that this was introduced by this patch, but once you're at it:
AFAICTnbd_client_handshake()callsyank_unregister_instance()on some error path(s); I assume this needs to go too, otherwise it's called twice (and asserts).Roman.
No, nbd_client_handshake() only calls yank_unregister_function(), not instance.
Signature · 2 lines
Best regards, Vladimir
Roman Kagan
Reviewed-byRe: [PATCH v3 02/33] block/nbd: fix how state is cleared on nbd_open() failure paths
Quoted 26 lines · Vladimir Sementsov-Ogievskiy
Quoted 25 lines · Roman Kagan
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
block/nbd.c +5
-0@@ -2305,20 +2301,23 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,return -EEXIST ; } + ret =nbd_process_options (bs, options, errp); +if (ret <0 ) { + goto fail; + } +/* block/nbd.c static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 2316
Quoted 10 lines
block/nbd.c +2
-2* establish TCP connection, return error if it fails * TODO: Configurable retry-until-timeout behaviour. */ if (nbd_establish_connection (bs, s->saddr, errp) <0 ) { -yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name)); -return -ECONNREFUSED ; + ret = -ECONNREFUSED ; + goto fail; } ret =nbd_client_handshake (bs, errp);Not that this was introduced by this patch, but once you're at it:
AFAICTnbd_client_handshake()callsyank_unregister_instance()on some error path(s); I assume this needs to go too, otherwise it's called twice (and asserts).
Quoted 4 lines
Quoted 2 lines
Roman.
No,
nbd_client_handshake()only callsyank_unregister_function(), not instance.
Indeed. Sorry for confusion.
Re: [PATCH v3 02/33] block/nbd: fix how state is cleared on nbd_open() failure paths
In reply to Vladimir Sementsov-OgievskiyQuoted 12 lines · Vladimir Sementsov-Ogievskiy
We have two "return error" paths in
nbd_open()afternbd_process_options(). Actually we should callnbd_clear_bdrvstate()on these paths. Interesting thatnbd_process_options()callsnbd_clear_bdrvstate()by itself.Let's fix leaks and refactor things to be more obvious:
- intialize yank at top of
nbd_open()
- move yank cleanup tonbd_clear_bdrvstate()
- refactornbd_open()so that all failure paths except for yank-register goes throughnbd_clear_bdrvstate()
Quoted 5 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
PATCH v3 3/33 block/nbd: ensure ->connection_thread is always valid
In reply to Vladimir Sementsov-OgievskiyFrom: Roman Kagan <[email protected]>
Simplify lifetime management of BDRVNBDState->connect_thread by delaying the possible cleanup of it until the BDRVNBDState itself goes away.
This also reverts
0267101af6 "block/nbd: fix possible use after free of s->connect_thread"
as now s->connect_thread can't be cleared until the very end.
--- block/nbd.c | 56 ++++++++++++++++++++--------------------------------- 1 file changed, 21 insertions(+), 35 deletions(-)
block/nbd.c +21 -35
@@ -144,17 +144,31 @@ typedef struct BDRVNBDState {NBDConnectThread *connect_thread; } BDRVNBDState; +static void nbd_free_connect_thread (NBDConnectThread *thr);static int nbd_establish_connection (BlockDriverState *bs,SocketAddress *saddr,Error **errp);static int nbd_co_establish_connection (BlockDriverState *bs,Error **errp); -static void nbd_co_establish_connection_cancel (BlockDriverState *bs, -bool detach); +static void nbd_co_establish_connection_cancel (BlockDriverState *bs);static int nbd_client_handshake (BlockDriverState *bs,Error **errp);static void nbd_yank (void *opaque);static void nbd_clear_bdrvstate (BlockDriverState *bs) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque; +NBDConnectThread *thr = s->connect_thread; +bool thr_running; + +qemu_mutex_lock (&thr->mutex); + thr_running = thr->state ==CONNECT_THREAD_RUNNING ; +if (thr_running) { + thr->state =CONNECT_THREAD_RUNNING_DETACHED ; + } +qemu_mutex_unlock (&thr->mutex); + +/* the runaway thread will clean it up itself */ +if (!thr_running) { +nbd_free_connect_thread (thr); + }yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name));
@@ -297,7 +311,7 @@ static void coroutine_fn nbd_client_co_drain_begin(BlockDriverState *bs)qemu_co_sleep_wake (s->connection_co_sleep_ns_state); } -nbd_co_establish_connection_cancel (bs, false); +nbd_co_establish_connection_cancel (bs);reconnect_delay_timer_del (s);
@@ -337,7 +351,7 @@ static void nbd_teardown_connection(BlockDriverState *bs)if (s->connection_co_sleep_ns_state) {qemu_co_sleep_wake (s->connection_co_sleep_ns_state); } -nbd_co_establish_connection_cancel (bs, true); +nbd_co_establish_connection_cancel (bs); }if (qemu_in_coroutine ()) { s->teardown_co =qemu_coroutine_self ();
@@ -448,11 +462,6 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)BDRVNBDState *s = bs->opaque;NBDConnectThread *thr = s->connect_thread; -if (!thr) { -/* detached */ -return -1 ; - } -qemu_mutex_lock (&thr->mutex);switch (thr->state) {
@@ -496,12 +505,6 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp) s->wait_connect = true;qemu_coroutine_yield (); -if (!s->connect_thread) { -/* detached */ -return -1 ; - } -assert (thr == s->connect_thread); -qemu_mutex_lock (&thr->mutex);switch (thr->state) {
@@ -549,18 +552,12 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)* nbd_co_establish_connection_cancel * Cancel nbd_co_establish_connection asynchronously: it will finish soon, to * allow drained section to begin. -* -* If detach is true, also cleanup the state (or if thread is running, move it -* to CONNECT_THREAD_RUNNING_DETACHED state). s->connect_thread becomes NULL if -* detach is true. */ -static void nbd_co_establish_connection_cancel (BlockDriverState *bs, -bool detach) +static void nbd_co_establish_connection_cancel (BlockDriverState *bs) {BDRVNBDState *s = bs->opaque;NBDConnectThread *thr = s->connect_thread;bool wake = false; -bool do_free = false;qemu_mutex_lock (&thr->mutex);
@@ -571,21 +568,10 @@ static void nbd_co_establish_connection_cancel(BlockDriverState *bs, s->wait_connect = false; wake = true; } -if (detach) { - thr->state =CONNECT_THREAD_RUNNING_DETACHED ; - s->connect_thread =NULL ; - } - }else if (detach) { - do_free = true; }qemu_mutex_unlock (&thr->mutex); -if (do_free) { -nbd_free_connect_thread (thr); - s->connect_thread =NULL ; - } -if (wake) {aio_co_wake (s->connection_co); }
@@ -2306,6 +2292,8 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } +nbd_init_connect_thread (s); +/* * establish TCP connection, return error if it fails * TODO: Configurable retry-until-timeout behaviour.
@@ -2322,8 +2310,6 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,/* successfully connected */ s->state =NBD_CLIENT_CONNECTED ; -nbd_init_connect_thread (s); - s->connection_co =qemu_coroutine_create (nbd_connection_entry, s);bdrv_inc_in_flight (bs);aio_co_schedule (bdrv_get_aio_context (bs), s->connection_co);
Signature · 1 line
2.29.2
Re: [PATCH v3 03/33] block/nbd: ensure ->connection_thread is always valid
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
From: Roman Kagan <[email protected]>
Simplify lifetime management of
BDRVNBDState->connect_threadby delaying the possible cleanup of it until the BDRVNBDState itself goes away.This also reverts
0267101af6"block/nbd: fix possible use after free ofs->connect_thread"
as nows->connect_threadcan't be cleared until the very end.Signed-off-by Roman Kagan <[email protected]>[vsementsov: rebase, revert 0267101af6 changes]Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 56 ++++++++++++++++++++--------------------------------- 1 file changed, 21 insertions(+), 35 deletions(-)static void
nbd_clear_bdrvstate(BlockDriverState *bs){ BDRVNBDState *s = (BDRVNBDState *)bs->opaque;block/nbd.c +1
-0+NBDConnectThread *thr = s->connect_thread;
block/nbd.c typedef struct BDRVNBDState { · line 168
Quoted 10 lines
block/nbd.c +10
-0+bool thr_running; + +qemu_mutex_lock (&thr->mutex); + thr_running = thr->state ==CONNECT_THREAD_RUNNING ; +if (thr_running) { + thr->state =CONNECT_THREAD_RUNNING_DETACHED ; + } +qemu_mutex_unlock (&thr->mutex); + +/* the runaway thread will clean it up itself */
s/clean it up/clean up/
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 4/33 block/nbd: nbd_client_handshake(): fix leak of s->ioc
In reply to Vladimir Sementsov-Ogievskiy--- block/nbd.c | 2 ++ 1 file changed, 2 insertions(+)
block/nbd.c +2 -0
@@ -1891,6 +1891,8 @@ static int nbd_client_handshake(BlockDriverState *bs, Error **errp) nbd_yank, bs);object_unref (OBJECT (s->sioc)); s->sioc =NULL ; +object_unref (OBJECT (s->ioc)); + s->ioc =NULL ;return ret; }
Signature · 1 line
2.29.2
Re: [PATCH v3 04/33] block/nbd: nbd_client_handshake(): fix leak of s->ioc
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 2 ++ 1 file changed, 2 insertions(+)
PATCH v3 5/33 block/nbd: BDRVNBDState: drop unused connect_err and connect_status
In reply to Vladimir Sementsov-OgievskiyThese fields are write-only. Drop them.
--- block/nbd.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
block/nbd.c +2 -10
@@ -121,8 +121,6 @@ typedef struct BDRVNBDState {bool wait_drained_end;int in_flight;NBDClientState state; -int connect_status; -Error *connect_err;bool wait_in_flight;QEMUTimer *reconnect_delay_timer;
@@ -580,7 +578,6 @@ static void nbd_co_establish_connection_cancel(BlockDriverState *bs) static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {int ret; -Error *local_err =NULL ;if (!nbd_client_connecting (s)) {return ;
@@ -621,14 +618,14 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) s->ioc =NULL ; } -if (nbd_co_establish_connection (s->bs, &local_err) <0 ) { +if (nbd_co_establish_connection (s->bs,NULL ) <0 ) { ret = -ECONNREFUSED ; goto out; }bdrv_dec_in_flight (s->bs); - ret =nbd_client_handshake (s->bs, &local_err); + ret =nbd_client_handshake (s->bs,NULL );if (s->drained) { s->wait_drained_end = true;
@@ -643,11 +640,6 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s)bdrv_inc_in_flight (s->bs); out: - s->connect_status = ret; -error_free (s->connect_err); - s->connect_err =NULL ; -error_propagate (&s->connect_err, local_err); -if (ret >=0 ) {/* successfully connected */ s->state =NBD_CLIENT_CONNECTED ;
Signature · 1 line
2.29.2
Re: [PATCH v3 05/33] block/nbd: BDRVNBDState: drop unused connect_err and connect_status
Quoted 8 lines · Vladimir Sementsov-Ogievskiy
These fields are write-only. Drop them.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Roman Kagan <[email protected]>--- block/nbd.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 6/33 util/async: aio_co_schedule(): support reschedule in same ctx
In reply to Vladimir Sementsov-OgievskiyWith the following patch we want to call wake coroutine from thread. And it doesn't work with aio_co_wake:
Assume we have no iothreads.
Assume we have a coroutine A, which waits in the yield point for external aio_co_wake(), and no progress can be done until it happen. Main thread is in blocking aio_poll() (for example, in blk_read()).
Now, in a separate thread we do aio_co_wake(). It calls aio_co_enter(), which goes through last "else" branch and do aio_context_acquire(ctx).
Now we have a deadlock, as aio_poll() will not release the context lock until some progress is done, and progress can't be done until aio_co_wake() wake the coroutine A. And it can't because it wait for aio_context_acquire().
Still, aio_co_schedule() works well in parallel with blocking aio_poll(). So we want use it. Let's add a possibility of rescheduling coroutine in same ctx where it was yield'ed.
Fetch co->ctx in same way as it is done in aio_co_wake().
--- include/block/aio.h | 2 +- util/async.c | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-)
include/block/aio.h +1 -1
@@ -643,7 +643,7 @@ static inline bool aio_node_check(AioContext *ctx, bool is_external)/** * aio_co_schedule: -* @ctx: the aio context +* @ctx: the aio context, if NULL, the current ctx of @co will be used. * @co: the coroutine * * Start a coroutine on a remote AioContext.
util/async.c +8 -0
@@ -545,6 +545,14 @@ fail:void aio_co_schedule (AioContext *ctx,Coroutine *co) { +if (!ctx) { +/* +* Read coroutine before co->ctx. Matches smp_wmb in +* qemu_coroutine_enter. +*/ +smp_read_barrier_depends (); + ctx =qatomic_read (&co->ctx); + }trace_aio_co_schedule (ctx, co);const char *scheduled =qatomic_cmpxchg (&co->scheduled,NULL , __func__);
Signature · 1 line
2.29.2
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
Quoted 46 lines · Vladimir Sementsov-Ogievskiy
With the following patch we want to call wake coroutine from thread. And it doesn't work with
aio_co_wake:
Assume we have no iothreads.
Assume we have a coroutine A, which waits in the yield point for externalaio_co_wake(), and no progress can be done until it happen. Main thread is in blockingaio_poll()(for example, inblk_read()).Now, in a separate thread we do
aio_co_wake(). It callsaio_co_enter(), which goes through last "else" branch and doaio_context_acquire(ctx).Now we have a deadlock, as
aio_poll()will not release the context lock until some progress is done, and progress can't be done untilaio_co_wake()wake the coroutine A. And it can't because it wait foraio_context_acquire().Still,
aio_co_schedule()works well in parallel with blockingaio_poll(). So we want use it. Let's add a possibility of rescheduling coroutine in same ctx where it was yield'ed.Fetch
co->ctxin same way as it is done inaio_co_wake().Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/aio.h | 2 +- util/async.c | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-)include/block/aio.h +1
-1@@ -643,7 +643,7 @@ static inline bool aio_node_check(AioContext *ctx, bool is_external)/** * aio_co_schedule: -* @ctx: the aio context +* @ctx: the aio context, if NULL, the current ctx of @co will be used. * @co: the coroutine * * Start a coroutine on a remote AioContext. util/async.c +0
-0@@ -545,6 +545,14 @@ fail:
util/async.c fail: · line 555
Quoted 10 lines
util/async.c +8
-0void aio_co_schedule (AioContext *ctx,Coroutine *co) { +if (!ctx) { +/* +* Read coroutine before co->ctx. Matches smp_wmb in +* qemu_coroutine_enter. +*/ +smp_read_barrier_depends (); + ctx =qatomic_read (&co->ctx); + }
I'd rather not extend this interface, but add a new one on top. And document how it's different from aio_co_wake().
Roman.
Quoted 6 lines
util/async.c +0
-0trace_aio_co_schedule (ctx, co);const char *scheduled =qatomic_cmpxchg (&co->scheduled,NULL , __func__);Signature · 1 line
2.29.2
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
Quoted 47 lines · Roman Kagan
Quoted 46 lines · Vladimir Sementsov-Ogievskiy
With the following patch we want to call wake coroutine from thread. And it doesn't work with
aio_co_wake:
Assume we have no iothreads.
Assume we have a coroutine A, which waits in the yield point for externalaio_co_wake(), and no progress can be done until it happen. Main thread is in blockingaio_poll()(for example, inblk_read()).Now, in a separate thread we do
aio_co_wake(). It callsaio_co_enter(), which goes through last "else" branch and doaio_context_acquire(ctx).Now we have a deadlock, as
aio_poll()will not release the context lock until some progress is done, and progress can't be done untilaio_co_wake()wake the coroutine A. And it can't because it wait foraio_context_acquire().Still,
aio_co_schedule()works well in parallel with blockingaio_poll(). So we want use it. Let's add a possibility of rescheduling coroutine in same ctx where it was yield'ed.Fetch
co->ctxin same way as it is done inaio_co_wake().Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/aio.h | 2 +- util/async.c | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-)include/block/aio.h +1
-1@@ -643,7 +643,7 @@ static inline bool aio_node_check(AioContext *ctx, bool is_external)/** * aio_co_schedule: -* @ctx: the aio context +* @ctx: the aio context, if NULL, the current ctx of @co will be used. * @co: the coroutine * * Start a coroutine on a remote AioContext. util/async.c +0
-0@@ -545,6 +545,14 @@ fail:
Quoted 14 lines
util/async.c fail: · line 555
Quoted 10 lines
util/async.c +8
-0void aio_co_schedule (AioContext *ctx,Coroutine *co) { +if (!ctx) { +/* +* Read coroutine before co->ctx. Matches smp_wmb in +* qemu_coroutine_enter. +*/ +smp_read_barrier_depends (); + ctx =qatomic_read (&co->ctx); + }I'd rather not extend this interface, but add a new one on top. And document how it's different from
aio_co_wake().
Agree, that's better. Will do.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
In reply to Vladimir Sementsov-OgievskiyQuoted 9 lines · Vladimir Sementsov-Ogievskiy
With the following patch we want to call wake coroutine from thread. And it doesn't work with
aio_co_wake:
Assume we have no iothreads.
Assume we have a coroutine A, which waits in the yield point for externalaio_co_wake(), and no progress can be done until it happen. Main thread is in blockingaio_poll()(for example, inblk_read()).Now, in a separate thread we do
aio_co_wake(). It callsaio_co_enter(), which goes through last "else" branch and doaio_context_acquire(ctx).
I don't understand. Why doesn't aio_co_enter go through the ctx != qemu_get_current_aio_context() branch and just do aio_co_schedule? That was at least the idea behind aio_co_wake and aio_co_enter.
Paolo
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
Quoted 8 lines · Paolo Bonzini
Quoted 7 lines · Vladimir Sementsov-Ogievskiy
With the following patch we want to call wake coroutine from thread. And it doesn't work with
aio_co_wake:
Assume we have no iothreads.
Assume we have a coroutine A, which waits in the yield point for externalaio_co_wake(), and no progress can be done until it happen. Main thread is in blockingaio_poll()(for example, inblk_read()).
Quoted 5 lines
Quoted 2 lines
Now, in a separate thread we do
aio_co_wake(). It callsaio_co_enter(), which goes through last "else" branch and doaio_context_acquire(ctx).I don't understand. Why doesn't
aio_co_entergo through the ctx !=qemu_get_current_aio_context()branch and just do aio_co_schedule? That was at least the idea behindaio_co_wakeandaio_co_enter.
Because ctx is exactly qemu_get_current_aio_context(), as we are not in iothread but in nbd connection thread. So, qemu_get_current_aio_context() returns qemu_aio_context.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines
Quoted 1 line
I don't understand. Why doesn't
aio_co_entergo through the ctx !=qemu_get_current_aio_context()branch and just doaio_co_schedule? That was at least the idea behindaio_co_wakeandaio_co_enter.Because ctx is exactly
qemu_get_current_aio_context(), as we are not in iothread but in nbd connection thread. So,qemu_get_current_aio_context()returnsqemu_aio_context.
So the problem is that threads other than the main thread and the I/O thread should not return qemu_aio_context. The vCPU thread may need to return it when running with BQL taken, though.
Something like this (untested):
include/block/aio.h +4 -1
@@ -691,10 +691,13 @@ void aio_co_enter(AioContext *ctx, struct Coroutine *co);* Return the AioContext whose event loop runs in the current thread. * * If called from an IOThread this will be the IOThread's AioContext. If -* called from another thread it will be the main loop AioContext. +* called from the main thread or with the "big QEMU lock" taken it +* will be the main loop AioContext. */ AioContext *qemu_get_current_aio_context (void ); +void qemu_set_current_aio_context (AioContext *ctx); +/** * aio_context_setup: * @ctx: the aio context
iothread.c +15 -3
@@ -39,11 +39,23 @@ DECLARE_CLASS_CHECKERS(IOThreadClass, IOTHREAD,#define IOTHREAD_POLL_MAX_NS_DEFAULT 0ULL#endif -static __threadIOThread *my_iothread; +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ +assert (!my_aiocontext); + my_aiocontext = ctx; +}AioContext *qemu_get_current_aio_context (void ) { -return my_iothread ? my_iothread->ctx :qemu_get_aio_context (); +if (my_aiocontext) { +return my_aiocontext; + } +if (qemu_mutex_iothread_locked ()) { +return qemu_get_aio_context (); + } +return NULL ; }static void *iothread_run (void *opaque)
@@ -56,7 +68,7 @@ static void *iothread_run(void *opaque)* in this new thread uses glib. */ g_main_context_push_thread_default (iothread->worker_context); - my_iothread = iothread; +qemu_set_current_aio_context (iothread->ctx); iothread->thread_id =qemu_get_thread_id ();qemu_sem_post (&iothread->init_done_sem);
stubs/iothread.c +4 -0
@@ -6,3 +6,7 @@ AioContext *qemu_get_current_aio_context(void) {return qemu_get_aio_context (); } + +void qemu_set_current_aio_context (AioContext *ctx) +{ +}
tests/unit/iothread.c +16 -3
@@ -30,13 +30,26 @@ struct IOThread {bool stopping; }; -static __threadIOThread *my_iothread; +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ +assert (!my_aiocontext); + my_aiocontext = ctx; +}AioContext *qemu_get_current_aio_context (void ) { -return my_iothread ? my_iothread->ctx :qemu_get_aio_context (); +if (my_aiocontext) { +return my_aiocontext; + } +if (qemu_mutex_iothread_locked ()) { +return qemu_get_aio_context (); + } +return NULL ; } +static void iothread_init_gcontext (IOThread *iothread) {GSource *source;
@@ -54,7 +67,7 @@ static void *iothread_run(void *opaque)rcu_register_thread (); - my_iothread = iothread; +qemu_set_current_aio_context (iothread->ctx);qemu_mutex_lock (&iothread->init_done_lock); iothread->ctx =aio_context_new (&error_abort);
util/main-loop.c +1 -0
@@ -170,6 +170,7 @@ int qemu_init_main_loop(Error **errp)if (!qemu_aio_context) {return -EMFILE ; } +qemu_set_current_aio_context (qemu_aio_context); qemu_notify_bh =qemu_bh_new (notify_event_cb,NULL ); gpollfds =g_array_new (FALSE, FALSE,sizeof (GPollFD)); src =aio_get_g_source (qemu_aio_context);
If it works for you, I can post it as a formal patch.
Paolo
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
Quoted 8 lines · Paolo Bonzini
Quoted 7 lines · Vladimir Sementsov-Ogievskiy
Quoted 6 lines
Quoted 1 line
I don't understand. Why doesn't
aio_co_entergo through the ctx !=qemu_get_current_aio_context()branch and just doaio_co_schedule? That was at least the idea behindaio_co_wakeandaio_co_enter.
Quoted 7 lines
Quoted 3 lines
Because ctx is exactly
qemu_get_current_aio_context(), as we are not in iothread but in nbd connection thread. So,qemu_get_current_aio_context()returnsqemu_aio_context.So the problem is that threads other than the main thread and the I/O thread should not return
qemu_aio_context. The vCPU thread may need to return it when running with BQL taken, though.
I'm not sure this is the only case.
AFAICS your patch has basically the same effect as Vladimir's patch "util/async: aio_co_enter(): do aio_co_schedule in general case"
(https://lore.kernel.org/qemu-devel/[email protected]/).
That one was found to break e.g. aio=threads cases. I guessed it implicitly relied upon aio_co_enter() acquiring the aio_context but we didn't dig further to pinpoint the exact scenario.
Roman.
Quoted 120 lines
Something like this (untested):
include/block/aio.h +4
-1@@ -691,10 +691,13 @@ void aio_co_enter(AioContext *ctx, struct Coroutine *co);* Return the AioContext whose event loop runs in the current thread. * * If called from an IOThread this will be the IOThread's AioContext. If -* called from another thread it will be the main loop AioContext. +* called from the main thread or with the "big QEMU lock" taken it +* will be the main loop AioContext. */ AioContext *qemu_get_current_aio_context (void ); +void qemu_set_current_aio_context (AioContext *ctx); +/** * aio_context_setup: * @ctx: the aio context iothread.c +15
-3@@ -39,11 +39,23 @@ DECLARE_CLASS_CHECKERS(IOThreadClass, IOTHREAD,#define IOTHREAD_POLL_MAX_NS_DEFAULT 0ULL#endif -static __threadIOThread *my_iothread; +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ +assert (!my_aiocontext); + my_aiocontext = ctx; +}AioContext *qemu_get_current_aio_context (void ) { -return my_iothread ? my_iothread->ctx :qemu_get_aio_context (); +if (my_aiocontext) { +return my_aiocontext; + } +if (qemu_mutex_iothread_locked ()) { +return qemu_get_aio_context (); + } +return NULL ; }static void *iothread_run (void *opaque)@@ -56,7 +68,7 @@ static void *iothread_run(void *opaque)* in this new thread uses glib. */ g_main_context_push_thread_default (iothread->worker_context); - my_iothread = iothread; +qemu_set_current_aio_context (iothread->ctx); iothread->thread_id =qemu_get_thread_id ();qemu_sem_post (&iothread->init_done_sem);stubs/iothread.c +4
-0@@ -6,3 +6,7 @@ AioContext *qemu_get_current_aio_context(void) {return qemu_get_aio_context (); } + +void qemu_set_current_aio_context (AioContext *ctx) +{ +}tests/unit/iothread.c +16
-3@@ -30,13 +30,26 @@ struct IOThread {bool stopping; }; -static __threadIOThread *my_iothread; +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ +assert (!my_aiocontext); + my_aiocontext = ctx; +}AioContext *qemu_get_current_aio_context (void ) { -return my_iothread ? my_iothread->ctx :qemu_get_aio_context (); +if (my_aiocontext) { +return my_aiocontext; + } +if (qemu_mutex_iothread_locked ()) { +return qemu_get_aio_context (); + } +return NULL ; } +static void iothread_init_gcontext (IOThread *iothread) {GSource *source;@@ -54,7 +67,7 @@ static void *iothread_run(void *opaque)rcu_register_thread (); - my_iothread = iothread; +qemu_set_current_aio_context (iothread->ctx);qemu_mutex_lock (&iothread->init_done_lock); iothread->ctx =aio_context_new (&error_abort);util/main-loop.c +1
-0@@ -170,6 +170,7 @@ int qemu_init_main_loop(Error **errp)if (!qemu_aio_context) {return -EMFILE ; } +qemu_set_current_aio_context (qemu_aio_context); qemu_notify_bh =qemu_bh_new (notify_event_cb,NULL ); gpollfds =g_array_new (FALSE, FALSE,sizeof (GPollFD)); src =aio_get_g_source (qemu_aio_context);If it works for you, I can post it as a formal patch.
Paolo
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
Quoted 7 lines · Roman Kagan
AFAICS your patch has basically the same effect as Vladimir's patch "util/async:
aio_co_enter(): doaio_co_schedulein general case"
(https://lore.kernel.org/qemu-devel/[email protected]/).
That one was found to break e.g. aio=threads cases. I guessed it implicitly relied uponaio_co_enter()acquiring theaio_contextbut we didn't dig further to pinpoint the exact scenario.
That one is much more intrusive, because it goes through a bottom half unnecessarily in the case of aio_co_wake being called from an I/O callback (or from another bottom half). I'll test my patch with aio=threads.
Paolo
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
In reply to Paolo BonziniQuoted 139 lines · Paolo Bonzini
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines
Quoted 1 line
I don't understand. Why doesn't
aio_co_entergo through the ctx !=qemu_get_current_aio_context()branch and just doaio_co_schedule? That was at least the idea behindaio_co_wakeandaio_co_enter.Because ctx is exactly
qemu_get_current_aio_context(), as we are not in iothread but in nbd connection thread. So,qemu_get_current_aio_context()returnsqemu_aio_context.So the problem is that threads other than the main thread and the I/O thread should not return qemu_aio_context. The vCPU thread may need to return it when running with BQL taken, though.
Something like this (untested):
include/block/aio.h +4
-1@@ -691,10 +691,13 @@ void aio_co_enter(AioContext *ctx, struct Coroutine *co);* Return the AioContext whose event loop runs in the current thread. * * If called from an IOThread this will be the IOThread's AioContext. If -* called from another thread it will be the main loop AioContext. +* called from the main thread or with the "big QEMU lock" taken it +* will be the main loop AioContext. */ AioContext *qemu_get_current_aio_context(void); +void qemu_set_current_aio_context (AioContext *ctx); + /*** aio_context_setup: * @ctx: the aio context iothread.c +15
-3@@ -39,11 +39,23 @@ DECLARE_CLASS_CHECKERS(IOThreadClass, IOTHREAD, #define IOTHREAD_POLL_MAX_NS_DEFAULT 0ULL #endif -static __threadIOThread *my_iothread; +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ + assert(!my_aiocontext); + my_aiocontext = ctx; +} AioContext *qemu_get_current_aio_context(void) { - return my_iothread ? my_iothread->ctx : qemu_get_aio_context(); + if (my_aiocontext) { + return my_aiocontext; + } + if (qemu_mutex_iothread_locked()) { + return qemu_get_aio_context(); + } + return NULL; } static void *iothread_run(void *opaque)@@ -56,7 +68,7 @@ static void *iothread_run(void *opaque)* in this new thread uses glib. */ g_main_context_push_thread_default(iothread->worker_context); - my_iothread = iothread; + qemu_set_current_aio_context(iothread->ctx); iothread->thread_id = qemu_get_thread_id(); qemu_sem_post(&iothread->init_done_sem);stubs/iothread.c +4
-0@@ -6,3 +6,7 @@ AioContext *qemu_get_current_aio_context(void) { return qemu_get_aio_context(); } + +void qemu_set_current_aio_context (AioContext *ctx) +{ +}tests/unit/iothread.c +16
-3@@ -30,13 +30,26 @@ struct IOThread { bool stopping; }; -static __threadIOThread *my_iothread; +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ + assert(!my_aiocontext); + my_aiocontext = ctx; +} AioContext *qemu_get_current_aio_context(void) { - return my_iothread ? my_iothread->ctx : qemu_get_aio_context(); + if (my_aiocontext) { + return my_aiocontext; + } + if (qemu_mutex_iothread_locked()) { + return qemu_get_aio_context(); + } + return NULL; } + static void iothread_init_gcontext(IOThread *iothread) { GSource *source;@@ -54,7 +67,7 @@ static void *iothread_run(void *opaque) rcu_register_thread(); - my_iothread = iothread; + qemu_set_current_aio_context(iothread->ctx); qemu_mutex_lock(&iothread->init_done_lock); iothread->ctx = aio_context_new(&error_abort);util/main-loop.c +1
-0@@ -170,6 +170,7 @@ int qemu_init_main_loop(Error **errp) if (!qemu_aio_context) { return -EMFILE; } + qemu_set_current_aio_context(qemu_aio_context); qemu_notify_bh = qemu_bh_new(notify_event_cb, NULL); gpollfds = g_array_new(FALSE, FALSE, sizeof(GPollFD)); src = aio_get_g_source(qemu_aio_context);
Quoted 2 lines
If it works for you, I can post it as a formal patch.
This doesn't work for iotests.. qemu-io goes through version in stub. It works if I add:
stubs/iothread.c +12 -1
@@ -2,7 +2,18 @@#include "block/aio.h" #include "qemu/main-loop.h" +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ +assert (!my_aiocontext); + my_aiocontext = ctx; +} +AioContext *qemu_get_current_aio_context (void ) { -return qemu_get_aio_context (); +if (my_aiocontext) { +return my_aiocontext; + } +return NULL ; }
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
Quoted 141 lines · Vladimir Sementsov-Ogievskiy
Quoted 140 lines · Paolo Bonzini
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines
Quoted 1 line
I don't understand. Why doesn't
aio_co_entergo through the ctx !=qemu_get_current_aio_context()branch and just doaio_co_schedule? That was at least the idea behindaio_co_wakeandaio_co_enter.Because ctx is exactly
qemu_get_current_aio_context(), as we are not in iothread but in nbd connection thread. So,qemu_get_current_aio_context()returnsqemu_aio_context.So the problem is that threads other than the main thread and the I/O thread should not return qemu_aio_context. The vCPU thread may need to return it when running with BQL taken, though.
Something like this (untested):
include/block/aio.h +4
-1@@ -691,10 +691,13 @@ void aio_co_enter(AioContext *ctx, structCoroutine *co); * Return the AioContext whose event loop runs in the current thread. * * If called from an IOThread this will be the IOThread's AioContext. If -* called from another thread it will be the main loop AioContext. +* called from the main thread or with the "big QEMU lock" taken it +* will be the main loop AioContext. */ AioContext *qemu_get_current_aio_context(void); +void qemu_set_current_aio_context (AioContext *ctx); + /*** aio_context_setup: * @ctx: the aio context
iothread.c +15
-3@@ -39,11 +39,23 @@ DECLARE_CLASS_CHECKERS(IOThreadClass, IOTHREAD, #define IOTHREAD_POLL_MAX_NS_DEFAULT 0ULL #endif -static __threadIOThread *my_iothread; +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ + assert(!my_aiocontext); + my_aiocontext = ctx; +} AioContext *qemu_get_current_aio_context(void) { - return my_iothread ? my_iothread->ctx : qemu_get_aio_context(); + if (my_aiocontext) { + return my_aiocontext; + } + if (qemu_mutex_iothread_locked()) { + return qemu_get_aio_context(); + } + return NULL; } static void *iothread_run(void *opaque)@@ -56,7 +68,7 @@ static void *iothread_run(void *opaque)* in this new thread uses glib. */ g_main_context_push_thread_default(iothread->worker_context); - my_iothread = iothread; + qemu_set_current_aio_context(iothread->ctx); iothread->thread_id = qemu_get_thread_id(); qemu_sem_post(&iothread->init_done_sem);stubs/iothread.c +4
-0@@ -6,3 +6,7 @@ AioContext *qemu_get_current_aio_context(void) { return qemu_get_aio_context(); } + +void qemu_set_current_aio_context (AioContext *ctx) +{ +}tests/unit/iothread.c +16
-3@@ -30,13 +30,26 @@ struct IOThread { bool stopping; }; -static __threadIOThread *my_iothread; +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ + assert(!my_aiocontext); + my_aiocontext = ctx; +} AioContext *qemu_get_current_aio_context(void) { - return my_iothread ? my_iothread->ctx : qemu_get_aio_context(); + if (my_aiocontext) { + return my_aiocontext; + } + if (qemu_mutex_iothread_locked()) { + return qemu_get_aio_context(); + } + return NULL; } + static void iothread_init_gcontext(IOThread *iothread) { GSource *source;@@ -54,7 +67,7 @@ static void *iothread_run(void *opaque) rcu_register_thread(); - my_iothread = iothread; + qemu_set_current_aio_context(iothread->ctx); qemu_mutex_lock(&iothread->init_done_lock); iothread->ctx = aio_context_new(&error_abort);util/main-loop.c +1
-0@@ -170,6 +170,7 @@ int qemu_init_main_loop(Error **errp) if (!qemu_aio_context) { return -EMFILE; } + qemu_set_current_aio_context(qemu_aio_context); qemu_notify_bh = qemu_bh_new(notify_event_cb, NULL); gpollfds = g_array_new(FALSE, FALSE, sizeof(GPollFD)); src = aio_get_g_source(qemu_aio_context);
Quoted 4 lines
Quoted 2 lines
If it works for you, I can post it as a formal patch.
This doesn't work for iotests.. qemu-io goes through version in stub. It works if I add:
Great, thanks. I'll try the patch, also with the test that broke with the earlier version, and post if it works.
Paolo
Quoted 27 lines
stubs/iothread.c +12
-1@@ -2,7 +2,18 @@ #include "block/aio.h" #include "qemu/main-loop.h" +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ + assert(!my_aiocontext); + my_aiocontext = ctx; +} + AioContext *qemu_get_current_aio_context(void) { - return qemu_get_aio_context(); + if (my_aiocontext) { + return my_aiocontext; + } + return NULL; }
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
Quoted 141 lines · Paolo Bonzini
Quoted 140 lines · Vladimir Sementsov-Ogievskiy
Quoted 139 lines · Paolo Bonzini
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines
Quoted 1 line
I don't understand. Why doesn't
aio_co_entergo through the ctx !=qemu_get_current_aio_context()branch and just doaio_co_schedule? That was at least the idea behindaio_co_wakeandaio_co_enter.Because ctx is exactly
qemu_get_current_aio_context(), as we are not in iothread but in nbd connection thread. So,qemu_get_current_aio_context()returnsqemu_aio_context.So the problem is that threads other than the main thread and the I/O thread should not return qemu_aio_context. The vCPU thread may need to return it when running with BQL taken, though.
Something like this (untested):
include/block/aio.h +4
-1@@ -691,10 +691,13 @@ void aio_co_enter(AioContext *ctx, struct Coroutine *co);* Return the AioContext whose event loop runs in the current thread. * * If called from an IOThread this will be the IOThread's AioContext. If -* called from another thread it will be the main loop AioContext. +* called from the main thread or with the "big QEMU lock" taken it +* will be the main loop AioContext. */ AioContext *qemu_get_current_aio_context(void); +void qemu_set_current_aio_context (AioContext *ctx); + /*** aio_context_setup: * @ctx: the aio context iothread.c +15
-3@@ -39,11 +39,23 @@ DECLARE_CLASS_CHECKERS(IOThreadClass, IOTHREAD, #define IOTHREAD_POLL_MAX_NS_DEFAULT 0ULL #endif -static __threadIOThread *my_iothread; +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ + assert(!my_aiocontext); + my_aiocontext = ctx; +} AioContext *qemu_get_current_aio_context(void) { - return my_iothread ? my_iothread->ctx : qemu_get_aio_context(); + if (my_aiocontext) { + return my_aiocontext; + } + if (qemu_mutex_iothread_locked()) { + return qemu_get_aio_context(); + } + return NULL; } static void *iothread_run(void *opaque)@@ -56,7 +68,7 @@ static void *iothread_run(void *opaque)* in this new thread uses glib. */ g_main_context_push_thread_default(iothread->worker_context); - my_iothread = iothread; + qemu_set_current_aio_context(iothread->ctx); iothread->thread_id = qemu_get_thread_id(); qemu_sem_post(&iothread->init_done_sem);stubs/iothread.c +4
-0@@ -6,3 +6,7 @@ AioContext *qemu_get_current_aio_context(void) { return qemu_get_aio_context(); } + +void qemu_set_current_aio_context (AioContext *ctx) +{ +}tests/unit/iothread.c +16
-3@@ -30,13 +30,26 @@ struct IOThread { bool stopping; }; -static __threadIOThread *my_iothread; +static __threadAioContext *my_aiocontext; + +void qemu_set_current_aio_context (AioContext *ctx) +{ + assert(!my_aiocontext); + my_aiocontext = ctx; +} AioContext *qemu_get_current_aio_context(void) { - return my_iothread ? my_iothread->ctx : qemu_get_aio_context(); + if (my_aiocontext) { + return my_aiocontext; + } + if (qemu_mutex_iothread_locked()) { + return qemu_get_aio_context(); + } + return NULL; } + static void iothread_init_gcontext(IOThread *iothread) { GSource *source;@@ -54,7 +67,7 @@ static void *iothread_run(void *opaque) rcu_register_thread(); - my_iothread = iothread; + qemu_set_current_aio_context(iothread->ctx); qemu_mutex_lock(&iothread->init_done_lock); iothread->ctx = aio_context_new(&error_abort);util/main-loop.c +1
-0@@ -170,6 +170,7 @@ int qemu_init_main_loop(Error **errp) if (!qemu_aio_context) { return -EMFILE; } + qemu_set_current_aio_context(qemu_aio_context); qemu_notify_bh = qemu_bh_new(notify_event_cb, NULL); gpollfds = g_array_new(FALSE, FALSE, sizeof(GPollFD)); src = aio_get_g_source(qemu_aio_context);
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
If it works for you, I can post it as a formal patch.
This doesn't work for iotests.. qemu-io goes through version in stub. It works if I add:
Great, thanks. I'll try the patch, also with the test that broke with the earlier version, and post if it works.
Thanks, I'll base v4 of nbd patches on it.
I now run make check. test-aio-multithread crashes on assertion:
(gdb) bt
#0 0x00007f4af8d839d5 in raise () from /lib64/libc.so.6 #1 0x00007f4af8d6c8a4 in abort () from /lib64/libc.so.6 #2 0x00007f4af8d6c789 in __assert_fail_base.cold () from /lib64/libc.so.6 #3 0x00007f4af8d7c026 in __assert_fail () from /lib64/libc.so.6 #4 0x000055daebfdab95 in aio_poll (ctx=0x7f4ae0000b60, blocking=true) at ../util/aio-posix.c:567 #5 0x000055daebea096c in iothread_run (opaque=0x55daed81bc90) at ../tests/unit/iothread.c:91 #6 0x000055daebfc6c4a in qemu_thread_start (args=0x55daed7d9940) at ../util/qemu-thread-posix.c:521 #7 0x00007f4af8f1a3f9 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4af8e47b53 in clone () from /lib64/libc.so.6
(gdb) fr 4
#4 0x000055daebfdab95 in aio_poll (ctx=0x7f4ae0000b60, blocking=true) at ../util/aio-posix.c:567
567 assert(in_aio_context_home_thread(ctx == iohandler_get_aio_context() ?
(gdb) list
562 * 563 * aio_poll() may only be called in the AioContext's thread. iohandler_ctx 564 * is special in that it runs in the main thread, but that thread's context 565 * is qemu_aio_context. 566 */
567 assert(in_aio_context_home_thread(ctx == iohandler_get_aio_context() ?
568 qemu_get_aio_context() : ctx));
569
570 qemu_lockcnt_inc(&ctx->list_lock);
571
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 06/33] util/async: aio_co_schedule(): support reschedule in same ctx
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
Thanks, I'll base v4 of nbd patches on it.
I now run make check. test-aio-multithread crashes on assertion:
With the patch I've sent it doesn't, so hopefully you can go ahead.
Paolo
Quoted 23 lines
(gdb) bt
#00x00007f4af8d839d5in raise () from/lib64/libc.so.6
#10x00007f4af8d6c8a4in abort () from/lib64/libc.so.6
#20x00007f4af8d6c789in __assert_fail_base.cold () from/lib64/libc.so.6
#30x00007f4af8d7c026in__assert_fail() from/lib64/libc.so.6
#40x000055daebfdab95inaio_poll(ctx=0x7f4ae0000b60, blocking=true) at../util/aio-posix.c:567#50x000055daebea096ciniothread_run(opaque=0x55daed81bc90) at../tests/unit/iothread.c:91#60x000055daebfc6c4ainqemu_thread_start(args=0x55daed7d9940) at../util/qemu-thread-posix.c:521#70x00007f4af8f1a3f9instart_thread() from/lib64/libpthread.so.0
#80x00007f4af8e47b53in clone () from/lib64/libc.so.6
(gdb) fr 4
#40x000055daebfdab95inaio_poll(ctx=0x7f4ae0000b60, blocking=true) at../util/aio-posix.c:567567 assert(in_aio_context_home_thread(ctx ==iohandler_get_aio_context()? (gdb) list
562 *
563 *aio_poll()may only be called in the AioContext's thread.iohandler_ctx564 * is special in that it runs in the main thread, but that thread's context 565 * isqemu_aio_context.
566 */
567 assert(in_aio_context_home_thread(ctx ==iohandler_get_aio_context()? 568qemu_get_aio_context(): ctx));
569
570qemu_lockcnt_inc(&ctx->list_lock);
PATCH v3 7/33 block/nbd: simplify waking of nbd_co_establish_connection()
In reply to Vladimir Sementsov-OgievskiyInstead of connect_bh, bh_ctx and wait_connect fields we can live with only one link to waiting coroutine, protected by mutex.
So new logic is:
nbd_co_establish_connection() sets wait_co under mutex, release the mutex and do yield(). Note, that wait_co may be scheduled by thread immediately after unlocking the mutex. Still, in main thread (or iothread) we'll not reach the code for entering the coroutine until the yield() so we are safe.
Both connect_thread_func() and nbd_co_establish_connection_cancel() do the following to handle wait_co:
Under mutex, if thr->wait_co is not NULL, call aio_co_wake() (which never tries to acquire aio context since previous commit, so we are safe to do it under thr->mutex) and set thr->wait_co to NULL. This way we protect ourselves of scheduling it twice.
Also this commit make nbd_co_establish_connection() less connected to bs (we have generic pointer to the coroutine, not use s->connection_co directly). So, we are on the way of splitting connection API out of nbd.c (which is overcomplicated now).
--- block/nbd.c | 49 +++++++++---------------------------------------- 1 file changed, 9 insertions(+), 40 deletions(-)
block/nbd.c +9 -40
@@ -87,12 +87,6 @@ typedef enum NBDConnectThreadState { typedef struct NBDConnectThread { /* Initialization constants */ SocketAddress *saddr; /* address to connect to */ - /* - * Bottom half to schedule on completion. Scheduled only if bh_ctx is not - * NULL - */ - QEMUBHFunc *bh_func; - void *bh_opaque; /* * Result of last attempt. Valid in FAIL and SUCCESS states.
@@ -101,10 +95,10 @@ typedef struct NBDConnectThread {QIOChannelSocket *sioc;Error *err; -/* state and bh_ctx are protected by mutex */ QemuMutex mutex; +/* All further fields are protected by mutex */ NBDConnectThreadState state;/* current state of the thread */ -AioContext *bh_ctx;/* where to schedule bh (NULL means don't schedule) */ +Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ } NBDConnectThread; typedef struct BDRVNBDState {
@@ -138,7 +132,6 @@ typedef struct BDRVNBDState {char *x_dirty_bitmap;bool alloc_depth; -bool wait_connect;NBDConnectThread *connect_thread; } BDRVNBDState;
@@ -374,15 +367,6 @@ static bool nbd_client_connecting_wait(BDRVNBDState *s)return qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ; } -static void connect_bh (void *opaque) -{ -BDRVNBDState *state = opaque; - -assert (state->wait_connect); - state->wait_connect = false; -aio_co_wake (state->connection_co); -} -static void nbd_init_connect_thread (BDRVNBDState *s) { s->connect_thread =g_new (NBDConnectThread,1 );
@@ -390,8 +374,6 @@ static void nbd_init_connect_thread(BDRVNBDState *s) *s->connect_thread = (NBDConnectThread ) { .saddr =QAPI_CLONE (SocketAddress, s->saddr), .state =CONNECT_THREAD_NONE , - .bh_func = connect_bh, - .bh_opaque = s, };qemu_mutex_init (&s->connect_thread->mutex);
@@ -429,11 +411,9 @@ static void *connect_thread_func(void *opaque)switch (thr->state) {case CONNECT_THREAD_RUNNING : thr->state = ret <0 ?CONNECT_THREAD_FAIL :CONNECT_THREAD_SUCCESS ; -if (thr->bh_ctx) { -aio_bh_schedule_oneshot (thr->bh_ctx, thr->bh_func, thr->bh_opaque); - -/* play safe, don't reuse bh_ctx on further connection attempts */ - thr->bh_ctx =NULL ; +if (thr->wait_co) { +aio_co_schedule (NULL , thr->wait_co); + thr->wait_co =NULL ; }break ;case CONNECT_THREAD_RUNNING_DETACHED :
@@ -487,20 +467,14 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)abort (); } - thr->bh_ctx =qemu_get_current_aio_context (); + thr->wait_co =qemu_coroutine_self ();qemu_mutex_unlock (&thr->mutex); -/* * We are going to wait for connect-thread finish, but * nbd_client_co_drain_begin() can interrupt. -* -* Note that wait_connect variable is not visible for connect-thread. It -* doesn't need mutex protection, it used only inside home aio context of -* bs. */ - s->wait_connect = true;qemu_coroutine_yield ();qemu_mutex_lock (&thr->mutex);
@@ -555,24 +529,19 @@ static void nbd_co_establish_connection_cancel(BlockDriverState *bs) {BDRVNBDState *s = bs->opaque;NBDConnectThread *thr = s->connect_thread; -bool wake = false;qemu_mutex_lock (&thr->mutex);if (thr->state ==CONNECT_THREAD_RUNNING ) {/* We can cancel only in running state, when bh is not yet scheduled */ - thr->bh_ctx =NULL ; -if (s->wait_connect) { - s->wait_connect = false; - wake = true; +if (thr->wait_co) { +aio_co_schedule (NULL , thr->wait_co); + thr->wait_co =NULL ; } }qemu_mutex_unlock (&thr->mutex); -if (wake) { -aio_co_wake (s->connection_co); - } } static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s)
Signature · 1 line
2.29.2
Re: [PATCH v3 07/33] block/nbd: simplify waking of nbd_co_establish_connection()
Quoted 130 lines · Vladimir Sementsov-Ogievskiy
Instead of
connect_bh,bh_ctxandwait_connectfields we can live with only one link to waiting coroutine, protected by mutex.So new logic is:
nbd_co_establish_connection()setswait_counder mutex, release the mutex and doyield(). Note, thatwait_comay be scheduled by thread immediately after unlocking the mutex. Still, in main thread (or iothread) we'll not reach the code for entering the coroutine until theyield()so we are safe.Both
connect_thread_func()andnbd_co_establish_connection_cancel()do the following to handlewait_co:Under mutex, if
thr->wait_cois not NULL, callaio_co_wake()(which never tries to acquire aio context since previous commit, so we are safe to do it underthr->mutex) and setthr->wait_coto NULL. This way we protect ourselves of scheduling it twice.Also this commit make
nbd_co_establish_connection()less connected to bs (we have generic pointer to the coroutine, not uses->connection_codirectly). So, we are on the way of splitting connection API out ofnbd.c(which is overcomplicated now).Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 49 +++++++++---------------------------------------- 1 file changed, 9 insertions(+), 40 deletions(-)block/nbd.c +6
-32@@ -87,12 +87,6 @@ typedef enum NBDConnectThreadState { typedef struct NBDConnectThread { /* Initialization constants */ SocketAddress *saddr; /* address to connect to */ - /* - * Bottom half to schedule on completion. Scheduled only if bh_ctx is not - * NULL - */ - QEMUBHFunc *bh_func; - void *bh_opaque; /* * Result of last attempt. Valid in FAIL and SUCCESS states.@@ -101,10 +95,10 @@ typedef struct NBDConnectThread {QIOChannelSocket *sioc;Error *err; -/* state and bh_ctx are protected by mutex */ QemuMutex mutex; +/* All further fields are protected by mutex */ NBDConnectThreadState state;/* current state of the thread */ -AioContext *bh_ctx;/* where to schedule bh (NULL means don't schedule) */ +Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ } NBDConnectThread; typedef struct BDRVNBDState {@@ -138,7 +132,6 @@ typedef struct BDRVNBDState {char *x_dirty_bitmap;bool alloc_depth; -bool wait_connect;NBDConnectThread *connect_thread; } BDRVNBDState;@@ -374,15 +367,6 @@ static bool nbd_client_connecting_wait(BDRVNBDState *s)return qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ; } -static void connect_bh (void *opaque) -{ -BDRVNBDState *state = opaque; - -assert (state->wait_connect); - state->wait_connect = false; -aio_co_wake (state->connection_co); -} -static void nbd_init_connect_thread (BDRVNBDState *s) { s->connect_thread =g_new (NBDConnectThread,1 );@@ -390,8 +374,6 @@ static void nbd_init_connect_thread(BDRVNBDState *s) *s->connect_thread = (NBDConnectThread ) { .saddr =QAPI_CLONE (SocketAddress, s->saddr), .state =CONNECT_THREAD_NONE , - .bh_func = connect_bh, - .bh_opaque = s, };qemu_mutex_init (&s->connect_thread->mutex);@@ -429,11 +411,9 @@ static void *connect_thread_func(void *opaque)switch (thr->state) {case CONNECT_THREAD_RUNNING : thr->state = ret <0 ?CONNECT_THREAD_FAIL :CONNECT_THREAD_SUCCESS ; -if (thr->bh_ctx) { -aio_bh_schedule_oneshot (thr->bh_ctx, thr->bh_func, thr->bh_opaque); - -/* play safe, don't reuse bh_ctx on further connection attempts */ - thr->bh_ctx =NULL ; +if (thr->wait_co) { +aio_co_schedule (NULL , thr->wait_co); + thr->wait_co =NULL ; }break ;case CONNECT_THREAD_RUNNING_DETACHED :@@ -487,20 +467,14 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)abort (); } - thr->bh_ctx =qemu_get_current_aio_context (); + thr->wait_co =qemu_coroutine_self ();qemu_mutex_unlock (&thr->mutex); -/* * We are going to wait for connect-thread finish, but * nbd_client_co_drain_begin() can interrupt. -* -* Note that wait_connect variable is not visible for connect-thread. It -* doesn't need mutex protection, it used only inside home aio context of -* bs. */ - s->wait_connect = true;qemu_coroutine_yield ();qemu_mutex_lock (&thr->mutex);
block/nbd.c static void nbd_co_establish_connection_cancel(BlockDriverState *bs) · line 535
Quoted 8 lines
block/nbd.c +0
-1@@ -555,24 +529,19 @@ static void nbd_co_establish_connection_cancel(BlockDriverState *bs) {BDRVNBDState *s = bs->opaque;NBDConnectThread *thr = s->connect_thread; -bool wake = false;qemu_mutex_lock (&thr->mutex);if (thr->state ==CONNECT_THREAD_RUNNING ) {
This check looks redundant and can probably go. This patch may be quite appropriate for this: the logic becomes even more straightforward.
block/nbd.c static void nbd_co_establish_connection_cancel(BlockDriverState *bs) · line 538
Quoted 7 lines
block/nbd.c +2
-4/* We can cancel only in running state, when bh is not yet scheduled */ - thr->bh_ctx =NULL ; -if (s->wait_connect) { - s->wait_connect = false; - wake = true; +if (thr->wait_co) { +aio_co_schedule (NULL , thr->wait_co);
This will probably be replaced by a new function per our discussion of the previous patch.
Note however that if that one doesn't fly for whatever reason you can retain the aio_context on NBDConnectThread and pass it explicitly into aio_co_schedule here.
Roman.
Quoted 12 lines
block/nbd.c +1
-3+ thr->wait_co =NULL ; } }qemu_mutex_unlock (&thr->mutex); -if (wake) { -aio_co_wake (s->connection_co); - } } static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s)
Re: [PATCH v3 07/33] block/nbd: simplify waking of nbd_co_establish_connection()
In reply to Vladimir Sementsov-OgievskiyOn Fri, Apr 16, 2021 at 11:08:45AM +0300, Vladimir Sementsov-Ogievskiy wrote:
Grammar suggestions:
Quoted 2 lines
Instead of
connect_bh,bh_ctxandwait_connectfields we can live with only one link to waiting coroutine, protected by mutex.
Instead of managing connect_bh, bh_ctx, and wait_connect fields, we can use a single link to the waiting coroutine with proper mutex protection.
Quoted 8 lines
So new logic is:
nbd_co_establish_connection()setswait_counder mutex, release the mutex and doyield(). Note, thatwait_comay be scheduled by thread immediately after unlocking the mutex. Still, in main thread (or iothread) we'll not reach the code for entering the coroutine until theyield()so we are safe.
nbd_co_establish_connection() sets wait_co under the mutex, releases the mutex, then yield()s. Note that wait_co may be scheduled by the thread immediately after unlocking the mutex. Still, the main thread (or iothread) will not reach the code for entering the coroutine until the yield(), so we are safe.
Quoted 8 lines
Both
connect_thread_func()andnbd_co_establish_connection_cancel()do the following to handlewait_co:Under mutex, if
thr->wait_cois not NULL, callaio_co_wake()(which never tries to acquire aio context since previous commit, so we are safe to do it underthr->mutex) and setthr->wait_coto NULL. This way we protect ourselves of scheduling it twice.
Under the mutex, if thr_wait_co is not NULL, call aio_co_wake() (the previous commit ensures it never tries to acquire the aio context, so we are safe even while under thr->mutex), then sets thr->wait_co to NULL. This way, we avoid scheduling the coroutine twice.
Quoted 5 lines
Also this commit make
nbd_co_establish_connection()less connected to bs (we have generic pointer to the coroutine, not uses->connection_codirectly). So, we are on the way of splitting connection API out ofnbd.c(which is overcomplicated now).
Also, this commit reduces the dependence of nbd_co_establish_connection() on the internals of bs (we now use a generic pointer to the coroutine, instead of direct use of s->connection_co). This is a step towards splitting the connection API out of nbd.c.
Quoted 10 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 49 +++++++++---------------------------------------- 1 file changed, 9 insertions(+), 40 deletions(-)+++
b/block/nbd.c
block/nbd.c typedef struct NBDConnectThread { · line 101
Quoted 9 lines
block/nbd.c +2
-2@@ -101,10 +95,10 @@ typedef struct NBDConnectThread {QIOChannelSocket *sioc;Error *err; -/* state and bh_ctx are protected by mutex */ QemuMutex mutex; +/* All further fields are protected by mutex */ NBDConnectThreadState state;/* current state of the thread */ -AioContext *bh_ctx;/* where to schedule bh (NULL means don't schedule) */ +Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */
I'm not sure if that comment is the most legible, but I'm not coming up with an alternative. Maybe:
/*
* if non-NULL, which coroutine to wake in
* nbd_co_establish_connection() after yield()
*/
But the simplification looks nice, and I didn't spot any obvious problems with the refactoring.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 8/33 block/nbd: drop thr->state
In reply to Vladimir Sementsov-OgievskiyWe don't need all these states. The code refactored to use two boolean variables looks simpler.
--- block/nbd.c | 125 ++++++++++++++-------------------------------------- 1 file changed, 34 insertions(+), 91 deletions(-)
block/nbd.c +34 -91
@@ -66,24 +66,6 @@ typedef enum NBDClientState {NBD_CLIENT_QUIT } NBDClientState; -typedef enum NBDConnectThreadState { -/* No thread, no pending results */ -CONNECT_THREAD_NONE , - -/* Thread is running, no results for now */ -CONNECT_THREAD_RUNNING , - -/* -* Thread is running, but requestor exited. Thread should close -* the new socket and free the connect state on exit. -*/ -CONNECT_THREAD_RUNNING_DETACHED , - -/* Thread finished, results are stored in a state */ -CONNECT_THREAD_FAIL , -CONNECT_THREAD_SUCCESS -}NBDConnectThreadState ; - typedef struct NBDConnectThread { /* Initialization constants */ SocketAddress *saddr; /* address to connect to */
@@ -97,7 +79,8 @@ typedef struct NBDConnectThread {QemuMutex mutex;/* All further fields are protected by mutex */ -NBDConnectThreadState state;/* current state of the thread */ +bool running;/* thread is running now */ +bool detached;/* thread is detached and should cleanup the state */ Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ } NBDConnectThread;
@@ -147,17 +130,17 @@ static void nbd_clear_bdrvstate(BlockDriverState *bs) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque;NBDConnectThread *thr = s->connect_thread; -bool thr_running; +bool do_free;qemu_mutex_lock (&thr->mutex); - thr_running = thr->state ==CONNECT_THREAD_RUNNING ; -if (thr_running) { - thr->state =CONNECT_THREAD_RUNNING_DETACHED ; +if (thr->running) { + thr->detached = true; } + do_free = !thr->running && !thr->detached;qemu_mutex_unlock (&thr->mutex);/* the runaway thread will clean it up itself */ -if (!thr_running) { +if (do_free) {nbd_free_connect_thread (thr); }
@@ -373,7 +356,6 @@ static void nbd_init_connect_thread(BDRVNBDState *s) *s->connect_thread = (NBDConnectThread ) { .saddr =QAPI_CLONE (SocketAddress, s->saddr), - .state =CONNECT_THREAD_NONE , };qemu_mutex_init (&s->connect_thread->mutex);
@@ -408,20 +390,13 @@ static void *connect_thread_func(void *opaque)qemu_mutex_lock (&thr->mutex); -switch (thr->state) { -case CONNECT_THREAD_RUNNING : - thr->state = ret <0 ?CONNECT_THREAD_FAIL :CONNECT_THREAD_SUCCESS ; -if (thr->wait_co) { -aio_co_schedule (NULL , thr->wait_co); - thr->wait_co =NULL ; - } -break ; -case CONNECT_THREAD_RUNNING_DETACHED : - do_free = true; -break ; -default : -abort (); +assert (thr->running); + thr->running = false; +if (thr->wait_co) { +aio_co_schedule (NULL , thr->wait_co); + thr->wait_co =NULL ; } + do_free = thr->detached;qemu_mutex_unlock (&thr->mutex);
@@ -435,36 +410,24 @@ static void *connect_thread_func(void *opaque) static int coroutine_fnnbd_co_establish_connection (BlockDriverState *bs,Error **errp) { -int ret;QemuThread thread;BDRVNBDState *s = bs->opaque;NBDConnectThread *thr = s->connect_thread; +assert (!s->sioc); +qemu_mutex_lock (&thr->mutex); -switch (thr->state) { -case CONNECT_THREAD_FAIL : -case CONNECT_THREAD_NONE : +if (!thr->running) { +if (thr->sioc) { +/* Previous attempt finally succeeded in background */ + goto out; + } + thr->running = true;error_free (thr->err); thr->err =NULL ; - thr->state =CONNECT_THREAD_RUNNING ;qemu_thread_create (&thread,"nbd-connect" , connect_thread_func, thr,QEMU_THREAD_DETACHED ); -break ; -case CONNECT_THREAD_SUCCESS : -/* Previous attempt finally succeeded in background */ - thr->state =CONNECT_THREAD_NONE ; - s->sioc = thr->sioc; - thr->sioc =NULL ; -yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), - nbd_yank, bs); -qemu_mutex_unlock (&thr->mutex); -return 0 ; -case CONNECT_THREAD_RUNNING : -/* Already running, will wait */ -break ; -default : -abort (); } thr->wait_co =qemu_coroutine_self ();
@@ -479,10 +442,15 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)qemu_mutex_lock (&thr->mutex); -switch (thr->state) { -case CONNECT_THREAD_SUCCESS : -case CONNECT_THREAD_FAIL : - thr->state =CONNECT_THREAD_NONE ; +out: +if (thr->running) { +/* +* Obviously, drained section wants to start. Report the attempt as +* failed. Still connect thread is executing in background, and its +* result may be used for next connection attempt. +*/ +error_setg (errp,"Connection attempt cancelled by other operation" ); + }else {error_propagate (errp, thr->err); thr->err =NULL ; s->sioc = thr->sioc;
@@ -491,33 +459,11 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), nbd_yank, bs); } - ret = (s->sioc ?0 :-1 ); -break ; -case CONNECT_THREAD_RUNNING : -case CONNECT_THREAD_RUNNING_DETACHED : -/* -* Obviously, drained section wants to start. Report the attempt as -* failed. Still connect thread is executing in background, and its -* result may be used for next connection attempt. -*/ - ret =-1 ; -error_setg (errp,"Connection attempt cancelled by other operation" ); -break ; - -case CONNECT_THREAD_NONE : -/* -* Impossible. We've seen this thread running. So it should be -* running or at least give some results. -*/ -abort (); - -default : -abort (); }qemu_mutex_unlock (&thr->mutex); -return ret; +return s->sioc ?0 :-1 ; }/*
@@ -532,12 +478,9 @@ static void nbd_co_establish_connection_cancel(BlockDriverState *bs)qemu_mutex_lock (&thr->mutex); -if (thr->state ==CONNECT_THREAD_RUNNING ) { -/* We can cancel only in running state, when bh is not yet scheduled */ -if (thr->wait_co) { -aio_co_schedule (NULL , thr->wait_co); - thr->wait_co =NULL ; - } +if (thr->wait_co) { +aio_co_schedule (NULL , thr->wait_co); + thr->wait_co =NULL ; }qemu_mutex_unlock (&thr->mutex);
Signature · 1 line
2.29.2
Re: [PATCH v3 08/33] block/nbd: drop thr->state
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
We don't need all these states. The code refactored to use two boolean variables looks simpler.
Indeed.
Quoted 50 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 125 ++++++++++++++-------------------------------------- 1 file changed, 34 insertions(+), 91 deletions(-)block/nbd.c +2
-20@@ -66,24 +66,6 @@ typedef enum NBDClientState {NBD_CLIENT_QUIT } NBDClientState; -typedef enum NBDConnectThreadState { -/* No thread, no pending results */ -CONNECT_THREAD_NONE , - -/* Thread is running, no results for now */ -CONNECT_THREAD_RUNNING , - -/* -* Thread is running, but requestor exited. Thread should close -* the new socket and free the connect state on exit. -*/ -CONNECT_THREAD_RUNNING_DETACHED , - -/* Thread finished, results are stored in a state */ -CONNECT_THREAD_FAIL , -CONNECT_THREAD_SUCCESS -}NBDConnectThreadState ; - typedef struct NBDConnectThread { /* Initialization constants */ SocketAddress *saddr; /* address to connect to */@@ -97,7 +79,8 @@ typedef struct NBDConnectThread {QemuMutex mutex;/* All further fields are protected by mutex */ -NBDConnectThreadState state;/* current state of the thread */ +bool running;/* thread is running now */ +bool detached;/* thread is detached and should cleanup the state */ Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ } NBDConnectThread;@@ -147,17 +130,17 @@ static void nbd_clear_bdrvstate(BlockDriverState *bs) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque;NBDConnectThread *thr = s->connect_thread; -bool thr_running;
block/nbd.c static void nbd_clear_bdrvstate(BlockDriverState *bs) · line 139
Quoted 10 lines
block/nbd.c +4
-3+bool do_free;qemu_mutex_lock (&thr->mutex); - thr_running = thr->state ==CONNECT_THREAD_RUNNING ; -if (thr_running) { - thr->state =CONNECT_THREAD_RUNNING_DETACHED ; +if (thr->running) { + thr->detached = true; } + do_free = !thr->running && !thr->detached;
This is redundant. You can unconditionally set ->detached and only depend on ->running for the rest of this function.
Quoted 93 lines
block/nbd.c +15
-36qemu_mutex_unlock (&thr->mutex);/* the runaway thread will clean it up itself */ -if (!thr_running) { +if (do_free) {nbd_free_connect_thread (thr); }@@ -373,7 +356,6 @@ static void nbd_init_connect_thread(BDRVNBDState *s) *s->connect_thread = (NBDConnectThread ) { .saddr =QAPI_CLONE (SocketAddress, s->saddr), - .state =CONNECT_THREAD_NONE , };qemu_mutex_init (&s->connect_thread->mutex);@@ -408,20 +390,13 @@ static void *connect_thread_func(void *opaque)qemu_mutex_lock (&thr->mutex); -switch (thr->state) { -case CONNECT_THREAD_RUNNING : - thr->state = ret <0 ?CONNECT_THREAD_FAIL :CONNECT_THREAD_SUCCESS ; -if (thr->wait_co) { -aio_co_schedule (NULL , thr->wait_co); - thr->wait_co =NULL ; - } -break ; -case CONNECT_THREAD_RUNNING_DETACHED : - do_free = true; -break ; -default : -abort (); +assert (thr->running); + thr->running = false; +if (thr->wait_co) { +aio_co_schedule (NULL , thr->wait_co); + thr->wait_co =NULL ; } + do_free = thr->detached;qemu_mutex_unlock (&thr->mutex);@@ -435,36 +410,24 @@ static void *connect_thread_func(void *opaque) static int coroutine_fnnbd_co_establish_connection (BlockDriverState *bs,Error **errp) { -int ret;QemuThread thread;BDRVNBDState *s = bs->opaque;NBDConnectThread *thr = s->connect_thread; +assert (!s->sioc); +qemu_mutex_lock (&thr->mutex); -switch (thr->state) { -case CONNECT_THREAD_FAIL : -case CONNECT_THREAD_NONE : +if (!thr->running) { +if (thr->sioc) { +/* Previous attempt finally succeeded in background */ + goto out; + } + thr->running = true;error_free (thr->err); thr->err =NULL ; - thr->state =CONNECT_THREAD_RUNNING ;qemu_thread_create (&thread,"nbd-connect" , connect_thread_func, thr,QEMU_THREAD_DETACHED ); -break ; -case CONNECT_THREAD_SUCCESS : -/* Previous attempt finally succeeded in background */ - thr->state =CONNECT_THREAD_NONE ; - s->sioc = thr->sioc; - thr->sioc =NULL ; -yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), - nbd_yank, bs); -qemu_mutex_unlock (&thr->mutex); -return 0 ; -case CONNECT_THREAD_RUNNING : -/* Already running, will wait */ -break ; -default : -abort (); } thr->wait_co =qemu_coroutine_self ();@@ -479,10 +442,15 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)qemu_mutex_lock (&thr->mutex); -switch (thr->state) {
block/nbd.c nbd_co_establish_connection(BlockDriverState *bs, Error **errp) · line 451
Quoted 10 lines
block/nbd.c +7
-3-case CONNECT_THREAD_SUCCESS : -case CONNECT_THREAD_FAIL : - thr->state =CONNECT_THREAD_NONE ; +out: +if (thr->running) { +/* +* Obviously, drained section wants to start. Report the attempt as +* failed. Still connect thread is executing in background, and its +* result may be used for next connection attempt. +*/
I must admit this wasn't quite obvious to me when I started looking at this code. While you're moving this comment, can you please consider rephrasing it? How about this:
/*
* The connection attempt was canceled and the coroutine resumed
* before the connection thread finished its job. Report the
* attempt as failed, but leave the connection thread running,
* to reuse it for the next connection attempt.
*/
Quoted 43 lines
block/nbd.c +3
-23+error_setg (errp,"Connection attempt cancelled by other operation" ); + }else {error_propagate (errp, thr->err); thr->err =NULL ; s->sioc = thr->sioc;@@ -491,33 +459,11 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), nbd_yank, bs); } - ret = (s->sioc ?0 :-1 ); -break ; -case CONNECT_THREAD_RUNNING : -case CONNECT_THREAD_RUNNING_DETACHED : -/* -* Obviously, drained section wants to start. Report the attempt as -* failed. Still connect thread is executing in background, and its -* result may be used for next connection attempt. -*/ - ret =-1 ; -error_setg (errp,"Connection attempt cancelled by other operation" ); -break ; - -case CONNECT_THREAD_NONE : -/* -* Impossible. We've seen this thread running. So it should be -* running or at least give some results. -*/ -abort (); - -default : -abort (); }qemu_mutex_unlock (&thr->mutex); -return ret; +return s->sioc ?0 :-1 ; }/* @@ -532,12 +478,9 @@ static void nbd_co_establish_connection_cancel(BlockDriverState *bs)qemu_mutex_lock (&thr->mutex);
block/nbd.c static void nbd_co_establish_connection_cancel(BlockDriverState *bs) · line 483
Quoted 10 lines
block/nbd.c +3
-6-if (thr->state ==CONNECT_THREAD_RUNNING ) { -/* We can cancel only in running state, when bh is not yet scheduled */ -if (thr->wait_co) { -aio_co_schedule (NULL , thr->wait_co); - thr->wait_co =NULL ; - } +if (thr->wait_co) { +aio_co_schedule (NULL , thr->wait_co); + thr->wait_co =NULL ;
Ah, so you get rid of this redundant check in this patch. Fine by me; please disregard my comment on this matter to the previous patch, then.
Roman.
Quoted 6 lines
block/nbd.c +0
-0}qemu_mutex_unlock (&thr->mutex);Signature · 1 line
2.29.2
Re: [PATCH v3 08/33] block/nbd: drop thr->state
Quoted 58 lines · Roman Kagan
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
We don't need all these states. The code refactored to use two boolean variables looks simpler.
Indeed.
Quoted 52 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 125 ++++++++++++++-------------------------------------- 1 file changed, 34 insertions(+), 91 deletions(-)block/nbd.c +3
-20@@ -66,24 +66,6 @@ typedef enum NBDClientState {NBD_CLIENT_QUIT } NBDClientState; -typedef enum NBDConnectThreadState { -/* No thread, no pending results */ -CONNECT_THREAD_NONE , - -/* Thread is running, no results for now */ -CONNECT_THREAD_RUNNING , - -/* -* Thread is running, but requestor exited. Thread should close -* the new socket and free the connect state on exit. -*/ -CONNECT_THREAD_RUNNING_DETACHED , - -/* Thread finished, results are stored in a state */ -CONNECT_THREAD_FAIL , -CONNECT_THREAD_SUCCESS -}NBDConnectThreadState ; - typedef struct NBDConnectThread { /* Initialization constants */ SocketAddress *saddr; /* address to connect to */@@ -97,7 +79,8 @@ typedef struct NBDConnectThread {QemuMutex mutex;/* All further fields are protected by mutex */ -NBDConnectThreadState state;/* current state of the thread */ +bool running;/* thread is running now */ +bool detached;/* thread is detached and should cleanup the state */ Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ } NBDConnectThread;@@ -147,17 +130,17 @@ static void nbd_clear_bdrvstate(BlockDriverState *bs) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque;NBDConnectThread *thr = s->connect_thread; -bool thr_running; +bool do_free;
Quoted 11 lines
block/nbd.c static void nbd_clear_bdrvstate(BlockDriverState *bs) · line 139
Quoted 8 lines
block/nbd.c +3
-3qemu_mutex_lock (&thr->mutex); - thr_running = thr->state ==CONNECT_THREAD_RUNNING ; -if (thr_running) { - thr->state =CONNECT_THREAD_RUNNING_DETACHED ; +if (thr->running) { + thr->detached = true; } + do_free = !thr->running && !thr->detached;This is redundant. You can unconditionally set ->detached and only depend on ->running for the rest of this function.
Still, I don't want to set ->detached unconditionally, just to not break its meaning (even before freeing). And that fact that detached is set only once worth assertion. So, I think:
assert(!thr->detached);
if (thr->running) {
thr->detached = true;
} else {
do_free = true;
}
Quoted 109 lines
Quoted 94 lines
block/nbd.c +15
-36qemu_mutex_unlock (&thr->mutex);/* the runaway thread will clean it up itself */ -if (!thr_running) { +if (do_free) {nbd_free_connect_thread (thr); }@@ -373,7 +356,6 @@ static void nbd_init_connect_thread(BDRVNBDState *s) *s->connect_thread = (NBDConnectThread ) { .saddr =QAPI_CLONE (SocketAddress, s->saddr), - .state =CONNECT_THREAD_NONE , };qemu_mutex_init (&s->connect_thread->mutex);@@ -408,20 +390,13 @@ static void *connect_thread_func(void *opaque)qemu_mutex_lock (&thr->mutex); -switch (thr->state) { -case CONNECT_THREAD_RUNNING : - thr->state = ret <0 ?CONNECT_THREAD_FAIL :CONNECT_THREAD_SUCCESS ; -if (thr->wait_co) { -aio_co_schedule (NULL , thr->wait_co); - thr->wait_co =NULL ; - } -break ; -case CONNECT_THREAD_RUNNING_DETACHED : - do_free = true; -break ; -default : -abort (); +assert (thr->running); + thr->running = false; +if (thr->wait_co) { +aio_co_schedule (NULL , thr->wait_co); + thr->wait_co =NULL ; } + do_free = thr->detached;qemu_mutex_unlock (&thr->mutex);@@ -435,36 +410,24 @@ static void *connect_thread_func(void *opaque) static int coroutine_fnnbd_co_establish_connection (BlockDriverState *bs,Error **errp) { -int ret;QemuThread thread;BDRVNBDState *s = bs->opaque;NBDConnectThread *thr = s->connect_thread; +assert (!s->sioc); +qemu_mutex_lock (&thr->mutex); -switch (thr->state) { -case CONNECT_THREAD_FAIL : -case CONNECT_THREAD_NONE : +if (!thr->running) { +if (thr->sioc) { +/* Previous attempt finally succeeded in background */ + goto out; + } + thr->running = true;error_free (thr->err); thr->err =NULL ; - thr->state =CONNECT_THREAD_RUNNING ;qemu_thread_create (&thread,"nbd-connect" , connect_thread_func, thr,QEMU_THREAD_DETACHED ); -break ; -case CONNECT_THREAD_SUCCESS : -/* Previous attempt finally succeeded in background */ - thr->state =CONNECT_THREAD_NONE ; - s->sioc = thr->sioc; - thr->sioc =NULL ; -yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), - nbd_yank, bs); -qemu_mutex_unlock (&thr->mutex); -return 0 ; -case CONNECT_THREAD_RUNNING : -/* Already running, will wait */ -break ; -default : -abort (); } thr->wait_co =qemu_coroutine_self ();@@ -479,10 +442,15 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)qemu_mutex_lock (&thr->mutex); -switch (thr->state) {block/nbd.c nbd_co_establish_connection(BlockDriverState *bs, Error **errp) · line 451
Quoted 10 lines
block/nbd.c +7
-3-case CONNECT_THREAD_SUCCESS : -case CONNECT_THREAD_FAIL : - thr->state =CONNECT_THREAD_NONE ; +out: +if (thr->running) { +/* +* Obviously, drained section wants to start. Report the attempt as +* failed. Still connect thread is executing in background, and its +* result may be used for next connection attempt. +*/ I must admit this wasn't quite obvious to me when I started looking at this code. While you're moving this comment, can you please consider rephrasing it? How about this:
Quoted 6 lines
/*
* The connection attempt was canceled and the coroutine resumed
* before the connection thread finished its job. Report the
* attempt as failed, but leave the connection thread running,
* to reuse it for the next connection attempt.
*/
Yes agree. Moreover, when code is moved to separate file, it shouldn't care _why_ it is cancelled, and comment should not mention drained sections. It's cancelled because user cancel it.
Quoted 65 lines
Quoted 44 lines
block/nbd.c +3
-23+error_setg (errp,"Connection attempt cancelled by other operation" ); + }else {error_propagate (errp, thr->err); thr->err =NULL ; s->sioc = thr->sioc;@@ -491,33 +459,11 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), nbd_yank, bs); } - ret = (s->sioc ?0 :-1 ); -break ; -case CONNECT_THREAD_RUNNING : -case CONNECT_THREAD_RUNNING_DETACHED : -/* -* Obviously, drained section wants to start. Report the attempt as -* failed. Still connect thread is executing in background, and its -* result may be used for next connection attempt. -*/ - ret =-1 ; -error_setg (errp,"Connection attempt cancelled by other operation" ); -break ; - -case CONNECT_THREAD_NONE : -/* -* Impossible. We've seen this thread running. So it should be -* running or at least give some results. -*/ -abort (); - -default : -abort (); }qemu_mutex_unlock (&thr->mutex); -return ret; +return s->sioc ?0 :-1 ; }/* @@ -532,12 +478,9 @@ static void nbd_co_establish_connection_cancel(BlockDriverState *bs)qemu_mutex_lock (&thr->mutex);block/nbd.c static void nbd_co_establish_connection_cancel(BlockDriverState *bs) · line 483
Quoted 10 lines
block/nbd.c +3
-6-if (thr->state ==CONNECT_THREAD_RUNNING ) { -/* We can cancel only in running state, when bh is not yet scheduled */ -if (thr->wait_co) { -aio_co_schedule (NULL , thr->wait_co); - thr->wait_co =NULL ; - } +if (thr->wait_co) { +aio_co_schedule (NULL , thr->wait_co); + thr->wait_co =NULL ;Ah, so you get rid of this redundant check in this patch. Fine by me; please disregard my comment on this matter to the previous patch, then.
Roman.
Quoted 6 lines
block/nbd.c +0
-0}qemu_mutex_unlock (&thr->mutex);Signature · 1 line
2.29.2
Signature · 2 lines
Best regards, Vladimir
PATCH v3 9/33 block/nbd: bs-independent interface for nbd_co_establish_connection()
In reply to Vladimir Sementsov-OgievskiyWe are going to split connection code to separate file. Now we are ready to give nbd_co_establish_connection() clean and bs-independent interface.
--- block/nbd.c | 49 +++++++++++++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 18 deletions(-)
block/nbd.c +31 -18
@@ -121,7 +121,8 @@ typedef struct BDRVNBDState {static void nbd_free_connect_thread (NBDConnectThread *thr);static int nbd_establish_connection (BlockDriverState *bs,SocketAddress *saddr,Error **errp); -static int nbd_co_establish_connection (BlockDriverState *bs,Error **errp); +static coroutine_fn QIOChannelSocket * +nbd_co_establish_connection (NBDConnectThread *thr, Error **errp);static void nbd_co_establish_connection_cancel (BlockDriverState *bs);static int nbd_client_handshake (BlockDriverState *bs,Error **errp);static void nbd_yank (void *opaque);
@@ -407,22 +408,36 @@ static void *connect_thread_func(void *opaque)return NULL ; } -static int coroutine_fn -nbd_co_establish_connection (BlockDriverState *bs,Error **errp) +/* +* Get a new connection in context of @thr: +* if thread is running, wait for completion +* if thread is already succeeded in background, and user didn't get the +* result, just return it now +* otherwise if thread is not running, start a thread and wait for completion +*/ +static coroutine_fn QIOChannelSocket * +nbd_co_establish_connection (NBDConnectThread *thr,Error **errp) { +QIOChannelSocket *sioc =NULL ;QemuThread thread; -BDRVNBDState *s = bs->opaque; -NBDConnectThread *thr = s->connect_thread; - -assert (!s->sioc);qemu_mutex_lock (&thr->mutex); +/* +* Don't call nbd_co_establish_connection() in several coroutines in +* parallel. Only one call at once is supported. +*/ +assert (!thr->wait_co); +if (!thr->running) {if (thr->sioc) {/* Previous attempt finally succeeded in background */ - goto out; + sioc =g_steal_pointer (&thr->sioc); +qemu_mutex_unlock (&thr->mutex); + +return sioc; } + thr->running = true;error_free (thr->err); thr->err =NULL ;
@@ -436,13 +451,12 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)/* * We are going to wait for connect-thread finish, but -* nbd_client_co_drain_begin() can interrupt. +* nbd_co_establish_connection_cancel() can interrupt. */ qemu_coroutine_yield ();qemu_mutex_lock (&thr->mutex); -out:if (thr->running) {/* * Obviously, drained section wants to start. Report the attempt as
@@ -453,17 +467,12 @@ out: }else {error_propagate (errp, thr->err); thr->err =NULL ; - s->sioc = thr->sioc; - thr->sioc =NULL ; -if (s->sioc) { -yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), - nbd_yank, bs); - } + sioc =g_steal_pointer (&thr->sioc); }qemu_mutex_unlock (&thr->mutex); -return s->sioc ?0 :-1 ; +return sioc; }/*
@@ -530,11 +539,15 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) s->ioc =NULL ; } -if (nbd_co_establish_connection (s->bs,NULL ) <0 ) { + s->sioc =nbd_co_establish_connection (s->connect_thread,NULL ); +if (!s->sioc) { ret = -ECONNREFUSED ; goto out; } +yank_register_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), nbd_yank, + s->bs); +bdrv_dec_in_flight (s->bs); ret =nbd_client_handshake (s->bs,NULL );
Signature · 1 line
2.29.2
Re: [PATCH v3 09/33] block/nbd: bs-independent interface for nbd_co_establish_connection()
Quoted 1 line · Vladimir Sementsov-Ogievskiy
We are going to split connection code to separate file. Now we are
to a separate
Quoted 10 lines
ready to give
nbd_co_establish_connection()clean and bs-independent interface.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Roman Kagan <[email protected]>--- block/nbd.c | 49 +++++++++++++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 18 deletions(-)
block/nbd.c static void *connect_thread_func(void *opaque) · line 413
Quoted 5 lines
block/nbd.c +3
-2-static int coroutine_fn -nbd_co_establish_connection (BlockDriverState *bs, Error **errp) +/* +* Get a new connection in context of @thr: +* if thread is running, wait for completion
if the thread is running,...
Quoted 1 line
+ * if thread is already succeeded in background, and user didn't get the
if the thread already succeeded in the background,...
Quoted 2 lines
+ * result, just return it now
+ * otherwise if thread is not running, start a thread and wait for completion
otherwise, the thread is not running, so start...
Quoted 15 lines
+ */
block/nbd.c +5
-4+static coroutine_fn QIOChannelSocket * +nbd_co_establish_connection (NBDConnectThread *thr,Error **errp) { +QIOChannelSocket *sioc =NULL ;QemuThread thread; -BDRVNBDState *s = bs->opaque; -NBDConnectThread *thr = s->connect_thread; - -assert (!s->sioc);qemu_mutex_lock (&thr->mutex); +/* +* Don't call nbd_co_establish_connection() in several coroutines in
block/nbd.c static void *connect_thread_func(void *opaque) · line 436
Quoted 10 lines
block/nbd.c +6
-1+* parallel. Only one call at once is supported. +*/ +assert (!thr->wait_co); +if (!thr->running) {if (thr->sioc) {/* Previous attempt finally succeeded in background */ - goto out; + sioc =g_steal_pointer (&thr->sioc); +qemu_mutex_unlock (&thr->mutex);
Worth using QEMU_LOCK_GUARD() here?
block/nbd.c static void *connect_thread_func(void *opaque) · line 443
Quoted 7 lines
block/nbd.c +3
-0+ +return sioc; } + thr->running = true;error_free (thr->err); thr->err =NULL ;
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 09/33] block/nbd: bs-independent interface for nbd_co_establish_connection()
Quoted 45 lines · Eric Blake
Quoted 1 line · Vladimir Sementsov-Ogievskiy
We are going to split connection code to separate file. Now we are
to a separate
Quoted 10 lines
ready to give
nbd_co_establish_connection()clean and bs-independent interface.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Roman Kagan <[email protected]>--- block/nbd.c | 49 +++++++++++++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 18 deletions(-)block/nbd.c static void *connect_thread_func(void *opaque) · line 413
Quoted 5 lines
block/nbd.c +3
-2-static int coroutine_fn -nbd_co_establish_connection (BlockDriverState *bs, Error **errp) +/* +* Get a new connection in context of @thr: +* if thread is running, wait for completion if the thread is running,...
Quoted 1 line
+ * if thread is already succeeded in background, and user didn't get the
if the thread already succeeded in the background,...
Quoted 2 lines
+ * result, just return it now
+ * otherwise if thread is not running, start a thread and wait for completionotherwise, the thread is not running, so start...
Quoted 13 lines
+ */
block/nbd.c +3
-4+static coroutine_fn QIOChannelSocket * +nbd_co_establish_connection (NBDConnectThread *thr,Error **errp) { +QIOChannelSocket *sioc =NULL ;QemuThread thread; -BDRVNBDState *s = bs->opaque; -NBDConnectThread *thr = s->connect_thread; - -assert (!s->sioc);qemu_mutex_lock (&thr->mutex);
Quoted 14 lines
block/nbd.c static void *connect_thread_func(void *opaque) · line 436
Quoted 12 lines
block/nbd.c +8
-1+/* +* Don't call nbd_co_establish_connection() in several coroutines in +* parallel. Only one call at once is supported. +*/ +assert (!thr->wait_co); +if (!thr->running) {if (thr->sioc) {/* Previous attempt finally succeeded in background */ - goto out; + sioc =g_steal_pointer (&thr->sioc); +qemu_mutex_unlock (&thr->mutex);Worth using
QEMU_LOCK_GUARD()here?
Refactored together with other critical sections in patch 15
Quoted 11 lines
block/nbd.c static void *connect_thread_func(void *opaque) · line 443
Quoted 8 lines
block/nbd.c +3
-0+ +return sioc; } + thr->running = true;error_free (thr->err); thr->err =NULL ;Reviewed-by Eric Blake <[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH v3 10/33 block/nbd: make nbd_co_establish_connection_cancel() bs-independent
In reply to Vladimir Sementsov-Ogievskiynbd_co_establish_connection_cancel() actually needs only pointer to NBDConnectThread. So, make it clean.
--- block/nbd.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-)
block/nbd.c +7 -9
@@ -123,7 +123,7 @@ static int nbd_establish_connection(BlockDriverState *bs, SocketAddress *saddr, Error **errp); static coroutine_fn QIOChannelSocket *nbd_co_establish_connection (NBDConnectThread *thr, Error **errp); -static void nbd_co_establish_connection_cancel (BlockDriverState *bs); +static void nbd_co_establish_connection_cancel (NBDConnectThread *thr);static int nbd_client_handshake (BlockDriverState *bs,Error **errp);static void nbd_yank (void *opaque);
@@ -286,7 +286,7 @@ static void coroutine_fn nbd_client_co_drain_begin(BlockDriverState *bs)qemu_co_sleep_wake (s->connection_co_sleep_ns_state); } -nbd_co_establish_connection_cancel (bs); +nbd_co_establish_connection_cancel (s->connect_thread);reconnect_delay_timer_del (s);
@@ -326,7 +326,7 @@ static void nbd_teardown_connection(BlockDriverState *bs)if (s->connection_co_sleep_ns_state) {qemu_co_sleep_wake (s->connection_co_sleep_ns_state); } -nbd_co_establish_connection_cancel (bs); +nbd_co_establish_connection_cancel (s->connect_thread); }if (qemu_in_coroutine ()) { s->teardown_co =qemu_coroutine_self ();
@@ -477,14 +477,12 @@ nbd_co_establish_connection(NBDConnectThread *thr, Error **errp)/* * nbd_co_establish_connection_cancel -* Cancel nbd_co_establish_connection asynchronously: it will finish soon, to -* allow drained section to begin. +* Cancel nbd_co_establish_connection() asynchronously. Note, that it doesn't +* stop the thread itself neither close the socket. It just safely wakes +* nbd_co_establish_connection() sleeping in the yield(). */ -static void nbd_co_establish_connection_cancel (BlockDriverState *bs) +static void nbd_co_establish_connection_cancel (NBDConnectThread *thr) { -BDRVNBDState *s = bs->opaque; -NBDConnectThread *thr = s->connect_thread; -qemu_mutex_lock (&thr->mutex);if (thr->wait_co) {
Signature · 1 line
2.29.2
Re: [PATCH v3 10/33] block/nbd: make nbd_co_establish_connection_cancel() bs-independent
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
nbd_co_establish_connection_cancel()actually needs only pointer to NBDConnectThread. So, make it clean.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Roman Kagan <[email protected]>--- block/nbd.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-)block/nbd.c +0
-0
block/nbd.c nbd_co_establish_connection(NBDConnectThread *thr, Error **errp) · line 482
Quoted 8 lines
block/nbd.c +3
-2/* * nbd_co_establish_connection_cancel -* Cancel nbd_co_establish_connection asynchronously: it will finish soon, to -* allow drained section to begin. +* Cancel nbd_co_establish_connection() asynchronously. Note, that it doesn't +* stop the thread itself neither close the socket. It just safely wakes +* nbd_co_establish_connection() sleeping in the yield().
Grammar suggestion:
Note that this function neither directly stops the thread nor closes the socket, but rather safely wakes nbd_co_establish_connection() which is sleeping in yield(), triggering subsequent cleanup there.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 11/33 block/nbd: rename NBDConnectThread to NBDClientConnection
In reply to Vladimir Sementsov-OgievskiyWe are going to move connection code to own file and want clear names and APIs.
The structure is shared between user and (possibly) several runs of connect-thread. So it's wrong to call it "thread". Let's rename to something more generic.
Appropriately rename connect_thread and thr variables to conn.
--- block/nbd.c | 137 ++++++++++++++++++++++++++-------------------------- 1 file changed, 68 insertions(+), 69 deletions(-)
block/nbd.c +68 -69
@@ -66,7 +66,7 @@ typedef enum NBDClientState {NBD_CLIENT_QUIT } NBDClientState; -typedef struct NBDConnectThread { +typedef struct NBDClientConnection { /* Initialization constants */ SocketAddress *saddr; /* address to connect to */
@@ -82,7 +82,7 @@ typedef struct NBDConnectThread {bool running;/* thread is running now */ bool detached;/* thread is detached and should cleanup the state */ Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ -} NBDConnectThread; +} NBDClientConnection; typedef struct BDRVNBDState { QIOChannelSocket *sioc; /* The master data channel */
@@ -115,34 +115,34 @@ typedef struct BDRVNBDState {char *x_dirty_bitmap;bool alloc_depth; -NBDConnectThread *connect_thread; +NBDClientConnection *conn; } BDRVNBDState; -static void nbd_free_connect_thread (NBDConnectThread *thr); +static void nbd_free_connect_thread (NBDClientConnection *conn);static int nbd_establish_connection (BlockDriverState *bs,SocketAddress *saddr,Error **errp); static coroutine_fn QIOChannelSocket * -nbd_co_establish_connection (NBDConnectThread *thr, Error **errp); -static void nbd_co_establish_connection_cancel (NBDConnectThread *thr); +nbd_co_establish_connection (NBDClientConnection *conn, Error **errp); +static void nbd_co_establish_connection_cancel (NBDClientConnection *conn);static int nbd_client_handshake (BlockDriverState *bs,Error **errp);static void nbd_yank (void *opaque);static void nbd_clear_bdrvstate (BlockDriverState *bs) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque; -NBDConnectThread *thr = s->connect_thread; +NBDClientConnection *conn = s->conn;bool do_free; -qemu_mutex_lock (&thr->mutex); -if (thr->running) { - thr->detached = true; +qemu_mutex_lock (&conn->mutex); +if (conn->running) { + conn->detached = true; } - do_free = !thr->running && !thr->detached; -qemu_mutex_unlock (&thr->mutex); + do_free = !conn->running && !conn->detached; +qemu_mutex_unlock (&conn->mutex);/* the runaway thread will clean it up itself */ if (do_free) { -nbd_free_connect_thread (thr); +nbd_free_connect_thread (conn); }yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name));
@@ -286,7 +286,7 @@ static void coroutine_fn nbd_client_co_drain_begin(BlockDriverState *bs)qemu_co_sleep_wake (s->connection_co_sleep_ns_state); } -nbd_co_establish_connection_cancel (s->connect_thread); +nbd_co_establish_connection_cancel (s->conn);reconnect_delay_timer_del (s);
@@ -326,7 +326,7 @@ static void nbd_teardown_connection(BlockDriverState *bs)if (s->connection_co_sleep_ns_state) {qemu_co_sleep_wake (s->connection_co_sleep_ns_state); } -nbd_co_establish_connection_cancel (s->connect_thread); +nbd_co_establish_connection_cancel (s->conn); }if (qemu_in_coroutine ()) { s->teardown_co =qemu_coroutine_self ();
@@ -353,101 +353,101 @@ static bool nbd_client_connecting_wait(BDRVNBDState *s)static void nbd_init_connect_thread (BDRVNBDState *s) { - s->connect_thread =g_new (NBDConnectThread,1 ); + s->conn =g_new (NBDClientConnection,1 ); - *s->connect_thread = (NBDConnectThread ) { + *s->conn = (NBDClientConnection ) { .saddr =QAPI_CLONE (SocketAddress, s->saddr), }; -qemu_mutex_init (&s->connect_thread->mutex); +qemu_mutex_init (&s->conn->mutex); } -static void nbd_free_connect_thread (NBDConnectThread *thr) +static void nbd_free_connect_thread (NBDClientConnection *conn) { -if (thr->sioc) { -qio_channel_close (QIO_CHANNEL (thr->sioc),NULL ); -object_unref (OBJECT (thr->sioc)); +if (conn->sioc) { +qio_channel_close (QIO_CHANNEL (conn->sioc),NULL ); +object_unref (OBJECT (conn->sioc)); } -error_free (thr->err); -qapi_free_SocketAddress (thr->saddr); -g_free (thr); +error_free (conn->err); +qapi_free_SocketAddress (conn->saddr); +g_free (conn); }static void *connect_thread_func (void *opaque) { -NBDConnectThread *thr = opaque; +NBDClientConnection *conn = opaque;int ret;bool do_free = false; - thr->sioc =qio_channel_socket_new (); + conn->sioc =qio_channel_socket_new (); -error_free (thr->err); - thr->err =NULL ; - ret =qio_channel_socket_connect_sync (thr->sioc, thr->saddr, &thr->err); +error_free (conn->err); + conn->err =NULL ; + ret =qio_channel_socket_connect_sync (conn->sioc, conn->saddr, &conn->err);if (ret <0 ) { -object_unref (OBJECT (thr->sioc)); - thr->sioc =NULL ; +object_unref (OBJECT (conn->sioc)); + conn->sioc =NULL ; } -qemu_mutex_lock (&thr->mutex); +qemu_mutex_lock (&conn->mutex); -assert (thr->running); - thr->running = false; -if (thr->wait_co) { -aio_co_schedule (NULL , thr->wait_co); - thr->wait_co =NULL ; +assert (conn->running); + conn->running = false; +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; } - do_free = thr->detached; + do_free = conn->detached; -qemu_mutex_unlock (&thr->mutex); +qemu_mutex_unlock (&conn->mutex);if (do_free) { -nbd_free_connect_thread (thr); +nbd_free_connect_thread (conn); }return NULL ; }/* -* Get a new connection in context of @thr: +* Get a new connection in context of @conn: * if thread is running, wait for completion * if thread is already succeeded in background, and user didn't get the * result, just return it now * otherwise if thread is not running, start a thread and wait for completion */ static coroutine_fn QIOChannelSocket * -nbd_co_establish_connection (NBDConnectThread *thr,Error **errp) +nbd_co_establish_connection (NBDClientConnection *conn,Error **errp) {QIOChannelSocket *sioc =NULL ;QemuThread thread; -qemu_mutex_lock (&thr->mutex); +qemu_mutex_lock (&conn->mutex);/* * Don't call nbd_co_establish_connection() in several coroutines in * parallel. Only one call at once is supported. */ -assert (!thr->wait_co); +assert (!conn->wait_co); -if (!thr->running) { -if (thr->sioc) { +if (!conn->running) { +if (conn->sioc) {/* Previous attempt finally succeeded in background */ - sioc =g_steal_pointer (&thr->sioc); -qemu_mutex_unlock (&thr->mutex); + sioc =g_steal_pointer (&conn->sioc); +qemu_mutex_unlock (&conn->mutex);return sioc; } - thr->running = true; -error_free (thr->err); - thr->err =NULL ; + conn->running = true; +error_free (conn->err); + conn->err =NULL ;qemu_thread_create (&thread,"nbd-connect" , - connect_thread_func, thr,QEMU_THREAD_DETACHED ); + connect_thread_func, conn,QEMU_THREAD_DETACHED ); } - thr->wait_co =qemu_coroutine_self (); + conn->wait_co =qemu_coroutine_self (); -qemu_mutex_unlock (&thr->mutex); +qemu_mutex_unlock (&conn->mutex);/* * We are going to wait for connect-thread finish, but
@@ -455,9 +455,9 @@ nbd_co_establish_connection(NBDConnectThread *thr, Error **errp)*/ qemu_coroutine_yield (); -qemu_mutex_lock (&thr->mutex); +qemu_mutex_lock (&conn->mutex); -if (thr->running) { +if (conn->running) {/* * Obviously, drained section wants to start. Report the attempt as * failed. Still connect thread is executing in background, and its
@@ -465,12 +465,12 @@ nbd_co_establish_connection(NBDConnectThread *thr, Error **errp)*/ error_setg (errp,"Connection attempt cancelled by other operation" ); }else { -error_propagate (errp, thr->err); - thr->err =NULL ; - sioc =g_steal_pointer (&thr->sioc); +error_propagate (errp, conn->err); + conn->err =NULL ; + sioc =g_steal_pointer (&conn->sioc); } -qemu_mutex_unlock (&thr->mutex); +qemu_mutex_unlock (&conn->mutex);return sioc; }
@@ -481,17 +481,16 @@ nbd_co_establish_connection(NBDConnectThread *thr, Error **errp)* stop the thread itself neither close the socket. It just safely wakes * nbd_co_establish_connection() sleeping in the yield(). */ -static void nbd_co_establish_connection_cancel (NBDConnectThread *thr) +static void nbd_co_establish_connection_cancel (NBDClientConnection *conn) { -qemu_mutex_lock (&thr->mutex); +qemu_mutex_lock (&conn->mutex); -if (thr->wait_co) { -aio_co_schedule (NULL , thr->wait_co); - thr->wait_co =NULL ; +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; } -qemu_mutex_unlock (&thr->mutex); - +qemu_mutex_unlock (&conn->mutex); } static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s)
@@ -537,7 +536,7 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) s->ioc =NULL ; } - s->sioc =nbd_co_establish_connection (s->connect_thread,NULL ); + s->sioc =nbd_co_establish_connection (s->conn,NULL );if (!s->sioc) { ret = -ECONNREFUSED ; goto out;
Signature · 1 line
2.29.2
Re: [PATCH v3 11/33] block/nbd: rename NBDConnectThread to NBDClientConnection
Quoted 7 lines · Vladimir Sementsov-Ogievskiy
We are going to move connection code to own file and want clear names and APIs.
The structure is shared between user and (possibly) several runs of connect-thread. So it's wrong to call it "thread". Let's rename to something more generic.
Quoted 6 lines
Appropriately rename
connect_threadand thr variables to conn.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 137 ++++++++++++++++++++++++++-------------------------- 1 file changed, 68 insertions(+), 69 deletions(-)
Re: [PATCH v3 11/33] block/nbd: rename NBDConnectThread to NBDClientConnection
In reply to Vladimir Sementsov-OgievskiyQuoted 2 lines · Vladimir Sementsov-Ogievskiy
We are going to move connection code to own file and want clear names and APIs.
We are going to move the connection code to its own file, and want clear names and APIs first.
Quoted 5 lines
The structure is shared between user and (possibly) several runs of connect-thread. So it's wrong to call it "thread". Let's rename to something more generic.
Quoted 7 lines
Appropriately rename
connect_threadand thr variables to conn.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 137 ++++++++++++++++++++++++++-------------------------- 1 file changed, 68 insertions(+), 69 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 12/33 block/nbd: introduce nbd_client_connection_new()
In reply to Vladimir Sementsov-OgievskiyThis is the last step of creating bs-independent nbd connection interface. With next commit we can finally move it to separate file.
--- block/nbd.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
block/nbd.c +9 -6
@@ -351,15 +351,18 @@ static bool nbd_client_connecting_wait(BDRVNBDState *s)return qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ; } -static void nbd_init_connect_thread (BDRVNBDState *s) +static NBDClientConnection * +nbd_client_connection_new (const SocketAddress *saddr) { - s->conn =g_new (NBDClientConnection,1 ); +NBDClientConnection *conn =g_new (NBDClientConnection,1 ); - *s->conn = (NBDClientConnection ) { - .saddr =QAPI_CLONE (SocketAddress, s->saddr), + *conn = (NBDClientConnection ) { + .saddr =QAPI_CLONE (SocketAddress, saddr), }; -qemu_mutex_init (&s->conn->mutex); +qemu_mutex_init (&conn->mutex); + +return conn; }static void nbd_free_connect_thread (NBDClientConnection *conn)
@@ -2208,7 +2211,7 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } -nbd_init_connect_thread (s); + s->conn =nbd_client_connection_new (s->saddr);/* * establish TCP connection, return error if it fails
Signature · 1 line
2.29.2
Re: [PATCH v3 12/33] block/nbd: introduce nbd_client_connection_new()
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
This is the last step of creating bs-independent nbd connection interface. With next commit we can finally move it to separate file.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Roman Kagan <[email protected]>--- block/nbd.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 13/33 block/nbd: introduce nbd_client_connection_release()
In reply to Vladimir Sementsov-Ogievskiy--- block/nbd.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-)
block/nbd.c +26 -17
@@ -118,7 +118,7 @@ typedef struct BDRVNBDState {NBDClientConnection *conn; } BDRVNBDState; -static void nbd_free_connect_thread (NBDClientConnection *conn); +static void nbd_client_connection_release (NBDClientConnection *conn);static int nbd_establish_connection (BlockDriverState *bs,SocketAddress *saddr,Error **errp); static coroutine_fn QIOChannelSocket *
@@ -130,20 +130,9 @@ static void nbd_yank(void *opaque);static void nbd_clear_bdrvstate (BlockDriverState *bs) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque; -NBDClientConnection *conn = s->conn; -bool do_free; - -qemu_mutex_lock (&conn->mutex); -if (conn->running) { - conn->detached = true; - } - do_free = !conn->running && !conn->detached; -qemu_mutex_unlock (&conn->mutex); -/* the runaway thread will clean it up itself */ -if (do_free) { -nbd_free_connect_thread (conn); - } +nbd_client_connection_release (s->conn); + s->conn =NULL ;yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name));
@@ -365,7 +354,7 @@ nbd_client_connection_new(const SocketAddress *saddr)return conn; } -static void nbd_free_connect_thread (NBDClientConnection *conn) +static void nbd_client_connection_do_free (NBDClientConnection *conn) {if (conn->sioc) {qio_channel_close (QIO_CHANNEL (conn->sioc),NULL );
@@ -379,8 +368,8 @@ static void nbd_free_connect_thread(NBDClientConnection *conn)static void *connect_thread_func (void *opaque) {NBDClientConnection *conn = opaque; +bool do_free;int ret; -bool do_free = false; conn->sioc =qio_channel_socket_new ();
@@ -405,12 +394,32 @@ static void *connect_thread_func(void *opaque)qemu_mutex_unlock (&conn->mutex);if (do_free) { -nbd_free_connect_thread (conn); +nbd_client_connection_do_free (conn); }return NULL ; } +static void nbd_client_connection_release (NBDClientConnection *conn) +{ +bool do_free; + +if (!conn) { +return ; + } + +qemu_mutex_lock (&conn->mutex); +if (conn->running) { + conn->detached = true; + } + do_free = !conn->running && !conn->detached; +qemu_mutex_unlock (&conn->mutex); + +if (do_free) { +nbd_client_connection_do_free (conn); + } +} +/* * Get a new connection in context of @conn: * if thread is running, wait for completion
Signature · 1 line
2.29.2
Re: [PATCH v3 13/33] block/nbd: introduce nbd_client_connection_release()
Quoted 51 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-)block/nbd.c +4
-15@@ -118,7 +118,7 @@ typedef struct BDRVNBDState {NBDClientConnection *conn; } BDRVNBDState; -static void nbd_free_connect_thread (NBDClientConnection *conn); +static void nbd_client_connection_release (NBDClientConnection *conn);static int nbd_establish_connection (BlockDriverState *bs,SocketAddress *saddr,Error **errp); static coroutine_fn QIOChannelSocket *@@ -130,20 +130,9 @@ static void nbd_yank(void *opaque);static void nbd_clear_bdrvstate (BlockDriverState *bs) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque; -NBDClientConnection *conn = s->conn; -bool do_free; - -qemu_mutex_lock (&conn->mutex); -if (conn->running) { - conn->detached = true; - } - do_free = !conn->running && !conn->detached; -qemu_mutex_unlock (&conn->mutex); -/* the runaway thread will clean it up itself */ -if (do_free) { -nbd_free_connect_thread (conn); - } +nbd_client_connection_release (s->conn); + s->conn =NULL ;yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name));@@ -365,7 +354,7 @@ nbd_client_connection_new(const SocketAddress *saddr)return conn; } -static void nbd_free_connect_thread (NBDClientConnection *conn) +static void nbd_client_connection_do_free (NBDClientConnection *conn) {if (conn->sioc) {qio_channel_close (QIO_CHANNEL (conn->sioc),NULL );
block/nbd.c static void nbd_free_connect_thread(NBDClientConnection *conn) · line 373
Quoted 7 lines
block/nbd.c +1
-1@@ -379,8 +368,8 @@ static void nbd_free_connect_thread(NBDClientConnection *conn)static void *connect_thread_func (void *opaque) {NBDClientConnection *conn = opaque; +bool do_free;int ret; -bool do_free = false;
This hunk belongs in patch 8.
Roman.
Quoted 39 lines
block/nbd.c +21
-1conn->sioc =qio_channel_socket_new ();@@ -405,12 +394,32 @@ static void *connect_thread_func(void *opaque)qemu_mutex_unlock (&conn->mutex);if (do_free) { -nbd_free_connect_thread (conn); +nbd_client_connection_do_free (conn); }return NULL ; } +static void nbd_client_connection_release (NBDClientConnection *conn) +{ +bool do_free; + +if (!conn) { +return ; + } + +qemu_mutex_lock (&conn->mutex); +if (conn->running) { + conn->detached = true; + } + do_free = !conn->running && !conn->detached; +qemu_mutex_unlock (&conn->mutex); + +if (do_free) { +nbd_client_connection_do_free (conn); + } +} +/* * Get a new connection in context of @conn: * if thread is running, wait for completion Signature · 1 line
2.29.2
Re: [PATCH v3 13/33] block/nbd: introduce nbd_client_connection_release()
Quoted 46 lines · Roman Kagan
Quoted 45 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-)block/nbd.c +3
-14@@ -118,7 +118,7 @@ typedef struct BDRVNBDState {NBDClientConnection *conn; } BDRVNBDState; -static void nbd_free_connect_thread (NBDClientConnection *conn); +static void nbd_client_connection_release (NBDClientConnection *conn);static int nbd_establish_connection (BlockDriverState *bs,SocketAddress *saddr,Error **errp); static coroutine_fn QIOChannelSocket *@@ -130,20 +130,9 @@ static void nbd_yank(void *opaque);static void nbd_clear_bdrvstate (BlockDriverState *bs) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque; -NBDClientConnection *conn = s->conn; -bool do_free; - -qemu_mutex_lock (&conn->mutex); -if (conn->running) { - conn->detached = true; - } - do_free = !conn->running && !conn->detached; -qemu_mutex_unlock (&conn->mutex); -/* the runaway thread will clean it up itself */ -if (do_free) { -nbd_free_connect_thread (conn); - } +nbd_client_connection_release (s->conn); + s->conn =NULL ;yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name));@@ -365,7 +354,7 @@ nbd_client_connection_new(const SocketAddress *saddr)return conn; }
Quoted 16 lines
block/nbd.c static void nbd_free_connect_thread(NBDClientConnection *conn) · line 373
Quoted 13 lines
block/nbd.c +2
-2-static void nbd_free_connect_thread (NBDClientConnection *conn) +static void nbd_client_connection_do_free (NBDClientConnection *conn) {if (conn->sioc) {qio_channel_close (QIO_CHANNEL (conn->sioc),NULL );@@ -379,8 +368,8 @@ static void nbd_free_connect_thread(NBDClientConnection *conn)static void *connect_thread_func (void *opaque) {NBDClientConnection *conn = opaque; +bool do_free;int ret; -bool do_free = false;This hunk belongs in patch 8.
Agree
Quoted 40 lines
Quoted 40 lines
block/nbd.c +21
-1conn->sioc =qio_channel_socket_new ();@@ -405,12 +394,32 @@ static void *connect_thread_func(void *opaque)qemu_mutex_unlock (&conn->mutex);if (do_free) { -nbd_free_connect_thread (conn); +nbd_client_connection_do_free (conn); }return NULL ; } +static void nbd_client_connection_release (NBDClientConnection *conn) +{ +bool do_free; + +if (!conn) { +return ; + } + +qemu_mutex_lock (&conn->mutex); +if (conn->running) { + conn->detached = true; + } + do_free = !conn->running && !conn->detached; +qemu_mutex_unlock (&conn->mutex); + +if (do_free) { +nbd_client_connection_do_free (conn); + } +} +/* * Get a new connection in context of @conn: * if thread is running, wait for completion Signature · 1 line
2.29.2
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 13/33] block/nbd: introduce nbd_client_connection_release()
In reply to Vladimir Sementsov-OgievskiyQuoted 4 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-)
Commit message said what, but not why. Presumably this is one more bit of refactoring to make the upcoming file split in a later patch easier. But patch 12/33 said it was the last step before a new file, and this patch isn't yet at a new file. Missing some continuity in your commit messages?
block/nbd.c typedef struct BDRVNBDState { · line 121
Quoted 11 lines
block/nbd.c +1
-1@@ -118,7 +118,7 @@ typedef struct BDRVNBDState {NBDClientConnection *conn; } BDRVNBDState; -static void nbd_free_connect_thread (NBDClientConnection *conn); +static void nbd_client_connection_release (NBDClientConnection *conn);
Is it necessary for a forward declaration, or can you just implement the new function prior to its users?
At any rate, the refactoring looks sane.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 13/33] block/nbd: introduce nbd_client_connection_release()
Quoted 11 lines · Eric Blake
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-)Commit message said what, but not why. Presumably this is one more bit of refactoring to make the upcoming file split in a later patch easier. But patch 12/33 said it was the last step before a new file, and this patch isn't yet at a new file. Missing some continuity in your commit messages?
Seems like one more small additional step after last step )
Quoted 10 lines
Quoted 10 lines
block/nbd.c +0
-0@@ -118,7 +118,7 @@ typedef struct BDRVNBDState {NBDClientConnection *conn; } BDRVNBDState;
Quoted 5 lines
block/nbd.c typedef struct BDRVNBDState { · line 121
Quoted 2 lines
block/nbd.c +1
-1-static void nbd_free_connect_thread (NBDClientConnection *conn); +static void nbd_client_connection_release (NBDClientConnection *conn);Is it necessary for a forward declaration, or can you just implement the new function prior to its users?
Hmm, seems it could be easily moved.
Quoted 3 lines
At any rate, the refactoring looks sane.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 13/33] block/nbd: introduce nbd_client_connection_release()
In reply to Eric BlakeQuoted 21 lines · Eric Blake
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-)Commit message said what, but not why. Presumably this is one more bit of refactoring to make the upcoming file split in a later patch easier. But patch 12/33 said it was the last step before a new file, and this patch isn't yet at a new file. Missing some continuity in your commit messages?
Quoted 9 lines
block/nbd.c +0
-0@@ -118,7 +118,7 @@ typedef struct BDRVNBDState {NBDClientConnection *conn; } BDRVNBDState;
Quoted 6 lines
block/nbd.c typedef struct BDRVNBDState { · line 121
Quoted 2 lines
block/nbd.c +1
-1-static void nbd_free_connect_thread (NBDClientConnection *conn); +static void nbd_client_connection_release (NBDClientConnection *conn);Is it necessary for a forward declaration, or can you just implement the new function prior to its users?
Actually, otherwise we'll need a forward declaration for nbd_client_connection_do_free(). Anyway, this all doesn't make real sense before moving to separate file
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 13/33] block/nbd: introduce nbd_client_connection_release()
Quoted 28 lines · Vladimir Sementsov-Ogievskiy
Quoted 13 lines · Eric Blake
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-)Commit message said what, but not why. Presumably this is one more bit of refactoring to make the upcoming file split in a later patch easier. But patch 12/33 said it was the last step before a new file, and this patch isn't yet at a new file. Missing some continuity in your commit messages?
Quoted 1 line
Quoted 13 lines
block/nbd.c typedef struct BDRVNBDState { · line 120
Quoted 9 lines
block/nbd.c +1
-1@@ -118,7 +118,7 @@ typedef struct BDRVNBDState {NBDClientConnection *conn; } BDRVNBDState; -static void nbd_free_connect_thread (NBDClientConnection *conn); +static void nbd_client_connection_release (NBDClientConnection *conn);Is it necessary for a forward declaration, or can you just implement the new function prior to its users?
Quoted 1 line
Actually, otherwise we'll need a forward declaration for
nbd_client_connection_do_free(). Anyway, this all doesn't make real sense before moving to separate file
Fair enough.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 14/33 nbd: move connection code from block/nbd to nbd/client-connection
In reply to Vladimir Sementsov-OgievskiyWe now have bs-independent connection API, which consists of four functions:
nbd_client_connection_new()
nbd_client_connection_unref()
nbd_co_establish_connection()
nbd_co_establish_connection_cancel()
Move them to a separate file together with NBDClientConnection structure which becomes private to the new API.
--- include/block/nbd.h | 11 +++ block/nbd.c | 187 ----------------------------------- nbd/client-connection.c | 212 ++++++++++++++++++++++++++++++++++++++++ nbd/meson.build | 1 + 4 files changed, 224 insertions(+), 187 deletions(-) create mode 100644 nbd/client-connection.c
include/block/nbd.h +11 -0
@@ -406,4 +406,15 @@ const char *nbd_info_lookup(uint16_t info);const char *nbd_cmd_lookup (uint16_t info);const char *nbd_err_lookup (int err); +/* nbd/client-connection.c */ +typedef struct NBDClientConnection NBDClientConnection ; + +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr); +void nbd_client_connection_release (NBDClientConnection *conn); + +QIOChannelSocket *coroutine_fn +nbd_co_establish_connection (NBDClientConnection *conn, Error **errp); + +void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn); +#endif
block/nbd.c +0 -187
207 lines not on this page · raw message
nbd/client-connection.c new file +212 -0
212 lines not on this page · raw message
nbd/meson.build +1 -0
@@ -1,5 +1,6 @@ block_ss.add(files( 'client.c', + 'client-connection.c', 'common.c', )) blockdev_ss.add(files(
Signature · 1 line
2.29.2
Re: [PATCH v3 14/33] nbd: move connection code from block/nbd to nbd/client-connection
Quoted 291 lines · Vladimir Sementsov-Ogievskiy
We now have bs-independent connection API, which consists of four functions:
nbd_client_connection_new()
nbd_client_connection_unref()
nbd_co_establish_connection()
nbd_co_establish_connection_cancel()Move them to a separate file together with NBDClientConnection structure which becomes private to the new API.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 11 +++ block/nbd.c | 187 ----------------------------------- nbd/client-connection.c | 212 ++++++++++++++++++++++++++++++++++++++++ nbd/meson.build | 1 + 4 files changed, 224 insertions(+), 187 deletions(-) create mode 100644 nbd/client-connection.cinclude/block/nbd.h +11
-0@@ -406,4 +406,15 @@ const char *nbd_info_lookup(uint16_t info);const char *nbd_cmd_lookup (uint16_t info);const char *nbd_err_lookup (int err); +/* nbd/client-connection.c */ +typedef struct NBDClientConnection NBDClientConnection ; + +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr); +void nbd_client_connection_release (NBDClientConnection *conn); + +QIOChannelSocket *coroutine_fn +nbd_co_establish_connection (NBDClientConnection *conn, Error **errp); + +void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn); +#endif block/nbd.c +0
-187@@ -66,24 +66,6 @@ typedef enum NBDClientState {NBD_CLIENT_QUIT } NBDClientState; -typedef struct NBDClientConnection { -/* Initialization constants */ -SocketAddress *saddr;/* address to connect to */ - -/* -* Result of last attempt. Valid in FAIL and SUCCESS states. -* If you want to steal error, don't forget to set pointer to NULL. -*/ -QIOChannelSocket *sioc; -Error *err; - -QemuMutex mutex; -/* All further fields are protected by mutex */ -bool running;/* thread is running now */ -bool detached;/* thread is detached and should cleanup the state */ -Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ -}NBDClientConnection ; - typedef struct BDRVNBDState { QIOChannelSocket *sioc; /* The master data channel */ QIOChannel *ioc; /* The current I/O channel which may differ (eg TLS) */@@ -118,12 +100,8 @@ typedef struct BDRVNBDState {NBDClientConnection *conn; } BDRVNBDState; -static void nbd_client_connection_release (NBDClientConnection *conn);static int nbd_establish_connection (BlockDriverState *bs,SocketAddress *saddr,Error **errp); -static coroutine_fn QIOChannelSocket * -nbd_co_establish_connection (NBDClientConnection *conn, Error **errp); -static void nbd_co_establish_connection_cancel (NBDClientConnection *conn);static int nbd_client_handshake (BlockDriverState *bs,Error **errp);static void nbd_yank (void *opaque);@@ -340,171 +318,6 @@ static bool nbd_client_connecting_wait(BDRVNBDState *s)return qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ; } -static NBDClientConnection * -nbd_client_connection_new (const SocketAddress *saddr) -{ -NBDClientConnection *conn =g_new (NBDClientConnection,1 ); - - *conn = (NBDClientConnection ) { - .saddr =QAPI_CLONE (SocketAddress, saddr), - }; - -qemu_mutex_init (&conn->mutex); - -return conn; -} - -static void nbd_client_connection_do_free (NBDClientConnection *conn) -{ -if (conn->sioc) { -qio_channel_close (QIO_CHANNEL (conn->sioc),NULL ); -object_unref (OBJECT (conn->sioc)); - } -error_free (conn->err); -qapi_free_SocketAddress (conn->saddr); -g_free (conn); -} - -static void *connect_thread_func (void *opaque) -{ -NBDClientConnection *conn = opaque; -bool do_free; -int ret; - - conn->sioc =qio_channel_socket_new (); - -error_free (conn->err); - conn->err =NULL ; - ret =qio_channel_socket_connect_sync (conn->sioc, conn->saddr, &conn->err); -if (ret <0 ) { -object_unref (OBJECT (conn->sioc)); - conn->sioc =NULL ; - } - -qemu_mutex_lock (&conn->mutex); - -assert (conn->running); - conn->running = false; -if (conn->wait_co) { -aio_co_schedule (NULL , conn->wait_co); - conn->wait_co =NULL ; - } - do_free = conn->detached; - -qemu_mutex_unlock (&conn->mutex); - -if (do_free) { -nbd_client_connection_do_free (conn); - } - -return NULL ; -} - -static void nbd_client_connection_release (NBDClientConnection *conn) -{ -bool do_free; - -if (!conn) { -return ; - } - -qemu_mutex_lock (&conn->mutex); -if (conn->running) { - conn->detached = true; - } - do_free = !conn->running && !conn->detached; -qemu_mutex_unlock (&conn->mutex); - -if (do_free) { -nbd_client_connection_do_free (conn); - } -} - -/* -* Get a new connection in context of @conn: -* if thread is running, wait for completion -* if thread is already succeeded in background, and user didn't get the -* result, just return it now -* otherwise if thread is not running, start a thread and wait for completion -*/ -static coroutine_fn QIOChannelSocket * -nbd_co_establish_connection (NBDClientConnection *conn,Error **errp) -{ -QIOChannelSocket *sioc =NULL ; -QemuThread thread; - -qemu_mutex_lock (&conn->mutex); - -/* -* Don't call nbd_co_establish_connection() in several coroutines in -* parallel. Only one call at once is supported. -*/ -assert (!conn->wait_co); - -if (!conn->running) { -if (conn->sioc) { -/* Previous attempt finally succeeded in background */ - sioc =g_steal_pointer (&conn->sioc); -qemu_mutex_unlock (&conn->mutex); - -return sioc; - } - - conn->running = true; -error_free (conn->err); - conn->err =NULL ; -qemu_thread_create (&thread,"nbd-connect" , - connect_thread_func, conn,QEMU_THREAD_DETACHED ); - } - - conn->wait_co =qemu_coroutine_self (); - -qemu_mutex_unlock (&conn->mutex); - -/* -* We are going to wait for connect-thread finish, but -* nbd_co_establish_connection_cancel() can interrupt. -*/ -qemu_coroutine_yield (); - -qemu_mutex_lock (&conn->mutex); - -if (conn->running) { -/* -* Obviously, drained section wants to start. Report the attempt as -* failed. Still connect thread is executing in background, and its -* result may be used for next connection attempt. -*/ -error_setg (errp,"Connection attempt cancelled by other operation" ); - }else { -error_propagate (errp, conn->err); - conn->err =NULL ; - sioc =g_steal_pointer (&conn->sioc); - } - -qemu_mutex_unlock (&conn->mutex); - -return sioc; -} - -/* -* nbd_co_establish_connection_cancel -* Cancel nbd_co_establish_connection() asynchronously. Note, that it doesn't -* stop the thread itself neither close the socket. It just safely wakes -* nbd_co_establish_connection() sleeping in the yield(). -*/ -static void nbd_co_establish_connection_cancel (NBDClientConnection *conn) -{ -qemu_mutex_lock (&conn->mutex); - -if (conn->wait_co) { -aio_co_schedule (NULL , conn->wait_co); - conn->wait_co =NULL ; - } - -qemu_mutex_unlock (&conn->mutex); -} - static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {int ret;nbd/client-connection.c new file +31
-0@@ -0,0 +1,212 @@ +/* +* QEMU Block driver for NBD +* +* Copyright (c) 2021 Virtuozzo International GmbH. +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +* THE SOFTWARE. +*/ + +#include "qemu/osdep.h" + +#include "block/nbd.h" + +#include "qapi/qapi-visit-sockets.h" +#include "qapi/clone-visitor.h" +
nbd/client-connection.c line 41
Quoted 10 lines
nbd/client-connection.c new file +10
-0+struct NBDClientConnection { +/* Initialization constants */ +SocketAddress *saddr;/* address to connect to */ + +/* +* Result of last attempt. Valid in FAIL and SUCCESS states. +* If you want to steal error, don't forget to set pointer to NULL. +*/ +QIOChannelSocket *sioc; +Error *err;
These two are also manipulated under the mutex. Consider also updating the comment: both these pointers are to be "stolen" by the caller, with the former being valid when the connection succeeds and the latter otherwise.
Roman.
Quoted 185 lines
nbd/client-connection.c new file +171
-0+ +QemuMutex mutex; +/* All further fields are protected by mutex */ +bool running;/* thread is running now */ +bool detached;/* thread is detached and should cleanup the state */ +Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ +}; + +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr) +{ +NBDClientConnection *conn =g_new (NBDClientConnection,1 ); + + *conn = (NBDClientConnection ) { + .saddr =QAPI_CLONE (SocketAddress, saddr), + }; + +qemu_mutex_init (&conn->mutex); + +return conn; +} + +static void nbd_client_connection_do_free (NBDClientConnection *conn) +{ +if (conn->sioc) { +qio_channel_close (QIO_CHANNEL (conn->sioc),NULL ); +object_unref (OBJECT (conn->sioc)); + } +error_free (conn->err); +qapi_free_SocketAddress (conn->saddr); +g_free (conn); +} + +static void *connect_thread_func (void *opaque) +{ +NBDClientConnection *conn = opaque; +bool do_free; +int ret; + + conn->sioc =qio_channel_socket_new (); + +error_free (conn->err); + conn->err =NULL ; + ret =qio_channel_socket_connect_sync (conn->sioc, conn->saddr, &conn->err); +if (ret <0 ) { +object_unref (OBJECT (conn->sioc)); + conn->sioc =NULL ; + } + +qemu_mutex_lock (&conn->mutex); + +assert (conn->running); + conn->running = false; +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; + } + do_free = conn->detached; + +qemu_mutex_unlock (&conn->mutex); + +if (do_free) { +nbd_client_connection_do_free (conn); + } + +return NULL ; +} + +void nbd_client_connection_release (NBDClientConnection *conn) +{ +bool do_free; + +if (!conn) { +return ; + } + +qemu_mutex_lock (&conn->mutex); +if (conn->running) { + conn->detached = true; + } + do_free = !conn->running && !conn->detached; +qemu_mutex_unlock (&conn->mutex); + +if (do_free) { +nbd_client_connection_do_free (conn); + } +} + +/* +* Get a new connection in context of @conn: +* if thread is running, wait for completion +* if thread is already succeeded in background, and user didn't get the +* result, just return it now +* otherwise if thread is not running, start a thread and wait for completion +*/ +QIOChannelSocket *coroutine_fn +nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) +{ +QIOChannelSocket *sioc =NULL ; +QemuThread thread; + +qemu_mutex_lock (&conn->mutex); + +/* +* Don't call nbd_co_establish_connection() in several coroutines in +* parallel. Only one call at once is supported. +*/ +assert (!conn->wait_co); + +if (!conn->running) { +if (conn->sioc) { +/* Previous attempt finally succeeded in background */ + sioc =g_steal_pointer (&conn->sioc); +qemu_mutex_unlock (&conn->mutex); + +return sioc; + } + + conn->running = true; +error_free (conn->err); + conn->err =NULL ; +qemu_thread_create (&thread,"nbd-connect" , + connect_thread_func, conn,QEMU_THREAD_DETACHED ); + } + + conn->wait_co =qemu_coroutine_self (); + +qemu_mutex_unlock (&conn->mutex); + +/* +* We are going to wait for connect-thread finish, but +* nbd_co_establish_connection_cancel() can interrupt. +*/ +qemu_coroutine_yield (); + +qemu_mutex_lock (&conn->mutex); + +if (conn->running) { +/* +* Obviously, drained section wants to start. Report the attempt as +* failed. Still connect thread is executing in background, and its +* result may be used for next connection attempt. +*/ +error_setg (errp,"Connection attempt cancelled by other operation" ); + }else { +error_propagate (errp, conn->err); + conn->err =NULL ; + sioc =g_steal_pointer (&conn->sioc); + } + +qemu_mutex_unlock (&conn->mutex); + +return sioc; +} + +/* +* nbd_co_establish_connection_cancel +* Cancel nbd_co_establish_connection() asynchronously. Note, that it doesn't +* stop the thread itself neither close the socket. It just safely wakes +* nbd_co_establish_connection() sleeping in the yield(). +*/ +void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn) +{ +qemu_mutex_lock (&conn->mutex); + +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; + } + +qemu_mutex_unlock (&conn->mutex); +}nbd/meson.build +1
-0@@ -1,5 +1,6 @@ block_ss.add(files( 'client.c', + 'client-connection.c', 'common.c', )) blockdev_ss.add(files(Signature · 1 line
2.29.2
Re: [PATCH v3 14/33] nbd: move connection code from block/nbd to nbd/client-connection
Quoted 299 lines · Roman Kagan
Quoted 298 lines · Vladimir Sementsov-Ogievskiy
We now have bs-independent connection API, which consists of four functions:
nbd_client_connection_new()
nbd_client_connection_unref()
nbd_co_establish_connection()
nbd_co_establish_connection_cancel()Move them to a separate file together with NBDClientConnection structure which becomes private to the new API.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 11 +++ block/nbd.c | 187 ----------------------------------- nbd/client-connection.c | 212 ++++++++++++++++++++++++++++++++++++++++ nbd/meson.build | 1 + 4 files changed, 224 insertions(+), 187 deletions(-)create mode 100644
nbd/client-connection.cinclude/block/nbd.h +11
-0@@ -406,4 +406,15 @@ const char *nbd_info_lookup(uint16_t info);const char *nbd_cmd_lookup (uint16_t info);const char *nbd_err_lookup (int err); +/* nbd/client-connection.c */ +typedef struct NBDClientConnection NBDClientConnection ; + +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr); +void nbd_client_connection_release (NBDClientConnection *conn); + +QIOChannelSocket *coroutine_fn +nbd_co_establish_connection (NBDClientConnection *conn, Error **errp); + +void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn); +#endif block/nbd.c +0
-187@@ -66,24 +66,6 @@ typedef enum NBDClientState {NBD_CLIENT_QUIT } NBDClientState; -typedef struct NBDClientConnection { -/* Initialization constants */ -SocketAddress *saddr;/* address to connect to */ - -/* -* Result of last attempt. Valid in FAIL and SUCCESS states. -* If you want to steal error, don't forget to set pointer to NULL. -*/ -QIOChannelSocket *sioc; -Error *err; - -QemuMutex mutex; -/* All further fields are protected by mutex */ -bool running;/* thread is running now */ -bool detached;/* thread is detached and should cleanup the state */ -Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ -}NBDClientConnection ; - typedef struct BDRVNBDState { QIOChannelSocket *sioc; /* The master data channel */ QIOChannel *ioc; /* The current I/O channel which may differ (eg TLS) */@@ -118,12 +100,8 @@ typedef struct BDRVNBDState {NBDClientConnection *conn; } BDRVNBDState; -static void nbd_client_connection_release (NBDClientConnection *conn);static int nbd_establish_connection (BlockDriverState *bs,SocketAddress *saddr,Error **errp); -static coroutine_fn QIOChannelSocket * -nbd_co_establish_connection (NBDClientConnection *conn, Error **errp); -static void nbd_co_establish_connection_cancel (NBDClientConnection *conn);static int nbd_client_handshake (BlockDriverState *bs,Error **errp);static void nbd_yank (void *opaque);@@ -340,171 +318,6 @@ static bool nbd_client_connecting_wait(BDRVNBDState *s)return qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ; } -static NBDClientConnection * -nbd_client_connection_new (const SocketAddress *saddr) -{ -NBDClientConnection *conn =g_new (NBDClientConnection,1 ); - - *conn = (NBDClientConnection ) { - .saddr =QAPI_CLONE (SocketAddress, saddr), - }; - -qemu_mutex_init (&conn->mutex); - -return conn; -} - -static void nbd_client_connection_do_free (NBDClientConnection *conn) -{ -if (conn->sioc) { -qio_channel_close (QIO_CHANNEL (conn->sioc),NULL ); -object_unref (OBJECT (conn->sioc)); - } -error_free (conn->err); -qapi_free_SocketAddress (conn->saddr); -g_free (conn); -} - -static void *connect_thread_func (void *opaque) -{ -NBDClientConnection *conn = opaque; -bool do_free; -int ret; - - conn->sioc =qio_channel_socket_new (); - -error_free (conn->err); - conn->err =NULL ; - ret =qio_channel_socket_connect_sync (conn->sioc, conn->saddr, &conn->err); -if (ret <0 ) { -object_unref (OBJECT (conn->sioc)); - conn->sioc =NULL ; - } - -qemu_mutex_lock (&conn->mutex); - -assert (conn->running); - conn->running = false; -if (conn->wait_co) { -aio_co_schedule (NULL , conn->wait_co); - conn->wait_co =NULL ; - } - do_free = conn->detached; - -qemu_mutex_unlock (&conn->mutex); - -if (do_free) { -nbd_client_connection_do_free (conn); - } - -return NULL ; -} - -static void nbd_client_connection_release (NBDClientConnection *conn) -{ -bool do_free; - -if (!conn) { -return ; - } - -qemu_mutex_lock (&conn->mutex); -if (conn->running) { - conn->detached = true; - } - do_free = !conn->running && !conn->detached; -qemu_mutex_unlock (&conn->mutex); - -if (do_free) { -nbd_client_connection_do_free (conn); - } -} - -/* -* Get a new connection in context of @conn: -* if thread is running, wait for completion -* if thread is already succeeded in background, and user didn't get the -* result, just return it now -* otherwise if thread is not running, start a thread and wait for completion -*/ -static coroutine_fn QIOChannelSocket * -nbd_co_establish_connection (NBDClientConnection *conn,Error **errp) -{ -QIOChannelSocket *sioc =NULL ; -QemuThread thread; - -qemu_mutex_lock (&conn->mutex); - -/* -* Don't call nbd_co_establish_connection() in several coroutines in -* parallel. Only one call at once is supported. -*/ -assert (!conn->wait_co); - -if (!conn->running) { -if (conn->sioc) { -/* Previous attempt finally succeeded in background */ - sioc =g_steal_pointer (&conn->sioc); -qemu_mutex_unlock (&conn->mutex); - -return sioc; - } - - conn->running = true; -error_free (conn->err); - conn->err =NULL ; -qemu_thread_create (&thread,"nbd-connect" , - connect_thread_func, conn,QEMU_THREAD_DETACHED ); - } - - conn->wait_co =qemu_coroutine_self (); - -qemu_mutex_unlock (&conn->mutex); - -/* -* We are going to wait for connect-thread finish, but -* nbd_co_establish_connection_cancel() can interrupt. -*/ -qemu_coroutine_yield (); - -qemu_mutex_lock (&conn->mutex); - -if (conn->running) { -/* -* Obviously, drained section wants to start. Report the attempt as -* failed. Still connect thread is executing in background, and its -* result may be used for next connection attempt. -*/ -error_setg (errp,"Connection attempt cancelled by other operation" ); - }else { -error_propagate (errp, conn->err); - conn->err =NULL ; - sioc =g_steal_pointer (&conn->sioc); - } - -qemu_mutex_unlock (&conn->mutex); - -return sioc; -} - -/* -* nbd_co_establish_connection_cancel -* Cancel nbd_co_establish_connection() asynchronously. Note, that it doesn't -* stop the thread itself neither close the socket. It just safely wakes -* nbd_co_establish_connection() sleeping in the yield(). -*/ -static void nbd_co_establish_connection_cancel (NBDClientConnection *conn) -{ -qemu_mutex_lock (&conn->mutex); - -if (conn->wait_co) { -aio_co_schedule (NULL , conn->wait_co); - conn->wait_co =NULL ; - } - -qemu_mutex_unlock (&conn->mutex); -} - static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {int ret;nbd/client-connection.c new file +38
-0@@ -0,0 +1,212 @@ +/* +* QEMU Block driver for NBD +* +* Copyright (c) 2021 Virtuozzo International GmbH. +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +* THE SOFTWARE. +*/ + +#include "qemu/osdep.h" + +#include "block/nbd.h" + +#include "qapi/qapi-visit-sockets.h" +#include "qapi/clone-visitor.h" + +struct NBDClientConnection { +/* Initialization constants */ +SocketAddress *saddr;/* address to connect to */ + +/* +* Result of last attempt. Valid in FAIL and SUCCESS states. +* If you want to steal error, don't forget to set pointer to NULL.
Quoted 9 lines
nbd/client-connection.c line 41
Quoted 3 lines
nbd/client-connection.c new file +3
-0+*/ +QIOChannelSocket *sioc; +Error *err;These two are also manipulated under the mutex. Consider also updating the comment: both these pointers are to be "stolen" by the caller, with the former being valid when the connection succeeds and the latter otherwise.
Hmm. I should move mutex and "All further" comment above these two fields.
Ok, I'll think on updating the comment (probably as an additional patch, to keep this as a simple movement). I don't like to document that they are stolen by caller(). For me it sounds like caller is user of the interface. And caller of nbd_co_establish_connection() doesn't stole anything: the structure is private now..
Quoted 186 lines
Quoted 186 lines
nbd/client-connection.c new file +171
-0+ +QemuMutex mutex; +/* All further fields are protected by mutex */ +bool running;/* thread is running now */ +bool detached;/* thread is detached and should cleanup the state */ +Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ +}; + +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr) +{ +NBDClientConnection *conn =g_new (NBDClientConnection,1 ); + + *conn = (NBDClientConnection ) { + .saddr =QAPI_CLONE (SocketAddress, saddr), + }; + +qemu_mutex_init (&conn->mutex); + +return conn; +} + +static void nbd_client_connection_do_free (NBDClientConnection *conn) +{ +if (conn->sioc) { +qio_channel_close (QIO_CHANNEL (conn->sioc),NULL ); +object_unref (OBJECT (conn->sioc)); + } +error_free (conn->err); +qapi_free_SocketAddress (conn->saddr); +g_free (conn); +} + +static void *connect_thread_func (void *opaque) +{ +NBDClientConnection *conn = opaque; +bool do_free; +int ret; + + conn->sioc =qio_channel_socket_new (); + +error_free (conn->err); + conn->err =NULL ; + ret =qio_channel_socket_connect_sync (conn->sioc, conn->saddr, &conn->err); +if (ret <0 ) { +object_unref (OBJECT (conn->sioc)); + conn->sioc =NULL ; + } + +qemu_mutex_lock (&conn->mutex); + +assert (conn->running); + conn->running = false; +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; + } + do_free = conn->detached; + +qemu_mutex_unlock (&conn->mutex); + +if (do_free) { +nbd_client_connection_do_free (conn); + } + +return NULL ; +} + +void nbd_client_connection_release (NBDClientConnection *conn) +{ +bool do_free; + +if (!conn) { +return ; + } + +qemu_mutex_lock (&conn->mutex); +if (conn->running) { + conn->detached = true; + } + do_free = !conn->running && !conn->detached; +qemu_mutex_unlock (&conn->mutex); + +if (do_free) { +nbd_client_connection_do_free (conn); + } +} + +/* +* Get a new connection in context of @conn: +* if thread is running, wait for completion +* if thread is already succeeded in background, and user didn't get the +* result, just return it now +* otherwise if thread is not running, start a thread and wait for completion +*/ +QIOChannelSocket *coroutine_fn +nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) +{ +QIOChannelSocket *sioc =NULL ; +QemuThread thread; + +qemu_mutex_lock (&conn->mutex); + +/* +* Don't call nbd_co_establish_connection() in several coroutines in +* parallel. Only one call at once is supported. +*/ +assert (!conn->wait_co); + +if (!conn->running) { +if (conn->sioc) { +/* Previous attempt finally succeeded in background */ + sioc =g_steal_pointer (&conn->sioc); +qemu_mutex_unlock (&conn->mutex); + +return sioc; + } + + conn->running = true; +error_free (conn->err); + conn->err =NULL ; +qemu_thread_create (&thread,"nbd-connect" , + connect_thread_func, conn,QEMU_THREAD_DETACHED ); + } + + conn->wait_co =qemu_coroutine_self (); + +qemu_mutex_unlock (&conn->mutex); + +/* +* We are going to wait for connect-thread finish, but +* nbd_co_establish_connection_cancel() can interrupt. +*/ +qemu_coroutine_yield (); + +qemu_mutex_lock (&conn->mutex); + +if (conn->running) { +/* +* Obviously, drained section wants to start. Report the attempt as +* failed. Still connect thread is executing in background, and its +* result may be used for next connection attempt. +*/ +error_setg (errp,"Connection attempt cancelled by other operation" ); + }else { +error_propagate (errp, conn->err); + conn->err =NULL ; + sioc =g_steal_pointer (&conn->sioc); + } + +qemu_mutex_unlock (&conn->mutex); + +return sioc; +} + +/* +* nbd_co_establish_connection_cancel +* Cancel nbd_co_establish_connection() asynchronously. Note, that it doesn't +* stop the thread itself neither close the socket. It just safely wakes +* nbd_co_establish_connection() sleeping in the yield(). +*/ +void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn) +{ +qemu_mutex_lock (&conn->mutex); + +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; + } + +qemu_mutex_unlock (&conn->mutex); +}nbd/meson.build +1
-0@@ -1,5 +1,6 @@ block_ss.add(files( 'client.c', + 'client-connection.c', 'common.c', )) blockdev_ss.add(files(Signature · 1 line
2.29.2
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 14/33] nbd: move connection code from block/nbd to nbd/client-connection
Quoted 17 lines · Vladimir Sementsov-Ogievskiy
Quoted 7 lines
Quoted 7 lines
nbd/client-connection.c new file +7
-0+struct NBDClientConnection { + /* Initialization constants */ + SocketAddress *saddr; /* address to connect to */ + + /* +* Result of last attempt. Valid in FAIL and SUCCESS states. +* If you want to steal error, don't forget to set pointer to NULL. Quoted 9 lines
Quoted 3 lines
+ */
+ QIOChannelSocket *sioc;
+ Error *err;These two are also manipulated under the mutex. Consider also updating the comment: both these pointers are to be "stolen" by the caller, with the former being valid when the connection succeeds and the latter otherwise.
Quoted 3 lines
Hmm. I should move mutex and "All further" comment above these two fields.
Ok, I'll think on updating the comment (probably as an additional patch, to keep this as a simple movement). I don't like to document that they are stolen by
caller(). For me it sounds like caller is user of the interface. And caller ofnbd_co_establish_connection()doesn't stole anything: the structure is private now..
Finally, I decided to improve the comment as part of "[PATCH v3 08/33] block/nbd: drop thr->state" commit, as "FAIL and SUCCESS states" string becomes outdated when we drop these states.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 14/33] nbd: move connection code from block/nbd to nbd/client-connection
In reply to Vladimir Sementsov-OgievskiyQuoted 21 lines · Vladimir Sementsov-Ogievskiy
We now have bs-independent connection API, which consists of four functions:
nbd_client_connection_new()
nbd_client_connection_unref()
nbd_co_establish_connection()
nbd_co_establish_connection_cancel()Move them to a separate file together with NBDClientConnection structure which becomes private to the new API.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 11 +++ block/nbd.c | 187 ----------------------------------- nbd/client-connection.c | 212 ++++++++++++++++++++++++++++++++++++++++ nbd/meson.build | 1 ++++
b/include/block/nbd.hinclude/block/nbd.h +0
-0@@ -406,4 +406,15 @@ const char *nbd_info_lookup(uint16_t info);const char *nbd_cmd_lookup (uint16_t info);
include/block/nbd.h const char *nbd_info_lookup(uint16_t info); · line 416
Quoted 10 lines
include/block/nbd.h +8
-0const char *nbd_err_lookup (int err); +/* nbd/client-connection.c */ +typedef struct NBDClientConnection NBDClientConnection ; + +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr); +void nbd_client_connection_release (NBDClientConnection *conn); + +QIOChannelSocket *coroutine_fn +nbd_co_establish_connection (NBDClientConnection *conn, Error **errp);
To me, the placement of coroutine_fn looks a bit odd here, like it is the name of a pointer declaration. But I see that we have precedence for doing it that way (such as block.c:bdrv_co_enter()); the difference being that none of the other locations split the return type on one line and the function name on another. I don't really have any changes to suggest, though, so I'm fine keeping it the way you wrote.
Review const char *nbd_info_lookup(uint16_t info); · line 417
Quoted 1 line
(quoted diff) +1
-0+++ b/nbd/client-connection.c
There may be fallout in v4 based on what you tweak in the code that got moved here, but the split to a new file looks sane to me.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 15/33 nbd/client-connection: use QEMU_LOCK_GUARD
In reply to Vladimir Sementsov-Ogievskiy--- nbd/client-connection.c | 94 ++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 52 deletions(-)
nbd/client-connection.c +42 -52
@@ -87,17 +87,16 @@ static void *connect_thread_func(void *opaque) conn->sioc =NULL ; } -qemu_mutex_lock (&conn->mutex); - -assert (conn->running); - conn->running = false; -if (conn->wait_co) { -aio_co_schedule (NULL , conn->wait_co); - conn->wait_co =NULL ; + WITH_QEMU_LOCK_GUARD(&conn->mutex) { +assert (conn->running); + conn->running = false; +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; + } } do_free = conn->detached; -qemu_mutex_unlock (&conn->mutex);if (do_free) {nbd_client_connection_do_free (conn);
@@ -136,61 +135,54 @@ void nbd_client_connection_release(NBDClientConnection *conn) QIOChannelSocket *coroutine_fn nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) { -QIOChannelSocket *sioc =NULL ;QemuThread thread; -qemu_mutex_lock (&conn->mutex); - -/* -* Don't call nbd_co_establish_connection() in several coroutines in -* parallel. Only one call at once is supported. -*/ -assert (!conn->wait_co); - -if (!conn->running) { -if (conn->sioc) { -/* Previous attempt finally succeeded in background */ - sioc =g_steal_pointer (&conn->sioc); -qemu_mutex_unlock (&conn->mutex); - -return sioc; + WITH_QEMU_LOCK_GUARD(&conn->mutex) { +/* +* Don't call nbd_co_establish_connection() in several coroutines in +* parallel. Only one call at once is supported. +*/ +assert (!conn->wait_co); + +if (!conn->running) { +if (conn->sioc) { +/* Previous attempt finally succeeded in background */ +return g_steal_pointer (&conn->sioc); + } + + conn->running = true; +error_free (conn->err); + conn->err =NULL ; +qemu_thread_create (&thread,"nbd-connect" , + connect_thread_func, conn,QEMU_THREAD_DETACHED ); } - conn->running = true; -error_free (conn->err); - conn->err =NULL ; -qemu_thread_create (&thread,"nbd-connect" , - connect_thread_func, conn,QEMU_THREAD_DETACHED ); + conn->wait_co =qemu_coroutine_self (); } - conn->wait_co =qemu_coroutine_self (); - -qemu_mutex_unlock (&conn->mutex); -/* * We are going to wait for connect-thread finish, but * nbd_co_establish_connection_cancel() can interrupt. */ qemu_coroutine_yield (); -qemu_mutex_lock (&conn->mutex); - -if (conn->running) { -/* -* Obviously, drained section wants to start. Report the attempt as -* failed. Still connect thread is executing in background, and its -* result may be used for next connection attempt. -*/ -error_setg (errp,"Connection attempt cancelled by other operation" ); - }else { -error_propagate (errp, conn->err); - conn->err =NULL ; - sioc =g_steal_pointer (&conn->sioc); + WITH_QEMU_LOCK_GUARD(&conn->mutex) { +if (conn->running) { +/* +* Obviously, drained section wants to start. Report the attempt as +* failed. Still connect thread is executing in background, and its +* result may be used for next connection attempt. +*/ +error_setg (errp,"Connection attempt cancelled by other operation" ); +return NULL ; + }else { +error_propagate (errp, conn->err); + conn->err =NULL ; +return g_steal_pointer (&conn->sioc); + } } -qemu_mutex_unlock (&conn->mutex); - -return sioc; +abort ();/* unreachable */ }/*
@@ -201,12 +193,10 @@ nbd_co_establish_connection(NBDClientConnection *conn, Error **errp)*/ void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn) { -qemu_mutex_lock (&conn->mutex); +QEMU_LOCK_GUARD (&conn->mutex);if (conn->wait_co) {aio_co_schedule (NULL , conn->wait_co); conn->wait_co =NULL ; } - -qemu_mutex_unlock (&conn->mutex); }
Signature · 1 line
2.29.2
Re: [PATCH v3 15/33] nbd/client-connection: use QEMU_LOCK_GUARD
Quoted 19 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client-connection.c | 94 ++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 52 deletions(-)nbd/client-connection.c +0
-6@@ -87,17 +87,16 @@ static void *connect_thread_func(void *opaque) conn->sioc =NULL ; } -qemu_mutex_lock (&conn->mutex); - -assert (conn->running); - conn->running = false; -if (conn->wait_co) { -aio_co_schedule (NULL , conn->wait_co);
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 98
Quoted 10 lines
nbd/client-connection.c +7
-1- conn->wait_co =NULL ; + WITH_QEMU_LOCK_GUARD(&conn->mutex) { +assert (conn->running); + conn->running = false; +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; + } } do_free = conn->detached;
->detached is now accessed outside the mutex
Quoted 118 lines
nbd/client-connection.c +35
-45-qemu_mutex_unlock (&conn->mutex);if (do_free) {nbd_client_connection_do_free (conn);@@ -136,61 +135,54 @@ void nbd_client_connection_release(NBDClientConnection *conn) QIOChannelSocket *coroutine_fn nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) { -QIOChannelSocket *sioc =NULL ;QemuThread thread; -qemu_mutex_lock (&conn->mutex); - -/* -* Don't call nbd_co_establish_connection() in several coroutines in -* parallel. Only one call at once is supported. -*/ -assert (!conn->wait_co); - -if (!conn->running) { -if (conn->sioc) { -/* Previous attempt finally succeeded in background */ - sioc =g_steal_pointer (&conn->sioc); -qemu_mutex_unlock (&conn->mutex); - -return sioc; + WITH_QEMU_LOCK_GUARD(&conn->mutex) { +/* +* Don't call nbd_co_establish_connection() in several coroutines in +* parallel. Only one call at once is supported. +*/ +assert (!conn->wait_co); + +if (!conn->running) { +if (conn->sioc) { +/* Previous attempt finally succeeded in background */ +return g_steal_pointer (&conn->sioc); + } + + conn->running = true; +error_free (conn->err); + conn->err =NULL ; +qemu_thread_create (&thread,"nbd-connect" , + connect_thread_func, conn,QEMU_THREAD_DETACHED ); } - conn->running = true; -error_free (conn->err); - conn->err =NULL ; -qemu_thread_create (&thread,"nbd-connect" , - connect_thread_func, conn,QEMU_THREAD_DETACHED ); + conn->wait_co =qemu_coroutine_self (); } - conn->wait_co =qemu_coroutine_self (); - -qemu_mutex_unlock (&conn->mutex); -/* * We are going to wait for connect-thread finish, but * nbd_co_establish_connection_cancel() can interrupt. */ qemu_coroutine_yield (); -qemu_mutex_lock (&conn->mutex); - -if (conn->running) { -/* -* Obviously, drained section wants to start. Report the attempt as -* failed. Still connect thread is executing in background, and its -* result may be used for next connection attempt. -*/ -error_setg (errp,"Connection attempt cancelled by other operation" ); - }else { -error_propagate (errp, conn->err); - conn->err =NULL ; - sioc =g_steal_pointer (&conn->sioc); + WITH_QEMU_LOCK_GUARD(&conn->mutex) { +if (conn->running) { +/* +* Obviously, drained section wants to start. Report the attempt as +* failed. Still connect thread is executing in background, and its +* result may be used for next connection attempt. +*/ +error_setg (errp,"Connection attempt cancelled by other operation" ); +return NULL ; + }else { +error_propagate (errp, conn->err); + conn->err =NULL ; +return g_steal_pointer (&conn->sioc); + } } -qemu_mutex_unlock (&conn->mutex); - -return sioc; +abort ();/* unreachable */ }/* @@ -201,12 +193,10 @@ nbd_co_establish_connection(NBDClientConnection *conn, Error **errp)*/ void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn) { -qemu_mutex_lock (&conn->mutex); +QEMU_LOCK_GUARD (&conn->mutex);if (conn->wait_co) {aio_co_schedule (NULL , conn->wait_co); conn->wait_co =NULL ; } - -qemu_mutex_unlock (&conn->mutex); }Signature · 1 line
2.29.2
Re: [PATCH v3 15/33] nbd/client-connection: use QEMU_LOCK_GUARD
Quoted 24 lines · Roman Kagan
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client-connection.c | 94 ++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 52 deletions(-)nbd/client-connection.c +3
-7@@ -87,17 +87,16 @@ static void *connect_thread_func(void *opaque) conn->sioc =NULL ; } -qemu_mutex_lock (&conn->mutex); - -assert (conn->running); - conn->running = false; -if (conn->wait_co) { -aio_co_schedule (NULL , conn->wait_co); - conn->wait_co =NULL ; + WITH_QEMU_LOCK_GUARD(&conn->mutex) { +assert (conn->running); + conn->running = false;
Quoted 8 lines
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 98
Quoted 6 lines
nbd/client-connection.c +4
-0+if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; + } } do_free = conn->detached;->detached is now accessed outside the mutex
Oops. Will fix.
Quoted 119 lines
Quoted 119 lines
nbd/client-connection.c +35
-45-qemu_mutex_unlock (&conn->mutex);if (do_free) {nbd_client_connection_do_free (conn);@@ -136,61 +135,54 @@ void nbd_client_connection_release(NBDClientConnection *conn) QIOChannelSocket *coroutine_fn nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) { -QIOChannelSocket *sioc =NULL ;QemuThread thread; -qemu_mutex_lock (&conn->mutex); - -/* -* Don't call nbd_co_establish_connection() in several coroutines in -* parallel. Only one call at once is supported. -*/ -assert (!conn->wait_co); - -if (!conn->running) { -if (conn->sioc) { -/* Previous attempt finally succeeded in background */ - sioc =g_steal_pointer (&conn->sioc); -qemu_mutex_unlock (&conn->mutex); - -return sioc; + WITH_QEMU_LOCK_GUARD(&conn->mutex) { +/* +* Don't call nbd_co_establish_connection() in several coroutines in +* parallel. Only one call at once is supported. +*/ +assert (!conn->wait_co); + +if (!conn->running) { +if (conn->sioc) { +/* Previous attempt finally succeeded in background */ +return g_steal_pointer (&conn->sioc); + } + + conn->running = true; +error_free (conn->err); + conn->err =NULL ; +qemu_thread_create (&thread,"nbd-connect" , + connect_thread_func, conn,QEMU_THREAD_DETACHED ); } - conn->running = true; -error_free (conn->err); - conn->err =NULL ; -qemu_thread_create (&thread,"nbd-connect" , - connect_thread_func, conn,QEMU_THREAD_DETACHED ); + conn->wait_co =qemu_coroutine_self (); } - conn->wait_co =qemu_coroutine_self (); - -qemu_mutex_unlock (&conn->mutex); -/* * We are going to wait for connect-thread finish, but * nbd_co_establish_connection_cancel() can interrupt. */ qemu_coroutine_yield (); -qemu_mutex_lock (&conn->mutex); - -if (conn->running) { -/* -* Obviously, drained section wants to start. Report the attempt as -* failed. Still connect thread is executing in background, and its -* result may be used for next connection attempt. -*/ -error_setg (errp,"Connection attempt cancelled by other operation" ); - }else { -error_propagate (errp, conn->err); - conn->err =NULL ; - sioc =g_steal_pointer (&conn->sioc); + WITH_QEMU_LOCK_GUARD(&conn->mutex) { +if (conn->running) { +/* +* Obviously, drained section wants to start. Report the attempt as +* failed. Still connect thread is executing in background, and its +* result may be used for next connection attempt. +*/ +error_setg (errp,"Connection attempt cancelled by other operation" ); +return NULL ; + }else { +error_propagate (errp, conn->err); + conn->err =NULL ; +return g_steal_pointer (&conn->sioc); + } } -qemu_mutex_unlock (&conn->mutex); - -return sioc; +abort ();/* unreachable */ }/* @@ -201,12 +193,10 @@ nbd_co_establish_connection(NBDClientConnection *conn, Error **errp)*/ void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn) { -qemu_mutex_lock (&conn->mutex); +QEMU_LOCK_GUARD (&conn->mutex);if (conn->wait_co) {aio_co_schedule (NULL , conn->wait_co); conn->wait_co =NULL ; } - -qemu_mutex_unlock (&conn->mutex); }Signature · 1 line
2.29.2
Signature · 2 lines
Best regards, Vladimir
PATCH v3 16/33 nbd/client-connection: add possibility of negotiation
In reply to Vladimir Sementsov-OgievskiyAdd arguments and logic to support nbd negotiation in the same thread after successful connection.
--- include/block/nbd.h | 9 +++- block/nbd.c | 4 +- nbd/client-connection.c | 105 ++++++++++++++++++++++++++++++++++++++-- 3 files changed, 109 insertions(+), 9 deletions(-)
include/block/nbd.h +7 -2
@@ -409,11 +409,16 @@ const char *nbd_err_lookup(int err);/* nbd/client-connection.c */ typedef struct NBDClientConnection NBDClientConnection ; -NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr); +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr, +bool do_negotiation, +const char *export_name, +const char *x_dirty_bitmap, +QCryptoTLSCreds *tlscreds);void nbd_client_connection_release (NBDClientConnection *conn); QIOChannelSocket *coroutine_fn -nbd_co_establish_connection (NBDClientConnection *conn, Error **errp); +nbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, + QIOChannel **ioc, Error **errp); void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn);
block/nbd.c +2 -2
@@ -361,7 +361,7 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) s->ioc =NULL ; } - s->sioc =nbd_co_establish_connection (s->conn,NULL ); + s->sioc =nbd_co_establish_connection (s->conn,NULL ,NULL ,NULL );if (!s->sioc) { ret = -ECONNREFUSED ; goto out;
@@ -2033,7 +2033,7 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } - s->conn =nbd_client_connection_new (s->saddr); + s->conn =nbd_client_connection_new (s->saddr, false,NULL ,NULL ,NULL );/* * establish TCP connection, return error if it fails
nbd/client-connection.c +100 -5
179 lines not on this page · raw message
Signature · 1 line
2.29.2
Re: [PATCH v3 16/33] nbd/client-connection: add possibility of negotiation
Quoted 110 lines · Vladimir Sementsov-Ogievskiy
Add arguments and logic to support nbd negotiation in the same thread after successful connection.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 9 +++- block/nbd.c | 4 +- nbd/client-connection.c | 105 ++++++++++++++++++++++++++++++++++++++-- 3 files changed, 109 insertions(+), 9 deletions(-)include/block/nbd.h +7
-2@@ -409,11 +409,16 @@ const char *nbd_err_lookup(int err);/* nbd/client-connection.c */ typedef struct NBDClientConnection NBDClientConnection ; -NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr); +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr, +bool do_negotiation, +const char *export_name, +const char *x_dirty_bitmap, +QCryptoTLSCreds *tlscreds);void nbd_client_connection_release (NBDClientConnection *conn); QIOChannelSocket *coroutine_fn -nbd_co_establish_connection (NBDClientConnection *conn, Error **errp); +nbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, + QIOChannel **ioc, Error **errp); void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn);block/nbd.c +2
-2@@ -361,7 +361,7 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) s->ioc =NULL ; } - s->sioc =nbd_co_establish_connection (s->conn,NULL ); + s->sioc =nbd_co_establish_connection (s->conn,NULL ,NULL ,NULL );if (!s->sioc) { ret = -ECONNREFUSED ; goto out;@@ -2033,7 +2033,7 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } - s->conn =nbd_client_connection_new (s->saddr); + s->conn =nbd_client_connection_new (s->saddr, false,NULL ,NULL ,NULL );/* * establish TCP connection, return error if it fails nbd/client-connection.c +20
-2@@ -30,14 +30,19 @@#include "qapi/clone-visitor.h" struct NBDClientConnection { -/* Initialization constants */ +/* Initialization constants, never change */ SocketAddress *saddr;/* address to connect to */ +QCryptoTLSCreds *tlscreds; +NBDExportInfo initial_info; +bool do_negotiation;/* * Result of last attempt. Valid in FAIL and SUCCESS states. * If you want to steal error, don't forget to set pointer to NULL. */ +NBDExportInfo updated_info;QIOChannelSocket *sioc; +QIOChannel *ioc;Error *err;QemuMutex mutex;@@ -47,12 +52,25 @@ struct NBDClientConnection {Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ }; -NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr) +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr, +bool do_negotiation, +const char *export_name, +const char *x_dirty_bitmap, +QCryptoTLSCreds *tlscreds) {NBDClientConnection *conn =g_new (NBDClientConnection,1 ); +object_ref (OBJECT (tlscreds)); *conn = (NBDClientConnection ) { .saddr =QAPI_CLONE (SocketAddress, saddr), + .tlscreds = tlscreds, + .do_negotiation = do_negotiation, + + .initial_info.request_sizes = true, + .initial_info.structured_reply = true, + .initial_info.base_allocation = true, + .initial_info.x_dirty_bitmap =g_strdup (x_dirty_bitmap), + .initial_info.name =g_strdup (export_name ?:"" ) };qemu_mutex_init (&conn->mutex);@@ -68,9 +86,59 @@ static void nbd_client_connection_do_free(NBDClientConnection *conn) }error_free (conn->err);
nbd/client-connection.c static void nbd_client_connection_do_free(NBDClientConnection *conn) · line 97
Quoted 10 lines
nbd/client-connection.c +6
-0qapi_free_SocketAddress (conn->saddr); +object_unref (OBJECT (conn->tlscreds)); +g_free (conn->initial_info.x_dirty_bitmap); +g_free (conn->initial_info.name);g_free (conn); } +/* +* Connect to @addr and do NBD negotiation if @info is not null. If @tlscreds +* given @outioc is provided. @outioc is provided only on success. The call may
s/given/are given/
s/provided/returned/g
nbd/client-connection.c static void nbd_client_connection_do_free(NBDClientConnection *conn) · line 98
Quoted 1 line
nbd/client-connection.c +1
-0+* be cancelled in parallel by simply qio_channel_shutdown(sioc).
I assume by "in parallel" you mean "from another thread", I'd suggest to spell this out. I'm also wondering how safe it really is. In general sockets should be fine with concurrent send()/recv() and shutdown(): the sender/receiver will be woken up with an error. Dunno if it's true for an arbitrary qio_channel. Also it may be worth documenting that the code path that cancels must leave all the cleanup up to the negotiation code, otherwise it risks conflicting.
Quoted 68 lines
nbd/client-connection.c +51
-1+*/ +static int nbd_connect (QIOChannelSocket *sioc,SocketAddress *addr, +NBDExportInfo *info,QCryptoTLSCreds *tlscreds, +QIOChannel **outioc,Error **errp) +{ +int ret; + +if (outioc) { + *outioc =NULL ; + } + + ret =qio_channel_socket_connect_sync (sioc, addr, errp); +if (ret <0 ) { +return ret; + } + +if (!info) { +return 0 ; + } + + ret =nbd_receive_negotiate (NULL ,QIO_CHANNEL (sioc), tlscreds, + tlscreds ? addr->u.inet.host :NULL , + outioc, info, errp); +if (ret <0 ) { +/* +* nbd_receive_negotiate() may setup tls ioc and return it even on +* failure path. In this case we should use it instead of original +* channel. +*/ +if (outioc && *outioc) { +qio_channel_close (QIO_CHANNEL (*outioc),NULL ); +object_unref (OBJECT (*outioc)); + *outioc =NULL ; + }else { +qio_channel_close (QIO_CHANNEL (sioc),NULL ); + } + +return ret; + } + +return 0 ; +} +static void *connect_thread_func (void *opaque) {NBDClientConnection *conn = opaque;@@ -81,12 +149,19 @@ static void *connect_thread_func(void *opaque)error_free (conn->err); conn->err =NULL ; - ret =qio_channel_socket_connect_sync (conn->sioc, conn->saddr, &conn->err); + conn->updated_info = conn->initial_info; + + ret =nbd_connect (conn->sioc, conn->saddr, + conn->do_negotiation ? &conn->updated_info :NULL , + conn->tlscreds, &conn->ioc, &conn->err);if (ret <0 ) {object_unref (OBJECT (conn->sioc)); conn->sioc =NULL ; } + conn->updated_info.x_dirty_bitmap =NULL ; + conn->updated_info.name =NULL ; + WITH_QEMU_LOCK_GUARD(&conn->mutex) {assert (conn->running); conn->running = false;
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 98
Quoted 6 lines
nbd/client-connection.c +1
-1@@ -94,8 +169,8 @@ static void *connect_thread_func(void *opaque)aio_co_schedule (NULL , conn->wait_co); conn->wait_co =NULL ; } + do_free = conn->detached; } - do_free = conn->detached;
This looks like the response to my earlier comment ;) This hunk just needs to be squashed into the previous patch.
Quoted 54 lines
nbd/client-connection.c +21
-1if (do_free) {@@ -131,12 +206,24 @@ void nbd_client_connection_release(NBDClientConnection *conn)* if thread is already succeeded in background, and user didn't get the * result, just return it now * otherwise if thread is not running, start a thread and wait for completion +* +* If @info is not NULL, also do nbd-negotiation after successful connection. +* In this case info is used only as out parameter, and is fully initialized by +* nbd_co_establish_connection(). "IN" fields of info as well as related only to +* nbd_receive_export_list() would be zero (see description of NBDExportInfo in +* include/block/nbd.h). */ QIOChannelSocket *coroutine_fn -nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) +nbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, + QIOChannel **ioc, Error **errp) {QemuThread thread; +if (conn->do_negotiation) { +assert (info); +assert (ioc); + } + WITH_QEMU_LOCK_GUARD(&conn->mutex) {/* * Don't call nbd_co_establish_connection() in several coroutines in @@ -147,6 +234,10 @@ nbd_co_establish_connection(NBDClientConnection *conn, Error **errp)if (!conn->running) {if (conn->sioc) {/* Previous attempt finally succeeded in background */ +if (conn->do_negotiation) { + *ioc =g_steal_pointer (&conn->ioc); +memcpy (info, &conn->updated_info,sizeof (*info)); + }return g_steal_pointer (&conn->sioc); }@@ -178,6 +269,10 @@ nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) }else {error_propagate (errp, conn->err); conn->err =NULL ; +if (conn->sioc && conn->do_negotiation) { + *ioc =g_steal_pointer (&conn->ioc); +memcpy (info, &conn->updated_info,sizeof (*info)); + }return g_steal_pointer (&conn->sioc); } }Signature · 1 line
2.29.2
Re: [PATCH v3 16/33] nbd/client-connection: add possibility of negotiation
Quoted 125 lines · Roman Kagan
Quoted 110 lines · Vladimir Sementsov-Ogievskiy
Add arguments and logic to support nbd negotiation in the same thread after successful connection.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 9 +++- block/nbd.c | 4 +- nbd/client-connection.c | 105 ++++++++++++++++++++++++++++++++++++++-- 3 files changed, 109 insertions(+), 9 deletions(-)include/block/nbd.h +7
-2@@ -409,11 +409,16 @@ const char *nbd_err_lookup(int err);/* nbd/client-connection.c */ typedef struct NBDClientConnection NBDClientConnection ; -NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr); +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr, +bool do_negotiation, +const char *export_name, +const char *x_dirty_bitmap, +QCryptoTLSCreds *tlscreds);void nbd_client_connection_release (NBDClientConnection *conn); QIOChannelSocket *coroutine_fn -nbd_co_establish_connection (NBDClientConnection *conn, Error **errp); +nbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, + QIOChannel **ioc, Error **errp); void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn);block/nbd.c +2
-2@@ -361,7 +361,7 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) s->ioc =NULL ; } - s->sioc =nbd_co_establish_connection (s->conn,NULL ); + s->sioc =nbd_co_establish_connection (s->conn,NULL ,NULL ,NULL );if (!s->sioc) { ret = -ECONNREFUSED ; goto out;@@ -2033,7 +2033,7 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } - s->conn =nbd_client_connection_new (s->saddr); + s->conn =nbd_client_connection_new (s->saddr, false,NULL ,NULL ,NULL );/* * establish TCP connection, return error if it fails nbd/client-connection.c +20
-2@@ -30,14 +30,19 @@#include "qapi/clone-visitor.h" struct NBDClientConnection { -/* Initialization constants */ +/* Initialization constants, never change */ SocketAddress *saddr;/* address to connect to */ +QCryptoTLSCreds *tlscreds; +NBDExportInfo initial_info; +bool do_negotiation;/* * Result of last attempt. Valid in FAIL and SUCCESS states. * If you want to steal error, don't forget to set pointer to NULL. */ +NBDExportInfo updated_info;QIOChannelSocket *sioc; +QIOChannel *ioc;Error *err;QemuMutex mutex;@@ -47,12 +52,25 @@ struct NBDClientConnection {Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ }; -NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr) +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr, +bool do_negotiation, +const char *export_name, +const char *x_dirty_bitmap, +QCryptoTLSCreds *tlscreds) {NBDClientConnection *conn =g_new (NBDClientConnection,1 ); +object_ref (OBJECT (tlscreds)); *conn = (NBDClientConnection ) { .saddr =QAPI_CLONE (SocketAddress, saddr), + .tlscreds = tlscreds, + .do_negotiation = do_negotiation, + + .initial_info.request_sizes = true, + .initial_info.structured_reply = true, + .initial_info.base_allocation = true, + .initial_info.x_dirty_bitmap =g_strdup (x_dirty_bitmap), + .initial_info.name =g_strdup (export_name ?:"" ) };qemu_mutex_init (&conn->mutex);@@ -68,9 +86,59 @@ static void nbd_client_connection_do_free(NBDClientConnection *conn) }error_free (conn->err);nbd/client-connection.c static void nbd_client_connection_do_free(NBDClientConnection *conn) · line 97
Quoted 10 lines
nbd/client-connection.c +6
-0qapi_free_SocketAddress (conn->saddr); +object_unref (OBJECT (conn->tlscreds)); +g_free (conn->initial_info.x_dirty_bitmap); +g_free (conn->initial_info.name);g_free (conn); } +/* +* Connect to @addr and do NBD negotiation if @info is not null. If @tlscreds +* given @outioc is provided. @outioc is provided only on success. The call may s/given/are given/
s/provided/returned/g
Quoted 7 lines
nbd/client-connection.c static void nbd_client_connection_do_free(NBDClientConnection *conn) · line 98
Quoted 1 line
nbd/client-connection.c +1
-0+* be cancelled in parallel by simply qio_channel_shutdown(sioc). I assume by "in parallel" you mean "from another thread", I'd suggest to spell this out. I'm also wondering how safe it really is. In general sockets should be fine with concurrent send()/recv() and
shutdown(): the sender/receiver will be woken up with an error. Dunno if it's true for an arbitraryqio_channel.
Hmm.. Good notice. I'll look at it.
Quoted 136 lines
Also it may be worth documenting that the
code path that cancels must leave all the cleanup up to the negotiation code, otherwise it risks conflicting.Quoted 68 lines
nbd/client-connection.c +51
-1+*/ +static int nbd_connect (QIOChannelSocket *sioc,SocketAddress *addr, +NBDExportInfo *info,QCryptoTLSCreds *tlscreds, +QIOChannel **outioc,Error **errp) +{ +int ret; + +if (outioc) { + *outioc =NULL ; + } + + ret =qio_channel_socket_connect_sync (sioc, addr, errp); +if (ret <0 ) { +return ret; + } + +if (!info) { +return 0 ; + } + + ret =nbd_receive_negotiate (NULL ,QIO_CHANNEL (sioc), tlscreds, + tlscreds ? addr->u.inet.host :NULL , + outioc, info, errp); +if (ret <0 ) { +/* +* nbd_receive_negotiate() may setup tls ioc and return it even on +* failure path. In this case we should use it instead of original +* channel. +*/ +if (outioc && *outioc) { +qio_channel_close (QIO_CHANNEL (*outioc),NULL ); +object_unref (OBJECT (*outioc)); + *outioc =NULL ; + }else { +qio_channel_close (QIO_CHANNEL (sioc),NULL ); + } + +return ret; + } + +return 0 ; +} +static void *connect_thread_func (void *opaque) {NBDClientConnection *conn = opaque;@@ -81,12 +149,19 @@ static void *connect_thread_func(void *opaque)error_free (conn->err); conn->err =NULL ; - ret =qio_channel_socket_connect_sync (conn->sioc, conn->saddr, &conn->err); + conn->updated_info = conn->initial_info; + + ret =nbd_connect (conn->sioc, conn->saddr, + conn->do_negotiation ? &conn->updated_info :NULL , + conn->tlscreds, &conn->ioc, &conn->err);if (ret <0 ) {object_unref (OBJECT (conn->sioc)); conn->sioc =NULL ; } + conn->updated_info.x_dirty_bitmap =NULL ; + conn->updated_info.name =NULL ; + WITH_QEMU_LOCK_GUARD(&conn->mutex) {assert (conn->running); conn->running = false;nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 98
Quoted 6 lines
nbd/client-connection.c +1
-1@@ -94,8 +169,8 @@ static void *connect_thread_func(void *opaque)aio_co_schedule (NULL , conn->wait_co); conn->wait_co =NULL ; } + do_free = conn->detached; } - do_free = conn->detached;This looks like the response to my earlier comment ;) This hunk just needs to be squashed into the previous patch.
Quoted 54 lines
nbd/client-connection.c +21
-1if (do_free) {@@ -131,12 +206,24 @@ void nbd_client_connection_release(NBDClientConnection *conn)* if thread is already succeeded in background, and user didn't get the * result, just return it now * otherwise if thread is not running, start a thread and wait for completion +* +* If @info is not NULL, also do nbd-negotiation after successful connection. +* In this case info is used only as out parameter, and is fully initialized by +* nbd_co_establish_connection(). "IN" fields of info as well as related only to +* nbd_receive_export_list() would be zero (see description of NBDExportInfo in +* include/block/nbd.h). */ QIOChannelSocket *coroutine_fn -nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) +nbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, + QIOChannel **ioc, Error **errp) {QemuThread thread; +if (conn->do_negotiation) { +assert (info); +assert (ioc); + } + WITH_QEMU_LOCK_GUARD(&conn->mutex) {/* * Don't call nbd_co_establish_connection() in several coroutines in @@ -147,6 +234,10 @@ nbd_co_establish_connection(NBDClientConnection *conn, Error **errp)if (!conn->running) {if (conn->sioc) {/* Previous attempt finally succeeded in background */ +if (conn->do_negotiation) { + *ioc =g_steal_pointer (&conn->ioc); +memcpy (info, &conn->updated_info,sizeof (*info)); + }return g_steal_pointer (&conn->sioc); }@@ -178,6 +269,10 @@ nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) }else {error_propagate (errp, conn->err); conn->err =NULL ; +if (conn->sioc && conn->do_negotiation) { + *ioc =g_steal_pointer (&conn->ioc); +memcpy (info, &conn->updated_info,sizeof (*info)); + }return g_steal_pointer (&conn->sioc); } }Signature · 1 line
2.29.2
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 16/33] nbd/client-connection: add possibility of negotiation
Quoted 114 lines · Vladimir Sementsov-Ogievskiy
Quoted 113 lines · Roman Kagan
Quoted 96 lines · Vladimir Sementsov-Ogievskiy
Add arguments and logic to support nbd negotiation in the same thread after successful connection.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
include/block/nbd.h | 9 +++-
block/nbd.c | 4 +-
nbd/client-connection.c| 105 ++++++++++++++++++++++++++++++++++++++--3 files changed, 109 insertions(+), 9 deletions(-)include/block/nbd.h +5
-1@@ -409,11 +409,16 @@ const char *nbd_err_lookup(int err); /* nbd/client-connection.c */ typedef struct NBDClientConnection NBDClientConnection; -NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr); +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr, + bool do_negotiation, + const char *export_name, + const char *x_dirty_bitmap, + QCryptoTLSCreds *tlscreds); void nbd_client_connection_release(NBDClientConnection *conn);QIOChannelSocket
*coroutine_fninclude/block/nbd.h +2
-1-nbd_co_establish_connection (NBDClientConnection *conn, Error **errp); +nbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, + QIOChannel **ioc, Error **errp);void
coroutine_fnnbd_co_establish_connection_cancel(NBDClientConnection *conn);block/nbd.c +2
-2@@ -361,7 +361,7 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) s->ioc = NULL; } - s->sioc = nbd_co_establish_connection(s->conn, NULL); + s->sioc = nbd_co_establish_connection(s->conn, NULL, NULL, NULL); if (!s->sioc) { ret = -ECONNREFUSED; goto out;@@ -2033,7 +2033,7 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } - s->conn = nbd_client_connection_new(s->saddr); + s->conn = nbd_client_connection_new(s->saddr, false, NULL, NULL, NULL); /** establish TCP connection, return error if it fails nbd/client-connection.c +1
-1@@ -30,14 +30,19 @@ #include "qapi/clone-visitor.h" struct NBDClientConnection { - /* Initialization constants */ + /* Initialization constants, never change */SocketAddress *saddr; /* address to connect to */
+ QCryptoTLSCreds *tlscreds;
+ NBDExportInfoinitial_info;
+ booldo_negotiation;
/*
* Result of last attempt. Valid in FAIL and SUCCESS states.
* If you want to steal error, don't forget to set pointer to NULL. */
+ NBDExportInfoupdated_info;
QIOChannelSocket *sioc;
+ QIOChannel *ioc;
Error *err;
QemuMutex mutex;nbd/client-connection.c +0
-0@@ -47,12 +52,25 @@ struct NBDClientConnection {Coroutine
*wait_co; /*nbd_co_establish_connection()wait inyield()*/ };nbd/client-connection.c +5
-1-NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr) +NBDClientConnection *nbd_client_connection_new (const SocketAddress *saddr, + bool do_negotiation, + const char *export_name, + const char *x_dirty_bitmap, + QCryptoTLSCreds *tlscreds){
NBDClientConnection *conn =g_new(NBDClientConnection, 1); + object_ref(OBJECT(tlscreds));
*conn = (NBDClientConnection) {
.saddr =QAPI_CLONE(SocketAddress, saddr),
+ .tlscreds = tlscreds,
+ .do_negotiation =do_negotiation,
+
+ .initial_info.request_sizes = true,
+ .initial_info.structured_reply = true,
+ .initial_info.base_allocation = true,
+ .initial_info.x_dirty_bitmap =g_strdup(x_dirty_bitmap), + .initial_info.name =g_strdup(export_name ?: "")};
qemu_mutex_init(&conn->mutex);nbd/client-connection.c +0
-0@@ -68,9 +86,59 @@ static void nbd_client_connection_do_free(NBDClientConnection *conn) }nbd/client-connection.c static void nbd_client_connection_do_free(NBDClientConnection *conn) · line 96
Quoted 10 lines
nbd/client-connection.c +6
-0error_free(conn->err); qapi_free_SocketAddress(conn->saddr); + object_unref(OBJECT(conn->tlscreds)); + g_free(conn->initial_info.x_dirty_bitmap); + g_free(conn->initial_info.name); g_free(conn); } +/* +* Connect to @addr and do NBD negotiation if @info is not null. If @tlscreds +* given @outioc is provided. @outioc is provided only on success. The call may s/given/are given/
s/provided/returned/gQuoted 1 line
nbd/client-connection.c +1
-0+* be cancelled in parallel by simply qio_channel_shutdown(sioc).
Quoted 7 lines
Quoted 5 lines
I assume by "in parallel" you mean "from another thread", I'd suggest to spell this out. I'm also wondering how safe it really is. In general sockets should be fine with concurrent send()/recv() and
shutdown(): the sender/receiver will be woken up with an error. Dunno if it's true for an arbitraryqio_channel.Hmm.. Good notice. I'll look at it.
At least, it should be safe, as in documentation of qio_channel_shutdown() I read:
This function is thread-safe, terminates quickly and does not block
Quoted 133 lines
Quoted 131 lines
Also it may be worth documenting that the
code path that cancels must leave all the cleanup up to the negotiation code, otherwise it risks conflicting.Quoted 66 lines
nbd/client-connection.c +51
-1+*/ +static int nbd_connect (QIOChannelSocket *sioc,SocketAddress *addr, + NBDExportInfo *info, QCryptoTLSCreds *tlscreds, + QIOChannel **outioc, Error **errp) +{ + int ret; + + if (outioc) { + *outioc = NULL; + } + + ret = qio_channel_socket_connect_sync(sioc, addr, errp); + if (ret < 0) { + return ret; + } + + if (!info) { + return 0; + } + + ret = nbd_receive_negotiate(NULL, QIO_CHANNEL(sioc), tlscreds, + tlscreds ? addr->u.inet.host : NULL, + outioc, info, errp); + if (ret < 0) { + /* + * nbd_receive_negotiate() may setup tls ioc and return it even on + * failure path. In this case we should use it instead of original + * channel. + */ + if (outioc && *outioc) { + qio_channel_close(QIO_CHANNEL(*outioc), NULL); + object_unref(OBJECT(*outioc)); + *outioc = NULL; + } else { + qio_channel_close(QIO_CHANNEL(sioc), NULL); + } + + return ret; + } + + return 0; +} + static void *connect_thread_func(void *opaque) { NBDClientConnection *conn = opaque;@@ -81,12 +149,19 @@ static void *connect_thread_func(void *opaque) error_free(conn->err); conn->err = NULL; - ret = qio_channel_socket_connect_sync(conn->sioc, conn->saddr, &conn->err); + conn->updated_info = conn->initial_info; + + ret = nbd_connect(conn->sioc, conn->saddr, + conn->do_negotiation ? &conn->updated_info : NULL, + conn->tlscreds, &conn->ioc, &conn->err); if (ret < 0) { object_unref(OBJECT(conn->sioc)); conn->sioc = NULL; } + conn->updated_info.x_dirty_bitmap = NULL; + conn->updated_info.name = NULL; + WITH_QEMU_LOCK_GUARD(&conn->mutex) { assert(conn->running); conn->running = false;nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 98
Quoted 6 lines
nbd/client-connection.c +1
-1@@ -94,8 +169,8 @@ static void *connect_thread_func(void *opaque) aio_co_schedule(NULL, conn->wait_co); conn->wait_co = NULL; } + do_free = conn->detached; } - do_free = conn->detached;This looks like the response to my earlier comment ;) This hunk just needs to be squashed into the previous patch.
Quoted 50 lines
nbd/client-connection.c +6
-0if (do_free) {@@ -131,12 +206,24 @@ void nbd_client_connection_release(NBDClientConnection *conn)* if thread is already succeeded in background, and user didn't get the * result, just return it now * otherwise if thread is not running, start a thread and wait for completion +* +* If @info is not NULL, also do nbd-negotiation after successful connection. +* In this case info is used only as out parameter, and is fully initialized by +* nbd_co_establish_connection(). "IN" fields of info as well as related only to +* nbd_receive_export_list() would be zero (see description of NBDExportInfo in +* include/block/nbd.h). */ QIOChannelSocket
*coroutine_fnnbd/client-connection.c +2
-1-nbd_co_establish_connection (NBDClientConnection *conn, Error **errp) +nbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, + QIOChannel **ioc, Error **errp){
QemuThread thread;
+ if (conn->do_negotiation) {
+ assert(info);
+ assert(ioc);
+ }
+
WITH_QEMU_LOCK_GUARD(&conn->mutex){
/*
* Don't callnbd_co_establish_connection()in several coroutines innbd/client-connection.c +8
-0@@ -147,6 +234,10 @@ nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) if (!conn->running) { if (conn->sioc) { /* Previous attempt finally succeeded in background */ + if (conn->do_negotiation) { + *ioc = g_steal_pointer(&conn->ioc); + memcpy(info, &conn->updated_info, sizeof(*info)); + } return g_steal_pointer(&conn->sioc); }@@ -178,6 +269,10 @@ nbd_co_establish_connection(NBDClientConnection *conn, Error **errp) } else { error_propagate(errp, conn->err); conn->err = NULL; + if (conn->sioc && conn->do_negotiation) { + *ioc = g_steal_pointer(&conn->ioc); + memcpy(info, &conn->updated_info, sizeof(*info)); + } return g_steal_pointer(&conn->sioc); } }Signature · 1 line
2.29.2
Signature · 2 lines
Best regards, Vladimir
PATCH v3 17/33 nbd/client-connection: implement connection retry
In reply to Vladimir Sementsov-OgievskiyAdd an option for thread to retry connection until success. We'll use nbd/client-connection both for reconnect and for initial connection in nbd_open(), so we need a possibility to use same NBDClientConnection instance to connect once in nbd_open() and then use retry semantics for reconnect.
--- include/block/nbd.h | 2 ++ nbd/client-connection.c | 55 +++++++++++++++++++++++++++++------------ 2 files changed, 41 insertions(+), 16 deletions(-)
include/block/nbd.h +2 -0
@@ -409,6 +409,8 @@ const char *nbd_err_lookup(int err);/* nbd/client-connection.c */ typedef struct NBDClientConnection NBDClientConnection ; +void nbd_client_connection_enable_retry (NBDClientConnection *conn); + NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr, bool do_negotiation, const char *export_name,
nbd/client-connection.c +39 -16
82 lines not on this page · raw message
Signature · 1 line
2.29.2
Re: [PATCH v3 17/33] nbd/client-connection: implement connection retry
Quoted 75 lines · Vladimir Sementsov-Ogievskiy
Add an option for thread to retry connection until success. We'll use nbd/client-connection both for reconnect and for initial connection in
nbd_open(), so we need a possibility to use same NBDClientConnection instance to connect once innbd_open()and then use retry semantics for reconnect.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 2 ++ nbd/client-connection.c | 55 +++++++++++++++++++++++++++++------------ 2 files changed, 41 insertions(+), 16 deletions(-)include/block/nbd.h +2
-0@@ -409,6 +409,8 @@ const char *nbd_err_lookup(int err);/* nbd/client-connection.c */ typedef struct NBDClientConnection NBDClientConnection ; +void nbd_client_connection_enable_retry (NBDClientConnection *conn); + NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr, bool do_negotiation, const char *export_name,nbd/client-connection.c +28
-0@@ -36,6 +36,8 @@ struct NBDClientConnection {NBDExportInfo initial_info;bool do_negotiation; +bool do_retry; +/* * Result of last attempt. Valid in FAIL and SUCCESS states. * If you want to steal error, don't forget to set pointer to NULL. @@ -52,6 +54,15 @@ struct NBDClientConnection {Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ }; +/* +* The function isn't protected by any mutex, so call it when thread is not +* running. +*/ +void nbd_client_connection_enable_retry (NBDClientConnection *conn) +{ + conn->do_retry = true; +} + NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr, bool do_negotiation, const char *export_name,@@ -144,24 +155,37 @@ static void *connect_thread_func(void *opaque)NBDClientConnection *conn = opaque;bool do_free;int ret; +uint64_t timeout =1 ; +uint64_t max_timeout =16 ; + +while (true) { + conn->sioc =qio_channel_socket_new (); + +error_free (conn->err); + conn->err =NULL ; + conn->updated_info = conn->initial_info; + + ret =nbd_connect (conn->sioc, conn->saddr, + conn->do_negotiation ? &conn->updated_info :NULL , + conn->tlscreds, &conn->ioc, &conn->err); + conn->updated_info.x_dirty_bitmap =NULL ; + conn->updated_info.name =NULL ; + +if (ret <0 ) {
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 184
Quoted 10 lines
nbd/client-connection.c +10
-0+object_unref (OBJECT (conn->sioc)); + conn->sioc =NULL ; +if (conn->do_retry) { +sleep (timeout); +if (timeout < max_timeout) { + timeout *=2 ; + } +continue ; + } + }
How is it supposed to get canceled?
Roman.
Quoted 32 lines
nbd/client-connection.c +1
-16- conn->sioc =qio_channel_socket_new (); - -error_free (conn->err); - conn->err =NULL ; - conn->updated_info = conn->initial_info; - - ret =nbd_connect (conn->sioc, conn->saddr, - conn->do_negotiation ? &conn->updated_info :NULL , - conn->tlscreds, &conn->ioc, &conn->err); -if (ret <0 ) { -object_unref (OBJECT (conn->sioc)); - conn->sioc =NULL ; +break ; } - conn->updated_info.x_dirty_bitmap =NULL ; - conn->updated_info.name =NULL ; - WITH_QEMU_LOCK_GUARD(&conn->mutex) {assert (conn->running); conn->running = false;@@ -172,7 +196,6 @@ static void *connect_thread_func(void *opaque) do_free = conn->detached; } -if (do_free) {nbd_client_connection_do_free (conn); }Signature · 1 line
2.29.2
Re: [PATCH v3 17/33] nbd/client-connection: implement connection retry
Quoted 80 lines · Roman Kagan
Quoted 79 lines · Vladimir Sementsov-Ogievskiy
Add an option for thread to retry connection until success. We'll use nbd/client-connection both for reconnect and for initial connection in
nbd_open(), so we need a possibility to use same NBDClientConnection instance to connect once innbd_open()and then use retry semantics for reconnect.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 2 ++ nbd/client-connection.c | 55 +++++++++++++++++++++++++++++------------ 2 files changed, 41 insertions(+), 16 deletions(-)include/block/nbd.h +2
-0@@ -409,6 +409,8 @@ const char *nbd_err_lookup(int err);/* nbd/client-connection.c */ typedef struct NBDClientConnection NBDClientConnection ; +void nbd_client_connection_enable_retry (NBDClientConnection *conn); + NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr, bool do_negotiation, const char *export_name,nbd/client-connection.c +32
-0@@ -36,6 +36,8 @@ struct NBDClientConnection {NBDExportInfo initial_info;bool do_negotiation; +bool do_retry; +/* * Result of last attempt. Valid in FAIL and SUCCESS states. * If you want to steal error, don't forget to set pointer to NULL. @@ -52,6 +54,15 @@ struct NBDClientConnection {Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ }; +/* +* The function isn't protected by any mutex, so call it when thread is not +* running. +*/ +void nbd_client_connection_enable_retry (NBDClientConnection *conn) +{ + conn->do_retry = true; +} + NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr, bool do_negotiation, const char *export_name,@@ -144,24 +155,37 @@ static void *connect_thread_func(void *opaque)NBDClientConnection *conn = opaque;bool do_free;int ret; +uint64_t timeout =1 ; +uint64_t max_timeout =16 ; + +while (true) { + conn->sioc =qio_channel_socket_new (); + +error_free (conn->err); + conn->err =NULL ; + conn->updated_info = conn->initial_info; + + ret =nbd_connect (conn->sioc, conn->saddr, + conn->do_negotiation ? &conn->updated_info :NULL , + conn->tlscreds, &conn->ioc, &conn->err); + conn->updated_info.x_dirty_bitmap =NULL ; + conn->updated_info.name =NULL ; + +if (ret <0 ) { +object_unref (OBJECT (conn->sioc)); + conn->sioc =NULL ; +if (conn->do_retry) { +sleep (timeout);
Quoted 9 lines
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 184
Quoted 6 lines
nbd/client-connection.c +6
-0+if (timeout < max_timeout) { + timeout *=2 ; + } +continue ; + } + }How is it supposed to get canceled?
Next commit does it
Quoted 32 lines
Quoted 32 lines
nbd/client-connection.c +1
-16- conn->sioc =qio_channel_socket_new (); - -error_free (conn->err); - conn->err =NULL ; - conn->updated_info = conn->initial_info; - - ret =nbd_connect (conn->sioc, conn->saddr, - conn->do_negotiation ? &conn->updated_info :NULL , - conn->tlscreds, &conn->ioc, &conn->err); -if (ret <0 ) { -object_unref (OBJECT (conn->sioc)); - conn->sioc =NULL ; +break ; } - conn->updated_info.x_dirty_bitmap =NULL ; - conn->updated_info.name =NULL ; - WITH_QEMU_LOCK_GUARD(&conn->mutex) {assert (conn->running); conn->running = false;@@ -172,7 +196,6 @@ static void *connect_thread_func(void *opaque) do_free = conn->detached; } -if (do_free) {nbd_client_connection_do_free (conn); }Signature · 1 line
2.29.2
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 17/33] nbd/client-connection: implement connection retry
In reply to Vladimir Sementsov-OgievskiyQuoted 1 line · Vladimir Sementsov-Ogievskiy
Add an option for thread to retry connection until success. We'll use
for a thread to retry connection until it succeeds.
Quoted 22 lines
nbd/client-connection both for reconnect and for initial connection in
nbd_open(), so we need a possibility to use same NBDClientConnection instance to connect once innbd_open()and then use retry semantics for reconnect.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 2 ++ nbd/client-connection.c | 55 +++++++++++++++++++++++++++++------------ 2 files changed, 41 insertions(+), 16 deletions(-)+++
b/nbd/client-connection.cnbd/client-connection.c +2
-0@@ -36,6 +36,8 @@ struct NBDClientConnection {NBDExportInfo initial_info;bool do_negotiation; +bool do_retry; +/* * Result of last attempt. Valid in FAIL and SUCCESS states. * If you want to steal error, don't forget to set pointer to NULL.
nbd/client-connection.c struct NBDClientConnection { · line 58
Quoted 5 lines
nbd/client-connection.c +2
-0@@ -52,6 +54,15 @@ struct NBDClientConnection {Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ }; +/* +* The function isn't protected by any mutex, so call it when thread is not
so only call it when the thread is not yet running
or maybe even
only call it when the client connection attempt has not yet started
Quoted 19 lines
nbd/client-connection.c +12
-0+* running. +*/ +void nbd_client_connection_enable_retry (NBDClientConnection *conn) +{ + conn->do_retry = true; +} + NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr, bool do_negotiation, const char *export_name,@@ -144,24 +155,37 @@ static void *connect_thread_func(void *opaque)NBDClientConnection *conn = opaque;bool do_free;int ret; +uint64_t timeout =1 ; +uint64_t max_timeout =16 ; + +while (true) { + conn->sioc =qio_channel_socket_new ();
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 172
Quoted 10 lines
nbd/client-connection.c +10
-0+ +error_free (conn->err); + conn->err =NULL ; + conn->updated_info = conn->initial_info; + + ret =nbd_connect (conn->sioc, conn->saddr, + conn->do_negotiation ? &conn->updated_info :NULL , + conn->tlscreds, &conn->ioc, &conn->err); + conn->updated_info.x_dirty_bitmap =NULL ; + conn->updated_info.name =NULL ;
I'm not quite sure I follow the allocation here: if we passed in &conn->updated_info which got modified in-place by nbd_connect, then are we risking a memory leak by ignoring the x_dirty_bitmap and name set by that call?
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 178
Quoted 6 lines
nbd/client-connection.c +6
-0+ +if (ret <0 ) { +object_unref (OBJECT (conn->sioc)); + conn->sioc =NULL ; +if (conn->do_retry) { +sleep (timeout);
This is a bare sleep in a function not marked as coroutine_fn. Do we need to instead use coroutine sleep for better response to an early exit if initialization is taking too long?
Quoted 29 lines
nbd/client-connection.c +7
-15+if (timeout < max_timeout) { + timeout *=2 ; + } +continue ; + } + } - conn->sioc =qio_channel_socket_new (); - -error_free (conn->err); - conn->err =NULL ; - conn->updated_info = conn->initial_info; - - ret =nbd_connect (conn->sioc, conn->saddr, - conn->do_negotiation ? &conn->updated_info :NULL , - conn->tlscreds, &conn->ioc, &conn->err); -if (ret <0 ) { -object_unref (OBJECT (conn->sioc)); - conn->sioc =NULL ; +break ; } - conn->updated_info.x_dirty_bitmap =NULL ; - conn->updated_info.name =NULL ; - WITH_QEMU_LOCK_GUARD(&conn->mutex) {assert (conn->running); conn->running = false;
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 200
Quoted 6 lines
nbd/client-connection.c +0
-1@@ -172,7 +196,6 @@ static void *connect_thread_func(void *opaque) do_free = conn->detached; } -if (do_free) {nbd_client_connection_do_free (conn);
Spurious hunk?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 17/33] nbd/client-connection: implement connection retry
Quoted 65 lines · Eric Blake
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Add an option for thread to retry connection until success. We'll use
for a thread to retry connection until it succeeds.
Quoted 22 lines
nbd/client-connection both for reconnect and for initial connection in
nbd_open(), so we need a possibility to use same NBDClientConnection instance to connect once innbd_open()and then use retry semantics for reconnect.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 2 ++ nbd/client-connection.c | 55 +++++++++++++++++++++++++++++------------ 2 files changed, 41 insertions(+), 16 deletions(-)+++
b/nbd/client-connection.cnbd/client-connection.c +2
-0@@ -36,6 +36,8 @@ struct NBDClientConnection {NBDExportInfo initial_info;bool do_negotiation; +bool do_retry; +/* * Result of last attempt. Valid in FAIL and SUCCESS states. * If you want to steal error, don't forget to set pointer to NULL. nbd/client-connection.c struct NBDClientConnection { · line 58
Quoted 5 lines
nbd/client-connection.c +2
-0@@ -52,6 +54,15 @@ struct NBDClientConnection {Coroutine *wait_co;/* nbd_co_establish_connection() wait in yield() */ }; +/* +* The function isn't protected by any mutex, so call it when thread is not so only call it when the thread is not yet running
or maybe even
only call it when the client connection attempt has not yet started
Quoted 26 lines
nbd/client-connection.c +19
-0+* running. +*/ +void nbd_client_connection_enable_retry (NBDClientConnection *conn) +{ + conn->do_retry = true; +} + NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr, bool do_negotiation, const char *export_name,@@ -144,24 +155,37 @@ static void *connect_thread_func(void *opaque)NBDClientConnection *conn = opaque;bool do_free;int ret; +uint64_t timeout =1 ; +uint64_t max_timeout =16 ; + +while (true) { + conn->sioc =qio_channel_socket_new (); + +error_free (conn->err); + conn->err =NULL ; + conn->updated_info = conn->initial_info; + + ret = nbd_connect(conn->sioc, conn->saddr, + conn->do_negotiation ? &conn->updated_info : NULL,
Quoted 8 lines
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 172
Quoted 3 lines
nbd/client-connection.c +3
-0+ conn->tlscreds, &conn->ioc, &conn->err); + conn->updated_info.x_dirty_bitmap =NULL ; + conn->updated_info.name =NULL ;I'm not quite sure I follow the allocation here: if we passed in
&conn->updated_infowhich got modified in-place bynbd_connect, then are we risking a memory leak by ignoring thex_dirty_bitmapand name set by that call?
Yes, that looks strange :\. Will check when prepare new version and fix or leave a comment here.
Quoted 11 lines
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 178
Quoted 7 lines
nbd/client-connection.c +6
-0+ +if (ret <0 ) { +object_unref (OBJECT (conn->sioc)); + conn->sioc =NULL ; +if (conn->do_retry) { +sleep (timeout);This is a bare sleep in a function not marked as
coroutine_fn. Do we need to instead use coroutine sleep for better response to an early exit if initialization is taking too long?
We are in a separate, by-hand created thread, which knows nothing about coroutines, iothreads, aio contexts etc.. I think bare sleep is what should be here.
Quoted 33 lines
Quoted 33 lines
nbd/client-connection.c +7
-15+if (timeout < max_timeout) { + timeout *=2 ; + } +continue ; + } + } - conn->sioc =qio_channel_socket_new (); - -error_free (conn->err); - conn->err =NULL ; - conn->updated_info = conn->initial_info; - - ret =nbd_connect (conn->sioc, conn->saddr, - conn->do_negotiation ? &conn->updated_info :NULL , - conn->tlscreds, &conn->ioc, &conn->err); -if (ret <0 ) { -object_unref (OBJECT (conn->sioc)); - conn->sioc =NULL ; +break ; } - conn->updated_info.x_dirty_bitmap =NULL ; - conn->updated_info.name =NULL ; - WITH_QEMU_LOCK_GUARD(&conn->mutex) {assert (conn->running); conn->running = false;@@ -172,7 +196,6 @@ static void *connect_thread_func(void *opaque) do_free = conn->detached; }
Quoted 6 lines
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 200
Quoted 3 lines
nbd/client-connection.c +0
-1-if (do_free) {nbd_client_connection_do_free (conn);Spurious hunk?
wull drop
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 17/33] nbd/client-connection: implement connection retry
Quoted 73 lines · Vladimir Sementsov-Ogievskiy
Quoted 63 lines · Eric Blake
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Add an option for thread to retry connection until success. We'll use
for a thread to retry connection until it succeeds.
Quoted 21 lines
nbd/client-connection both for reconnect and for initial connection in
nbd_open(), so we need a possibility to use same NBDClientConnection instance to connect once innbd_open()and then use retry semantics for reconnect.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
include/block/nbd.h | 2 ++
nbd/client-connection.c| 55 +++++++++++++++++++++++++++++------------2 files changed, 41 insertions(+), 16 deletions(-)+++
b/nbd/client-connection.cnbd/client-connection.c +2
-0@@ -36,6 +36,8 @@ struct NBDClientConnection { NBDExportInfo initial_info; bool do_negotiation; + bool do_retry; + /** Result of last attempt. Valid in FAIL and SUCCESS states. * If you want to steal error, don't forget to set pointer to NULL. @@ -52,6 +54,15 @@ struct NBDClientConnection {Quoted 4 lines
Coroutine
*wait_co; /*nbd_co_establish_connection()wait inyield()*/ };
+/*
+ * The function isn't protected by any mutex, so call it when thread is notso only call it when the thread is not yet running
or maybe even
only call it when the client connection attempt has not yet started
Quoted 26 lines
+ * running.
+ */nbd/client-connection.c +5
-0+void nbd_client_connection_enable_retry (NBDClientConnection *conn) +{ + conn->do_retry = true; +} +NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr, bool
do_negotiation, const char*export_name,nbd/client-connection.c +12
-0@@ -144,24 +155,37 @@ static void *connect_thread_func(void *opaque) NBDClientConnection *conn = opaque; bool do_free; int ret; + uint64_t timeout = 1; + uint64_t max_timeout = 16; + + while (true) { + conn->sioc = qio_channel_socket_new(); + + error_free(conn->err); + conn->err = NULL; + conn->updated_info = conn->initial_info; + + ret = nbd_connect(conn->sioc, conn->saddr, + conn->do_negotiation ? &conn->updated_info : NULL,Quoted 8 lines
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 172
Quoted 3 lines
nbd/client-connection.c +3
-0+ conn->tlscreds, &conn->ioc, &conn->err); + conn->updated_info.x_dirty_bitmap = NULL; + conn->updated_info.name = NULL;I'm not quite sure I follow the allocation here: if we passed in
&conn->updated_infowhich got modified in-place bynbd_connect, then are we risking a memory leak by ignoring thex_dirty_bitmapand name set by that call?
Quoted 1 line
Yes, that looks strange :\. Will check when prepare new version and fix or leave a comment here.
x_dirty_bitmap and name are not set by nbd_connect, they are IN parameters of nbd_receive_negotiate(). Their allocations are owned by conn->initial_info. So, here we've copied pointers into conn->updated_info. And then zero out them, when they are not needed anymore (and actually, to not store them and not finally return to the user our internal allocations). I'll add a comment here.
Quoted 54 lines
Quoted 9 lines
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 178
Quoted 8 lines
nbd/client-connection.c +6
-0+ + if (ret < 0) { + object_unref(OBJECT(conn->sioc)); + conn->sioc = NULL; + if (conn->do_retry) { + sleep(timeout);Quoted 3 lines
This is a bare sleep in a function not marked as coroutine_fn. Do we need to instead use coroutine sleep for better response to an early exit if initialization is taking too long?
We are in a separate, by-hand created thread, which knows nothing about coroutines, iothreads, aio contexts etc.. I think bare sleep is what should be here.
Quoted 27 lines
Quoted 27 lines
nbd/client-connection.c +7
-15+ if (timeout < max_timeout) { + timeout *= 2; + } + continue; + } + } - conn->sioc = qio_channel_socket_new(); - - error_free(conn->err); - conn->err = NULL; - conn->updated_info = conn->initial_info; - - ret = nbd_connect(conn->sioc, conn->saddr, - conn->do_negotiation ? &conn->updated_info : NULL, - conn->tlscreds, &conn->ioc, &conn->err); - if (ret < 0) { - object_unref(OBJECT(conn->sioc)); - conn->sioc = NULL; + break; } - conn->updated_info.x_dirty_bitmap = NULL; - conn->updated_info.name = NULL; - WITH_QEMU_LOCK_GUARD(&conn->mutex) { assert(conn->running); conn->running = false;Quoted 9 lines
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 199
Quoted 6 lines
nbd/client-connection.c +0
-1@@ -172,7 +196,6 @@ static void *connect_thread_func(void *opaque) do_free = conn->detached; } - if (do_free) { nbd_client_connection_do_free(conn);Spurious hunk?
wull drop
Signature · 2 lines
Best regards, Vladimir
PATCH v3 18/33 nbd/client-connection: shutdown connection on release
In reply to Vladimir Sementsov-OgievskiyNow, when thread can do negotiation and retry, it may run relatively long. We need a mechanism to stop it, when user is not interested in result anymore. So, on nbd_client_connection_release() let's shutdown the socked, and do not retry connection if thread is detached.
--- nbd/client-connection.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-)
nbd/client-connection.c +26 -10
@@ -158,9 +158,13 @@ static void *connect_thread_func(void *opaque)uint64_t timeout =1 ;uint64_t max_timeout =16 ; -while (true) { +qemu_mutex_lock (&conn->mutex); +while (!conn->detached) { +assert (!conn->sioc); conn->sioc =qio_channel_socket_new (); +qemu_mutex_unlock (&conn->mutex); +error_free (conn->err); conn->err =NULL ; conn->updated_info = conn->initial_info;
@@ -171,14 +175,20 @@ static void *connect_thread_func(void *opaque) conn->updated_info.x_dirty_bitmap =NULL ; conn->updated_info.name =NULL ; +qemu_mutex_lock (&conn->mutex); +if (ret <0 ) {object_unref (OBJECT (conn->sioc)); conn->sioc =NULL ; -if (conn->do_retry) { +if (conn->do_retry && !conn->detached) { +qemu_mutex_unlock (&conn->mutex); +sleep (timeout);if (timeout < max_timeout) { timeout *=2 ; } + +qemu_mutex_lock (&conn->mutex);continue ; } }
@@ -186,15 +196,17 @@ static void *connect_thread_func(void *opaque)break ; } - WITH_QEMU_LOCK_GUARD(&conn->mutex) { -assert (conn->running); - conn->running = false; -if (conn->wait_co) { -aio_co_schedule (NULL , conn->wait_co); - conn->wait_co =NULL ; - } - do_free = conn->detached; +/* mutex is locked */ + +assert (conn->running); + conn->running = false; +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; } + do_free = conn->detached; + +qemu_mutex_unlock (&conn->mutex);if (do_free) {nbd_client_connection_do_free (conn);
@@ -215,6 +227,10 @@ void nbd_client_connection_release(NBDClientConnection *conn)if (conn->running) { conn->detached = true; } +if (conn->sioc) { +qio_channel_shutdown (QIO_CHANNEL (conn->sioc), +QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); + } do_free = !conn->running && !conn->detached;qemu_mutex_unlock (&conn->mutex);
Signature · 1 line
2.29.2
Re: [PATCH v3 18/33] nbd/client-connection: shutdown connection on release
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
Now, when thread can do negotiation and retry, it may run relatively long. We need a mechanism to stop it, when user is not interested in result anymore. So, on
nbd_client_connection_release()let's shutdown the socked, and do not retry connection if thread is detached.
This kinda answers my question to the previous patch about reconnect cancellation.
Quoted 59 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client-connection.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-)nbd/client-connection.c +13
-10@@ -158,9 +158,13 @@ static void *connect_thread_func(void *opaque)uint64_t timeout =1 ;uint64_t max_timeout =16 ; -while (true) { +qemu_mutex_lock (&conn->mutex); +while (!conn->detached) { +assert (!conn->sioc); conn->sioc =qio_channel_socket_new (); +qemu_mutex_unlock (&conn->mutex); +error_free (conn->err); conn->err =NULL ; conn->updated_info = conn->initial_info;@@ -171,14 +175,20 @@ static void *connect_thread_func(void *opaque) conn->updated_info.x_dirty_bitmap =NULL ; conn->updated_info.name =NULL ; +qemu_mutex_lock (&conn->mutex); +if (ret <0 ) {object_unref (OBJECT (conn->sioc)); conn->sioc =NULL ; -if (conn->do_retry) { +if (conn->do_retry && !conn->detached) { +qemu_mutex_unlock (&conn->mutex); +sleep (timeout);if (timeout < max_timeout) { timeout *=2 ; } + +qemu_mutex_lock (&conn->mutex);continue ; } }@@ -186,15 +196,17 @@ static void *connect_thread_func(void *opaque)break ; } - WITH_QEMU_LOCK_GUARD(&conn->mutex) { -assert (conn->running); - conn->running = false; -if (conn->wait_co) { -aio_co_schedule (NULL , conn->wait_co); - conn->wait_co =NULL ; - } - do_free = conn->detached; +/* mutex is locked */
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 209
Quoted 10 lines
nbd/client-connection.c +9
-0+ +assert (conn->running); + conn->running = false; +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ; } + do_free = conn->detached; + +qemu_mutex_unlock (&conn->mutex);
This basically reverts some hunks from patch 15 "nbd/client-connection:
use QEMU_LOCK_GUARD". How about dropping them there (they weren't an obvious improvement even then).
Roman.
Quoted 17 lines
nbd/client-connection.c +4
-0if (do_free) {nbd_client_connection_do_free (conn);@@ -215,6 +227,10 @@ void nbd_client_connection_release(NBDClientConnection *conn)if (conn->running) { conn->detached = true; } +if (conn->sioc) { +qio_channel_shutdown (QIO_CHANNEL (conn->sioc), +QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); + } do_free = !conn->running && !conn->detached;qemu_mutex_unlock (&conn->mutex);Signature · 1 line
2.29.2
Re: [PATCH v3 18/33] nbd/client-connection: shutdown connection on release
Quoted 74 lines · Roman Kagan
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
Now, when thread can do negotiation and retry, it may run relatively long. We need a mechanism to stop it, when user is not interested in result anymore. So, on
nbd_client_connection_release()let's shutdown the socked, and do not retry connection if thread is detached.This kinda answers my question to the previous patch about reconnect cancellation.
Quoted 65 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client-connection.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-)nbd/client-connection.c +19
-10@@ -158,9 +158,13 @@ static void *connect_thread_func(void *opaque)uint64_t timeout =1 ;uint64_t max_timeout =16 ; -while (true) { +qemu_mutex_lock (&conn->mutex); +while (!conn->detached) { +assert (!conn->sioc); conn->sioc =qio_channel_socket_new (); +qemu_mutex_unlock (&conn->mutex); +error_free (conn->err); conn->err =NULL ; conn->updated_info = conn->initial_info;@@ -171,14 +175,20 @@ static void *connect_thread_func(void *opaque) conn->updated_info.x_dirty_bitmap =NULL ; conn->updated_info.name =NULL ; +qemu_mutex_lock (&conn->mutex); +if (ret <0 ) {object_unref (OBJECT (conn->sioc)); conn->sioc =NULL ; -if (conn->do_retry) { +if (conn->do_retry && !conn->detached) { +qemu_mutex_unlock (&conn->mutex); +sleep (timeout);if (timeout < max_timeout) { timeout *=2 ; } + +qemu_mutex_lock (&conn->mutex);continue ; } }@@ -186,15 +196,17 @@ static void *connect_thread_func(void *opaque)break ; } - WITH_QEMU_LOCK_GUARD(&conn->mutex) { -assert (conn->running); - conn->running = false; -if (conn->wait_co) { -aio_co_schedule (NULL , conn->wait_co); - conn->wait_co =NULL ; - } - do_free = conn->detached; +/* mutex is locked */ + +assert (conn->running); + conn->running = false; +if (conn->wait_co) { +aio_co_schedule (NULL , conn->wait_co); + conn->wait_co =NULL ;
Quoted 9 lines
nbd/client-connection.c static void *connect_thread_func(void *opaque) · line 209
Quoted 4 lines
nbd/client-connection.c +3
-0} + do_free = conn->detached; + +qemu_mutex_unlock (&conn->mutex);This basically reverts some hunks from patch 15 "nbd/client-connection:
useQEMU_LOCK_GUARD". How about dropping them there (they weren't an obvious improvement even then).
OK, will do
Quoted 18 lines
Quoted 18 lines
nbd/client-connection.c +4
-0if (do_free) {nbd_client_connection_do_free (conn);@@ -215,6 +227,10 @@ void nbd_client_connection_release(NBDClientConnection *conn)if (conn->running) { conn->detached = true; } +if (conn->sioc) { +qio_channel_shutdown (QIO_CHANNEL (conn->sioc), +QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); + } do_free = !conn->running && !conn->detached;qemu_mutex_unlock (&conn->mutex);Signature · 1 line
2.29.2
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 18/33] nbd/client-connection: shutdown connection on release
In reply to Vladimir Sementsov-OgievskiyQuoted 1 line · Vladimir Sementsov-Ogievskiy
Now, when thread can do negotiation and retry, it may run relatively
when a thread
Quoted 1 line
long. We need a mechanism to stop it, when user is not interested in
when the user
Quoted 1 line
result anymore. So, on
nbd_client_connection_release()let's shutdown
in a result any more.
Quoted 1 line
the socked, and do not retry connection if thread is detached.
socket
Quoted 6 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client-connection.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 19/33 block/nbd: split nbd_handle_updated_info out of nbd_client_handshake()
In reply to Vladimir Sementsov-OgievskiyTo be reused in the following patch.
--- block/nbd.c | 99 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 57 insertions(+), 42 deletions(-)
block/nbd.c +57 -42
@@ -318,6 +318,50 @@ static bool nbd_client_connecting_wait(BDRVNBDState *s)return qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ; } +/* +* Check s->info updated by negotiation process. +* Update @bs correspondingly to new options. +*/ +static int nbd_handle_updated_info (BlockDriverState *bs,Error **errp) +{ +BDRVNBDState *s = (BDRVNBDState *)bs->opaque; +int ret; + +if (s->x_dirty_bitmap) { +if (!s->info.base_allocation) { +error_setg (errp,"requested x-dirty-bitmap %s not found" , + s->x_dirty_bitmap); +return -EINVAL ; + } +if (strcmp (s->x_dirty_bitmap,"qemu:allocation-depth" ) ==0 ) { + s->alloc_depth = true; + } + } + +if (s->info.flags &NBD_FLAG_READ_ONLY ) { + ret =bdrv_apply_auto_read_only (bs,"NBD export is read-only" , errp); +if (ret <0 ) { +return ret; + } + } + +if (s->info.flags &NBD_FLAG_SEND_FUA ) { + bs->supported_write_flags =BDRV_REQ_FUA ; + bs->supported_zero_flags |=BDRV_REQ_FUA ; + } + +if (s->info.flags &NBD_FLAG_SEND_WRITE_ZEROES ) { + bs->supported_zero_flags |=BDRV_REQ_MAY_UNMAP ; +if (s->info.flags &NBD_FLAG_SEND_FAST_ZERO ) { + bs->supported_zero_flags |=BDRV_REQ_NO_FALLBACK ; + } + } + +trace_nbd_client_handshake_success (s->export); + +return 0 ; +} + static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {int ret;
@@ -1579,49 +1623,13 @@ static int nbd_client_handshake(BlockDriverState *bs, Error **errp) s->sioc =NULL ;return ret; } -if (s->x_dirty_bitmap) { -if (!s->info.base_allocation) { -error_setg (errp,"requested x-dirty-bitmap %s not found" , - s->x_dirty_bitmap); - ret = -EINVAL ; - goto fail; - } -if (strcmp (s->x_dirty_bitmap,"qemu:allocation-depth" ) ==0 ) { - s->alloc_depth = true; - } - } -if (s->info.flags &NBD_FLAG_READ_ONLY ) { - ret =bdrv_apply_auto_read_only (bs,"NBD export is read-only" , errp); -if (ret <0 ) { - goto fail; - } - } -if (s->info.flags &NBD_FLAG_SEND_FUA ) { - bs->supported_write_flags =BDRV_REQ_FUA ; - bs->supported_zero_flags |=BDRV_REQ_FUA ; - } -if (s->info.flags &NBD_FLAG_SEND_WRITE_ZEROES ) { - bs->supported_zero_flags |=BDRV_REQ_MAY_UNMAP ; -if (s->info.flags &NBD_FLAG_SEND_FAST_ZERO ) { - bs->supported_zero_flags |=BDRV_REQ_NO_FALLBACK ; - } - } -if (!s->ioc) { - s->ioc =QIO_CHANNEL (s->sioc); -object_ref (OBJECT (s->ioc)); - } - -trace_nbd_client_handshake_success (s->export); - -return 0 ; - - fail: -/* -* We have connected, but must fail for other reasons. -* Send NBD_CMD_DISC as a courtesy to the server. -*/ - { + ret =nbd_handle_updated_info (bs, errp); +if (ret <0 ) { +/* +* We have connected, but must fail for other reasons. +* Send NBD_CMD_DISC as a courtesy to the server. +*/ NBDRequest request = { .type =NBD_CMD_DISC };nbd_send_request (s->ioc ?:QIO_CHANNEL (s->sioc), &request);
@@ -1635,6 +1643,13 @@ static int nbd_client_handshake(BlockDriverState *bs, Error **errp)return ret; } + +if (!s->ioc) { + s->ioc =QIO_CHANNEL (s->sioc); +object_ref (OBJECT (s->ioc)); + } + +return 0 ; }/*
Signature · 1 line
2.29.2
Re: [PATCH v3 19/33] block/nbd: split nbd_handle_updated_info out of nbd_client_handshake()
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
To be reused in the following patch.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 99 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 57 insertions(+), 42 deletions(-)
Re: [PATCH v3 19/33] block/nbd: split nbd_handle_updated_info out of nbd_client_handshake()
In reply to Vladimir Sementsov-OgievskiyQuoted 12 lines · Vladimir Sementsov-Ogievskiy
To be reused in the following patch.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 99 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 57 insertions(+), 42 deletions(-)block/nbd.c +0
-0
block/nbd.c static bool nbd_client_connecting_wait(BDRVNBDState *s) · line 322
Quoted 5 lines
block/nbd.c +2
-0@@ -318,6 +318,50 @@ static bool nbd_client_connecting_wait(BDRVNBDState *s)return qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ; } +/* +* Check s->info updated by negotiation process.
The parameter name is bs, not s; so this comment is a bit confusing...
block/nbd.c static bool nbd_client_connecting_wait(BDRVNBDState *s) · line 327
Quoted 5 lines
block/nbd.c +5
-0+* Update @bs correspondingly to new options. +*/ +static int nbd_handle_updated_info (BlockDriverState *bs,Error **errp) +{ +BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
...until here. Maybe rewrite the entire comment as:
Update @bs with information learned during a completed negotiation process. Return failure if the server's advertised options are incompatible with the client's needs.
Quoted 13 lines
block/nbd.c +13
-0+int ret; + +if (s->x_dirty_bitmap) { +if (!s->info.base_allocation) { +error_setg (errp,"requested x-dirty-bitmap %s not found" , + s->x_dirty_bitmap); +return -EINVAL ; + } +if (strcmp (s->x_dirty_bitmap,"qemu:allocation-depth" ) ==0 ) { + s->alloc_depth = true; + } + } +
block/nbd.c static bool nbd_client_connecting_wait(BDRVNBDState *s) · line 350
Quoted 10 lines
block/nbd.c +10
-0+if (s->info.flags &NBD_FLAG_READ_ONLY ) { + ret =bdrv_apply_auto_read_only (bs,"NBD export is read-only" , errp); +if (ret <0 ) { +return ret; + } + } + +if (s->info.flags &NBD_FLAG_SEND_FUA ) { + bs->supported_write_flags =BDRV_REQ_FUA ; + bs->supported_zero_flags |=BDRV_REQ_FUA ;
Code motion, so it is correct, but it looks odd to use = for one assignment and |= for the other. Using |= in both places would be more consistent.
block/nbd.c static int nbd_client_handshake(BlockDriverState *bs, Error **errp)
Quoted 18 lines
block/nbd.c +14
-0+ } + +if (s->info.flags &NBD_FLAG_SEND_WRITE_ZEROES ) { + bs->supported_zero_flags |=BDRV_REQ_MAY_UNMAP ; +if (s->info.flags &NBD_FLAG_SEND_FAST_ZERO ) { + bs->supported_zero_flags |=BDRV_REQ_NO_FALLBACK ; + } + } + +trace_nbd_client_handshake_success (s->export); + +return 0 ; +} + static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {int ret;@@ -1579,49 +1623,13 @@ static int nbd_client_handshake(BlockDriverState *bs, Error **errp)
As updating the comment doesn't affect code correctness,
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 19/33] block/nbd: split nbd_handle_updated_info out of nbd_client_handshake()
Quoted 52 lines · Eric Blake
Quoted 12 lines · Vladimir Sementsov-Ogievskiy
To be reused in the following patch.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 99 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 57 insertions(+), 42 deletions(-)block/nbd.c +0
-0block/nbd.c static bool nbd_client_connecting_wait(BDRVNBDState *s) · line 322
Quoted 5 lines
block/nbd.c +2
-0@@ -318,6 +318,50 @@ static bool nbd_client_connecting_wait(BDRVNBDState *s)return qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ; } +/* +* Check s->info updated by negotiation process. The parameter name is bs, not s; so this comment is a bit confusing...
block/nbd.c static bool nbd_client_connecting_wait(BDRVNBDState *s) · line 327
Quoted 5 lines
block/nbd.c +5
-0+* Update @bs correspondingly to new options. +*/ +static int nbd_handle_updated_info (BlockDriverState *bs,Error **errp) +{ +BDRVNBDState *s = (BDRVNBDState *)bs->opaque;...until here. Maybe rewrite the entire comment as:
Update @bs with information learned during a completed negotiation process. Return failure if the server's advertised options are incompatible with the client's needs.
Quoted 19 lines
block/nbd.c +19
-0+int ret; + +if (s->x_dirty_bitmap) { +if (!s->info.base_allocation) { +error_setg (errp,"requested x-dirty-bitmap %s not found" , + s->x_dirty_bitmap); +return -EINVAL ; + } +if (strcmp (s->x_dirty_bitmap,"qemu:allocation-depth" ) ==0 ) { + s->alloc_depth = true; + } + } + +if (s->info.flags &NBD_FLAG_READ_ONLY ) { + ret =bdrv_apply_auto_read_only (bs,"NBD export is read-only" , errp); +if (ret <0 ) { +return ret; + } + }
Quoted 8 lines
block/nbd.c static bool nbd_client_connecting_wait(BDRVNBDState *s) · line 350
Quoted 4 lines
block/nbd.c +4
-0+ +if (s->info.flags &NBD_FLAG_SEND_FUA ) { + bs->supported_write_flags =BDRV_REQ_FUA ; + bs->supported_zero_flags |=BDRV_REQ_FUA ;Code motion, so it is correct, but it looks odd to use = for one assignment and |= for the other. Using |= in both places would be more consistent.
Actually I see bugs here:
1. we should do =, not |=, as on reconnect info changes, so we should reset supported flags.
2. in-fligth requests that are in retying loops are not prepared to flags changing. I afraid, that some malicious server may even do some bad thing
Still, let's fix it after these series. To avoid more conflicts.
Quoted 23 lines
block/nbd.c static int nbd_client_handshake(BlockDriverState *bs, Error **errp)
Quoted 19 lines
block/nbd.c +14
-0+ } + +if (s->info.flags &NBD_FLAG_SEND_WRITE_ZEROES ) { + bs->supported_zero_flags |=BDRV_REQ_MAY_UNMAP ; +if (s->info.flags &NBD_FLAG_SEND_FAST_ZERO ) { + bs->supported_zero_flags |=BDRV_REQ_NO_FALLBACK ; + } + } + +trace_nbd_client_handshake_success (s->export); + +return 0 ; +} + static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {int ret;@@ -1579,49 +1623,13 @@ static int nbd_client_handshake(BlockDriverState *bs, Error **errp)As updating the comment doesn't affect code correctness,
Reviewed-by Eric Blake <[email protected]>
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 19/33] block/nbd: split nbd_handle_updated_info out of nbd_client_handshake()
Quoted 26 lines · Vladimir Sementsov-Ogievskiy
Quoted 17 lines
Quoted 17 lines
block/nbd.c +17
-0+if (s->x_dirty_bitmap) { +if (!s->info.base_allocation) { +error_setg (errp,"requested x-dirty-bitmap %s not found" , + s->x_dirty_bitmap); +return -EINVAL ; + } +if (strcmp (s->x_dirty_bitmap,"qemu:allocation-depth" ) ==0 ) { + s->alloc_depth = true; + } + } + +if (s->info.flags &NBD_FLAG_READ_ONLY ) { + ret =bdrv_apply_auto_read_only (bs,"NBD export is read-only" , errp); +if (ret <0 ) { +return ret; + } + }Quoted 8 lines
block/nbd.c static bool nbd_client_connecting_wait(BDRVNBDState *s) · line 350
Quoted 4 lines
block/nbd.c +4
-0+ +if (s->info.flags &NBD_FLAG_SEND_FUA ) { + bs->supported_write_flags =BDRV_REQ_FUA ; + bs->supported_zero_flags |=BDRV_REQ_FUA ;Code motion, so it is correct, but it looks odd to use = for one assignment and |= for the other. Using |= in both places would be more consistent.
Quoted 7 lines
Actually I see bugs here:
1. we should do =, not |=, as on reconnect info changes, so we should reset supported flags.
2. in-fligth requests that are in retying loops are not prepared to flags changing. I afraid, that some malicious server may even do some bad thing
Still, let's fix it after these series. To avoid more conflicts.
Oh, you raise some good points. And it's not just bs->*flags; qemu as server uses constant metacontext ids (base:allocation is always context 0), but even that might not be stable across reconnect. For example, with my proposed patch of adding qemu:joint-allocation metacontext, if the reason we have to reconnect is because the server is upgrading from qemu 6.0 to 6.1 temporarily bouncing the server, and the client was paying attention to qemu:dirty-bitmap:FOO, that context would now have a different id.
Yeah, making this code safer across potential changes in server information (either to fail the reconnect because the reconnected server dropped something we were previously depending on, or gracefully handling the downgrade, or ...) is worth leaving for a later series while we focus on the more immediate issue of making reconnect itself stable.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 20/33 block/nbd: use negotiation of NBDClientConnection
In reply to Vladimir Sementsov-OgievskiyUse a new possibility of negotiation in connect thread. Now we are on the way of simplifying connection_co. We want to move the whole reconnect code to NBDClientConnection. NBDClientConnection already updated to support negotiation and retry. Use now the first thing.
--- block/nbd.c | 44 ++++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 14 deletions(-)
block/nbd.c +30 -14
@@ -365,6 +365,7 @@ static int nbd_handle_updated_info(BlockDriverState *bs, Error **errp) static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {int ret; +AioContext *aio_context =bdrv_get_aio_context (s->bs);if (!nbd_client_connecting (s)) {return ;
@@ -405,30 +406,44 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) s->ioc =NULL ; } - s->sioc =nbd_co_establish_connection (s->conn,NULL ,NULL ,NULL ); + s->sioc =nbd_co_establish_connection (s->conn, &s->info, &s->ioc,NULL );if (!s->sioc) { ret = -ECONNREFUSED ; goto out; } +qio_channel_set_blocking (QIO_CHANNEL (s->sioc), false,NULL ); +qio_channel_attach_aio_context (QIO_CHANNEL (s->sioc), aio_context); +if (s->ioc) { +qio_channel_set_blocking (QIO_CHANNEL (s->ioc), false,NULL ); +qio_channel_attach_aio_context (QIO_CHANNEL (s->ioc), aio_context); + }else { + s->ioc =QIO_CHANNEL (s->sioc); +object_ref (OBJECT (s->ioc)); + } +yank_register_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), nbd_yank, s->bs); -bdrv_dec_in_flight (s->bs); + ret =nbd_handle_updated_info (s->bs,NULL ); +if (ret <0 ) { +/* +* We have connected, but must fail for other reasons. +* Send NBD_CMD_DISC as a courtesy to the server. +*/ +NBDRequest request = { .type =NBD_CMD_DISC }; - ret =nbd_client_handshake (s->bs,NULL ); +nbd_send_request (s->ioc, &request); -if (s->drained) { - s->wait_drained_end = true; -while (s->drained) { -/* -* We may be entered once from nbd_client_attach_aio_context_bh -* and then from nbd_client_co_drain_end. So here is a loop. -*/ -qemu_coroutine_yield (); - } +yank_unregister_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), + nbd_yank, s->bs); +object_unref (OBJECT (s->sioc)); + s->sioc =NULL ; +object_unref (OBJECT (s->ioc)); + s->ioc =NULL ; + +return ; } -bdrv_inc_in_flight (s->bs); out:if (ret >=0 ) {
@@ -2048,7 +2063,8 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } - s->conn =nbd_client_connection_new (s->saddr, false,NULL ,NULL ,NULL ); + s->conn =nbd_client_connection_new (s->saddr, true, s->export, + s->x_dirty_bitmap, s->tlscreds);/* * establish TCP connection, return error if it fails
Signature · 1 line
2.29.2
Re: [PATCH v3 20/33] block/nbd: use negotiation of NBDClientConnection
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
Use a new possibility of negotiation in connect thread. Now we are on the way of simplifying
connection_co. We want to move the whole reconnect code to NBDClientConnection. NBDClientConnection already updated to support negotiation and retry. Use now the first thing.
Maybe:
Now that we can opt in to negotiation as part of the client connection thread, use that to simplify connection_co. This is another step on the way to moving all reconnect code into NBDClientConnection.
Quoted 6 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 44 ++++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 14 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 21/33 qemu-socket: pass monitor link to socket_get_fd directly
In reply to Vladimir Sementsov-OgievskiyDetecting monitor by current coroutine works bad when we are not in coroutine context. And that's exactly so in nbd reconnect code, where qio_channel_socket_connect_sync() is called from thread.
Add a possibility to pass monitor by hand, to be used in the following commit.
--- include/io/channel-socket.h | 20 ++++++++++++++++++++ include/qemu/sockets.h | 2 +- io/channel-socket.c | 17 +++++++++++++---- tests/unit/test-util-sockets.c | 16 ++++++++-------- util/qemu-sockets.c | 10 +++++----- 5 files changed, 47 insertions(+), 18 deletions(-)
include/io/channel-socket.h +20 -0
@@ -78,6 +78,23 @@ qio_channel_socket_new_fd(int fd, Error **errp); +/** +* qio_channel_socket_connect_sync_mon: +* @ioc: the socket channel object +* @addr: the address to connect to +* @mon: current monitor. If NULL, it will be detected by +* current coroutine. +* @errp: pointer to a NULL-initialized error object +* +* Attempt to connect to the address @addr. This method +* will run in the foreground so the caller will not regain +* execution control until the connection is established or +* an error occurs. +*/ +int qio_channel_socket_connect_sync_mon (QIOChannelSocket *ioc, +SocketAddress *addr, +Monitor *mon, +Error **errp);/** * qio_channel_socket_connect_sync: * @ioc: the socket channel object
@@ -88,6 +105,9 @@ qio_channel_socket_new_fd(int fd,* will run in the foreground so the caller will not regain * execution control until the connection is established or * an error occurs. +* +* This a wrapper, calling qio_channel_socket_connect_sync_mon() +* with @mon=NULL. */ int qio_channel_socket_connect_sync(QIOChannelSocket *ioc, SocketAddress *addr,
include/qemu/sockets.h +1 -1
@@ -41,7 +41,7 @@ int unix_listen(const char *path, Error **errp);int unix_connect (const char *path,Error **errp);SocketAddress *socket_parse (const char *str,Error **errp); -int socket_connect (SocketAddress *addr,Error **errp); +int socket_connect (SocketAddress *addr,Monitor *mon,Error **errp);int socket_listen (SocketAddress *addr,int num,Error **errp);void socket_listen_cleanup (int fd,Error **errp);int socket_dgram (SocketAddress *remote,SocketAddress *local,Error **errp);
io/channel-socket.c +13 -4
@@ -135,14 +135,15 @@ qio_channel_socket_new_fd(int fd, } -int qio_channel_socket_connect_sync (QIOChannelSocket *ioc, -SocketAddress *addr, -Error **errp) +int qio_channel_socket_connect_sync_mon (QIOChannelSocket *ioc, +SocketAddress *addr, +Monitor *mon, +Error **errp) {int fd;trace_qio_channel_socket_connect_sync (ioc, addr); - fd =socket_connect (addr, errp); + fd =socket_connect (addr, mon, errp);if (fd <0 ) {trace_qio_channel_socket_connect_fail (ioc);return -1 ;
@@ -158,6 +159,14 @@ int qio_channel_socket_connect_sync(QIOChannelSocket *ioc, } +int qio_channel_socket_connect_sync (QIOChannelSocket *ioc, +SocketAddress *addr, +Error **errp) +{ +return qio_channel_socket_connect_sync_mon (ioc, addr,NULL , errp); +} + +static void qio_channel_socket_connect_worker (QIOTask *task,gpointer opaque) {
tests/unit/test-util-sockets.c +8 -8
@@ -90,7 +90,7 @@ static void test_socket_fd_pass_name_good(void) addr.type =SOCKET_ADDRESS_TYPE_FD ; addr.u.fd.str =g_strdup (mon_fdname); - fd =socket_connect (&addr, &error_abort); + fd =socket_connect (&addr,NULL , &error_abort); g_assert_cmpint(fd, !=, -1); g_assert_cmpint(fd, !=, mon_fd);close (fd);
@@ -122,7 +122,7 @@ static void test_socket_fd_pass_name_bad(void) addr.type =SOCKET_ADDRESS_TYPE_FD ; addr.u.fd.str =g_strdup (mon_fdname); - fd =socket_connect (&addr, &err); + fd =socket_connect (&addr,NULL , &err); g_assert_cmpint(fd, ==, -1);error_free_or_abort (&err);
@@ -149,7 +149,7 @@ static void test_socket_fd_pass_name_nomon(void) addr.type =SOCKET_ADDRESS_TYPE_FD ; addr.u.fd.str =g_strdup ("myfd" ); - fd =socket_connect (&addr, &err); + fd =socket_connect (&addr,NULL , &err); g_assert_cmpint(fd, ==, -1);error_free_or_abort (&err);
@@ -173,7 +173,7 @@ static void test_socket_fd_pass_num_good(void) addr.type =SOCKET_ADDRESS_TYPE_FD ; addr.u.fd.str =g_strdup_printf ("%d" , sfd); - fd =socket_connect (&addr, &error_abort); + fd =socket_connect (&addr,NULL , &error_abort); g_assert_cmpint(fd, ==, sfd); fd =socket_listen (&addr,1 , &error_abort);
@@ -195,7 +195,7 @@ static void test_socket_fd_pass_num_bad(void) addr.type =SOCKET_ADDRESS_TYPE_FD ; addr.u.fd.str =g_strdup_printf ("%d" , sfd); - fd =socket_connect (&addr, &err); + fd =socket_connect (&addr,NULL , &err); g_assert_cmpint(fd, ==, -1);error_free_or_abort (&err);
@@ -218,7 +218,7 @@ static void test_socket_fd_pass_num_nocli(void) addr.type =SOCKET_ADDRESS_TYPE_FD ; addr.u.fd.str =g_strdup_printf ("%d" ,STDOUT_FILENO ); - fd =socket_connect (&addr, &err); + fd =socket_connect (&addr,NULL , &err); g_assert_cmpint(fd, ==, -1);error_free_or_abort (&err);
@@ -247,10 +247,10 @@ static gpointer unix_client_thread_func(gpointer user_data)for (i =0 ; i <ABSTRACT_SOCKET_VARIANTS ; i++) {if (row->expect_connect[i]) { - fd =socket_connect (row->client[i], &error_abort); + fd =socket_connect (row->client[i],NULL , &error_abort); g_assert_cmpint(fd, >=, 0); }else { - fd =socket_connect (row->client[i], &err); + fd =socket_connect (row->client[i],NULL , &err); g_assert_cmpint(fd, ==, -1);error_free_or_abort (&err); }
util/qemu-sockets.c +5 -5
@@ -1116,9 +1116,9 @@ fail:return NULL ; } -static int socket_get_fd (const char *fdstr,int num,Error **errp) +static int socket_get_fd (const char *fdstr,int num,Monitor *mon,Error **errp) { -Monitor *cur_mon =monitor_cur (); +Monitor *cur_mon = mon ?:monitor_cur ();int fd;if (num !=1 ) {error_setg_errno (errp,EINVAL ,"socket_get_fd: too many connections" );
@@ -1145,7 +1145,7 @@ static int socket_get_fd(const char *fdstr, int num, Error **errp)return fd; } -int socket_connect (SocketAddress *addr,Error **errp) +int socket_connect (SocketAddress *addr,Monitor *mon,Error **errp) {int fd;
@@ -1159,7 +1159,7 @@ int socket_connect(SocketAddress *addr, Error **errp)break ;case SOCKET_ADDRESS_TYPE_FD : - fd =socket_get_fd (addr->u.fd.str,1 , errp); + fd =socket_get_fd (addr->u.fd.str,1 , mon, errp);break ;case SOCKET_ADDRESS_TYPE_VSOCK :
@@ -1187,7 +1187,7 @@ int socket_listen(SocketAddress *addr, int num, Error **errp)break ;case SOCKET_ADDRESS_TYPE_FD : - fd =socket_get_fd (addr->u.fd.str, num, errp); + fd =socket_get_fd (addr->u.fd.str, num,NULL , errp);break ;case SOCKET_ADDRESS_TYPE_VSOCK :
Signature · 1 line
2.29.2
Re: [PATCH v3 21/33] qemu-socket: pass monitor link to socket_get_fd directly
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Detecting monitor by current coroutine works bad when we are not in coroutine context. And that's exactly so in nbd reconnect code, where
qio_channel_socket_connect_sync()is called from thread.Add a possibility to pass monitor by hand, to be used in the following commit.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/io/channel-socket.h | 20 ++++++++++++++++++++ include/qemu/sockets.h | 2 +- io/channel-socket.c | 17 +++++++++++++---- tests/unit/test-util-sockets.c | 16 ++++++++-------- util/qemu-sockets.c | 10 +++++----- 5 files changed, 47 insertions(+), 18 deletions(-)include/io/channel-socket.h +7
-0@@ -78,6 +78,23 @@ qio_channel_socket_new_fd(int fd, Error **errp); +/** +* qio_channel_socket_connect_sync_mon: +* @ioc: the socket channel object +* @addr: the address to connect to +* @mon: current monitor. If NULL, it will be detected by +* current coroutine. +* @errp: pointer to a NULL-initialized error object
include/io/channel-socket.h qio_channel_socket_new_fd(int fd, · line 97
Quoted 10 lines
include/io/channel-socket.h +10
-0+* +* Attempt to connect to the address @addr. This method +* will run in the foreground so the caller will not regain +* execution control until the connection is established or +* an error occurs. +*/ +int qio_channel_socket_connect_sync_mon (QIOChannelSocket *ioc, +SocketAddress *addr, +Monitor *mon, +Error **errp);
I don't really like exposing the concept of the QEMU monitor in the IO layer APIs. IMHO these ought to remain completely separate subsystems from the API pov, and we ought to fix this problem by making monitor_cur() work better in the scenario required.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v3 21/33] qemu-socket: pass monitor link to socket_get_fd directly
Quoted 39 lines · Daniel P. Berrangé
Quoted 38 lines · Vladimir Sementsov-Ogievskiy
Detecting monitor by current coroutine works bad when we are not in coroutine context. And that's exactly so in nbd reconnect code, where
qio_channel_socket_connect_sync()is called from thread.Add a possibility to pass monitor by hand, to be used in the following commit.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/io/channel-socket.h | 20 ++++++++++++++++++++ include/qemu/sockets.h | 2 +- io/channel-socket.c | 17 +++++++++++++---- tests/unit/test-util-sockets.c | 16 ++++++++-------- util/qemu-sockets.c | 10 +++++----- 5 files changed, 47 insertions(+), 18 deletions(-)include/io/channel-socket.h +14
-0@@ -78,6 +78,23 @@ qio_channel_socket_new_fd(int fd, Error **errp); +/** +* qio_channel_socket_connect_sync_mon: +* @ioc: the socket channel object +* @addr: the address to connect to +* @mon: current monitor. If NULL, it will be detected by +* current coroutine. +* @errp: pointer to a NULL-initialized error object +* +* Attempt to connect to the address @addr. This method +* will run in the foreground so the caller will not regain +* execution control until the connection is established or +* an error occurs. +*/ +int qio_channel_socket_connect_sync_mon(QIOChannelSocket *ioc,
Quoted 9 lines
include/io/channel-socket.h qio_channel_socket_new_fd(int fd, · line 97
Quoted 3 lines
include/io/channel-socket.h +3
-0+ SocketAddress *addr, + Monitor *mon, + Error **errp);I don't really like exposing the concept of the QEMU monitor in the IO layer APIs. IMHO these ought to remain completely separate subsystems from the API pov, and we ought to fix this problem by making
monitor_cur()work better in the scenario required.
Hmm..
I can add thread_mon hash-table to monitor/monitor.c, so we can set monitor for current thread in same way like for coroutine. And monitor_cur will look first at coroutine->monitor hash map and then to thread->monitor. Then, I'll pass needed monitor link to my specific thread, and thread will call monitor_set_cur_for_thread(), an then qio_channel_socket_connect_sync() will work correctly.
David, Markus, is it OK?
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 21/33] qemu-socket: pass monitor link to socket_get_fd directly
In reply to Daniel P. BerrangéQuoted 38 lines · Daniel P. Berrangé
Quoted 37 lines · Vladimir Sementsov-Ogievskiy
Detecting monitor by current coroutine works bad when we are not in coroutine context. And that's exactly so in nbd reconnect code, where
qio_channel_socket_connect_sync()is called from thread.Add a possibility to pass monitor by hand, to be used in the following commit.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/io/channel-socket.h | 20 ++++++++++++++++++++ include/qemu/sockets.h | 2 +- io/channel-socket.c | 17 +++++++++++++---- tests/unit/test-util-sockets.c | 16 ++++++++-------- util/qemu-sockets.c | 10 +++++----- 5 files changed, 47 insertions(+), 18 deletions(-)include/io/channel-socket.h +13
-0@@ -78,6 +78,23 @@ qio_channel_socket_new_fd(int fd, Error **errp); +/** +* qio_channel_socket_connect_sync_mon: +* @ioc: the socket channel object +* @addr: the address to connect to +* @mon: current monitor. If NULL, it will be detected by +* current coroutine. +* @errp: pointer to a NULL-initialized error object +* +* Attempt to connect to the address @addr. This method +* will run in the foreground so the caller will not regain +* execution control until the connection is established or +* an error occurs. +*/
Quoted 8 lines
include/io/channel-socket.h qio_channel_socket_new_fd(int fd, · line 97
Quoted 4 lines
include/io/channel-socket.h +4
-0+int qio_channel_socket_connect_sync_mon (QIOChannelSocket *ioc, +SocketAddress *addr, +Monitor *mon, +Error **errp);I don't really like exposing the concept of the QEMU monitor in the IO layer APIs. IMHO these ought to remain completely separate subsystems from the API pov,
Agreed.
Quoted 2 lines
and we ought to fix this problem by
makingmonitor_cur()work better in the scenario required.
Would it make sense instead to resolve the fdstr into actual file descriptor number in the context where monitor_cur() works and makes sense, prior to passing it to the connection thread?
Roman.
Re: [PATCH v3 21/33] qemu-socket: pass monitor link to socket_get_fd directly
Quoted 48 lines · Roman Kagan
Quoted 38 lines · Daniel P. Berrangé
Quoted 37 lines · Vladimir Sementsov-Ogievskiy
Detecting monitor by current coroutine works bad when we are not in coroutine context. And that's exactly so in nbd reconnect code, where
qio_channel_socket_connect_sync()is called from thread.Add a possibility to pass monitor by hand, to be used in the following commit.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/io/channel-socket.h | 20 ++++++++++++++++++++ include/qemu/sockets.h | 2 +- io/channel-socket.c | 17 +++++++++++++---- tests/unit/test-util-sockets.c | 16 ++++++++-------- util/qemu-sockets.c | 10 +++++----- 5 files changed, 47 insertions(+), 18 deletions(-)include/io/channel-socket.h +13
-0@@ -78,6 +78,23 @@ qio_channel_socket_new_fd(int fd, Error **errp); +/** +* qio_channel_socket_connect_sync_mon: +* @ioc: the socket channel object +* @addr: the address to connect to +* @mon: current monitor. If NULL, it will be detected by +* current coroutine. +* @errp: pointer to a NULL-initialized error object +* +* Attempt to connect to the address @addr. This method +* will run in the foreground so the caller will not regain +* execution control until the connection is established or +* an error occurs. +*/ Quoted 8 lines
include/io/channel-socket.h qio_channel_socket_new_fd(int fd, · line 97
Quoted 4 lines
include/io/channel-socket.h +4
-0+int qio_channel_socket_connect_sync_mon (QIOChannelSocket *ioc, +SocketAddress *addr, +Monitor *mon, +Error **errp);I don't really like exposing the concept of the QEMU monitor in the IO layer APIs. IMHO these ought to remain completely separate subsystems from the API pov,
Quoted 8 lines
Agreed.
Quoted 2 lines
and we ought to fix this problem by
makingmonitor_cur()work better in the scenario required.Would it make sense instead to resolve the fdstr into actual file descriptor number in the context where
monitor_cur()works and makes sense, prior to passing it to the connection thread?
Yes, arguably the root problem is caused by the util/qemu-sockets.c code directly referring to the current monitor. This has resultde in the slightly strange scenario where we have two distinct semantics for the 'fd' SocketAddressType
@fd: decimal is for file descriptor number, otherwise a file descriptor name.
Named file descriptors are permitted in monitor commands, in combination with the 'getfd' command. Decimal file descriptors are permitted at startup or other contexts where no monitor context is active.
Now these distinct semantics kinda make sense from the POV of the management application, but we've let the dual semantics propagate all the way down our I/O stack.
Folowing your idea, we could have 'socket_address_resolve_monitor_fd' method which takes a 'SocketAddress' and returns a new 'SocketAddress' with the real FD filled in. We then call this method in any codepath which is getting a 'SocketAddress' struct from the monitor.
The util/qemu-sockets.c code then only has to think about real FDs, and the monitor_get_fd() call only occurs right at the top level.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v3 21/33] qemu-socket: pass monitor link to socket_get_fd directly
Quoted 72 lines · Daniel P. Berrangé
Quoted 48 lines · Roman Kagan
Quoted 38 lines · Daniel P. Berrangé
Quoted 37 lines · Vladimir Sementsov-Ogievskiy
Detecting monitor by current coroutine works bad when we are not in coroutine context. And that's exactly so in nbd reconnect code, where
qio_channel_socket_connect_sync()is called from thread.Add a possibility to pass monitor by hand, to be used in the following commit.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/io/channel-socket.h | 20 ++++++++++++++++++++ include/qemu/sockets.h | 2 +- io/channel-socket.c | 17 +++++++++++++---- tests/unit/test-util-sockets.c | 16 ++++++++-------- util/qemu-sockets.c | 10 +++++----- 5 files changed, 47 insertions(+), 18 deletions(-)include/io/channel-socket.h +13
-0@@ -78,6 +78,23 @@ qio_channel_socket_new_fd(int fd, Error **errp); +/** +* qio_channel_socket_connect_sync_mon: +* @ioc: the socket channel object +* @addr: the address to connect to +* @mon: current monitor. If NULL, it will be detected by +* current coroutine. +* @errp: pointer to a NULL-initialized error object +* +* Attempt to connect to the address @addr. This method +* will run in the foreground so the caller will not regain +* execution control until the connection is established or +* an error occurs. +*/ Quoted 8 lines
include/io/channel-socket.h qio_channel_socket_new_fd(int fd, · line 97
Quoted 4 lines
include/io/channel-socket.h +4
-0+int qio_channel_socket_connect_sync_mon (QIOChannelSocket *ioc, +SocketAddress *addr, +Monitor *mon, +Error **errp);I don't really like exposing the concept of the QEMU monitor in the IO layer APIs. IMHO these ought to remain completely separate subsystems from the API pov,
Quoted 8 lines
Agreed.
Quoted 2 lines
and we ought to fix this problem by
makingmonitor_cur()work better in the scenario required.Would it make sense instead to resolve the fdstr into actual file descriptor number in the context where
monitor_cur()works and makes sense, prior to passing it to the connection thread?Yes, arguably the root problem is caused by the
util/qemu-sockets.ccode directly referring to the current monitor. This has resultde in the slightly strange scenario where we have two distinct semantics for the 'fd' SocketAddressType@fd: decimal is for file descriptor number, otherwise a file descriptor name.
Named file descriptors are permitted in monitor commands, in combination with the 'getfd' command. Decimal file descriptors are permitted at startup or other contexts where no monitor context is active.Now these distinct semantics kinda make sense from the POV of the management application, but we've let the dual semantics propagate all the way down our I/O stack.
Quoted 8 lines
Folowing your idea, we could have '
socket_address_resolve_monitor_fd' method which takes a 'SocketAddress' and returns a new 'SocketAddress' with the real FD filled in. We then call this method in any codepath which is getting a 'SocketAddress' struct from the monitor.The
util/qemu-sockets.ccode then only has to think about real FDs, and themonitor_get_fd()call only occurs right at the top level.
Reasonable, I'll try this way
Signature · 2 lines
Best regards, Vladimir
PATCH v3 22/33 block/nbd: pass monitor directly to connection thread
In reply to Vladimir Sementsov-Ogievskiymonitor_cur() is used by socket_get_fd, but it doesn't work in connection thread. Let's monitor directly to cover this thing. We are going to unify connection establishing path in nbd_open and reconnect, so we should support fd-passing.
--- include/block/nbd.h | 3 ++- block/nbd.c | 5 ++++- nbd/client-connection.c | 11 +++++++---- 3 files changed, 13 insertions(+), 6 deletions(-)
include/block/nbd.h +2 -1
@@ -415,7 +415,8 @@ NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr,bool do_negotiation, const char *export_name, const char *x_dirty_bitmap, - QCryptoTLSCreds *tlscreds); + QCryptoTLSCreds *tlscreds, + Monitor *mon);void nbd_client_connection_release (NBDClientConnection *conn);QIOChannelSocket *coroutine_fn
block/nbd.c +4 -1
@@ -45,6 +45,8 @@#include "block/nbd.h" #include "block/block_int.h" +#include "monitor/monitor.h" +#include "qemu/yank.h" #define EN_OPTSTR ":exportname="
@@ -2064,7 +2066,8 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, } s->conn =nbd_client_connection_new (s->saddr, true, s->export, - s->x_dirty_bitmap, s->tlscreds); + s->x_dirty_bitmap, s->tlscreds, +monitor_cur ());/* * establish TCP connection, return error if it fails
nbd/client-connection.c +7 -4
@@ -37,6 +37,7 @@ struct NBDClientConnection {bool do_negotiation;bool do_retry; +Monitor *mon;/* * Result of last attempt. Valid in FAIL and SUCCESS states.
@@ -67,7 +68,8 @@ NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr,bool do_negotiation, const char *export_name, const char *x_dirty_bitmap, - QCryptoTLSCreds *tlscreds) + QCryptoTLSCreds *tlscreds, + Monitor *mon) { NBDClientConnection *conn = g_new(NBDClientConnection, 1);
@@ -76,6 +78,7 @@ NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr, .saddr = QAPI_CLONE(SocketAddress, saddr), .tlscreds = tlscreds, .do_negotiation = do_negotiation, + .mon = mon, .initial_info.request_sizes = true, .initial_info.structured_reply = true,
@@ -110,7 +113,7 @@ static void nbd_client_connection_do_free(NBDClientConnection *conn)*/ static int nbd_connect (QIOChannelSocket *sioc,SocketAddress *addr,NBDExportInfo *info,QCryptoTLSCreds *tlscreds, -QIOChannel **outioc,Error **errp) +QIOChannel **outioc,Monitor *mon,Error **errp) {int ret;
@@ -118,7 +121,7 @@ static int nbd_connect(QIOChannelSocket *sioc, SocketAddress *addr, *outioc =NULL ; } - ret =qio_channel_socket_connect_sync (sioc, addr, errp); + ret =qio_channel_socket_connect_sync_mon (sioc, addr, mon, errp);if (ret <0 ) {return ret; }
@@ -171,7 +174,7 @@ static void *connect_thread_func(void *opaque) ret =nbd_connect (conn->sioc, conn->saddr, conn->do_negotiation ? &conn->updated_info :NULL , - conn->tlscreds, &conn->ioc, &conn->err); + conn->tlscreds, &conn->ioc, conn->mon, &conn->err); conn->updated_info.x_dirty_bitmap =NULL ; conn->updated_info.name =NULL ;
Signature · 1 line
2.29.2
Re: [PATCH v3 22/33] block/nbd: pass monitor directly to connection thread
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
monitor_cur()is used bysocket_get_fd, but it doesn't work in connection thread. Let's monitor directly to cover this thing. We are going to unify connection establishing path innbd_openand reconnect, so we should support fd-passing.
Grammar suggestion:
Let's pass in the monitor directly to work around this. This gets us closer to unifing the path for establishing a connection in nbd_open and reconnect, by supporting fd-passing.
But given Dan's review on 21/33, I suspect you won't be using this patch in this form after all (instead, the caller of nbd_client_connection_new will use the new monitor_resolve_fd or whatever we call that, so that nbd_client_connection_new remains oblivious to the monitor).
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 22/33] block/nbd: pass monitor directly to connection thread
Quoted 13 lines · Eric Blake
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
monitor_cur()is used bysocket_get_fd, but it doesn't work in connection thread. Let's monitor directly to cover this thing. We are going to unify connection establishing path innbd_openand reconnect, so we should support fd-passing.Grammar suggestion:
Let's pass in the monitor directly to work around this. This gets us closer to unifing the path for establishing a connection in
nbd_openand reconnect, by supporting fd-passing.
Quoted 6 lines
But given Dan's review on 21/33, I suspect you won't be using this patch in this form after all (instead, the caller of
nbd_client_connection_newwill use the newmonitor_resolve_fdor whatever we call that, so thatnbd_client_connection_newremains oblivious to the monitor).
Yes.. I even have some patches for it locally. Seems I didn't send them, don't remember why :/ Will check tomorrow.
Signature · 2 lines
Best regards, Vladimir
PATCH v3 23/33 block/nbd: nbd_teardown_connection() don't touch s->sioc
In reply to Vladimir Sementsov-OgievskiyNegotiation during reconnect is now done in thread, and s->sioc is not available during negotiation. Negotiation in thread will be cancelled by nbd_client_connection_release() called from nbd_clear_bdrvstate(). So, we don't need this code chunk anymore.
--- block/nbd.c | 4 ---- 1 file changed, 4 deletions(-)
block/nbd.c +0 -4
@@ -284,10 +284,6 @@ static void nbd_teardown_connection(BlockDriverState *bs)if (s->ioc) {/* finish any pending coroutines */ qio_channel_shutdown (s->ioc,QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); - }else if (s->sioc) { -/* abort negotiation */ -qio_channel_shutdown (QIO_CHANNEL (s->sioc),QIO_CHANNEL_SHUTDOWN_BOTH , -NULL ); } s->state =NBD_CLIENT_QUIT ;
Signature · 1 line
2.29.2
Re: [PATCH v3 23/33] block/nbd: nbd_teardown_connection() don't touch s->sioc
On Fri, Apr 16, 2021 at 11:09:01AM +0300, Vladimir Sementsov-Ogievskiy wrote:
For the subject line, might read better as:
block/nbd: don't touch s->sioc in nbd_teardown_connection()
Quoted 1 line
Negotiation during reconnect is now done in thread, and
s->siocis not
in a thread
Quoted 8 lines
available during negotiation. Negotiation in thread will be cancelled by
nbd_client_connection_release()called fromnbd_clear_bdrvstate(). So, we don't need this code chunk anymore.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 4 ---- 1 file changed, 4 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 24/33 block/nbd: drop BDRVNBDState::sioc
In reply to Vladimir Sementsov-OgievskiyCurrently sioc pointer is used just to pass from socket-connection to nbd negotiation. Drop the field, and use local variables instead. With next commit we'll update nbd/client-connection.c to behave appropriately (return only top-most ioc, not two channels).
--- block/nbd.c | 98 ++++++++++++++++++++++++++--------------------------- 1 file changed, 48 insertions(+), 50 deletions(-)
block/nbd.c +48 -50
@@ -69,8 +69,7 @@ typedef enum NBDClientState { } NBDClientState; typedef struct BDRVNBDState { - QIOChannelSocket *sioc; /* The master data channel */ - QIOChannel *ioc; /* The current I/O channel which may differ (eg TLS) */ + QIOChannel *ioc; /* The current I/O channel */ NBDExportInfo info; CoMutex send_mutex;
@@ -102,9 +101,11 @@ typedef struct BDRVNBDState {NBDClientConnection *conn; } BDRVNBDState; -static int nbd_establish_connection (BlockDriverState *bs,SocketAddress *saddr, -Error **errp); -static int nbd_client_handshake (BlockDriverState *bs,Error **errp); +static QIOChannelSocket *nbd_establish_connection (BlockDriverState *bs, +SocketAddress *saddr, +Error **errp); +static int nbd_client_handshake (BlockDriverState *bs,QIOChannelSocket *sioc, +Error **errp);static void nbd_yank (void *opaque);static void nbd_clear_bdrvstate (BlockDriverState *bs)
@@ -364,6 +365,7 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {int ret;AioContext *aio_context =bdrv_get_aio_context (s->bs); +QIOChannelSocket *sioc;if (!nbd_client_connecting (s)) {return ;
@@ -398,27 +400,26 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s)qio_channel_detach_aio_context (QIO_CHANNEL (s->ioc));yank_unregister_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), nbd_yank, s->bs); -object_unref (OBJECT (s->sioc)); - s->sioc =NULL ;object_unref (OBJECT (s->ioc)); s->ioc =NULL ; } - s->sioc =nbd_co_establish_connection (s->conn, &s->info, &s->ioc,NULL ); -if (!s->sioc) { + sioc =nbd_co_establish_connection (s->conn, &s->info, &s->ioc,NULL ); +if (!sioc) { ret = -ECONNREFUSED ; goto out; } -qio_channel_set_blocking (QIO_CHANNEL (s->sioc), false,NULL ); -qio_channel_attach_aio_context (QIO_CHANNEL (s->sioc), aio_context);if (s->ioc) { -qio_channel_set_blocking (QIO_CHANNEL (s->ioc), false,NULL ); -qio_channel_attach_aio_context (QIO_CHANNEL (s->ioc), aio_context); +/* sioc is referenced by s->ioc */ +object_unref (OBJECT (sioc)); }else { - s->ioc =QIO_CHANNEL (s->sioc); -object_ref (OBJECT (s->ioc)); + s->ioc =QIO_CHANNEL (sioc); } + sioc =NULL ; + +qio_channel_set_blocking (QIO_CHANNEL (s->ioc), false,NULL ); +qio_channel_attach_aio_context (QIO_CHANNEL (s->ioc), aio_context);yank_register_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), nbd_yank, s->bs);
@@ -435,8 +436,6 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s)yank_unregister_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), nbd_yank, s->bs); -object_unref (OBJECT (s->sioc)); - s->sioc =NULL ;object_unref (OBJECT (s->ioc)); s->ioc =NULL ;
@@ -571,8 +570,6 @@ static coroutine_fn void nbd_connection_entry(void *opaque)qio_channel_detach_aio_context (QIO_CHANNEL (s->ioc));yank_unregister_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), nbd_yank, s->bs); -object_unref (OBJECT (s->sioc)); - s->sioc =NULL ;object_unref (OBJECT (s->ioc)); s->ioc =NULL ; }
@@ -1571,7 +1568,7 @@ static void nbd_yank(void *opaque)BDRVNBDState *s = (BDRVNBDState *)bs->opaque;qatomic_store_release (&s->state,NBD_CLIENT_QUIT ); -qio_channel_shutdown (QIO_CHANNEL (s->sioc),QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); +qio_channel_shutdown (QIO_CHANNEL (s->ioc),QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); }static void nbd_client_close (BlockDriverState *bs)
@@ -1586,57 +1583,64 @@ static void nbd_client_close(BlockDriverState *bs)nbd_teardown_connection (bs); } -static int nbd_establish_connection (BlockDriverState *bs, -SocketAddress *saddr, -Error **errp) +static QIOChannelSocket *nbd_establish_connection (BlockDriverState *bs, +SocketAddress *saddr, +Error **errp) {ERRP_GUARD (); -BDRVNBDState *s = (BDRVNBDState *)bs->opaque; +QIOChannelSocket *sioc; - s->sioc =qio_channel_socket_new (); -qio_channel_set_name (QIO_CHANNEL (s->sioc),"nbd-client" ); + sioc =qio_channel_socket_new (); +qio_channel_set_name (QIO_CHANNEL (sioc),"nbd-client" ); -qio_channel_socket_connect_sync (s->sioc, saddr, errp); +qio_channel_socket_connect_sync (sioc, saddr, errp);if (*errp) { -object_unref (OBJECT (s->sioc)); - s->sioc =NULL ; -return -1 ; +object_unref (OBJECT (sioc)); +return NULL ; }yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), nbd_yank, bs); -qio_channel_set_delay (QIO_CHANNEL (s->sioc), false); +qio_channel_set_delay (QIO_CHANNEL (sioc), false); -return 0 ; +return sioc; } -/* nbd_client_handshake takes ownership on s->sioc. On failure it's unref'ed. */ -static int nbd_client_handshake (BlockDriverState *bs,Error **errp) +/* nbd_client_handshake takes ownership on sioc. */ +static int nbd_client_handshake (BlockDriverState *bs,QIOChannelSocket *sioc, +Error **errp) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque;AioContext *aio_context =bdrv_get_aio_context (bs);int ret;trace_nbd_client_handshake (s->export); -qio_channel_set_blocking (QIO_CHANNEL (s->sioc), false,NULL ); -qio_channel_attach_aio_context (QIO_CHANNEL (s->sioc), aio_context); +qio_channel_set_blocking (QIO_CHANNEL (sioc), false,NULL ); +qio_channel_attach_aio_context (QIO_CHANNEL (sioc), aio_context); s->info.request_sizes = true; s->info.structured_reply = true; s->info.base_allocation = true; s->info.x_dirty_bitmap =g_strdup (s->x_dirty_bitmap); s->info.name =g_strdup (s->export ?:"" ); - ret =nbd_receive_negotiate (aio_context,QIO_CHANNEL (s->sioc), s->tlscreds, + ret =nbd_receive_negotiate (aio_context,QIO_CHANNEL (sioc), s->tlscreds, s->hostname, &s->ioc, &s->info, errp);g_free (s->info.x_dirty_bitmap);g_free (s->info.name);if (ret <0 ) {yank_unregister_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), nbd_yank, bs); -object_unref (OBJECT (s->sioc)); - s->sioc =NULL ; +object_unref (OBJECT (sioc));return ret; } +if (s->ioc) { +/* sioc is referenced by s->ioc */ +object_unref (OBJECT (sioc)); + }else { + s->ioc =QIO_CHANNEL (sioc); + } + sioc =NULL ; + ret =nbd_handle_updated_info (bs, errp);if (ret <0 ) {/*
@@ -1645,23 +1649,15 @@ static int nbd_client_handshake(BlockDriverState *bs, Error **errp)*/ NBDRequest request = { .type =NBD_CMD_DISC }; -nbd_send_request (s->ioc ?:QIO_CHANNEL (s->sioc), &request); +nbd_send_request (s->ioc, &request);yank_unregister_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), nbd_yank, bs); -object_unref (OBJECT (s->sioc)); - s->sioc =NULL ;object_unref (OBJECT (s->ioc)); s->ioc =NULL ; -return ret; } -if (!s->ioc) { - s->ioc =QIO_CHANNEL (s->sioc); -object_ref (OBJECT (s->ioc)); - } -return 0 ; }
@@ -2047,6 +2043,7 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, {int ret;BDRVNBDState *s = (BDRVNBDState *)bs->opaque; +QIOChannelSocket *sioc; s->bs = bs;qemu_co_mutex_init (&s->send_mutex);
@@ -2069,12 +2066,13 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,* establish TCP connection, return error if it fails * TODO: Configurable retry-until-timeout behaviour. */ -if (nbd_establish_connection (bs, s->saddr, errp) <0 ) { + sioc =nbd_establish_connection (bs, s->saddr, errp); +if (!sioc) { ret = -ECONNREFUSED ; goto fail; } - ret =nbd_client_handshake (bs, errp); + ret =nbd_client_handshake (bs, sioc, errp);if (ret <0 ) { goto fail; }
Signature · 1 line
2.29.2
Re: [PATCH v3 24/33] block/nbd: drop BDRVNBDState::sioc
Quoted 10 lines · Vladimir Sementsov-Ogievskiy
Currently sioc pointer is used just to pass from socket-connection to nbd negotiation. Drop the field, and use local variables instead. With next commit we'll update
nbd/client-connection.cto behave appropriately (return only top-most ioc, not two channels).Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 98 ++++++++++++++++++++++++++--------------------------- 1 file changed, 48 insertions(+), 50 deletions(-)
It's a bit hard to review 's->ioc' vs. 'sioc' when reading aloud ;)
But the change looks reasonable, and I'm not spotting any memory leak in the refactoring.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 25/33 nbd/client-connection: return only one io channel
In reply to Vladimir Sementsov-Ogievskiyblock/nbd doesn't need underlying sioc channel anymore. So, we can update nbd/client-connection interface to return only one top-most io channel, which is more straight forward.
--- include/block/nbd.h | 4 ++-- block/nbd.c | 13 ++----------- nbd/client-connection.c | 33 +++++++++++++++++++++++++-------- 3 files changed, 29 insertions(+), 21 deletions(-)
include/block/nbd.h +2 -2
@@ -419,9 +419,9 @@ NBDClientConnection *nbd_client_connection_new(const SocketAddress *saddr, Monitor *mon);void nbd_client_connection_release (NBDClientConnection *conn); -QIOChannelSocket *coroutine_fn +QIOChannel *coroutine_fnnbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, - QIOChannel **ioc, Error **errp); + Error **errp); void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn);
block/nbd.c +2 -11
@@ -365,7 +365,6 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {int ret;AioContext *aio_context =bdrv_get_aio_context (s->bs); -QIOChannelSocket *sioc;if (!nbd_client_connecting (s)) {return ;
@@ -404,20 +403,12 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) s->ioc =NULL ; } - sioc =nbd_co_establish_connection (s->conn, &s->info, &s->ioc,NULL ); -if (!sioc) { + s->ioc =nbd_co_establish_connection (s->conn, &s->info,NULL ); +if (!s->ioc) { ret = -ECONNREFUSED ; goto out; } -if (s->ioc) { -/* sioc is referenced by s->ioc */ -object_unref (OBJECT (sioc)); - }else { - s->ioc =QIO_CHANNEL (sioc); - } - sioc =NULL ; -qio_channel_set_blocking (QIO_CHANNEL (s->ioc), false,NULL );qio_channel_attach_aio_context (QIO_CHANNEL (s->ioc), aio_context);
nbd/client-connection.c +25 -8
@@ -255,15 +255,15 @@ void nbd_client_connection_release(NBDClientConnection *conn)* nbd_receive_export_list() would be zero (see description of NBDExportInfo in * include/block/nbd.h). */ -QIOChannelSocket *coroutine_fn +QIOChannel *coroutine_fnnbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, - QIOChannel **ioc, Error **errp) + Error **errp) { +QIOChannel *ioc;QemuThread thread;if (conn->do_negotiation) {assert (info); -assert (ioc); } WITH_QEMU_LOCK_GUARD(&conn->mutex) {
@@ -277,10 +277,17 @@ nbd_co_establish_connection(NBDClientConnection *conn, NBDExportInfo *info,if (conn->sioc) {/* Previous attempt finally succeeded in background */ if (conn->do_negotiation) { - *ioc =g_steal_pointer (&conn->ioc); + ioc =g_steal_pointer (&conn->ioc);memcpy (info, &conn->updated_info,sizeof (*info)); } -return g_steal_pointer (&conn->sioc); +if (ioc) { +/* TLS channel now has own reference to parent */ +object_unref (OBJECT (conn->sioc)); + }else { + ioc =QIO_CHANNEL (conn->sioc); + } + conn->sioc =NULL ; +return ioc; } conn->running = true;
@@ -311,11 +318,21 @@ nbd_co_establish_connection(NBDClientConnection *conn, NBDExportInfo *info, }else {error_propagate (errp, conn->err); conn->err =NULL ; -if (conn->sioc && conn->do_negotiation) { - *ioc =g_steal_pointer (&conn->ioc); +if (!conn->sioc) { +return NULL ; + } +if (conn->do_negotiation) { + ioc =g_steal_pointer (&conn->ioc);memcpy (info, &conn->updated_info,sizeof (*info)); } -return g_steal_pointer (&conn->sioc); +if (ioc) { +/* TLS channel now has own reference to parent */ +object_unref (OBJECT (conn->sioc)); + }else { + ioc =QIO_CHANNEL (conn->sioc); + } + conn->sioc =NULL ; +return ioc; } }
Signature · 1 line
2.29.2
Re: [PATCH v3 25/33] nbd/client-connection: return only one io channel
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
block/nbd doesn't need underlying sioc channel anymore. So, we can update nbd/client-connection interface to return only one top-most io channel, which is more straight forward.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 4 ++-- block/nbd.c | 13 ++----------- nbd/client-connection.c | 33 +++++++++++++++++++++++++-------- 3 files changed, 29 insertions(+), 21 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 26/33 block-coroutine-wrapper: allow non bdrv_ prefix
In reply to Vladimir Sementsov-OgievskiyWe are going to reuse the script to generate a qcow2_ function in further commit. Prepare the script now.
--- scripts/block-coroutine-wrapper.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
scripts/block-coroutine-wrapper.py +4 -3
@@ -98,12 +98,13 @@ def snake_to_camel(func_name: str) -> str: def gen_wrapper(func: FuncDecl) -> str: - assert func.name.startswith('bdrv_') - assert not func.name.startswith('bdrv_co_') + assert not '_co_' in func.name assert func.return_type == 'int' assert func.args[0].type in ['BlockDriverState *', 'BdrvChild *'] - name = 'bdrv_co_' + func.name[5:] + subsystem, subname = func.name.split('_', 1) + + name = f'{subsystem}_co_{subname}' bs = 'bs' if func.args[0].type == 'BlockDriverState *' else 'child->bs' struct_name = snake_to_camel(name)
Signature · 1 line
2.29.2
Re: [PATCH v3 26/33] block-coroutine-wrapper: allow non bdrv_ prefix
Quoted 7 lines · Vladimir Sementsov-Ogievskiy
We are going to reuse the script to generate a
qcow2_function in further commit. Prepare the script now.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- scripts/block-coroutine-wrapper.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 26/33] block-coroutine-wrapper: allow non bdrv_ prefix
In reply to Vladimir Sementsov-OgievskiyQuoted 1 line · Vladimir Sementsov-Ogievskiy
We are going to reuse the script to generate a
qcow2_function in
Is qcow2_ right here? Looking ahead to patch 30/33, it looks like you meant nbd_, at least in the context of this series.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 26/33] block-coroutine-wrapper: allow non bdrv_ prefix
Quoted 6 lines · Eric Blake
Quoted 1 line · Vladimir Sementsov-Ogievskiy
We are going to reuse the script to generate a
qcow2_function inIs
qcow2_right here? Looking ahead to patch 30/33, it looks like you meantnbd_, at least in the context of this series.
will change. This because the patch was taken from my parallel qcow2 series
Signature · 2 lines
Best regards, Vladimir
PATCH v3 27/33 block/nbd: split nbd_co_do_establish_connection out of nbd_reconnect_attempt
In reply to Vladimir Sementsov-OgievskiySplit out part, that we want to reuse for nbd_open().
--- block/nbd.c | 79 +++++++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 38 deletions(-)
block/nbd.c +41 -38
@@ -361,11 +361,49 @@ static int nbd_handle_updated_info(BlockDriverState *bs, Error **errp)return 0 ; } -static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) +static int nbd_co_do_establish_connection (BlockDriverState *bs,Error **errp) { +BDRVNBDState *s = (BDRVNBDState *)bs->opaque;int ret; -AioContext *aio_context =bdrv_get_aio_context (s->bs); +assert (!s->ioc); + + s->ioc =nbd_co_establish_connection (s->conn, &s->info, errp); +if (!s->ioc) { +return -ECONNREFUSED ; + } + + ret =nbd_handle_updated_info (s->bs,NULL ); +if (ret <0 ) { +/* +* We have connected, but must fail for other reasons. +* Send NBD_CMD_DISC as a courtesy to the server. +*/ +NBDRequest request = { .type =NBD_CMD_DISC }; + +nbd_send_request (s->ioc, &request); + +object_unref (OBJECT (s->ioc)); + s->ioc =NULL ; + +return ret; + } + +qio_channel_set_blocking (s->ioc, false,NULL ); +qio_channel_attach_aio_context (s->ioc,bdrv_get_aio_context (bs)); + +yank_register_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), nbd_yank, + bs); + +/* successfully connected */ + s->state =NBD_CLIENT_CONNECTED ; +qemu_co_queue_restart_all (&s->free_sema); + +return 0 ; +} + +static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) +{if (!nbd_client_connecting (s)) {return ; }
@@ -403,42 +441,7 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) s->ioc =NULL ; } - s->ioc =nbd_co_establish_connection (s->conn, &s->info,NULL ); -if (!s->ioc) { - ret = -ECONNREFUSED ; - goto out; - } - -qio_channel_set_blocking (QIO_CHANNEL (s->ioc), false,NULL ); -qio_channel_attach_aio_context (QIO_CHANNEL (s->ioc), aio_context); - -yank_register_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), nbd_yank, - s->bs); - - ret =nbd_handle_updated_info (s->bs,NULL ); -if (ret <0 ) { -/* -* We have connected, but must fail for other reasons. -* Send NBD_CMD_DISC as a courtesy to the server. -*/ -NBDRequest request = { .type =NBD_CMD_DISC }; - -nbd_send_request (s->ioc, &request); - -yank_unregister_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), - nbd_yank, s->bs); -object_unref (OBJECT (s->ioc)); - s->ioc =NULL ; - -return ; - } - -out: -if (ret >=0 ) { -/* successfully connected */ - s->state =NBD_CLIENT_CONNECTED ; -qemu_co_queue_restart_all (&s->free_sema); - } +nbd_co_do_establish_connection (s->bs,NULL ); } static coroutine_fn void nbd_co_reconnect_loop(BDRVNBDState *s)
Signature · 1 line
2.29.2
Re: [PATCH v3 27/33] block/nbd: split nbd_co_do_establish_connection out of nbd_reconnect_attempt
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Split out part, that we want to reuse for
nbd_open().
Split out the part that we want to reuse for nbd_open().
Quoted 11 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 79 +++++++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 38 deletions(-)block/nbd.c +0
-0
block/nbd.c static int nbd_handle_updated_info(BlockDriverState *bs, Error **errp) · line 364
Quoted 5 lines
block/nbd.c +1
-1@@ -361,11 +361,49 @@ static int nbd_handle_updated_info(BlockDriverState *bs, Error **errp)return 0 ; } -static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) +static int nbd_co_do_establish_connection (BlockDriverState *bs,Error **errp)
Given the _co_ in the name, don't you need a coroutine_fn marker?
Otherwise looks sane.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 27/33] block/nbd: split nbd_co_do_establish_connection out of nbd_reconnect_attempt
Quoted 19 lines · Eric Blake
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Split out part, that we want to reuse for
nbd_open().Split out the part that we want to reuse for
nbd_open().Quoted 14 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 79 +++++++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 38 deletions(-)block/nbd.c +0
-0@@ -361,11 +361,49 @@ static int nbd_handle_updated_info(BlockDriverState *bs, Error **errp)return 0 ; }
Quoted 4 lines
block/nbd.c static int nbd_handle_updated_info(BlockDriverState *bs, Error **errp) · line 364
Quoted 2 lines
block/nbd.c +1
-1-static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) +static int nbd_co_do_establish_connection (BlockDriverState *bs,Error **errp)Given the _co_ in the name, don't you need a
coroutine_fnmarker?
Yes, strange that I've dropped it
Quoted 3 lines
Otherwise looks sane.
Signature · 2 lines
Best regards, Vladimir
PATCH v3 28/33 nbd/client-connection: do qio_channel_set_delay(false)
In reply to Vladimir Sementsov-Ogievskiynbd_open() does it (through nbd_establish_connection()). Actually we lost that call on reconnect path in 1dc4718d849e1a1fe "block/nbd: use non-blocking connect: fix vm hang on connect()" when we have introduced reconnect thread.
--- nbd/client-connection.c | 2 ++ 1 file changed, 2 insertions(+)
nbd/client-connection.c +2 -0
@@ -126,6 +126,8 @@ static int nbd_connect(QIOChannelSocket *sioc, SocketAddress *addr,return ret; } +qio_channel_set_delay (QIO_CHANNEL (sioc), false); +if (!info) {return 0 ; }
Signature · 1 line
2.29.2
Re: [PATCH v3 28/33] nbd/client-connection: do qio_channel_set_delay(false)
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
nbd_open()does it (throughnbd_establish_connection()). Actually we lost that call on reconnect path in1dc4718d849e1a1fe"block/nbd: use non-blocking connect: fix vm hang onconnect()" when we have introduced reconnect thread.
s/have introduced/introduced the/
Quoted 12 lines
Fixes 1dc4718d849e1a1fe665ce5241ed79048cfa2cfcSigned-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client-connection.c | 2 ++ 1 file changed, 2 insertions(+)nbd/client-connection.c +0
-0
nbd/client-connection.c static int nbd_connect(QIOChannelSocket *sioc, SocketAddress *addr, · line 133
Quoted 8 lines
nbd/client-connection.c +2
-0@@ -126,6 +126,8 @@ static int nbd_connect(QIOChannelSocket *sioc, SocketAddress *addr,return ret; } +qio_channel_set_delay (QIO_CHANNEL (sioc), false); +if (!info) {return 0 ; }
Is this bug fix something that can be cherry-picked in isolation, or does it depend too much on the rest of the series?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 28/33] nbd/client-connection: do qio_channel_set_delay(false)
Quoted 31 lines · Eric Blake
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
nbd_open()does it (throughnbd_establish_connection()). Actually we lost that call on reconnect path in1dc4718d849e1a1fe"block/nbd: use non-blocking connect: fix vm hang onconnect()" when we have introduced reconnect thread.s/have introduced/introduced the/
Quoted 12 lines
Fixes 1dc4718d849e1a1fe665ce5241ed79048cfa2cfcSigned-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client-connection.c | 2 ++ 1 file changed, 2 insertions(+)nbd/client-connection.c +0
-0nbd/client-connection.c static int nbd_connect(QIOChannelSocket *sioc, SocketAddress *addr, · line 133
Quoted 8 lines
nbd/client-connection.c +2
-0@@ -126,6 +126,8 @@ static int nbd_connect(QIOChannelSocket *sioc, SocketAddress *addr,return ret; } +qio_channel_set_delay (QIO_CHANNEL (sioc), false); +if (!info) {return 0 ; }Reviewed-by Eric Blake <[email protected]>
Quoted 3 lines
Is this bug fix something that can be cherry-picked in isolation, or does it depend too much on the rest of the series?
Will try to move to start of the series
Signature · 2 lines
Best regards, Vladimir
PATCH v3 29/33 nbd/client-connection: add option for non-blocking connection attempt
In reply to Vladimir Sementsov-OgievskiyWe'll need a possibility of non-blocking nbd_co_establish_connection(), so that it returns immediately, and it returns success only if connections was previously established in background.
--- include/block/nbd.h | 2 +- block/nbd.c | 2 +- nbd/client-connection.c | 8 +++++++- 3 files changed, 9 insertions(+), 3 deletions(-)
include/block/nbd.h +1 -1
@@ -421,7 +421,7 @@ void nbd_client_connection_release(NBDClientConnection *conn); QIOChannel *coroutine_fnnbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, - Error **errp); + bool blocking, Error **errp); void coroutine_fn nbd_co_establish_connection_cancel(NBDClientConnection *conn);
block/nbd.c +1 -1
@@ -368,7 +368,7 @@ static int nbd_co_do_establish_connection(BlockDriverState *bs, Error **errp)assert (!s->ioc); - s->ioc =nbd_co_establish_connection (s->conn, &s->info, errp); + s->ioc =nbd_co_establish_connection (s->conn, &s->info, true, errp);if (!s->ioc) {return -ECONNREFUSED ; }
nbd/client-connection.c +7 -1
@@ -251,6 +251,8 @@ void nbd_client_connection_release(NBDClientConnection *conn)* result, just return it now * otherwise if thread is not running, start a thread and wait for completion * +* If @blocking is false, don't wait for the thread, return immediately. +* * If @info is not NULL, also do nbd-negotiation after successful connection. * In this case info is used only as out parameter, and is fully initialized by * nbd_co_establish_connection(). "IN" fields of info as well as related only to
@@ -259,7 +261,7 @@ void nbd_client_connection_release(NBDClientConnection *conn)*/ QIOChannel *coroutine_fnnbd_co_establish_connection (NBDClientConnection *conn, NBDExportInfo *info, - Error **errp) + bool blocking, Error **errp) {QIOChannel *ioc;QemuThread thread;
@@ -299,6 +301,10 @@ nbd_co_establish_connection(NBDClientConnection *conn, NBDExportInfo *info, connect_thread_func, conn, QEMU_THREAD_DETACHED); } +if (!blocking) { +return NULL ; + } + conn->wait_co =qemu_coroutine_self (); }
Signature · 1 line
2.29.2
Re: [PATCH v3 29/33] nbd/client-connection: add option for non-blocking connection attempt
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
We'll need a possibility of non-blocking
nbd_co_establish_connection(), so that it returns immediately, and it returns success only if connections was previously established in background.
only if a connection was
Quoted 8 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 2 +- block/nbd.c | 2 +- nbd/client-connection.c | 8 +++++++- 3 files changed, 9 insertions(+), 3 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 30/33 block/nbd: reuse nbd_co_do_establish_connection() in nbd_open()
In reply to Vladimir Sementsov-OgievskiyThe only last step we need to reuse the function is coroutine-wrapper. nbd_open() may be called from non-coroutine context. So, generate the wrapper and use it.
--- block/coroutines.h | 6 +++ block/nbd.c | 101 ++------------------------------------------- 2 files changed, 10 insertions(+), 97 deletions(-)
block/coroutines.h +6 -0
@@ -66,4 +66,10 @@ int coroutine_fn bdrv_co_readv_vmstate(BlockDriverState *bs, int coroutine_fn bdrv_co_writev_vmstate(BlockDriverState *bs,QEMUIOVector *qiov,int64_t pos); +int generated_co_wrapper +nbd_do_establish_connection (BlockDriverState *bs, Error **errp); +int coroutine_fn +nbd_co_do_establish_connection (BlockDriverState *bs, Error **errp); + +#endif /* BLOCK_COROUTINES_INT_H */
block/nbd.c +4 -97
140 lines not on this page · raw message
Signature · 1 line
2.29.2
Re: [PATCH v3 30/33] block/nbd: reuse nbd_co_do_establish_connection() in nbd_open()
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
The only last step we need to reuse the function is coroutine-wrapper.
nbd_open()may be called from non-coroutine context. So, generate the wrapper and use it.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/coroutines.h | 6 +++ block/nbd.c | 101 ++------------------------------------------- 2 files changed, 10 insertions(+), 97 deletions(-)block/coroutines.h +0
-0
block/coroutines.h int coroutine_fn bdrv_co_readv_vmstate(BlockDriverState *bs, · line 72
Quoted 7 lines
block/coroutines.h +4
-0@@ -66,4 +66,10 @@ int coroutine_fn bdrv_co_readv_vmstate(BlockDriverState *bs, int coroutine_fn bdrv_co_writev_vmstate(BlockDriverState *bs,QEMUIOVector *qiov,int64_t pos); +int generated_co_wrapper +nbd_do_establish_connection (BlockDriverState *bs, Error **errp); +int coroutine_fn +nbd_co_do_establish_connection (BlockDriverState *bs, Error **errp);
Tagged coroutine_fn here,...
block/coroutines.h int coroutine_fn bdrv_co_readv_vmstate(BlockDriverState *bs,
Quoted 4 lines
block/coroutines.h +2
-1+++ b/block/nbd.c -static int nbd_co_do_establish_connection (BlockDriverState *bs,Error **errp) +int nbd_co_do_establish_connection (BlockDriverState *bs,Error **errp)
...but not here. Is it worth being consistent?
Quoted 6 lines
block/coroutines.h +0
-2@@ -2056,22 +1974,11 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, s->x_dirty_bitmap, s->tlscreds, monitor_cur()); -/* -* establish TCP connection, return error if it fails
block/coroutines.h static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 1977
Quoted 10 lines
block/coroutines.h +1
-9-* TODO: Configurable retry-until-timeout behaviour. -*/ - sioc =nbd_establish_connection (bs, s->saddr, errp); -if (!sioc) { - ret = -ECONNREFUSED ; - goto fail; - } - - ret =nbd_client_handshake (bs, sioc, errp); +/* TODO: Configurable retry-until-timeout behaviour.*/
Space before */
Nice diffstat, proving the refactoring worked.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 31/33 block/nbd: add nbd_clinent_connected() helper
In reply to Vladimir Sementsov-OgievskiyWe already have two similar helpers for other state. Let's add another one for convenience.
--- block/nbd.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)
block/nbd.c +14 -11
@@ -124,15 +124,20 @@ static void nbd_clear_bdrvstate(BlockDriverState *bs) s->x_dirty_bitmap =NULL ; } +static bool nbd_client_connected (BDRVNBDState *s) +{ +return qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTED ; +} +static void nbd_channel_error (BDRVNBDState *s,int ret) {if (ret == -EIO ) { -if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTED ) { +if (nbd_client_connected (s)) { s->state = s->reconnect_delay ?NBD_CLIENT_CONNECTING_WAIT :NBD_CLIENT_CONNECTING_NOWAIT ; } }else { -if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTED ) { +if (nbd_client_connected (s)) {qio_channel_shutdown (s->ioc,QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); } s->state =NBD_CLIENT_QUIT ;
@@ -230,7 +235,7 @@ static void nbd_client_attach_aio_context(BlockDriverState *bs,* s->connection_co is either yielded from nbd_receive_reply or from * nbd_co_reconnect_loop() */ -if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTED ) { +if (nbd_client_connected (s)) {qio_channel_attach_aio_context (QIO_CHANNEL (s->ioc), new_context); }
@@ -503,7 +508,7 @@ static coroutine_fn void nbd_connection_entry(void *opaque)nbd_co_reconnect_loop (s); } -if (qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTED ) { +if (!nbd_client_connected (s)) {continue ; }
@@ -582,7 +587,7 @@ static int nbd_co_send_request(BlockDriverState *bs,qemu_co_queue_wait (&s->free_sema, &s->send_mutex); } -if (qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTED ) { +if (!nbd_client_connected (s)) { rc = -EIO ; goto err; }
@@ -609,8 +614,7 @@ static int nbd_co_send_request(BlockDriverState *bs,if (qiov) {qio_channel_set_cork (s->ioc, true); rc =nbd_send_request (s->ioc, request); -if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTED && - rc >=0 ) { +if (nbd_clinet_connected (s) && rc >=0 ) {if (qio_channel_writev_all (s->ioc, qiov->iov, qiov->niov,NULL ) <0 ) { rc = -EIO ;
@@ -935,7 +939,7 @@ static coroutine_fn int nbd_co_do_receive_one_chunk( s->requests[i].receiving = true;qemu_coroutine_yield (); s->requests[i].receiving = false; -if (qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTED ) { +if (!nbd_client_connected (s)) {error_setg (errp,"Connection closed" );return -EIO ; }
@@ -1094,7 +1098,7 @@ static bool nbd_reply_chunk_iter_receive(BDRVNBDState *s,NBDReply local_reply;NBDStructuredReplyChunk *chunk;Error *local_err =NULL ; -if (qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTED ) { +if (!nbd_client_connected (s)) {error_setg (&local_err,"Connection closed" );nbd_iter_channel_error (iter, -EIO , &local_err); goto break_loop;
@@ -1119,8 +1123,7 @@ static bool nbd_reply_chunk_iter_receive(BDRVNBDState *s, }/* Do not execute the body of NBD_FOREACH_REPLY_CHUNK for simple reply. */ -if (nbd_reply_is_simple (reply) || -qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTED ) { +if (nbd_reply_is_simple (reply) || !nbd_client_connected (s)) { goto break_loop; }
Signature · 1 line
2.29.2
Re: [PATCH v3 31/33] block/nbd: add nbd_clinent_connected() helper
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
We already have two similar helpers for other state. Let's add another one for convenience.
Signed-off-by Vladimir Sementsov-Ogievskiy<[email protected]>
The whole usage of load-acquire for the connected state is completely meaningless, but that's not your fault. :) For now let's keep it as is, and we'll clean it up later.
Paolo
Re: [PATCH v3 31/33] block/nbd: add nbd_clinent_connected() helper
Quoted 8 lines · Paolo Bonzini
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
We already have two similar helpers for other state. Let's add another one for convenience.
Signed-off-by Vladimir Sementsov-Ogievskiy<[email protected]>The whole usage of load-acquire for the connected state is completely meaningless, but that's not your fault. :) For now let's keep it as is, and we'll clean it up later.
It would be nice, thanks!
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 31/33] block/nbd: add nbd_clinent_connected() helper
In reply to Vladimir Sementsov-OgievskiyOn Fri, Apr 16, 2021 at 11:09:09AM +0300, Vladimir Sementsov-Ogievskiy wrote:
In the subject: s/clinent/client/
Quoted 8 lines
We already have two similar helpers for other state. Let's add another one for convenience.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v3 32/33 block/nbd: safer transition to receiving request
In reply to Vladimir Sementsov-Ogievskiyreq->receiving is a flag of request being in one concrete yield point in nbd_co_do_receive_one_chunk().
Such kind of boolean flag is always better to unset before scheduling the coroutine, to avoid double scheduling. So, let's be more careful.
--- block/nbd.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
block/nbd.c +4 -2
@@ -152,6 +152,7 @@ static void nbd_recv_coroutines_wake_all(BDRVNBDState *s)NBDClientRequest *req = &s->requests[i];if (req->coroutine && req->receiving) { + req->receiving = false;aio_co_wake (req->coroutine); } }
@@ -552,6 +553,7 @@ static coroutine_fn void nbd_connection_entry(void *opaque)* connection_co happens through a bottom half, which can only * run after we yield. */ + s->requests[i].receiving = false;aio_co_wake (s->requests[i].coroutine);qemu_coroutine_yield (); }
@@ -614,7 +616,7 @@ static int nbd_co_send_request(BlockDriverState *bs,if (qiov) {qio_channel_set_cork (s->ioc, true); rc =nbd_send_request (s->ioc, request); -if (nbd_clinet_connected (s) && rc >=0 ) { +if (nbd_client_connected (s) && rc >=0 ) {if (qio_channel_writev_all (s->ioc, qiov->iov, qiov->niov,NULL ) <0 ) { rc = -EIO ;
@@ -938,7 +940,7 @@ static coroutine_fn int nbd_co_do_receive_one_chunk(/* Wait until we're woken up by nbd_connection_entry. */ s->requests[i].receiving = true;qemu_coroutine_yield (); - s->requests[i].receiving = false; +assert (!s->requests[i].receiving);if (!nbd_client_connected (s)) {error_setg (errp,"Connection closed" );return -EIO ;
Signature · 1 line
2.29.2
Re: [PATCH v3 32/33] block/nbd: safer transition to receiving request
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
req->receivingis a flag of request being in one concrete yield point innbd_co_do_receive_one_chunk().Such kind of boolean flag is always better to unset before scheduling the coroutine, to avoid double scheduling. So, let's be more careful.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
block/nbd.c static int nbd_co_send_request(BlockDriverState *bs, · line 619
Quoted 5 lines
block/nbd.c +1
-1@@ -614,7 +616,7 @@ static int nbd_co_send_request(BlockDriverState *bs,if (qiov) {qio_channel_set_cork (s->ioc, true); rc =nbd_send_request (s->ioc, request); -if (nbd_clinet_connected (s) && rc >=0 ) { +if (nbd_client_connected (s) && rc >=0 ) {
Ouch - typo fix in clinet seems unrelated in this fix; please hoist it into the correct point in the series so that we don't have the typo in the first place.
Otherwise,
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 32/33] block/nbd: safer transition to receiving request
Quoted 13 lines · Eric Blake
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
req->receivingis a flag of request being in one concrete yield point innbd_co_do_receive_one_chunk().Such kind of boolean flag is always better to unset before scheduling the coroutine, to avoid double scheduling. So, let's be more careful.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
Quoted 10 lines
block/nbd.c static int nbd_co_send_request(BlockDriverState *bs, · line 619
Quoted 6 lines
block/nbd.c +1
-1@@ -614,7 +616,7 @@ static int nbd_co_send_request(BlockDriverState *bs,if (qiov) {qio_channel_set_cork (s->ioc, true); rc =nbd_send_request (s->ioc, request); -if (nbd_clinet_connected (s) && rc >=0 ) { +if (nbd_client_connected (s) && rc >=0 ) {Ouch - typo fix in clinet seems unrelated in this fix; please hoist it into the correct point in the series so that we don't have the typo in the first place.
That also means that I didn't made my favorite "git rebase -x 'make -j9' master".. Not good, will fix of course.
Quoted 4 lines
Otherwise,
Reviewed-by Eric Blake <[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH v3 33/33 block/nbd: drop connection_co
In reply to Vladimir Sementsov-OgievskiyOK, that's a big rewrite of the logic.
Pre-patch we have an always running coroutine - connection_co. It does reply receiving and reconnecting. And it leads to a lot of difficult and unobvious code around drained sections and context switch. We also abuse bs->in_flight counter which is increased for connection_co and temporary decreased in points where we want to allow drained section to begin. One of these place is in another file: in nbd_read_eof() in nbd/client.c.
We also cancel reconnect and requests waiting for reconnect on drained begin which is not correct.
Let's finally drop this always running coroutine and go another way:
1. reconnect_attempt() goes to nbd_co_send_request and called under send_mutex
2. We do receive headers in request coroutine. But we also should dispatch replies for another pending requests. So, nbd_connection_entry() is turned into nbd_receive_replies(), which does reply dispatching until it receive another request headers, and returns when it receive the requested header.
3. All old staff around drained sections and context switch is dropped.
--- block/nbd.c | 376 ++++++++++++++++----------------------------------- nbd/client.c | 2 - 2 files changed, 119 insertions(+), 259 deletions(-)
block/nbd.c +119 -257
@@ -59,7 +59,7 @@typedef struct {Coroutine *coroutine;uint64_t offset;/* original offset of the request */ -bool receiving;/* waiting for connection_co? */ +bool receiving;/* waiting in first yield of nbd_receive_replies() */ }NBDClientRequest ; typedef enum NBDClientState {
@@ -75,14 +75,10 @@ typedef struct BDRVNBDState {CoMutex send_mutex;CoQueue free_sema; -Coroutine *connection_co; -Coroutine *teardown_co; -QemuCoSleepState *connection_co_sleep_ns_state; -bool drained; -bool wait_drained_end; +Coroutine *receive_co; +Coroutine *in_flight_waiter;int in_flight;NBDClientState state; -bool wait_in_flight;QEMUTimer *reconnect_delay_timer;
@@ -131,33 +127,20 @@ static bool nbd_client_connected(BDRVNBDState *s)static void nbd_channel_error (BDRVNBDState *s,int ret) { +if (nbd_client_connected (s)) { +qio_channel_shutdown (s->ioc,QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); + } +if (ret == -EIO ) {if (nbd_client_connected (s)) { s->state = s->reconnect_delay ?NBD_CLIENT_CONNECTING_WAIT :NBD_CLIENT_CONNECTING_NOWAIT ; } }else { -if (nbd_client_connected (s)) { -qio_channel_shutdown (s->ioc,QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); - } s->state =NBD_CLIENT_QUIT ; } } -static void nbd_recv_coroutines_wake_all (BDRVNBDState *s) -{ -int i; - -for (i =0 ; i <MAX_NBD_REQUESTS ; i++) { -NBDClientRequest *req = &s->requests[i]; - -if (req->coroutine && req->receiving) { - req->receiving = false; -aio_co_wake (req->coroutine); - } - } -} -static void reconnect_delay_timer_del (BDRVNBDState *s) {if (s->reconnect_delay_timer) {
@@ -194,117 +177,23 @@ static void reconnect_delay_timer_init(BDRVNBDState *s, uint64_t expire_time_ns)timer_mod (s->reconnect_delay_timer, expire_time_ns); } -static void nbd_client_detach_aio_context (BlockDriverState *bs) -{ -BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - -/* Timer is deleted in nbd_client_co_drain_begin() */ -assert (!s->reconnect_delay_timer); -/* -* If reconnect is in progress we may have no ->ioc. It will be -* re-instantiated in the proper aio context once the connection is -* reestablished. -*/ -if (s->ioc) { -qio_channel_detach_aio_context (QIO_CHANNEL (s->ioc)); - } -} - -static void nbd_client_attach_aio_context_bh (void *opaque) -{ -BlockDriverState *bs = opaque; -BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - -if (s->connection_co) { -/* -* The node is still drained, so we know the coroutine has yielded in -* nbd_read_eof(), the only place where bs->in_flight can reach 0, or -* it is entered for the first time. Both places are safe for entering -* the coroutine. -*/ -qemu_aio_coroutine_enter (bs->aio_context, s->connection_co); - } -bdrv_dec_in_flight (bs); -} - -static void nbd_client_attach_aio_context (BlockDriverState *bs, -AioContext *new_context) -{ -BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - -/* -* s->connection_co is either yielded from nbd_receive_reply or from -* nbd_co_reconnect_loop() -*/ -if (nbd_client_connected (s)) { -qio_channel_attach_aio_context (QIO_CHANNEL (s->ioc), new_context); - } - -bdrv_inc_in_flight (bs); - -/* -* Need to wait here for the BH to run because the BH must run while the -* node is still drained. -*/ -aio_wait_bh_oneshot (new_context, nbd_client_attach_aio_context_bh, bs); -} - -static void coroutine_fn nbd_client_co_drain_begin(BlockDriverState *bs) -{ -BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - - s->drained = true; -if (s->connection_co_sleep_ns_state) { -qemu_co_sleep_wake (s->connection_co_sleep_ns_state); - } - -nbd_co_establish_connection_cancel (s->conn); - -reconnect_delay_timer_del (s); - -if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ) { - s->state =NBD_CLIENT_CONNECTING_NOWAIT ; -qemu_co_queue_restart_all (&s->free_sema); - } -} - -static void coroutine_fn nbd_client_co_drain_end(BlockDriverState *bs) -{ -BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - - s->drained = false; -if (s->wait_drained_end) { - s->wait_drained_end = false; -aio_co_wake (s->connection_co); - } -} - -static void nbd_teardown_connection (BlockDriverState *bs) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque; +assert (!s->in_flight); +assert (!s->receive_co); +if (s->ioc) {/* finish any pending coroutines */ qio_channel_shutdown (s->ioc,QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); +yank_unregister_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), + nbd_yank, s->bs); +object_unref (OBJECT (s->ioc)); + s->ioc =NULL ; } s->state =NBD_CLIENT_QUIT ; -if (s->connection_co) { -if (s->connection_co_sleep_ns_state) { -qemu_co_sleep_wake (s->connection_co_sleep_ns_state); - } -nbd_co_establish_connection_cancel (s->conn); - } -if (qemu_in_coroutine ()) { - s->teardown_co =qemu_coroutine_self (); -/* connection_co resumes us when it terminates */ -qemu_coroutine_yield (); - s->teardown_co =NULL ; - }else { -BDRV_POLL_WHILE (bs, s->connection_co); - } -assert (!s->connection_co); }static bool nbd_client_connecting (BDRVNBDState *s)
@@ -367,10 +256,11 @@ int nbd_co_do_establish_connection(BlockDriverState *bs, Error **errp) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque;int ret; +bool blocking =nbd_client_connecting_wait (s);assert (!s->ioc); - s->ioc =nbd_co_establish_connection (s->conn, &s->info, true, errp); + s->ioc =nbd_co_establish_connection (s->conn, &s->info, blocking, errp);if (!s->ioc) {return -ECONNREFUSED ; }
@@ -404,6 +294,7 @@ int nbd_co_do_establish_connection(BlockDriverState *bs, Error **errp)return 0 ; } +/* called under s->send_mutex */ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {if (!nbd_client_connecting (s)) {
@@ -412,23 +303,29 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s)/* Wait for completion of all in-flight requests */ -qemu_co_mutex_lock (&s->send_mutex); - -while (s->in_flight >0 ) { -qemu_co_mutex_unlock (&s->send_mutex); -nbd_recv_coroutines_wake_all (s); - s->wait_in_flight = true; +if (s->in_flight) { + s->in_flight_waiter =qemu_coroutine_self ();qemu_coroutine_yield (); - s->wait_in_flight = false; -qemu_co_mutex_lock (&s->send_mutex); +assert (!s->in_flight_waiter); +assert (!s->in_flight); } -qemu_co_mutex_unlock (&s->send_mutex); -if (!nbd_client_connecting (s)) {return ; } +if (nbd_client_connecting_wait (s) && s->reconnect_delay && + !s->reconnect_delay_timer) + { +/* +* It's first reconnect attempt after switching to +* NBD_CLIENT_CONNECTING_WAIT +*/ +reconnect_delay_timer_init (s, +qemu_clock_get_ns (QEMU_CLOCK_REALTIME ) + + s->reconnect_delay *NANOSECONDS_PER_SECOND ); + } +/* * Now we are sure that nobody is accessing the channel, and no one will * try until we set the state to CONNECTED.
@@ -446,73 +343,34 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s)nbd_co_do_establish_connection (s->bs,NULL ); } -static coroutine_fn void nbd_co_reconnect_loop(BDRVNBDState *s) -{ -uint64_t timeout =1 *NANOSECONDS_PER_SECOND ; -uint64_t max_timeout =16 *NANOSECONDS_PER_SECOND ; - -if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ) { -reconnect_delay_timer_init (s,qemu_clock_get_ns (QEMU_CLOCK_REALTIME ) + - s->reconnect_delay *NANOSECONDS_PER_SECOND ); - } - -nbd_reconnect_attempt (s); - -while (nbd_client_connecting (s)) { -if (s->drained) { -bdrv_dec_in_flight (s->bs); - s->wait_drained_end = true; -while (s->drained) { -/* -* We may be entered once from nbd_client_attach_aio_context_bh -* and then from nbd_client_co_drain_end. So here is a loop. -*/ -qemu_coroutine_yield (); - } -bdrv_inc_in_flight (s->bs); - }else { -qemu_co_sleep_ns_wakeable (QEMU_CLOCK_REALTIME , timeout, - &s->connection_co_sleep_ns_state); -if (s->drained) { -continue ; - } -if (timeout < max_timeout) { - timeout *=2 ; - } - } - -nbd_reconnect_attempt (s); - } - -reconnect_delay_timer_del (s); -} - -static coroutine_fn void nbd_connection_entry(void *opaque) +static coroutine_fn void nbd_receive_replies(BDRVNBDState *s, uint64_t handle) { -BDRVNBDState *s = opaque;uint64_t i;int ret =0 ;Error *local_err =NULL ; -while (qatomic_load_acquire (&s->state) !=NBD_CLIENT_QUIT ) { -/* -* The NBD client can only really be considered idle when it has -* yielded from qio_channel_readv_all_eof(), waiting for data. This is -* the point where the additional scheduled coroutine entry happens -* after nbd_client_attach_aio_context(). -* -* Therefore we keep an additional in_flight reference all the time and -* only drop it temporarily here. -*/ + i =HANDLE_TO_INDEX (s, handle); +if (s->receive_co) { +assert (s->receive_co !=qemu_coroutine_self ()); -if (nbd_client_connecting (s)) { -nbd_co_reconnect_loop (s); - } +/* Another request coroutine is receiving now */ + s->requests[i].receiving = true; +qemu_coroutine_yield (); +assert (!s->requests[i].receiving); -if (!nbd_client_connected (s)) { -continue ; +if (s->receive_co !=qemu_coroutine_self ()) { +/* +* We are either failed or done, caller uses nbd_client_connected() +* to distinguish. +*/ +return ; } + } + +assert (s->receive_co ==0 || s->receive_co ==qemu_coroutine_self ()); + s->receive_co =qemu_coroutine_self (); +while (nbd_client_connected (s)) {assert (s->reply.handle ==0 ); ret =nbd_receive_reply (s->bs, s->ioc, &s->reply, &local_err);
@@ -522,8 +380,21 @@ static coroutine_fn void nbd_connection_entry(void *opaque) local_err =NULL ; }if (ret <=0 ) { -nbd_channel_error (s, ret ? ret : -EIO ); -continue ; + ret = ret ? ret : -EIO ; +nbd_channel_error (s, ret); + goto out; + } + +if (!nbd_client_connected (s)) { + ret = -EIO ; + goto out; + } + + i =HANDLE_TO_INDEX (s, s->reply.handle); + +if (s->reply.handle == handle) { + ret =0 ; + goto out; }/*
@@ -531,50 +402,49 @@ static coroutine_fn void nbd_connection_entry(void *opaque)* handler acts as a synchronization point and ensures that only * one coroutine is called until the reply finishes. */ - i =HANDLE_TO_INDEX (s, s->reply.handle);if (i >=MAX_NBD_REQUESTS || !s->requests[i].coroutine || !s->requests[i].receiving || (nbd_reply_is_structured (&s->reply) && !s->info.structured_reply)) {nbd_channel_error (s, -EINVAL ); -continue ; + ret = -EINVAL ; + goto out; } -/* -* We're woken up again by the request itself. Note that there -* is no race between yielding and reentering connection_co. This -* is because: -* -* - if the request runs on the same AioContext, it is only -* entered after we yield -* -* - if the request runs on a different AioContext, reentering -* connection_co happens through a bottom half, which can only -* run after we yield. -*/ s->requests[i].receiving = false;aio_co_wake (s->requests[i].coroutine);qemu_coroutine_yield (); } -qemu_co_queue_restart_all (&s->free_sema); -nbd_recv_coroutines_wake_all (s); -bdrv_dec_in_flight (s->bs); - - s->connection_co =NULL ; -if (s->ioc) { -qio_channel_detach_aio_context (QIO_CHANNEL (s->ioc)); -yank_unregister_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), - nbd_yank, s->bs); -object_unref (OBJECT (s->ioc)); - s->ioc =NULL ; - } +out: +if (ret <0 ) { + s->receive_co =NULL ; +for (i =0 ; i <MAX_NBD_REQUESTS ; i++) { +NBDClientRequest *req = &s->requests[i]; -if (s->teardown_co) { -aio_co_wake (s->teardown_co); +if (req->coroutine && req->receiving) { + req->receiving = false; +aio_co_wake (req->coroutine); + } + } + }else { +/* +* If there are still some receiving request, it should become next +* "receive_co" +*/ +for (i =0 ; i <MAX_NBD_REQUESTS ; i++) { +NBDClientRequest *req = &s->requests[i]; + +if (req->coroutine && req->receiving) { + req->receiving = false; + s->receive_co = req->coroutine; +aio_co_wake (req->coroutine); +return ; + } + } + s->receive_co =NULL ; } -aio_wait_kick (); } static int nbd_co_send_request(BlockDriverState *bs,
@@ -585,7 +455,15 @@ static int nbd_co_send_request(BlockDriverState *bs,int rc, i =-1 ;qemu_co_mutex_lock (&s->send_mutex); -while (s->in_flight ==MAX_NBD_REQUESTS ||nbd_client_connecting_wait (s)) { + +nbd_reconnect_attempt (s); + +if (!nbd_client_connected (s)) { +qemu_co_mutex_unlock (&s->send_mutex); +return -EIO ; + } + +while (s->in_flight ==MAX_NBD_REQUESTS ) {qemu_co_queue_wait (&s->free_sema, &s->send_mutex); }
@@ -636,10 +514,10 @@ err: s->requests[i].coroutine =NULL ; s->in_flight--; } -if (s->in_flight ==0 && s->wait_in_flight) { -aio_co_wake (s->connection_co); - }else { -qemu_co_queue_next (&s->free_sema); +if (s->in_flight ==0 && s->in_flight_waiter) { +Coroutine *co = s->in_flight_waiter; + s->in_flight_waiter =NULL ; +aio_co_wake (co); } }qemu_co_mutex_unlock (&s->send_mutex);
@@ -938,9 +816,7 @@ static coroutine_fn int nbd_co_do_receive_one_chunk( *request_ret =0 ;/* Wait until we're woken up by nbd_connection_entry. */ - s->requests[i].receiving = true; -qemu_coroutine_yield (); -assert (!s->requests[i].receiving); +nbd_receive_replies (s, handle);if (!nbd_client_connected (s)) {error_setg (errp,"Connection closed" );return -EIO ;
@@ -1033,13 +909,8 @@ static coroutine_fn int nbd_co_receive_one_chunk( } s->reply.handle =0 ; -if (s->connection_co && !s->wait_in_flight) { -/* -* We must check s->wait_in_flight, because we may entered by -* nbd_recv_coroutines_wake_all(), in this case we should not -* wake connection_co here, it will woken by last request. -*/ -aio_co_wake (s->connection_co); +if (s->receive_co) { +aio_co_wake (s->receive_co); }return ret;
@@ -1151,8 +1022,10 @@ break_loop:qemu_co_mutex_lock (&s->send_mutex); s->in_flight--; -if (s->in_flight ==0 && s->wait_in_flight) { -aio_co_wake (s->connection_co); +if (s->in_flight ==0 && s->in_flight_waiter) { +Coroutine *co = s->in_flight_waiter; + s->in_flight_waiter =NULL ; +aio_co_wake (co); }else {qemu_co_queue_next (&s->free_sema); }
@@ -1980,14 +1853,13 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, monitor_cur());/* TODO: Configurable retry-until-timeout behaviour.*/ + s->state =NBD_CLIENT_CONNECTING_WAIT ; ret =nbd_do_establish_connection (bs, errp);if (ret <0 ) { goto fail; } - s->connection_co =qemu_coroutine_create (nbd_connection_entry, s); -bdrv_inc_in_flight (bs); -aio_co_schedule (bdrv_get_aio_context (bs), s->connection_co); +nbd_client_connection_enable_retry (s->conn);return 0 ;
@@ -2141,6 +2013,8 @@ static void nbd_cancel_in_flight(BlockDriverState *bs) s->state =NBD_CLIENT_CONNECTING_NOWAIT ;qemu_co_queue_restart_all (&s->free_sema); } + +nbd_co_establish_connection_cancel (s->conn); }static BlockDriver bdrv_nbd = {
@@ -2161,10 +2035,6 @@ static BlockDriver bdrv_nbd = { .bdrv_refresh_limits = nbd_refresh_limits, .bdrv_co_truncate = nbd_co_truncate, .bdrv_getlength = nbd_getlength, - .bdrv_detach_aio_context = nbd_client_detach_aio_context, - .bdrv_attach_aio_context = nbd_client_attach_aio_context, - .bdrv_co_drain_begin = nbd_client_co_drain_begin, - .bdrv_co_drain_end = nbd_client_co_drain_end, .bdrv_refresh_filename = nbd_refresh_filename, .bdrv_co_block_status = nbd_client_co_block_status, .bdrv_dirname = nbd_dirname,
@@ -2190,10 +2060,6 @@ static BlockDriver bdrv_nbd_tcp = { .bdrv_refresh_limits = nbd_refresh_limits, .bdrv_co_truncate = nbd_co_truncate, .bdrv_getlength = nbd_getlength, - .bdrv_detach_aio_context = nbd_client_detach_aio_context, - .bdrv_attach_aio_context = nbd_client_attach_aio_context, - .bdrv_co_drain_begin = nbd_client_co_drain_begin, - .bdrv_co_drain_end = nbd_client_co_drain_end, .bdrv_refresh_filename = nbd_refresh_filename, .bdrv_co_block_status = nbd_client_co_block_status, .bdrv_dirname = nbd_dirname,
@@ -2219,10 +2085,6 @@ static BlockDriver bdrv_nbd_unix = { .bdrv_refresh_limits = nbd_refresh_limits, .bdrv_co_truncate = nbd_co_truncate, .bdrv_getlength = nbd_getlength, - .bdrv_detach_aio_context = nbd_client_detach_aio_context, - .bdrv_attach_aio_context = nbd_client_attach_aio_context, - .bdrv_co_drain_begin = nbd_client_co_drain_begin, - .bdrv_co_drain_end = nbd_client_co_drain_end, .bdrv_refresh_filename = nbd_refresh_filename, .bdrv_co_block_status = nbd_client_co_block_status, .bdrv_dirname = nbd_dirname,
nbd/client.c +0 -2
@@ -1434,9 +1434,7 @@ nbd_read_eof(BlockDriverState *bs, QIOChannel *ioc, void *buffer, size_t size, len =qio_channel_readv (ioc, &iov,1 , errp);if (len ==QIO_CHANNEL_ERR_BLOCK ) { -bdrv_dec_in_flight (bs);qio_channel_yield (ioc,G_IO_IN ); -bdrv_inc_in_flight (bs);continue ; }else if (len <0 ) {return -EIO ;
Signature · 1 line
2.29.2
Re: [PATCH v3 33/33] block/nbd: drop connection_co
Quoted 24 lines · Vladimir Sementsov-Ogievskiy
OK, that's a big rewrite of the logic.
Pre-patch we have an always running coroutine -
connection_co. It does reply receiving and reconnecting. And it leads to a lot of difficult and unobvious code around drained sections and context switch. We also abusebs->in_flightcounter which is increased forconnection_coand temporary decreased in points where we want to allow drained section to begin. One of these place is in another file: innbd_read_eof()innbd/client.c.We also cancel reconnect and requests waiting for reconnect on drained begin which is not correct.
Let's finally drop this always running coroutine and go another way:
1.
reconnect_attempt()goes tonbd_co_send_requestand called undersend_mutex2. We do receive headers in request coroutine. But we also should dispatch replies for another pending requests. So,
nbd_connection_entry()is turned intonbd_receive_replies(), which does reply dispatching until it receive another request headers, and returns when it receive the requested header.
Quoted 3 lines
3. All old staff around drained sections and context switch is dropped.
Signed-off-by Vladimir Sementsov-Ogievskiy<[email protected]>
Please consider this last patch as RFC for now:
1. It is complicated, and doesn't have good documentation. Please look through and ask everything that is not obvious, I'll explain. Don't waste your time trying to understand what is not clean.
2. I also failed to image, how to split the patch into smaller simple patches.. Ideas are welcome.
3. It actually reverts what was done in
commit 8c517de24a8a1dcbeb54e7e12b5b0fda42a90ace
Author: Vladimir Sementsov-Ogievskiy <[email protected]> Date: Thu Sep 3 22:02:58 2020 +0300
block/nbd: fix drain dead-lock because of nbd reconnect-delay
and I didn't check yet, does this dead-lock still here or not. Even if it still here I believe that nbd driver is a wrong place to workaround this bug, but I should check it first at least.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 33/33] block/nbd: drop connection_co
Quoted 37 lines · Vladimir Sementsov-Ogievskiy
Quoted 24 lines · Vladimir Sementsov-Ogievskiy
OK, that's a big rewrite of the logic.
Pre-patch we have an always running coroutine -
connection_co. It does reply receiving and reconnecting. And it leads to a lot of difficult and unobvious code around drained sections and context switch. We also abusebs->in_flightcounter which is increased forconnection_coand temporary decreased in points where we want to allow drained section to begin. One of these place is in another file: innbd_read_eof()innbd/client.c.We also cancel reconnect and requests waiting for reconnect on drained begin which is not correct.
Let's finally drop this always running coroutine and go another way:
1.
reconnect_attempt()goes tonbd_co_send_requestand called undersend_mutex2. We do receive headers in request coroutine. But we also should dispatch replies for another pending requests. So,
nbd_connection_entry()is turned intonbd_receive_replies(), which does reply dispatching until it receive another request headers, and returns when it receive the requested header.Quoted 3 lines
3. All old staff around drained sections and context switch is dropped.
Signed-off-by Vladimir Sementsov-Ogievskiy<[email protected]>Please consider this last patch as RFC for now:
1. It is complicated, and doesn't have good documentation. Please look through and ask everything that is not obvious, I'll explain. Don't waste your time trying to understand what is not clean.
2. I also failed to image, how to split the patch into smaller simple patches.. Ideas are welcome.
3. It actually reverts what was done in
Quoted 8 lines
commit
8c517de24a8a1dcbeb54e7e12b5b0fda42a90ace
Author: Vladimir Sementsov-Ogievskiy <[email protected]> Date: Thu Sep 3 22:02:58 2020 +0300block/nbd: fix drain dead-lock because of nbd reconnect-delay
and I didn't check yet, does this dead-lock still here or not. Even if it still here I believe that nbd driver is a wrong place to workaround this bug, but I should check it first at least.
4. As Roman said, there is a problem in new architecture: when guest is idle, we will not detect disconnect immediately but only on the next request from the guest. That may be considered as a degradation. Still, let's implement a kind of keep-alive on top of this series, some ideas:
- add an idle-timeout, and do simple NBD request by timeout, which will result in some expected error reply from the server.
- or add an idle coroutine, which will do endless "read" when there no requests. It will be a kind of old connection_co, but it will have only one function and will be extremely simple. And we may just cancel it on drained-begin and restart on drained-end.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v3 33/33] block/nbd: drop connection_co
In reply to Vladimir Sementsov-OgievskiyQuoted 65 lines · Vladimir Sementsov-Ogievskiy
OK, that's a big rewrite of the logic.
Pre-patch we have an always running coroutine -
connection_co. It does reply receiving and reconnecting. And it leads to a lot of difficult and unobvious code around drained sections and context switch. We also abusebs->in_flightcounter which is increased forconnection_coand temporary decreased in points where we want to allow drained section to begin. One of these place is in another file: innbd_read_eof()innbd/client.c.We also cancel reconnect and requests waiting for reconnect on drained begin which is not correct.
Let's finally drop this always running coroutine and go another way:
1.
reconnect_attempt()goes tonbd_co_send_requestand called undersend_mutex2. We do receive headers in request coroutine. But we also should dispatch replies for another pending requests. So,
nbd_connection_entry()is turned intonbd_receive_replies(), which does reply dispatching until it receive another request headers, and returns when it receive the requested header.3. All old staff around drained sections and context switch is dropped.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 376 ++++++++++++++++----------------------------------- nbd/client.c | 2 - 2 files changed, 119 insertions(+), 259 deletions(-)block/nbd.c +8
-17-static coroutine_fn void nbd_connection_entry(void *opaque) +static coroutine_fn void nbd_receive_replies(BDRVNBDState *s, uint64_t handle) { -BDRVNBDState *s = opaque;uint64_t i;int ret =0 ;Error *local_err =NULL ; -while (qatomic_load_acquire (&s->state) !=NBD_CLIENT_QUIT ) { -/* -* The NBD client can only really be considered idle when it has -* yielded from qio_channel_readv_all_eof(), waiting for data. This is -* the point where the additional scheduled coroutine entry happens -* after nbd_client_attach_aio_context(). -* -* Therefore we keep an additional in_flight reference all the time and -* only drop it temporarily here. -*/ + i =HANDLE_TO_INDEX (s, handle); +if (s->receive_co) { +assert (s->receive_co !=qemu_coroutine_self ()); -if (nbd_client_connecting (s)) { -nbd_co_reconnect_loop (s); - } +/* Another request coroutine is receiving now */ + s->requests[i].receiving = true; +qemu_coroutine_yield (); +assert (!s->requests[i].receiving); -if (!nbd_client_connected (s)) { -continue ;
block/nbd.c static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) · line 370
Quoted 10 lines
block/nbd.c +9
-0+if (s->receive_co !=qemu_coroutine_self ()) { +/* +* We are either failed or done, caller uses nbd_client_connected() +* to distinguish. +*/ +return ; } + } + +assert (s->receive_co ==0 || s->receive_co ==qemu_coroutine_self ());
s/0/NULL/ here
Quoted 20 lines
block/nbd.c +10
-2+ s->receive_co =qemu_coroutine_self (); +while (nbd_client_connected (s)) {assert (s->reply.handle ==0 ); ret =nbd_receive_reply (s->bs, s->ioc, &s->reply, &local_err);@@ -522,8 +380,21 @@ static coroutine_fn void nbd_connection_entry(void *opaque) local_err =NULL ; }if (ret <=0 ) { -nbd_channel_error (s, ret ? ret : -EIO ); -continue ; + ret = ret ? ret : -EIO ; +nbd_channel_error (s, ret); + goto out; + } + +if (!nbd_client_connected (s)) { + ret = -EIO ; + goto out;
block/nbd.c static coroutine_fn void nbd_connection_entry(void *opaque) · line 400
Quoted 10 lines
block/nbd.c +7
-0+ } + + i =HANDLE_TO_INDEX (s, s->reply.handle); + +if (s->reply.handle == handle) { + ret =0 ; + goto out; }/*
I know your followup said there is more work to do before v4, but I look forward to seeing it.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v3 33/33] block/nbd: drop connection_co
Quoted 108 lines · Eric Blake
Quoted 65 lines · Vladimir Sementsov-Ogievskiy
OK, that's a big rewrite of the logic.
Pre-patch we have an always running coroutine -
connection_co. It does reply receiving and reconnecting. And it leads to a lot of difficult and unobvious code around drained sections and context switch. We also abusebs->in_flightcounter which is increased forconnection_coand temporary decreased in points where we want to allow drained section to begin. One of these place is in another file: innbd_read_eof()innbd/client.c.We also cancel reconnect and requests waiting for reconnect on drained begin which is not correct.
Let's finally drop this always running coroutine and go another way:
1.
reconnect_attempt()goes tonbd_co_send_requestand called undersend_mutex2. We do receive headers in request coroutine. But we also should dispatch replies for another pending requests. So,
nbd_connection_entry()is turned intonbd_receive_replies(), which does reply dispatching until it receive another request headers, and returns when it receive the requested header.3. All old staff around drained sections and context switch is dropped.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd.c | 376 ++++++++++++++++----------------------------------- nbd/client.c | 2 - 2 files changed, 119 insertions(+), 259 deletions(-)block/nbd.c +8
-17-static coroutine_fn void nbd_connection_entry(void *opaque) +static coroutine_fn void nbd_receive_replies(BDRVNBDState *s, uint64_t handle) { -BDRVNBDState *s = opaque;uint64_t i;int ret =0 ;Error *local_err =NULL ; -while (qatomic_load_acquire (&s->state) !=NBD_CLIENT_QUIT ) { -/* -* The NBD client can only really be considered idle when it has -* yielded from qio_channel_readv_all_eof(), waiting for data. This is -* the point where the additional scheduled coroutine entry happens -* after nbd_client_attach_aio_context(). -* -* Therefore we keep an additional in_flight reference all the time and -* only drop it temporarily here. -*/ + i =HANDLE_TO_INDEX (s, handle); +if (s->receive_co) { +assert (s->receive_co !=qemu_coroutine_self ()); -if (nbd_client_connecting (s)) { -nbd_co_reconnect_loop (s); - } +/* Another request coroutine is receiving now */ + s->requests[i].receiving = true; +qemu_coroutine_yield (); +assert (!s->requests[i].receiving); -if (!nbd_client_connected (s)) { -continue ;block/nbd.c static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) · line 370
Quoted 10 lines
block/nbd.c +9
-0+if (s->receive_co !=qemu_coroutine_self ()) { +/* +* We are either failed or done, caller uses nbd_client_connected() +* to distinguish. +*/ +return ; } + } + +assert (s->receive_co ==0 || s->receive_co ==qemu_coroutine_self ());s/0/NULL/ here
Quoted 29 lines
block/nbd.c +17
-2+ s->receive_co =qemu_coroutine_self (); +while (nbd_client_connected (s)) {assert (s->reply.handle ==0 ); ret =nbd_receive_reply (s->bs, s->ioc, &s->reply, &local_err);@@ -522,8 +380,21 @@ static coroutine_fn void nbd_connection_entry(void *opaque) local_err =NULL ; }if (ret <=0 ) { -nbd_channel_error (s, ret ? ret : -EIO ); -continue ; + ret = ret ? ret : -EIO ; +nbd_channel_error (s, ret); + goto out; + } + +if (!nbd_client_connected (s)) { + ret = -EIO ; + goto out; + } + + i =HANDLE_TO_INDEX (s, s->reply.handle); + +if (s->reply.handle == handle) { + ret =0 ; + goto out; }
Quoted 5 lines
Review static coroutine_fn void nbd_connection_entry(void *opaque) · line 400
Quoted 1 line
(quoted diff) +0
-0/*I know your followup said there is more work to do before v4, but I look forward to seeing it.
Great thanks for reviewing this huge series! Now is my turn to make v4.
Signature · 2 lines
Best regards, Vladimir
Quoted 8 lines · Vladimir Sementsov-Ogievskiy
The series substitutes "[PATCH v2 00/10] block/nbd: move connection code to separate file"
Supersedes: <[email protected]>
so it's called v3
block/nbd.cis overcomplicated. These series is a big refactoring, which finally drops all the complications around drained sections and context switching, including abuse ofbs->in_flightcounter.
Quoted 6 lines
Also, at the end of the series we don't cancel reconnect on drained sections (and don't cancel requests waiting for reconnect on drained section begin), which fixes a problem reported by Roman.
The series is also available at tag up-nbd-client-connection-v3 in git https://src.openvz.org/scm/~vsementsov/qemu.git
I have independently done some rework of the connection state machine, mostly in order to use the QemuCoSleep API instead of aio_co_wake. In general it seems to be independent of this work. I'll review this series.
Paolo
Quoted 71 lines
v3:
Changes in first part of the series (main thing is not using refcnt, but instead (modified) Roman's patch):01-04: new
05: add Roman's r-b
06: new
07: now, newaio_co_schedule(NULL, thr->wait_co)is used 08: reworked, we now need also bool detached, as we don't have refcnt 09,10: add Roman's r-b
11: rebased, don't modifynbd_free_connect_thread()name at this point 12: add Roman's r-b
13: new
14: rebasedOther patches are new.
Roman Kagan (2):
block/nbd: fix channel object leak block/nbd: ensure ->connection_thread is always validVladimir Sementsov-Ogievskiy (31):
block/nbd: fix how state is cleared onnbd_open()failure paths block/nbd:nbd_client_handshake(): fix leak ofs->iocblock/nbd: BDRVNBDState: drop unusedconnect_errandconnect_statusutil/async:aio_co_schedule(): support reschedule in same ctx block/nbd: simplify waking ofnbd_co_establish_connection()block/nbd: dropthr->state
block/nbd: bs-independent interface fornbd_co_establish_connection()block/nbd: makenbd_co_establish_connection_cancel()bs-independent block/nbd: rename NBDConnectThread to NBDClientConnection block/nbd: introducenbd_client_connection_new()block/nbd: introducenbd_client_connection_release()
nbd: move connection code from block/nbd to nbd/client-connection nbd/client-connection: useQEMU_LOCK_GUARDnbd/client-connection: add possibility of negotiation nbd/client-connection: implement connection retry nbd/client-connection: shutdown connection on release block/nbd: splitnbd_handle_updated_infoout ofnbd_client_handshake()block/nbd: use negotiation of NBDClientConnection
qemu-socket: pass monitor link tosocket_get_fddirectly block/nbd: pass monitor directly to connection thread block/nbd:nbd_teardown_connection()don't touchs->siocblock/nbd: dropBDRVNBDState::sioc
nbd/client-connection: return only one io channel
block-coroutine-wrapper: allow nonbdrv_prefix
block/nbd: splitnbd_co_do_establish_connectionout of
nbd_reconnect_attempt
nbd/client-connection: doqio_channel_set_delay(false)nbd/client-connection: add option for non-blocking connection attempt block/nbd: reusenbd_co_do_establish_connection()innbd_open()block/nbd: addnbd_clinent_connected()helper
block/nbd: safer transition to receiving request block/nbd: dropconnection_coblock/coroutines.h | 6 + include/block/aio.h | 2 +- include/block/nbd.h | 19 + include/io/channel-socket.h | 20 + include/qemu/sockets.h | 2 +- block/nbd.c | 908 +++++++---------------------- io/channel-socket.c | 17 +- nbd/client-connection.c | 364 ++++++++++++ nbd/client.c | 2 - tests/unit/test-util-sockets.c | 16 +- util/async.c | 8 + util/qemu-sockets.c | 10 +- nbd/meson.build | 1 + scripts/block-coroutine-wrapper.py | 7 +- 14 files changed, 666 insertions(+), 716 deletions(-)create mode 100644
nbd/client-connection.c