Introduce 'yank' oob qmp command to recover from hanging qemu
Subsystems (2)
v11 cover Introduce 'yank' oob qmp command to recover from hanging qemu
Hello Everyone,
So here is v11.
@Eric Blake and @Marc-André Lureau: We still need ACKs for NBD and chardev.
Changes:
v11:
-squashed MAINTAINERS update into patch 1
-move qmp doc of yank before misc
-add title for qmp docs
-change "Since:" to 6.0
-add Reviewed-by tags
v10:
-moved from qapi/misc.json to qapi/yank.json
-rename 'blockdev' -> 'block-node'
-document difference betwen migration yank instance and migrate_cancel -better document return values of yank command
-better document yank_lock
-minor style and spelling fixes
v9:
-rebase onto master
-implemented new qmp api as proposed by Markus
v8:
-add Reviewed-by and Acked-by tags
-rebase onto master
-minor change to migration
-convert to meson
-change "Since:" to 5.2
-varios code style fixes (Markus Armbruster)
-point to oob restrictions in comment to yank_register_function
(Markus Armbruster)
-improve qmp documentation (Markus Armbruster)
-document oob suitability of qio_channel and io_shutdown (Markus Armbruster)
v7:
-yank_register_instance now returns error via Error **errp instead of aborting -dropped "chardev/char.c: Check for duplicate id before creating chardev"
v6:
-add Reviewed-by and Acked-by tags
-rebase on master
-lots of changes in nbd due to rebase
-only take maintainership of util/yank.c and include/qemu/yank.h (Daniel P. Berrangé) -fix a crash discovered by the newly added chardev test
-fix the test itself
v5:
-move yank.c to util/
-move yank.h to include/qemu/
-add license to yank.h
-use const char*
-nbd: use atomic_store_release and atomic_load_aqcuire -io-channel: ensure thread-safety and document it -add myself as maintainer for yank
v4:
-fix build errors...
v3:
-don't touch softmmu/vl.c, use __contructor__ attribute instead (Paolo Bonzini) -fix build errors
-rewrite migration patch so it actually passes all tests
v2:
-don't touch io/ code anymore
-always register yank functions
-'yank' now takes a list of instances to yank -'query-yank' returns a list of yankable instances
Overview:
Hello Everyone,
In many cases, if qemu has a network connection (qmp, migration, chardev, etc.) to some other server and that server dies or hangs, qemu hangs too. These patches introduce the new 'yank' out-of-band qmp command to recover from these kinds of hangs. The different subsystems register callbacks which get executed with the yank command. For example the callback can shutdown() a socket. This is intended for the colo use-case, but it can be used for other things too of course.
Regards,
Lukas Straub
Lukas Straub (7):
Introduce yank feature
block/nbd.c: Add yank feature chardev/char-socket.c: Add yank feature
migration: Add yank feature
io/channel-tls.c: make qio_channel_tls_shutdown thread-safe
io: Document qmp oob suitability of qio_channel_shutdown and
io_shutdown
tests/test-char.c: Wait for the chardev to connect in
char_socket_client_dupid_test
MAINTAINERS | 7 ++ block/nbd.c | 154 ++++++++++++++---------- chardev/char-socket.c | 35 ++++++ include/io/channel.h | 5 +- include/qemu/yank.h | 95 +++++++++++++++ io/channel-tls.c | 6 +- migration/channel.c | 13 ++ migration/migration.c | 25 ++++ migration/multifd.c | 10 ++ migration/qemu-file-channel.c | 7 ++ migration/savevm.c | 6 + qapi/meson.build | 1 + qapi/qapi-schema.json | 1 + qapi/yank.json | 119 +++++++++++++++++++ tests/test-char.c | 1 + util/meson.build | 1 + util/yank.c | 216 ++++++++++++++++++++++++++++++++++ 17 files changed, 638 insertions(+), 64 deletions(-) create mode 100644 include/qemu/yank.h create mode 100644 qapi/yank.json create mode 100644 util/yank.c
--
2.20.1
PATCH v11 1/7 Introduce yank feature
The yank feature allows to recover from hanging qemu by "yanking" at various parts. Other qemu systems can register themselves and multiple yank functions. Then all yank functions for selected instances can be called by the 'yank' out-of-band qmp command. Available instances can be queried by a 'query-yank' oob command.
--- MAINTAINERS | 7 ++ include/qemu/yank.h | 95 +++++++++++++++++++ qapi/meson.build | 1 + qapi/qapi-schema.json | 1 + qapi/yank.json | 119 +++++++++++++++++++++++ util/meson.build | 1 + util/yank.c | 216 ++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 440 insertions(+) create mode 100644 include/qemu/yank.h create mode 100644 qapi/yank.json create mode 100644 util/yank.c
MAINTAINERS +7 -0
@@ -2688,6 +2688,13 @@ F: util/uuid.c F: include/qemu/uuid.h F: tests/test-uuid.c +Yank feature +M: Lukas Straub <[email protected]> +S: Odd fixes +F: util/yank.c +F: include/qemu/yank.h +F: qapi/yank.json + COLO Framework M: zhanghailiang <[email protected]> S: Maintained
include/qemu/yank.h new file +95 -0
95 lines not on this page · raw message
qapi/meson.build +1 -0
@@ -47,6 +47,7 @@ qapi_all_modules = [ 'trace', 'transaction', 'ui', + 'yank', ] qapi_storage_daemon_modules = [
qapi/qapi-schema.json +1 -0
@@ -86,6 +86,7 @@ { 'include': 'machine.json' } { 'include': 'machine-target.json' } { 'include': 'replay.json' } +{ 'include': 'yank.json' } { 'include': 'misc.json' } { 'include': 'misc-target.json' } { 'include': 'audio.json' }
qapi/yank.json new file +119 -0
119 lines not on this page · raw message
util/meson.build +1 -0
@@ -50,6 +50,7 @@ endif if have_system util_ss.add(when: 'CONFIG_GIO', if_true: [files('dbus.c'), gio]) + util_ss.add(files('yank.c')) endif if have_block
util/yank.c new file +216 -0
216 lines not on this page · raw message
--
2.20.1
Re: [PATCH v11 1/7] Introduce yank feature
Quoted 1 line · Lukas Straub
The yank feature allows to recover from hanging qemu by "yanking"
"allows to $verb" is not idiomatic English, better is "allows $subject to verb" or "allows ${verb}ing". In this case, I suggest "The yank feature allows the recovery of a hung qemu by "yanking" at various parts".
Quoted 21 lines
at various parts. Other qemu systems can register themselves and multiple yank functions. Then all yank functions for selected instances can be called by the 'yank' out-of-band qmp command. Available instances can be queried by a 'query-yank' oob command.
Signed-off-by Lukas Straub <[email protected]>Acked-by Stefan Hajnoczi <[email protected]>Reviewed-by Markus Armbruster <[email protected]>---
qapi/yank.json new file +10
-0+# @YankInstanceType: +# +# An enumeration of yank instance types. See @YankInstance for more +# information. +# +# Since: 6.0 +## +{ 'enum': 'YankInstanceType', + 'data': [ 'block-node', 'chardev', 'migration' ] } +
qapi/yank.json line 55
Quoted 10 lines
qapi/yank.json new file +10
-0+## +# @YankInstance: +# +# A yank instance can be yanked with the @yank qmp command to recover from a +# hanging QEMU. +# +# Currently implemented yank instances: +# - nbd block device: +# Yanking it will shut down the connection to the nbd server without +# attempting to reconnect.
Mismatch in documentation; I presume it gets cleaned up later in the series, in which case I can live with this patch as-is.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v11 1/7] Introduce yank feature
Quoted 28 lines · Eric Blake
Quoted 1 line · Lukas Straub
The yank feature allows to recover from hanging qemu by "yanking"
"allows to $verb" is not idiomatic English, better is "allows $subject to verb" or "allows ${verb}ing". In this case, I suggest "The yank feature allows the recovery of a hung qemu by "yanking" at various parts".
Quoted 20 lines
at various parts. Other qemu systems can register themselves and multiple yank functions. Then all yank functions for selected instances can be called by the 'yank' out-of-band qmp command. Available instances can be queried by a 'query-yank' oob command.
Signed-off-by Lukas Straub <[email protected]>Acked-by Stefan Hajnoczi <[email protected]>Reviewed-by Markus Armbruster <[email protected]>---
qapi/yank.json new file +10
-0+# @YankInstanceType: +# +# An enumeration of yank instance types. See @YankInstance for more +# information. +# +# Since: 6.0 +## +{ 'enum': 'YankInstanceType', + 'data': [ 'block-node', 'chardev', 'migration' ] } +
Quoted 13 lines
qapi/yank.json line 55
Quoted 10 lines
qapi/yank.json new file +10
-0+## +# @YankInstance: +# +# A yank instance can be yanked with the @yank qmp command to recover from a +# hanging QEMU. +# +# Currently implemented yank instances: +# - nbd block device: +# Yanking it will shut down the connection to the nbd server without +# attempting to reconnect.Mismatch in documentation; I presume it gets cleaned up later in the series, in which case I can live with this patch as-is.
Oh, I see. 'block-node' refers to a block node being served over NBD. So if you have multiple NBD devices, you choose which one or more nodes are yanked, rather than blindly yanking all of them at once. I just found it odd that we mention 'nbd' here but not in the enum; on the other hand, we may have more than just NBD networked devices where other types of network-enabled block nodes (ceph, sheepdog, gluster...) might also add yank support.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v11 2/7 block/nbd.c: Add yank feature
In reply to Lukas StraubRegister a yank function which shuts down the socket and sets s->state = NBD_CLIENT_QUIT. This is the same behaviour as if an error occured.
--- block/nbd.c | 154 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 93 insertions(+), 61 deletions(-)
block/nbd.c +93 -61
@@ -35,6 +35,7 @@#include "qemu/option.h" #include "qemu/cutils.h" #include "qemu/main-loop.h" +#include "qemu/atomic.h" #include "qapi/qapi-visit-sockets.h" #include "qapi/qmp/qstring.h"
@@ -44,6 +45,8 @@#include "block/nbd.h" #include "block/block_int.h" +#include "qemu/yank.h" +#define EN_OPTSTR ":exportname="#define MAX_NBD_REQUESTS 16
@@ -141,14 +144,13 @@ typedef struct BDRVNBDState {NBDConnectThread *connect_thread; } BDRVNBDState; -static QIOChannelSocket *nbd_establish_connection (SocketAddress *saddr, -Error **errp); -static QIOChannelSocket *nbd_co_establish_connection (BlockDriverState *bs, -Error **errp); +static int nbd_establish_connection (BlockDriverState *bs,SocketAddress *saddr, +Error **errp); +static int nbd_co_establish_connection (BlockDriverState *bs,Error **errp);static void nbd_co_establish_connection_cancel (BlockDriverState *bs,bool detach); -static int nbd_client_handshake (BlockDriverState *bs,QIOChannelSocket *sioc, -Error **errp); +static int nbd_client_handshake (BlockDriverState *bs,Error **errp); +static void nbd_yank (void *opaque);static void nbd_clear_bdrvstate (BDRVNBDState *s) {
@@ -166,12 +168,12 @@ static void nbd_clear_bdrvstate(BDRVNBDState *s)static void nbd_channel_error (BDRVNBDState *s,int ret) {if (ret == -EIO ) { -if (s->state ==NBD_CLIENT_CONNECTED ) { +if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTED ) { s->state = s->reconnect_delay ?NBD_CLIENT_CONNECTING_WAIT :NBD_CLIENT_CONNECTING_NOWAIT ; } }else { -if (s->state ==NBD_CLIENT_CONNECTED ) { +if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTED ) {qio_channel_shutdown (s->ioc,QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); } s->state =NBD_CLIENT_QUIT ;
@@ -204,7 +206,7 @@ static void reconnect_delay_timer_cb(void *opaque) {BDRVNBDState *s = opaque; -if (s->state ==NBD_CLIENT_CONNECTING_WAIT ) { +if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ) { s->state =NBD_CLIENT_CONNECTING_NOWAIT ;while (qemu_co_enter_next (&s->free_sema,NULL )) {/* Resume all queued requests */
@@ -216,7 +218,7 @@ static void reconnect_delay_timer_cb(void *opaque)static void reconnect_delay_timer_init (BDRVNBDState *s,uint64_t expire_time_ns) { -if (s->state !=NBD_CLIENT_CONNECTING_WAIT ) { +if (qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTING_WAIT ) {return ; }
@@ -261,7 +263,7 @@ static void nbd_client_attach_aio_context(BlockDriverState *bs,* s->connection_co is either yielded from nbd_receive_reply or from * nbd_co_reconnect_loop() */ -if (s->state ==NBD_CLIENT_CONNECTED ) { +if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTED ) {qio_channel_attach_aio_context (QIO_CHANNEL (s->ioc), new_context); }
@@ -287,7 +289,7 @@ static void coroutine_fn nbd_client_co_drain_begin(BlockDriverState *bs)reconnect_delay_timer_del (s); -if (s->state ==NBD_CLIENT_CONNECTING_WAIT ) { +if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ) { s->state =NBD_CLIENT_CONNECTING_NOWAIT ;qemu_co_queue_restart_all (&s->free_sema); }
@@ -338,13 +340,14 @@ static void nbd_teardown_connection(BlockDriverState *bs)static bool nbd_client_connecting (BDRVNBDState *s) { -return s->state ==NBD_CLIENT_CONNECTING_WAIT || - s->state ==NBD_CLIENT_CONNECTING_NOWAIT ; +NBDClientState state =qatomic_load_acquire (&s->state); +return state ==NBD_CLIENT_CONNECTING_WAIT || + state ==NBD_CLIENT_CONNECTING_NOWAIT ; }static bool nbd_client_connecting_wait (BDRVNBDState *s) { -return s->state ==NBD_CLIENT_CONNECTING_WAIT ; +return qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ; }static void connect_bh (void *opaque)
@@ -424,12 +427,12 @@ static void *connect_thread_func(void *opaque)return NULL ; } -static QIOChannelSocket *coroutine_fn +static int coroutine_fnnbd_co_establish_connection (BlockDriverState *bs,Error **errp) { +int ret;QemuThread thread;BDRVNBDState *s = bs->opaque; -QIOChannelSocket *res;NBDConnectThread *thr = s->connect_thread;qemu_mutex_lock (&thr->mutex);
@@ -446,10 +449,12 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)case CONNECT_THREAD_SUCCESS :/* Previous attempt finally succeeded in background */ thr->state =CONNECT_THREAD_NONE ; - res = thr->sioc; + s->sioc = thr->sioc; thr->sioc =NULL ; +yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), + nbd_yank, bs);qemu_mutex_unlock (&thr->mutex); -return res; +return 0 ;case CONNECT_THREAD_RUNNING :/* Already running, will wait */ break ;
@@ -481,8 +486,13 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp) thr->state =CONNECT_THREAD_NONE ;error_propagate (errp, thr->err); thr->err =NULL ; - res = thr->sioc; + s->sioc = thr->sioc; thr->sioc =NULL ; +if (s->sioc) { +yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), + nbd_yank, bs); + } + ret = (s->sioc ?0 :-1 );break ;case CONNECT_THREAD_RUNNING :case CONNECT_THREAD_RUNNING_DETACHED :
@@ -491,7 +501,7 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)* failed. Still connect thread is executing in background, and its * result may be used for next connection attempt. */ - res =NULL ; + ret =-1 ;error_setg (errp,"Connection attempt cancelled by other operation" );break ;
@@ -508,7 +518,7 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)qemu_mutex_unlock (&thr->mutex); -return res; +return ret; }/*
@@ -561,7 +571,6 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) {int ret;Error *local_err =NULL ; -QIOChannelSocket *sioc;if (!nbd_client_connecting (s)) {return ;
@@ -594,21 +603,22 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s)/* Finalize previous connection if any */ if (s->ioc) {qio_channel_detach_aio_context (QIO_CHANNEL (s->ioc)); +yank_unregister_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), + nbd_yank, s->bs);object_unref (OBJECT (s->sioc)); s->sioc =NULL ;object_unref (OBJECT (s->ioc)); s->ioc =NULL ; } - sioc =nbd_co_establish_connection (s->bs, &local_err); -if (!sioc) { +if (nbd_co_establish_connection (s->bs, &local_err) <0 ) { ret = -ECONNREFUSED ; goto out; }bdrv_dec_in_flight (s->bs); - ret =nbd_client_handshake (s->bs, sioc, &local_err); + ret =nbd_client_handshake (s->bs, &local_err);if (s->drained) { s->wait_drained_end = true;
@@ -640,7 +650,7 @@ static coroutine_fn void nbd_co_reconnect_loop(BDRVNBDState *s)uint64_t timeout =1 *NANOSECONDS_PER_SECOND ;uint64_t max_timeout =16 *NANOSECONDS_PER_SECOND ; -if (s->state ==NBD_CLIENT_CONNECTING_WAIT ) { +if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTING_WAIT ) {reconnect_delay_timer_init (s,qemu_clock_get_ns (QEMU_CLOCK_REALTIME ) + s->reconnect_delay *NANOSECONDS_PER_SECOND ); }
@@ -683,7 +693,7 @@ static coroutine_fn void nbd_connection_entry(void *opaque)int ret =0 ;Error *local_err =NULL ; -while (s->state !=NBD_CLIENT_QUIT ) { +while (qatomic_load_acquire (&s->state) !=NBD_CLIENT_QUIT ) {/* * The NBD client can only really be considered idle when it has * yielded from qio_channel_readv_all_eof(), waiting for data. This is
@@ -698,7 +708,7 @@ static coroutine_fn void nbd_connection_entry(void *opaque)nbd_co_reconnect_loop (s); } -if (s->state !=NBD_CLIENT_CONNECTED ) { +if (qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTED ) {continue ; }
@@ -753,6 +763,8 @@ static coroutine_fn void nbd_connection_entry(void *opaque) s->connection_co =NULL ;if (s->ioc) {qio_channel_detach_aio_context (QIO_CHANNEL (s->ioc)); +yank_unregister_function (BLOCKDEV_YANK_INSTANCE (s->bs->node_name), + nbd_yank, s->bs);object_unref (OBJECT (s->sioc)); s->sioc =NULL ;object_unref (OBJECT (s->ioc));
@@ -777,7 +789,7 @@ static int nbd_co_send_request(BlockDriverState *bs,qemu_co_queue_wait (&s->free_sema, &s->send_mutex); } -if (s->state !=NBD_CLIENT_CONNECTED ) { +if (qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTED ) { rc = -EIO ; goto err; }
@@ -804,7 +816,8 @@ static int nbd_co_send_request(BlockDriverState *bs,if (qiov) {qio_channel_set_cork (s->ioc, true); rc =nbd_send_request (s->ioc, request); -if (rc >=0 && s->state ==NBD_CLIENT_CONNECTED ) { +if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTED && + rc >=0 ) {if (qio_channel_writev_all (s->ioc, qiov->iov, qiov->niov,NULL ) <0 ) { rc = -EIO ;
@@ -1129,7 +1142,7 @@ static coroutine_fn int nbd_co_do_receive_one_chunk( s->requests[i].receiving = true;qemu_coroutine_yield (); s->requests[i].receiving = false; -if (s->state !=NBD_CLIENT_CONNECTED ) { +if (qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTED ) {error_setg (errp,"Connection closed" );return -EIO ; }
@@ -1288,7 +1301,7 @@ static bool nbd_reply_chunk_iter_receive(BDRVNBDState *s,NBDReply local_reply;NBDStructuredReplyChunk *chunk;Error *local_err =NULL ; -if (s->state !=NBD_CLIENT_CONNECTED ) { +if (qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTED ) {error_setg (&local_err,"Connection closed" );nbd_iter_channel_error (iter, -EIO , &local_err); goto break_loop;
@@ -1313,7 +1326,8 @@ static bool nbd_reply_chunk_iter_receive(BDRVNBDState *s, }/* Do not execute the body of NBD_FOREACH_REPLY_CHUNK for simple reply. */ -if (nbd_reply_is_simple (reply) || s->state !=NBD_CLIENT_CONNECTED ) { +if (nbd_reply_is_simple (reply) || +qatomic_load_acquire (&s->state) !=NBD_CLIENT_CONNECTED ) { goto break_loop; }
@@ -1745,6 +1759,15 @@ static int nbd_client_reopen_prepare(BDRVReopenState *state,return 0 ; } +static void nbd_yank (void *opaque) +{ +BlockDriverState *bs = opaque; +BDRVNBDState *s = (BDRVNBDState *)bs->opaque; + +qatomic_store_release (&s->state,NBD_CLIENT_QUIT ); +qio_channel_shutdown (QIO_CHANNEL (s->sioc),QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); +} +static void nbd_client_close (BlockDriverState *bs) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
@@ -1757,52 +1780,53 @@ static void nbd_client_close(BlockDriverState *bs)nbd_teardown_connection (bs); } -static QIOChannelSocket *nbd_establish_connection (SocketAddress *saddr, -Error **errp) +static int nbd_establish_connection (BlockDriverState *bs, +SocketAddress *saddr, +Error **errp) {ERRP_GUARD (); -QIOChannelSocket *sioc; +BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - sioc =qio_channel_socket_new (); -qio_channel_set_name (QIO_CHANNEL (sioc),"nbd-client" ); + s->sioc =qio_channel_socket_new (); +qio_channel_set_name (QIO_CHANNEL (s->sioc),"nbd-client" ); -qio_channel_socket_connect_sync (sioc, saddr, errp); +qio_channel_socket_connect_sync (s->sioc, saddr, errp);if (*errp) { -object_unref (OBJECT (sioc)); -return NULL ; +object_unref (OBJECT (s->sioc)); + s->sioc =NULL ; +return -1 ; } -qio_channel_set_delay (QIO_CHANNEL (sioc), false); +yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), nbd_yank, bs); +qio_channel_set_delay (QIO_CHANNEL (s->sioc), false); -return sioc; +return 0 ; } -/* nbd_client_handshake takes ownership on sioc. On failure it is unref'ed. */ -static int nbd_client_handshake (BlockDriverState *bs,QIOChannelSocket *sioc, -Error **errp) +/* nbd_client_handshake takes ownership on s->sioc. On failure it's unref'ed. */ +static int nbd_client_handshake (BlockDriverState *bs,Error **errp) {BDRVNBDState *s = (BDRVNBDState *)bs->opaque;AioContext *aio_context =bdrv_get_aio_context (bs);int ret;trace_nbd_client_handshake (s->export); - - s->sioc = sioc; - -qio_channel_set_blocking (QIO_CHANNEL (sioc), false,NULL ); -qio_channel_attach_aio_context (QIO_CHANNEL (sioc), aio_context); +qio_channel_set_blocking (QIO_CHANNEL (s->sioc), false,NULL ); +qio_channel_attach_aio_context (QIO_CHANNEL (s->sioc), aio_context); s->info.request_sizes = true; s->info.structured_reply = true; s->info.base_allocation = true; s->info.x_dirty_bitmap =g_strdup (s->x_dirty_bitmap); s->info.name =g_strdup (s->export ?:"" ); - ret =nbd_receive_negotiate (aio_context,QIO_CHANNEL (sioc), s->tlscreds, + ret =nbd_receive_negotiate (aio_context,QIO_CHANNEL (s->sioc), s->tlscreds, s->hostname, &s->ioc, &s->info, errp);g_free (s->info.x_dirty_bitmap);g_free (s->info.name);if (ret <0 ) { -object_unref (OBJECT (sioc)); +yank_unregister_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), + nbd_yank, bs); +object_unref (OBJECT (s->sioc)); s->sioc =NULL ;return ret; }
@@ -1835,7 +1859,7 @@ static int nbd_client_handshake(BlockDriverState *bs, QIOChannelSocket *sioc, }if (!s->ioc) { - s->ioc =QIO_CHANNEL (sioc); + s->ioc =QIO_CHANNEL (s->sioc);object_ref (OBJECT (s->ioc)); }
@@ -1851,9 +1875,11 @@ static int nbd_client_handshake(BlockDriverState *bs, QIOChannelSocket *sioc, {NBDRequest request = { .type =NBD_CMD_DISC }; -nbd_send_request (s->ioc ?:QIO_CHANNEL (sioc), &request); +nbd_send_request (s->ioc ?:QIO_CHANNEL (s->sioc), &request); -object_unref (OBJECT (sioc)); +yank_unregister_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), + nbd_yank, bs); +object_unref (OBJECT (s->sioc)); s->sioc =NULL ;return ret;
@@ -2245,7 +2271,6 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, {int ret;BDRVNBDState *s = (BDRVNBDState *)bs->opaque; -QIOChannelSocket *sioc; ret =nbd_process_options (bs, options, errp);if (ret <0 ) {
@@ -2256,17 +2281,23 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,qemu_co_mutex_init (&s->send_mutex);qemu_co_queue_init (&s->free_sema); +yank_register_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name), errp); +if (*errp) { +return -EEXIST ; + } +/* * establish TCP connection, return error if it fails * TODO: Configurable retry-until-timeout behaviour. */ - sioc =nbd_establish_connection (s->saddr, errp); -if (!sioc) { +if (nbd_establish_connection (bs, s->saddr, errp) <0 ) { +yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name));return -ECONNREFUSED ; } - ret =nbd_client_handshake (bs, sioc, errp); + ret =nbd_client_handshake (bs, errp);if (ret <0 ) { +yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name));nbd_clear_bdrvstate (s);return ret; }
@@ -2326,6 +2357,7 @@ static void nbd_close(BlockDriverState *bs)BDRVNBDState *s = bs->opaque;nbd_client_close (bs); +yank_unregister_instance (BLOCKDEV_YANK_INSTANCE (bs->node_name));nbd_clear_bdrvstate (s); }
--
2.20.1
Re: [PATCH v11 2/7] block/nbd.c: Add yank feature
Quoted 3 lines · Lukas Straub
Register a yank function which shuts down the socket and sets
s->state=NBD_CLIENT_QUIT. This is the same behaviour as if an error occured.
occurred
Quoted 9 lines
Signed-off-by Lukas Straub <[email protected]>Acked-by Stefan Hajnoczi <[email protected]>--- block/nbd.c | 154 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 93 insertions(+), 61 deletions(-)
block/nbd.c static void nbd_clear_bdrvstate(BDRVNBDState *s) · line 171
Quoted 5 lines
block/nbd.c +1
-1@@ -166,12 +168,12 @@ static void nbd_clear_bdrvstate(BDRVNBDState *s)static void nbd_channel_error (BDRVNBDState *s,int ret) {if (ret == -EIO ) { -if (s->state ==NBD_CLIENT_CONNECTED ) { +if (qatomic_load_acquire (&s->state) ==NBD_CLIENT_CONNECTED ) {
This may have interesting interactions with Vladimir's latest patches to make NBD connection re-startable, but we'll sort that out as needed.
https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg07012.html
The patch seems big; I might have broken it into two pieces (conversion of existing logic to use qatomic_*() accesses instead of direct s->state manipulation, and then adding new logic). But I'm not going to hold up the series demanding for a split at this time.
Quoted 17 lines
block/nbd.c +2
-2@@ -424,12 +427,12 @@ static void *connect_thread_func(void *opaque)return NULL ; } -static QIOChannelSocket *coroutine_fn +static int coroutine_fnnbd_co_establish_connection (BlockDriverState *bs,Error **errp) { +int ret;QemuThread thread;BDRVNBDState *s = bs->opaque; -QIOChannelSocket *res;NBDConnectThread *thr = s->connect_thread;qemu_mutex_lock (&thr->mutex);@@ -446,10 +449,12 @@ nbd_co_establish_connection(BlockDriverState *bs, Error **errp)case CONNECT_THREAD_SUCCESS :
block/nbd.c nbd_co_establish_connection(BlockDriverState *bs, Error **errp) · line 457
Quoted 10 lines
block/nbd.c +4
-2/* Previous attempt finally succeeded in background */ thr->state =CONNECT_THREAD_NONE ; - res = thr->sioc; + s->sioc = thr->sioc; thr->sioc =NULL ; +yank_register_function (BLOCKDEV_YANK_INSTANCE (bs->node_name), + nbd_yank, bs);qemu_mutex_unlock (&thr->mutex); -return res; +return 0 ;
Looks sensible.
block/nbd.c static int nbd_client_reopen_prepare(BDRVReopenState *state, · line 1770
Quoted 13 lines
block/nbd.c +9
-0@@ -1745,6 +1759,15 @@ static int nbd_client_reopen_prepare(BDRVReopenState *state,return 0 ; } +static void nbd_yank (void *opaque) +{ +BlockDriverState *bs = opaque; +BDRVNBDState *s = (BDRVNBDState *)bs->opaque; + +qatomic_store_release (&s->state,NBD_CLIENT_QUIT ); +qio_channel_shutdown (QIO_CHANNEL (s->sioc),QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); +} +
Yep, that does indeed tell qemu to give up on the NBD socket right away.
And sorry it's taken me so long to actually stare at this series.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v11 2/7] block/nbd.c: Add yank feature
In reply to Lukas StraubQuoted 6 lines · Lukas Straub
Register a yank function which shuts down the socket and sets
s->state=NBD_CLIENT_QUIT. This is the same behaviour as if an error occured.Signed-off-by Lukas Straub<[email protected]>Acked-by Stefan Hajnoczi<[email protected]>
Hi! Could I ask, what's the reason for qatomic_load_acquire access to s->state? Is there same bug fixed? Or is it related somehow to new feature?
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v11 2/7] block/nbd.c: Add yank feature
Quoted 10 lines · On Wed, 2 Dec 2020 15:18:48 +0300 Vladimir Sementsov-Ogievskiy…
Quoted 6 lines · Lukas Straub
Register a yank function which shuts down the socket and sets
s->state=NBD_CLIENT_QUIT. This is the same behaviour as if an error occured.Signed-off-by Lukas Straub<[email protected]>Acked-by Stefan Hajnoczi<[email protected]>Hi! Could I ask, what's the reason for
qatomic_load_acquireaccess tos->state? Is there same bug fixed? Or is it related somehow to new feature?
Hi,
This is for the new feature, as the yank function runs in a separate thread.
Regards,
Lukas Straub
PATCH v11 3/7 chardev/char-socket.c: Add yank feature
In reply to Lukas StraubRegister a yank function to shutdown the socket on yank.
--- chardev/char-socket.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
chardev/char-socket.c +35 -0
@@ -34,6 +34,7 @@#include "qapi/error.h" #include "qapi/clone-visitor.h" #include "qapi/qapi-visit-sockets.h" +#include "qemu/yank.h" #include "chardev/char-io.h" #include "qom/object.h"
@@ -70,6 +71,7 @@ struct SocketChardev {size_t read_msgfds_num;int *write_msgfds;size_t write_msgfds_num; +bool registered_yank;SocketAddress *addr;bool is_listen;
@@ -415,6 +417,12 @@ static void tcp_chr_free_connection(Chardev *chr)tcp_set_msgfds (chr,NULL ,0 );remove_fd_in_watch (chr); +if (s->state ==TCP_CHARDEV_STATE_CONNECTING + || s->state ==TCP_CHARDEV_STATE_CONNECTED ) { +yank_unregister_function (CHARDEV_YANK_INSTANCE (chr->label), + yank_generic_iochannel, +QIO_CHANNEL (s->sioc)); + }object_unref (OBJECT (s->sioc)); s->sioc =NULL ;object_unref (OBJECT (s->ioc));
@@ -932,6 +940,9 @@ static int tcp_chr_add_client(Chardev *chr, int fd) }tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING );tcp_chr_set_client_ioc_name (chr, sioc); +yank_register_function (CHARDEV_YANK_INSTANCE (chr->label), + yank_generic_iochannel, +QIO_CHANNEL (sioc)); ret =tcp_chr_new_client (chr, sioc);object_unref (OBJECT (sioc));return ret;
@@ -946,6 +957,9 @@ static void tcp_chr_accept(QIONetListener *listener,tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING );tcp_chr_set_client_ioc_name (chr, cioc); +yank_register_function (CHARDEV_YANK_INSTANCE (chr->label), + yank_generic_iochannel, +QIO_CHANNEL (cioc));tcp_chr_new_client (chr, cioc); }
@@ -961,6 +975,9 @@ static int tcp_chr_connect_client_sync(Chardev *chr, Error **errp)object_unref (OBJECT (sioc));return -1 ; } +yank_register_function (CHARDEV_YANK_INSTANCE (chr->label), + yank_generic_iochannel, +QIO_CHANNEL (sioc));tcp_chr_new_client (chr, sioc);object_unref (OBJECT (sioc));return 0 ;
@@ -976,6 +993,9 @@ static void tcp_chr_accept_server_sync(Chardev *chr)tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING ); sioc =qio_net_listener_wait_client (s->listener);tcp_chr_set_client_ioc_name (chr, sioc); +yank_register_function (CHARDEV_YANK_INSTANCE (chr->label), + yank_generic_iochannel, +QIO_CHANNEL (sioc));tcp_chr_new_client (chr, sioc);object_unref (OBJECT (sioc)); }
@@ -1086,6 +1106,9 @@ static void char_socket_finalize(Object *obj)object_unref (OBJECT (s->tls_creds)); }g_free (s->tls_authz); +if (s->registered_yank) { +yank_unregister_instance (CHARDEV_YANK_INSTANCE (chr->label)); + }qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); }
@@ -1101,6 +1124,9 @@ static void qemu_chr_socket_connected(QIOTask *task, void *opaque)if (qio_task_propagate_error (task, &err)) {tcp_chr_change_state (s,TCP_CHARDEV_STATE_DISCONNECTED ); +yank_unregister_function (CHARDEV_YANK_INSTANCE (chr->label), + yank_generic_iochannel, +QIO_CHANNEL (sioc));check_report_connect_error (chr, err); goto cleanup; }
@@ -1134,6 +1160,9 @@ static void tcp_chr_connect_client_async(Chardev *chr)tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING ); sioc =qio_channel_socket_new ();tcp_chr_set_client_ioc_name (chr, sioc); +yank_register_function (CHARDEV_YANK_INSTANCE (chr->label), + yank_generic_iochannel, +QIO_CHANNEL (sioc));/* * Normally code would use the qio_channel_socket_connect_async * method which uses a QIOTask + qio_task_set_error internally
@@ -1376,6 +1405,12 @@ static void qmp_chardev_open_socket(Chardev *chr,qemu_chr_set_feature (chr,QEMU_CHAR_FEATURE_FD_PASS ); } +yank_register_instance (CHARDEV_YANK_INSTANCE (chr->label), errp); +if (*errp) { +return ; + } + s->registered_yank = true; +/* be isn't opened until we get a connection */ *be_opened = false;
--
2.20.1
PATCH v11 4/7 migration: Add yank feature
In reply to Lukas StraubRegister yank functions on sockets to shut them down.
--- migration/channel.c | 13 +++++++++++++ migration/migration.c | 25 +++++++++++++++++++++++++ migration/multifd.c | 10 ++++++++++ migration/qemu-file-channel.c | 7 +++++++ migration/savevm.c | 6 ++++++ 5 files changed, 61 insertions(+)
migration/channel.c +13 -0
@@ -18,6 +18,8 @@#include "trace.h" #include "qapi/error.h" #include "io/channel-tls.h" +#include "io/channel-socket.h" +#include "qemu/yank.h" /** * @migration_channel_process_incoming - Create new incoming migration channel
@@ -35,6 +37,11 @@ void migration_channel_process_incoming(QIOChannel *ioc)trace_migration_set_incoming_channel ( ioc,object_get_typename (OBJECT (ioc))); +if (object_dynamic_cast (OBJECT (ioc),TYPE_QIO_CHANNEL_SOCKET )) { +yank_register_function (MIGRATION_YANK_INSTANCE , yank_generic_iochannel, +QIO_CHANNEL (ioc)); + } +if (s->parameters.tls_creds && *s->parameters.tls_creds && !object_dynamic_cast(OBJECT(ioc),
@@ -67,6 +74,12 @@ void migration_channel_connect(MigrationState *s, ioc, object_get_typename(OBJECT(ioc)), hostname, error); if (!error) { + if (object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_SOCKET)) { + yank_register_function(MIGRATION_YANK_INSTANCE, + yank_generic_iochannel, + QIO_CHANNEL(ioc)); + } + if (s->parameters.tls_creds && *s->parameters.tls_creds && !object_dynamic_cast(OBJECT(ioc),
migration/migration.c +25 -0
@@ -56,6 +56,7 @@#include "net/announce.h" #include "qemu/queue.h" #include "multifd.h" +#include "qemu/yank.h" #ifdef CONFIG_VFIO #include "hw/vfio/vfio-common.h"
@@ -252,6 +253,8 @@ void migration_incoming_state_destroy(void)qapi_free_SocketAddressList (mis->socket_address_list); mis->socket_address_list =NULL ; } + +yank_unregister_instance (MIGRATION_YANK_INSTANCE ); }static void migrate_generate_event (int new_state)
@@ -429,8 +432,14 @@ void qemu_start_incoming_migration(const char *uri, Error **errp) {const char *p =NULL ; +yank_register_instance (MIGRATION_YANK_INSTANCE , errp); +if (*errp) { +return ; + } +qapi_event_send_migration (MIGRATION_STATUS_SETUP );if (!strcmp (uri,"defer" )) { +yank_unregister_instance (MIGRATION_YANK_INSTANCE );deferred_incoming_migration (errp); } else if (strstart(uri, "tcp:", &p) || strstart(uri, "unix:", NULL) ||
@@ -445,6 +454,7 @@ void qemu_start_incoming_migration(const char *uri, Error **errp) }else if (strstart (uri,"fd:" , &p)) {fd_start_incoming_migration (p, errp); }else { +yank_unregister_instance (MIGRATION_YANK_INSTANCE );error_setg (errp,"unknown migration protocol: %s" , uri); } }
@@ -1750,6 +1760,7 @@ static void migrate_fd_cleanup(MigrationState *s) }notifier_list_notify (&migration_state_notifiers, s);block_cleanup_parameters (s); +yank_unregister_instance (MIGRATION_YANK_INSTANCE ); }static void migrate_fd_cleanup_schedule (MigrationState *s)
@@ -2024,6 +2035,7 @@ void qmp_migrate_recover(const char *uri, Error **errp)* only re-setup the migration stream and poke existing migration * to continue using that newly established channel. */ +yank_unregister_instance (MIGRATION_YANK_INSTANCE );qemu_start_incoming_migration (uri, errp); }
@@ -2161,6 +2173,13 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,return ; } +if (!(has_resume && resume)) { +yank_register_instance (MIGRATION_YANK_INSTANCE , errp); +if (*errp) { +return ; + } + } +if (strstart (uri,"tcp:" , &p) ||strstart (uri,"unix:" ,NULL ) ||strstart (uri,"vsock:" ,NULL )) {
@@ -2174,6 +2193,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, }else if (strstart (uri,"fd:" , &p)) {fd_start_outgoing_migration (s, p, &local_err); }else { +if (!(has_resume && resume)) { +yank_unregister_instance (MIGRATION_YANK_INSTANCE ); + }error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"uri" ,"a valid migration protocol" ); migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
@@ -2183,6 +2205,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, }if (local_err) { +if (!(has_resume && resume)) { +yank_unregister_instance (MIGRATION_YANK_INSTANCE ); + }migrate_fd_error (s, local_err);error_propagate (errp, local_err);return ;
migration/multifd.c +10 -0
@@ -25,6 +25,9 @@#include "trace.h" #include "multifd.h" +#include "qemu/yank.h" +#include "io/channel-socket.h" +/* Multiple fd's */ #define MULTIFD_MAGIC 0x11223344U
@@ -974,6 +977,13 @@ int multifd_load_cleanup(Error **errp)for (i =0 ; i <migrate_multifd_channels (); i++) {MultiFDRecvParams *p = &multifd_recv_state->params[i]; +if (object_dynamic_cast (OBJECT (p->c),TYPE_QIO_CHANNEL_SOCKET ) + &&OBJECT (p->c)->ref ==1 ) { +yank_unregister_function (MIGRATION_YANK_INSTANCE , + yank_generic_iochannel, +QIO_CHANNEL (p->c)); + } +object_unref (OBJECT (p->c)); p->c =NULL ;qemu_mutex_destroy (&p->mutex);
migration/qemu-file-channel.c +7 -0
@@ -27,6 +27,7 @@#include "qemu-file.h" #include "io/channel-socket.h" #include "qemu/iov.h" +#include "qemu/yank.h" static ssize_t channel_writev_buffer(void *opaque,
@@ -104,6 +105,12 @@ static int channel_close(void *opaque, Error **errp)int ret;QIOChannel *ioc =QIO_CHANNEL (opaque); ret =qio_channel_close (ioc, errp); +if (object_dynamic_cast (OBJECT (ioc),TYPE_QIO_CHANNEL_SOCKET ) + &&OBJECT (ioc)->ref ==1 ) { +yank_unregister_function (MIGRATION_YANK_INSTANCE , + yank_generic_iochannel, +QIO_CHANNEL (ioc)); + }object_unref (OBJECT (ioc));return ret; }
migration/savevm.c +6 -0
@@ -62,6 +62,7 @@#include "migration/colo.h" #include "qemu/bitmap.h" #include "net/announce.h" +#include "qemu/yank.h" const unsigned int postcopy_ram_discard_version;
@@ -3008,6 +3009,11 @@ int load_snapshot(const char *name, Error **errp)qemu_system_reset (SHUTDOWN_CAUSE_NONE ); mis->from_src_file = f; +yank_register_instance (MIGRATION_YANK_INSTANCE , errp); +if (*errp) { + ret = -EINVAL ; + goto err_drain; + }aio_context_acquire (aio_context); ret =qemu_loadvm_state (f);migration_incoming_state_destroy ();
--
2.20.1
PATCH v11 5/7 io/channel-tls.c: make qio_channel_tls_shutdown thread-safe
In reply to Lukas StraubMake qio_channel_tls_shutdown thread-safe by using atomics when accessing tioc->shutdown.
--- io/channel-tls.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
io/channel-tls.c +4 -2
@@ -23,6 +23,7 @@#include "qemu/module.h" #include "io/channel-tls.h" #include "trace.h" +#include "qemu/atomic.h" static ssize_t qio_channel_tls_write_handler(const char *buf,
@@ -277,7 +278,8 @@ static ssize_t qio_channel_tls_readv(QIOChannel *ioc,return QIO_CHANNEL_ERR_BLOCK ; } }else if (errno ==ECONNABORTED && - (tioc->shutdown &QIO_CHANNEL_SHUTDOWN_READ )) { + (qatomic_load_acquire (&tioc->shutdown) & +QIO_CHANNEL_SHUTDOWN_READ )) {return 0 ; }
@@ -361,7 +363,7 @@ static int qio_channel_tls_shutdown(QIOChannel *ioc, {QIOChannelTLS *tioc =QIO_CHANNEL_TLS (ioc); - tioc->shutdown |= how; +qatomic_or (&tioc->shutdown, how);return qio_channel_shutdown (tioc->master, how, errp); }
--
2.20.1
PATCH v11 6/7 io: Document qmp oob suitability of qio_channel_shutdown and io_shutdown
In reply to Lukas StraubMigration and yank code assume that qio_channel_shutdown is thread -safe and can be called from qmp oob handler. Document this after checking the code.
--- include/io/channel.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
include/io/channel.h +4 -1
@@ -92,7 +92,8 @@ struct QIOChannel {* provide additional optional features. * * Consult the corresponding public API docs for a description -* of the semantics of each callback +* of the semantics of each callback. io_shutdown in particular +* must be thread-safe, terminate quickly and must not block. */ struct QIOChannelClass {ObjectClass parent;
@@ -510,6 +511,8 @@ int qio_channel_close(QIOChannel *ioc,* QIO_CHANNEL_FEATURE_SHUTDOWN prior to calling * this method. * +* This function is thread-safe, terminates quickly and does not block. +* * Returns: 0 on success, -1 on error */ int qio_channel_shutdown(QIOChannel *ioc,
--
2.20.1
PATCH v11 7/7 tests/test-char.c: Wait for the chardev to connect in char_socket_client_dupid_test
In reply to Lukas StraubA connecting chardev object has an additional reference by the connecting thread, so if the chardev is still connecting by the end of the test, then the chardev object won't be freed. This in turn means that the yank instance won't be unregistered and when running the next test-case yank_register_instance will abort, because the yank instance is already/still registered.
--- tests/test-char.c | 1 + 1 file changed, 1 insertion(+)
tests/test-char.c +1 -0
@@ -937,6 +937,7 @@ static void char_socket_client_dupid_test(gconstpointer opaque)g_assert_nonnull (opts); chr1 =qemu_chr_new_from_opts (opts,NULL , &error_abort);g_assert_nonnull (chr1); +qemu_chr_wait_connected (chr1, &error_abort); chr2 =qemu_chr_new_from_opts (opts,NULL , &local_err);g_assert_null (chr2);
--
2.20.1
Quoted 3 lines · Lukas Straub
Hello Everyone,
So here is v11.
@Eric Blake and @Marc-André Lureau: We still need ACKs for NBD and chardev.
Once we have them, I can take the series through my tree.