Introduce 'yank' oob qmp command to recover from hanging qemu
Subsystems (2)
PATCH v4 2/4 block/nbd.c: Add yank feature
In reply to Lukas StraubRegister a yank function which shuts down the socket and sets s->state = NBD_CLIENT_QUIT. This is the same behaviour as if an error occured.
--- Makefile.objs | 1 + block/nbd.c | 101 ++++++++++++++++++++++++++++++++------------------ 2 files changed, 65 insertions(+), 37 deletions(-)
Makefile.objs +1 -0
@@ -18,6 +18,7 @@ block-obj-y += block.o blockjob.o job.o block-obj-y += block/ scsi/ block-obj-y += qemu-io-cmds.o block-obj-$(CONFIG_REPLICATION) += replication.o +block-obj-y += yank.o block-obj-m = block/
block/nbd.c +64 -37
290 lines not on this page · raw message
--
2.20.1
v4 cover Introduce 'yank' oob qmp command to recover from hanging qemu
Hello Everyone,
In many cases, if qemu has a network connection (qmp, migration, chardev, etc.) to some other server and that server dies or hangs, qemu hangs too. These patches introduce the new 'yank' out-of-band qmp command to recover from these kinds of hangs. The different subsystems register callbacks which get executed with the yank command. For example the callback can shutdown() a socket. This is intended for the colo use-case, but it can be used for other things too of course.
Regards,
Lukas Straub
v4:
-fix build errors...
v3:
-don't touch softmmu/vl.c, use __contructor__ attribute instead (Paolo Bonzini) -fix build errors
-rewrite migration patch so it actually passes all tests
v2:
-don't touch io/ code anymore
-always register yank functions
-'yank' now takes a list of instances to yank -'query-yank' returns a list of yankable instances
Lukas Straub (4):
Introduce yank feature
block/nbd.c: Add yank feature
chardev/char-socket.c: Add yank feature
migration: Add yank feature
Makefile.objs | 3 + block/nbd.c | 101 ++++++++++++-------- chardev/char-socket.c | 24 +++++ migration/channel.c | 12 +++ migration/migration.c | 18 +++- migration/multifd.c | 10 ++ migration/qemu-file-channel.c | 6 ++ migration/savevm.c | 2 + qapi/misc.json | 45 +++++++++ tests/Makefile.include | 2 +- yank.c | 174 ++++++++++++++++++++++++++++++++++ yank.h | 67 +++++++++++++ 12 files changed, 425 insertions(+), 39 deletions(-) create mode 100644 yank.c create mode 100644 yank.h
--
2.20.1
PATCH v4 3/4 chardev/char-socket.c: Add yank feature
Register a yank function to shutdown the socket on yank.
--- Makefile.objs | 1 + chardev/char-socket.c | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+)
Makefile.objs +1 -0
@@ -10,6 +10,7 @@ qom-obj-y = qom/ ifeq ($(call lor,$(CONFIG_SOFTMMU),$(CONFIG_TOOLS)),y) chardev-obj-y = chardev/ +chardev-obj-y += yank.o authz-obj-y = authz/
chardev/char-socket.c +24 -0
@@ -34,6 +34,7 @@#include "qapi/error.h" #include "qapi/clone-visitor.h" #include "qapi/qapi-visit-sockets.h" +#include "yank.h" #include "chardev/char-io.h"
@@ -69,6 +70,7 @@ typedef struct {size_t read_msgfds_num;int *write_msgfds;size_t write_msgfds_num; +char *yank_name;SocketAddress *addr;bool is_listen;
@@ -409,6 +411,11 @@ static void tcp_chr_free_connection(Chardev *chr)tcp_set_msgfds (chr,NULL ,0 );remove_fd_in_watch (chr); +if (s->state ==TCP_CHARDEV_STATE_CONNECTING + || s->state ==TCP_CHARDEV_STATE_CONNECTED ) { +yank_unregister_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (s->sioc)); + }object_unref (OBJECT (s->sioc)); s->sioc =NULL ;object_unref (OBJECT (s->ioc));
@@ -912,6 +919,8 @@ static int tcp_chr_add_client(Chardev *chr, int fd) }tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING );tcp_chr_set_client_ioc_name (chr, sioc); +yank_register_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (sioc)); ret =tcp_chr_new_client (chr, sioc);object_unref (OBJECT (sioc));return ret;
@@ -926,6 +935,8 @@ static void tcp_chr_accept(QIONetListener *listener,tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING );tcp_chr_set_client_ioc_name (chr, cioc); +yank_register_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (cioc));tcp_chr_new_client (chr, cioc); }
@@ -941,6 +952,8 @@ static int tcp_chr_connect_client_sync(Chardev *chr, Error **errp)object_unref (OBJECT (sioc));return -1 ; } +yank_register_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (sioc));tcp_chr_new_client (chr, sioc);object_unref (OBJECT (sioc));return 0 ;
@@ -956,6 +969,8 @@ static void tcp_chr_accept_server_sync(Chardev *chr)tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING ); sioc =qio_net_listener_wait_client (s->listener);tcp_chr_set_client_ioc_name (chr, sioc); +yank_register_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (sioc));tcp_chr_new_client (chr, sioc);object_unref (OBJECT (sioc)); }
@@ -1066,6 +1081,8 @@ static void char_socket_finalize(Object *obj)object_unref (OBJECT (s->tls_creds)); }g_free (s->tls_authz); +yank_unregister_instance (s->yank_name); +g_free (s->yank_name);qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); }
@@ -1081,6 +1098,8 @@ static void qemu_chr_socket_connected(QIOTask *task, void *opaque)if (qio_task_propagate_error (task, &err)) {tcp_chr_change_state (s,TCP_CHARDEV_STATE_DISCONNECTED ); +yank_unregister_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (sioc));check_report_connect_error (chr, err);error_free (err); goto cleanup;
@@ -1115,6 +1134,8 @@ static void tcp_chr_connect_client_async(Chardev *chr)tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING ); sioc =qio_channel_socket_new ();tcp_chr_set_client_ioc_name (chr, sioc); +yank_register_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (sioc));/* * Normally code would use the qio_channel_socket_connect_async * method which uses a QIOTask + qio_task_set_error internally
@@ -1356,6 +1377,9 @@ static void qmp_chardev_open_socket(Chardev *chr,qemu_chr_set_feature (chr,QEMU_CHAR_FEATURE_FD_PASS ); } + s->yank_name =g_strconcat ("chardev:" , chr->label,NULL ); +yank_register_instance (s->yank_name); +/* be isn't opened until we get a connection */ *be_opened = false;
--
2.20.1
PATCH v4 1/4 Introduce yank feature
In reply to Lukas StraubThe yank feature allows to recover from hanging qemu by "yanking" at various parts. Other qemu systems can register themselves and multiple yank functions. Then all yank functions for selected instances can be called by the 'yank' out-of-band qmp command. Available instances can be queried by a 'query-yank' oob command.
--- qapi/misc.json | 45 +++++++++++++ yank.c | 174 +++++++++++++++++++++++++++++++++++++++++++++++++ yank.h | 67 +++++++++++++++++++ 3 files changed, 286 insertions(+) create mode 100644 yank.c create mode 100644 yank.h
qapi/misc.json +45 -0
@@ -1550,3 +1550,48 @@ ## { 'command': 'query-vm-generation-id', 'returns': 'GuidInfo' } +## +# @YankInstances: +# +# @instances: List of yank instances. +# +# Yank instances are named after the following schema: +# "blockdev:<node-name>", "chardev:<chardev-name>" and "migration" +# +# Since: 5.1 +## +{ 'struct': 'YankInstances', 'data': {'instances': ['str'] } } + +## +# @yank: +# +# Recover from hanging qemu by yanking the specified instances. +# +# Takes @YankInstances as argument. +# +# Returns: nothing. +# +# Example: +# +# -> { "execute": "yank", "arguments": { "instances": ["blockdev:nbd0"] } } +# <- { "return": {} } +# +# Since: 5.1 +## +{ 'command': 'yank', 'data': 'YankInstances', 'allow-oob': true } + +## +# @query-yank: +# +# Query yank instances. +# +# Returns: @YankInstances +# +# Example: +# +# -> { "execute": "query-yank" } +# <- { "return": { "instances": ["blockdev:nbd0"] } } +# +# Since: 5.1 +## +{ 'command': 'query-yank', 'returns': 'YankInstances', 'allow-oob': true }
yank.c new file +174 -0
174 lines not on this page · raw message
yank.h new file +67 -0
67 lines not on this page · raw message
--
2.20.1
PATCH v4 4/4 migration: Add yank feature
In reply to Lukas StraubRegister yank functions on sockets to shut them down.
--- Makefile.objs | 1 + migration/channel.c | 12 ++++++++++++ migration/migration.c | 18 +++++++++++++++++- migration/multifd.c | 10 ++++++++++ migration/qemu-file-channel.c | 6 ++++++ migration/savevm.c | 2 ++ tests/Makefile.include | 2 +- 7 files changed, 49 insertions(+), 2 deletions(-)
Makefile.objs +1 -0
@@ -58,6 +58,7 @@ common-obj-$(CONFIG_LINUX) += fsdev/ common-obj-y += accel/ common-obj-y += migration/ +common-obj-y += yank.o common-obj-y += audio/ common-obj-m += audio/
migration/channel.c +12 -0
@@ -18,6 +18,8 @@#include "trace.h" #include "qapi/error.h" #include "io/channel-tls.h" +#include "io/channel-socket.h" +#include "yank.h" /** * @migration_channel_process_incoming - Create new incoming migration channel
@@ -35,6 +37,11 @@ void migration_channel_process_incoming(QIOChannel *ioc)trace_migration_set_incoming_channel ( ioc,object_get_typename (OBJECT (ioc))); +if (object_dynamic_cast (OBJECT (ioc),TYPE_QIO_CHANNEL_SOCKET )) { +yank_register_function ((char *)"migration" , yank_generic_iochannel, +QIO_CHANNEL (ioc)); + } +if (s->parameters.tls_creds && *s->parameters.tls_creds && !object_dynamic_cast(OBJECT(ioc),
@@ -67,6 +74,11 @@ void migration_channel_connect(MigrationState *s, ioc, object_get_typename(OBJECT(ioc)), hostname, error); if (!error) { + if (object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_SOCKET)) { + yank_register_function((char *) "migration", yank_generic_iochannel, + QIO_CHANNEL(ioc)); + } + if (s->parameters.tls_creds && *s->parameters.tls_creds && !object_dynamic_cast(OBJECT(ioc),
migration/migration.c +17 -1
@@ -54,6 +54,7 @@#include "net/announce.h" #include "qemu/queue.h" #include "multifd.h" +#include "yank.h" #define MAX_THROTTLE (32 << 20)/* Migration transfer speed throttling */
@@ -231,6 +232,8 @@ void migration_incoming_state_destroy(void)qapi_free_SocketAddressList (mis->socket_address_list); mis->socket_address_list =NULL ; } + +yank_unregister_instance ((char *)"migration" ); }static void migrate_generate_event (int new_state)
@@ -362,7 +365,9 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)const char *p;qapi_event_send_migration (MIGRATION_STATUS_SETUP ); +yank_register_instance ((char *)"migration" );if (!strcmp (uri,"defer" )) { +yank_unregister_instance ((char *)"migration" );deferred_incoming_migration (errp); }else if (strstart (uri,"tcp:" , &p)) {tcp_start_incoming_migration (p, errp);
@@ -377,6 +382,7 @@ void qemu_start_incoming_migration(const char *uri, Error **errp) }else if (strstart (uri,"fd:" , &p)) {fd_start_incoming_migration (p, errp); }else { +yank_unregister_instance ((char *)"migration" );error_setg (errp,"unknown migration protocol: %s" , uri); } }
@@ -1632,6 +1638,7 @@ static void migrate_fd_cleanup(MigrationState *s) }notifier_list_notify (&migration_state_notifiers, s);block_cleanup_parameters (s); +yank_unregister_instance ((char *)"migration" ); }static void migrate_fd_cleanup_schedule (MigrationState *s)
@@ -1899,6 +1906,7 @@ void qmp_migrate_recover(const char *uri, Error **errp)* only re-setup the migration stream and poke existing migration * to continue using that newly established channel. */ +yank_unregister_instance ((char *)"migration" );qemu_start_incoming_migration (uri, errp); }
@@ -2035,7 +2043,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,/* Error detected, put into errp */ return ; } - +if (!(has_resume && resume)) { +yank_register_instance ((char *)"migration" ); + }if (strstart (uri,"tcp:" , &p)) {tcp_start_outgoing_migration (s, p, &local_err);#ifdef CONFIG_RDMA
@@ -2049,6 +2059,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, }else if (strstart (uri,"fd:" , &p)) {fd_start_outgoing_migration (s, p, &local_err); }else { +if (!(has_resume && resume)) { +yank_unregister_instance ((char *)"migration" ); + }error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"uri" ,"a valid migration protocol" ); migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
@@ -2058,6 +2071,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, }if (local_err) { +if (!(has_resume && resume)) { +yank_unregister_instance ((char *)"migration" ); + }migrate_fd_error (s, local_err);error_propagate (errp, local_err);return ;
migration/multifd.c +10 -0
@@ -24,6 +24,9 @@#include "trace.h" #include "multifd.h" +#include "yank.h" +#include "io/channel-socket.h" +/* Multiple fd's */ #define MULTIFD_MAGIC 0x11223344U
@@ -856,6 +859,13 @@ int multifd_load_cleanup(Error **errp)for (i =0 ; i <migrate_multifd_channels (); i++) {MultiFDRecvParams *p = &multifd_recv_state->params[i]; +if (object_dynamic_cast (OBJECT (p->c),TYPE_QIO_CHANNEL_SOCKET ) + &&OBJECT (p->c)->ref ==1 ) { +yank_unregister_function ((char *)"migration" , + yank_generic_iochannel, +QIO_CHANNEL (p->c)); + } +object_unref (OBJECT (p->c)); p->c =NULL ;qemu_mutex_destroy (&p->mutex);
migration/qemu-file-channel.c +6 -0
@@ -27,6 +27,7 @@#include "qemu-file.h" #include "io/channel-socket.h" #include "qemu/iov.h" +#include "yank.h" static ssize_t channel_writev_buffer(void *opaque,
@@ -104,6 +105,11 @@ static int channel_close(void *opaque, Error **errp)int ret;QIOChannel *ioc =QIO_CHANNEL (opaque); ret =qio_channel_close (ioc, errp); +if (object_dynamic_cast (OBJECT (ioc),TYPE_QIO_CHANNEL_SOCKET ) + &&OBJECT (ioc)->ref ==1 ) { +yank_unregister_function ((char *)"migration" , yank_generic_iochannel, +QIO_CHANNEL (ioc)); + }object_unref (OBJECT (ioc));return ret; }
migration/savevm.c +2 -0
@@ -63,6 +63,7 @@#include "migration/colo.h" #include "qemu/bitmap.h" #include "net/announce.h" +#include "yank.h" const unsigned int postcopy_ram_discard_version =0 ;
@@ -2897,6 +2898,7 @@ int load_snapshot(const char *name, Error **errp)qemu_system_reset (SHUTDOWN_CAUSE_NONE ); mis->from_src_file = f; +yank_register_instance ((char *)"migration" );aio_context_acquire (aio_context); ret =qemu_loadvm_state (f);migration_incoming_state_destroy ();
tests/Makefile.include +1 -1
@@ -442,7 +442,7 @@ tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \ $(test-qapi-obj-y) tests/test-vmstate$(EXESUF): tests/test-vmstate.o \ migration/vmstate.o migration/vmstate-types.o migration/qemu-file.o \ - migration/qemu-file-channel.o migration/qjson.o \ + migration/qemu-file-channel.o migration/qjson.o yank.o \ $(test-io-obj-y) tests/test-timed-average$(EXESUF): tests/test-timed-average.o $(test-util-obj-y) tests/test-base64$(EXESUF): tests/test-base64.o $(test-util-obj-y)
--
2.20.1
Quoted 11 lines · On Mon, 25 May 2020 17:44:12 +0200 Lukas Straub <[email protected]> wrote:
Hello Everyone,
In many cases, if qemu has a network connection (qmp, migration, chardev, etc.) to some other server and that server dies or hangs, qemu hangs too. These patches introduce the new 'yank' out-of-band qmp command to recover from these kinds of hangs. The different subsystems register callbacks which get executed with the yank command. For example the callback canshutdown()a socket. This is intended for the colo use-case, but it can be used for other things too of course.Regards,
Lukas Straub
Hello Everyone,
Can this be reviewed, it would be cool to have this in qemu 5.1.
Regards,
Lukas Straub
Re: [PATCH v4 2/4] block/nbd.c: Add yank feature
In reply to Lukas StraubQuoted 15 lines · Lukas Straub
Register a yank function which shuts down the socket and sets
s->state=NBD_CLIENT_QUIT. This is the same behaviour as if an error occured.Signed-off-by Lukas Straub <[email protected]>--- Makefile.objs | 1 + block/nbd.c | 101 ++++++++++++++++++++++++++++++++------------------ 2 files changed, 65 insertions(+), 37 deletions(-)Makefile.objs +0
-0
Makefile.objs block-obj-y += block.o blockjob.o job.o · line 21
Quoted 4 lines
Makefile.objs +1
-0@@ -18,6 +18,7 @@ block-obj-y += block.o blockjob.o job.o block-obj-y += block/ scsi/ block-obj-y += qemu-io-cmds.o block-obj-$(CONFIG_REPLICATION) += replication.o +block-obj-y += yank.o
Oh, I see this is repeated for migration and chardev code too.
Instead of doing this and relying on linker to merge duplicates, I think we should put yank.c into util/ and built it into util-obj-y, so it gets added to everything.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v4 3/4] chardev/char-socket.c: Add yank feature
In reply to Lukas StraubQuoted 7 lines · Lukas Straub
Register a yank function to shutdown the socket on yank.
Signed-off-by Lukas Straub <[email protected]>--- Makefile.objs | 1 + chardev/char-socket.c | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v4 1/4] Introduce yank feature
In reply to Lukas StraubQuoted 42 lines · Lukas Straub
The yank feature allows to recover from hanging qemu by "yanking" at various parts. Other qemu systems can register themselves and multiple yank functions. Then all yank functions for selected instances can be called by the 'yank' out-of-band qmp command. Available instances can be queried by a 'query-yank' oob command.
Signed-off-by Lukas Straub <[email protected]>--- qapi/misc.json | 45 +++++++++++++ yank.c | 174 +++++++++++++++++++++++++++++++++++++++++++++++++ yank.h | 67 +++++++++++++++++++ 3 files changed, 286 insertions(+) create mode 100644 yank.c create mode 100644 yank.hyank.c new file +27
-0+void yank_register_function (char *instance_name,YankFn *func,void *opaque) +{ +struct YankInstance *instance; +struct YankFuncAndParam *entry; + +qemu_mutex_lock (&lock); + instance =yank_find_instance (instance_name); +assert (instance); + + entry = g_slice_new(struct YankFuncAndParam); + entry->func = func; + entry->opaque = opaque; + +QLIST_INSERT_HEAD (&instance->yankfns, entry, next); +qemu_mutex_unlock (&lock); +} + +void yank_unregister_function (char *instance_name,YankFn *func,void *opaque) +{ +struct YankInstance *instance; +struct YankFuncAndParam *entry; + +qemu_mutex_lock (&lock); + instance =yank_find_instance (instance_name); +assert (instance); + + QLIST_FOREACH(entry, &instance->yankfns, next) {
yank.c line 113
Quoted 10 lines
yank.c new file +10
-0+if (entry->func == func && entry->opaque == opaque) { +QLIST_REMOVE (entry, next); + g_slice_free(struct YankFuncAndParam, entry); +qemu_mutex_unlock (&lock); +return ; + } + } + +abort (); +}
Since the NBD impl no longer needs to register multiple different functions on the same insance_nane, these methods could be be simplified, to only accept a single function, instead of keeping a whole list. This would avoid need to pass a function into the unregister() method at all.
I don't consider this a blocker though, so you pick whether to do this or not.
yank.h line 1
Quoted 7 lines
yank.h new file +1
-0@@ -0,0 +1,67 @@ +
Missing license header
Quoted 56 lines
yank.h new file +56
-0+#ifndef YANK_H +#define YANK_H + +typedef void (YankFn ) (void *opaque); + +/** +* yank_register_instance: Register a new instance. +* +* This registers a new instance for yanking. Must be called before any yank +* function is registered for this instance. +* +* This function is thread-safe. +* +* @instance_name: The globally unique name of the instance. +*/ +void yank_register_instance (char *instance_name); + +/** +* yank_unregister_instance: Unregister a instance. +* +* This unregisters a instance. Must be called only after every yank function +* of the instance has been unregistered. +* +* This function is thread-safe. +* +* @instance_name: The name of the instance. +*/ +void yank_unregister_instance (char *instance_name); + +/** +* yank_register_function: Register a yank function +* +* This registers a yank function. All limitations of qmp oob commands apply +* to the yank function as well. +* +* This function is thread-safe. +* +* @instance_name: The name of the instance +* @func: The yank function +* @opaque: Will be passed to the yank function +*/ +void yank_register_function (char *instance_name,YankFn *func,void *opaque); + +/** +* yank_unregister_function: Unregister a yank function +* +* This unregisters a yank function. +* +* This function is thread-safe. +* +* @instance_name: The name of the instance +* @func: func that was passed to yank_register_function +* @opaque: opaque that was passed to yank_register_function +*/ +void yank_unregister_function (char *instance_name,YankFn *func,void *opaque); +
yank.h line 67
Quoted 10 lines
yank.h new file +10
-0+/** +* yank_unregister_function: Generic yank function for iochannel +* +* This is a generic yank function which will call qio_channel_shutdown on the +* provided QIOChannel. +* +* @opaque: QIOChannel to shutdown +*/ +void yank_generic_iochannel (void *opaque); +#endif
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v4 2/4] block/nbd.c: Add yank feature
In reply to Lukas StraubQuoted 9 lines · Lukas Straub
Register a yank function which shuts down the socket and sets
s->state=NBD_CLIENT_QUIT. This is the same behaviour as if an error occured.Signed-off-by Lukas Straub <[email protected]>--- Makefile.objs | 1 + block/nbd.c | 101 ++++++++++++++++++++++++++++++++------------------ 2 files changed, 65 insertions(+), 37 deletions(-)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v4 4/4] migration: Add yank feature
In reply to Lukas StraubQuoted 12 lines · Lukas Straub
Register yank functions on sockets to shut them down.
Signed-off-by Lukas Straub <[email protected]>--- Makefile.objs | 1 + migration/channel.c | 12 ++++++++++++ migration/migration.c | 18 +++++++++++++++++- migration/multifd.c | 10 ++++++++++ migration/qemu-file-channel.c | 6 ++++++ migration/savevm.c | 2 ++ tests/Makefile.include | 2 +- 7 files changed, 49 insertions(+), 2 deletions(-)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v4 1/4] Introduce yank feature
In reply to Lukas StraubQuoted 11 lines · Lukas Straub
The yank feature allows to recover from hanging qemu by "yanking" at various parts. Other qemu systems can register themselves and multiple yank functions. Then all yank functions for selected instances can be called by the 'yank' out-of-band qmp command. Available instances can be queried by a 'query-yank' oob command.
Signed-off-by Lukas Straub <[email protected]>--- qapi/misc.json | 45 +++++++++++++ yank.c | 174 +++++++++++++++++++++++++++++++++++++++++++++++++ yank.h | 67 +++++++++++++++++++
As mentioned n the second patch, I think these should live in util/ and util/Makefile.objs should add them to util-obj-y
Ideally MAINTAINERS would be updated to list someone as the official maintainer too.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 12 lines · Lukas Straub
Quoted 9 lines · On Mon, 25 May 2020 17:44:12 +0200 Lukas Straub <[email protected]> wrote:
Hello Everyone,
In many cases, if qemu has a network connection (qmp, migration, chardev, etc.) to some other server and that server dies or hangs, qemu hangs too. These patches introduce the new 'yank' out-of-band qmp command to recover from these kinds of hangs. The different subsystems register callbacks which get executed with the yank command. For example the callback canshutdown()a socket. This is intended for the colo use-case, but it can be used for other things too of course.
Quoted 5 lines
Quoted 2 lines
Regards,
Lukas StraubHello Everyone,
Can this be reviewed, it would be cool to have this in qemu 5.1.
Please see my reply to a previous version. Code that executes in the oob environment needs to take special precautions, this needs to be documented so that yank API users know what the limitations are.
Stefan
Re: [PATCH v4 2/4] block/nbd.c: Add yank feature
In reply to Lukas Straubblock/nbd.c static int nbd_client_reopen_prepare(BDRVReopenState *state, · line 1415
Quoted 10 lines · Lukas Straub
block/nbd.c +6
-0@@ -1395,6 +1407,15 @@ static int nbd_client_reopen_prepare(BDRVReopenState *state,return 0 ; } +static void nbd_yank (void *opaque) +{ +BlockDriverState *bs = opaque; +BDRVNBDState *s = (BDRVNBDState *)bs->opaque; + +qio_channel_shutdown (QIO_CHANNEL (s->sioc),QIO_CHANNEL_SHUTDOWN_BOTH ,NULL );
qio_channel_shutdown() is not guaranteed to be thread-safe. Please document new assumptions that are being introduced.
Today we can more or less get away with it (although TLS sockets are a little iffy) because it boils down the a shutdown(2) system call. I think it would be okay to update the qio_channel_shutdown() and .io_shutdown() documentation to clarify that this is thread-safe.
block/nbd.c static int nbd_client_reopen_prepare(BDRVReopenState *state, · line 1416
Quoted 1 line
block/nbd.c +1
-0+atomic_set (&s->state,NBD_CLIENT_QUIT );
docs/devel/atomics.rst says:
No barriers are implied by ``atomic_read`` and ``atomic_set`` in either Linux or QEMU.
Other threads might not see the latest value of s->state because this is a weakly ordered memory access.
I haven't audited the NBD code in detail, but if you want the other threads to always see NBD_CLIENT_QUIT then s->state should be set before calling qio_channel_shutdown() using a stronger atomics API like atomic_load_acquire()/atomic_store_release().
Re: [PATCH v4 1/4] Introduce yank feature
In reply to Lukas StraubQuoted 1 line · Lukas Straub
+static struct YankInstance *yank_find_instance(char *name)
There are const char * -> char * casts in later patches. Please use const char * where possible. Callers shouldn't need to cast away const.
Re: [PATCH v4 1/4] Introduce yank feature
In reply to Daniel P. BerrangéQuoted 50 lines · On Tue, 16 Jun 2020 15:39:57 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 49 lines · Lukas Straub
The yank feature allows to recover from hanging qemu by "yanking" at various parts. Other qemu systems can register themselves and multiple yank functions. Then all yank functions for selected instances can be called by the 'yank' out-of-band qmp command. Available instances can be queried by a 'query-yank' oob command.
Signed-off-by Lukas Straub <[email protected]>--- qapi/misc.json | 45 +++++++++++++ yank.c | 174 +++++++++++++++++++++++++++++++++++++++++++++++++ yank.h | 67 +++++++++++++++++++ 3 files changed, 286 insertions(+) create mode 100644 yank.c create mode 100644 yank.hyank.c new file +34
-0+void yank_register_function (char *instance_name,YankFn *func,void *opaque) +{ +struct YankInstance *instance; +struct YankFuncAndParam *entry; + +qemu_mutex_lock (&lock); + instance =yank_find_instance (instance_name); +assert (instance); + + entry = g_slice_new(struct YankFuncAndParam); + entry->func = func; + entry->opaque = opaque; + +QLIST_INSERT_HEAD (&instance->yankfns, entry, next); +qemu_mutex_unlock (&lock); +} + +void yank_unregister_function (char *instance_name,YankFn *func,void *opaque) +{ +struct YankInstance *instance; +struct YankFuncAndParam *entry; + +qemu_mutex_lock (&lock); + instance =yank_find_instance (instance_name); +assert (instance); + + QLIST_FOREACH(entry, &instance->yankfns, next) { +if (entry->func == func && entry->opaque == opaque) { +QLIST_REMOVE (entry, next); + g_slice_free(struct YankFuncAndParam, entry); +qemu_mutex_unlock (&lock); +return ; + } + }
Quoted 8 lines
yank.c line 113
Quoted 3 lines
yank.c new file +3
-0+ +abort (); +}Since the NBD impl no longer needs to register multiple different functions on the same
insance_nane, these methods could be be simplified, to only accept a single function, instead of keeping a whole list. This would avoid need to pass a function into theunregister()method at all.
Multiple yank functions are still needed for multifd migration.
Quoted 4 lines
I don't consider this a blocker though, so you pick whether to do this or not.
Quoted 9 lines
yank.h line 1
Quoted 7 lines
yank.h new file +1
-0@@ -0,0 +1,67 @@ +Missing license header
I will fix that in the next version.
Thanks,
Lukas Straub
Quoted 72 lines
Quoted 57 lines
yank.h new file +56
-0+#ifndef YANK_H +#define YANK_H + +typedef void (YankFn ) (void *opaque); + +/** +* yank_register_instance: Register a new instance. +* +* This registers a new instance for yanking. Must be called before any yank +* function is registered for this instance. +* +* This function is thread-safe. +* +* @instance_name: The globally unique name of the instance. +*/ +void yank_register_instance (char *instance_name); + +/** +* yank_unregister_instance: Unregister a instance. +* +* This unregisters a instance. Must be called only after every yank function +* of the instance has been unregistered. +* +* This function is thread-safe. +* +* @instance_name: The name of the instance. +*/ +void yank_unregister_instance (char *instance_name); + +/** +* yank_register_function: Register a yank function +* +* This registers a yank function. All limitations of qmp oob commands apply +* to the yank function as well. +* +* This function is thread-safe. +* +* @instance_name: The name of the instance +* @func: The yank function +* @opaque: Will be passed to the yank function +*/ +void yank_register_function (char *instance_name,YankFn *func,void *opaque); + +/** +* yank_unregister_function: Unregister a yank function +* +* This unregisters a yank function. +* +* This function is thread-safe. +* +* @instance_name: The name of the instance +* @func: func that was passed to yank_register_function +* @opaque: opaque that was passed to yank_register_function +*/ +void yank_unregister_function (char *instance_name,YankFn *func,void *opaque); +yank.h line 67
Quoted 10 lines
yank.h new file +10
-0+/** +* yank_unregister_function: Generic yank function for iochannel +* +* This is a generic yank function which will call qio_channel_shutdown on the +* provided QIOChannel. +* +* @opaque: QIOChannel to shutdown +*/ +void yank_generic_iochannel (void *opaque); +#endif Regards,
Daniel
Re: [PATCH v4 2/4] block/nbd.c: Add yank feature
In reply to Daniel P. BerrangéQuoted 23 lines · On Tue, 16 Jun 2020 15:44:06 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 15 lines · Lukas Straub
Register a yank function which shuts down the socket and sets
s->state=NBD_CLIENT_QUIT. This is the same behaviour as if an error occured.Signed-off-by Lukas Straub <[email protected]>--- Makefile.objs | 1 + block/nbd.c | 101 ++++++++++++++++++++++++++++++++------------------ 2 files changed, 65 insertions(+), 37 deletions(-)Makefile.objs +0
-0Makefile.objs block-obj-y += block.o blockjob.o job.o · line 21
Quoted 4 lines
Makefile.objs +1
-0@@ -18,6 +18,7 @@ block-obj-y += block.o blockjob.o job.o block-obj-y += block/ scsi/ block-obj-y += qemu-io-cmds.o block-obj-$(CONFIG_REPLICATION) += replication.o +block-obj-y += yank.oOh, I see this is repeated for migration and chardev code too.
Quoted 3 lines
Instead of doing this and relying on linker to merge duplicates, I think we should put
yank.cinto util/ and built it into util-obj-y, so it gets added to everything.
Ok, I will do this in the next version.
Thanks,
Lukas Straub
Quoted 2 lines
Regards,
Daniel
Re: [PATCH v4 1/4] Introduce yank feature
In reply to Stefan HajnocziQuoted 5 lines · On Wed, 17 Jun 2020 16:12:40 +0100 Stefan Hajnoczi <[email protected]> wrote:
Quoted 1 line · Lukas Straub
+static struct YankInstance *yank_find_instance(char *name)
There are const char * -> char * casts in later patches. Please use const char * where possible. Callers shouldn't need to cast away const.
nbd and chardev generate the instance name dynamically so it needs to be char *, but in migration it's hardcoded.
Thanks,
Lukas Straub
Re: [PATCH v4 1/4] Introduce yank feature
Quoted 11 lines · Lukas Straub
Quoted 5 lines · On Wed, 17 Jun 2020 16:12:40 +0100 Stefan Hajnoczi <[email protected]> wrote:
Quoted 1 line · Lukas Straub
+static struct YankInstance *yank_find_instance(char *name)
There are const char * -> char * casts in later patches. Please use const char * where possible. Callers shouldn't need to cast away const.
nbd and chardev generate the instance name dynamically so it needs to be char *, but in migration it's hardcoded.
I think you're looking at it from the wrong perspective.
The yank_find_instance() method never modifies the 'name' paramater that it receives. Therefore it should be "const char *". Likewise for the other yank_*() methods in fact.
The caller can have a char *, or a const char * as suits its needs. Either can be passed into the yank_* methods and will gain const-ness from the POV of yank code.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v4 1/4] Introduce yank feature
In reply to Lukas StraubQuoted 62 lines · Lukas Straub
Quoted 50 lines · On Tue, 16 Jun 2020 15:39:57 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 49 lines · Lukas Straub
The yank feature allows to recover from hanging qemu by "yanking" at various parts. Other qemu systems can register themselves and multiple yank functions. Then all yank functions for selected instances can be called by the 'yank' out-of-band qmp command. Available instances can be queried by a 'query-yank' oob command.
Signed-off-by Lukas Straub <[email protected]>--- qapi/misc.json | 45 +++++++++++++ yank.c | 174 +++++++++++++++++++++++++++++++++++++++++++++++++ yank.h | 67 +++++++++++++++++++ 3 files changed, 286 insertions(+) create mode 100644 yank.c create mode 100644 yank.hyank.c new file +34
-0+void yank_register_function (char *instance_name,YankFn *func,void *opaque) +{ +struct YankInstance *instance; +struct YankFuncAndParam *entry; + +qemu_mutex_lock (&lock); + instance =yank_find_instance (instance_name); +assert (instance); + + entry = g_slice_new(struct YankFuncAndParam); + entry->func = func; + entry->opaque = opaque; + +QLIST_INSERT_HEAD (&instance->yankfns, entry, next); +qemu_mutex_unlock (&lock); +} + +void yank_unregister_function (char *instance_name,YankFn *func,void *opaque) +{ +struct YankInstance *instance; +struct YankFuncAndParam *entry; + +qemu_mutex_lock (&lock); + instance =yank_find_instance (instance_name); +assert (instance); + + QLIST_FOREACH(entry, &instance->yankfns, next) { +if (entry->func == func && entry->opaque == opaque) { +QLIST_REMOVE (entry, next); + g_slice_free(struct YankFuncAndParam, entry); +qemu_mutex_unlock (&lock); +return ; + } + }Quoted 8 lines
yank.c line 113
Quoted 3 lines
yank.c new file +3
-0+ +abort (); +}Since the NBD impl no longer needs to register multiple different functions on the same
insance_nane, these methods could be be simplified, to only accept a single function, instead of keeping a whole list. This would avoid need to pass a function into theunregister()method at all.
Quoted 1 line
Multiple yank functions are still needed for multifd migration.
Oh I missed that subtlety, so fine to ignore my suggestion.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v4 1/4] Introduce yank feature
In reply to Daniel P. BerrangéQuoted 15 lines · On Fri, 19 Jun 2020 17:52:40 +0100 Daniel P. Berrangé <[email protected]>…
Quoted 11 lines · Lukas Straub
Quoted 5 lines · On Wed, 17 Jun 2020 16:12:40 +0100 Stefan Hajnoczi <[email protected]> wrote:
Quoted 1 line · Lukas Straub
+static struct YankInstance *yank_find_instance(char *name)
There are const char * -> char * casts in later patches. Please use const char * where possible. Callers shouldn't need to cast away const.
nbd and chardev generate the instance name dynamically so it needs to be char *, but in migration it's hardcoded.
I think you're looking at it from the wrong perspective.
Quoted 7 lines
The
yank_find_instance()method never modifies the 'name' paramater that it receives. Therefore it should be "const char *". Likewise for the other yank_*() methods in fact.The caller can have a char *, or a const char * as suits its needs. Either can be passed into the yank_* methods and will gain const-ness from the POV of yank code.
Makes sense, I will change it in the next version.
Thanks,
Lukas Straub
Quoted 3 lines
Regards,
Daniel
Re: [PATCH v4 2/4] block/nbd.c: Add yank feature
In reply to Stefan HajnocziQuoted 15 lines · On Wed, 17 Jun 2020 16:09:09 +0100 Stefan Hajnoczi <[email protected]> wrote:
block/nbd.c static int nbd_client_reopen_prepare(BDRVReopenState *state, · line 1415
Quoted 10 lines · Lukas Straub
block/nbd.c +6
-0@@ -1395,6 +1407,15 @@ static int nbd_client_reopen_prepare(BDRVReopenState *state,return 0 ; } +static void nbd_yank (void *opaque) +{ +BlockDriverState *bs = opaque; +BDRVNBDState *s = (BDRVNBDState *)bs->opaque; + +qio_channel_shutdown (QIO_CHANNEL (s->sioc),QIO_CHANNEL_SHUTDOWN_BOTH ,NULL );
qio_channel_shutdown()is not guaranteed to be thread-safe. Please document new assumptions that are being introduced.
Quoted 4 lines
Today we can more or less get away with it (although TLS sockets are a little iffy) because it boils down the a shutdown(2) system call. I think it would be okay to update the
qio_channel_shutdown()and .io_shutdown() documentation to clarify that this is thread-safe.
Good idea, especially since the migration code already assumes this. I will do this in the next version.
Quoted 7 lines
block/nbd.c static int nbd_client_reopen_prepare(BDRVReopenState *state, · line 1416
Quoted 1 line
block/nbd.c +1
-0+atomic_set (&s->state,NBD_CLIENT_QUIT );
docs/devel/atomics.rstsays:No barriers are implied by ``atomic_read`` and ``atomic_set`` in either Linux or QEMU.
Quoted 7 lines
Other threads might not see the latest value of
s->statebecause this is a weakly ordered memory access.I haven't audited the NBD code in detail, but if you want the other threads to always see
NBD_CLIENT_QUITthens->stateshould be set before callingqio_channel_shutdown()using a stronger atomics API like atomic_load_acquire()/atomic_store_release().
You are right, I will change that in the next version.
Thanks,
Lukas Straub