Introduce 'yank' oob qmp command to recover from hanging qemu
Subsystems (2)
v3 cover Introduce 'yank' oob qmp command to recover from hanging qemu
Hello Everyone,
In many cases, if qemu has a network connection (qmp, migration, chardev, etc.) to some other server and that server dies or hangs, qemu hangs too. These patches introduce the new 'yank' out-of-band qmp command to recover from these kinds of hangs. The different subsystems register callbacks which get executed with the yank command. For example the callback can shutdown() a socket. This is intended for the colo use-case, but it can be used for other things too of course.
Regards,
Lukas Straub
v3:
-don't touch softmmu/vl.c, use __contructor__ attribute instead (Paolo Bonzini) -fix build errors
-rewrite migration patch so it actually passes all tests
v2:
-don't touch io/ code anymore
-always register yank functions
-'yank' now takes a list of instances to yank -'query-yank' returns a list of yankable instances
Lukas Straub (4):
Introduce yank feature
block/nbd.c: Add yank feature
chardev/char-socket.c: Add yank feature
migration: Add yank feature
Makefile.objs | 3 + block/nbd.c | 101 ++++++++++++-------- chardev/char-socket.c | 24 +++++ migration/channel.c | 12 +++ migration/migration.c | 18 +++- migration/multifd.c | 10 ++ migration/qemu-file-channel.c | 6 ++ qapi/misc.json | 45 +++++++++ yank.c | 174 ++++++++++++++++++++++++++++++++++ yank.h | 67 +++++++++++++ 10 files changed, 422 insertions(+), 38 deletions(-) create mode 100644 yank.c create mode 100644 yank.h
--
2.20.1
PATCH v3 1/4 Introduce yank feature
The yank feature allows to recover from hanging qemu by "yanking" at various parts. Other qemu systems can register themselves and multiple yank functions. Then all yank functions for selected instances can be called by the 'yank' out-of-band qmp command. Available instances can be queried by a 'query-yank' oob command.
--- qapi/misc.json | 45 +++++++++++++ yank.c | 174 +++++++++++++++++++++++++++++++++++++++++++++++++ yank.h | 67 +++++++++++++++++++ 3 files changed, 286 insertions(+) create mode 100644 yank.c create mode 100644 yank.h
qapi/misc.json +45 -0
@@ -1550,3 +1550,48 @@ ## { 'command': 'query-vm-generation-id', 'returns': 'GuidInfo' } +## +# @YankInstances: +# +# @instances: List of yank instances. +# +# Yank instances are named after the following schema: +# "blockdev:<node-name>", "chardev:<chardev-name>" and "migration" +# +# Since: 5.1 +## +{ 'struct': 'YankInstances', 'data': {'instances': ['str'] } } + +## +# @yank: +# +# Recover from hanging qemu by yanking the specified instances. +# +# Takes @YankInstances as argument. +# +# Returns: nothing. +# +# Example: +# +# -> { "execute": "yank", "arguments": { "instances": ["blockdev:nbd0"] } } +# <- { "return": {} } +# +# Since: 5.1 +## +{ 'command': 'yank', 'data': 'YankInstances', 'allow-oob': true } + +## +# @query-yank: +# +# Query yank instances. +# +# Returns: @YankInstances +# +# Example: +# +# -> { "execute": "query-yank" } +# <- { "return": { "instances": ["blockdev:nbd0"] } } +# +# Since: 5.1 +## +{ 'command': 'query-yank', 'returns': 'YankInstances', 'allow-oob': true }
yank.c new file +174 -0
174 lines not on this page · raw message
yank.h new file +67 -0
67 lines not on this page · raw message
--
2.20.1
PATCH v3 2/4 block/nbd.c: Add yank feature
In reply to Lukas StraubRegister a yank function which shuts down the socket and sets s->state = NBD_CLIENT_QUIT. This is the same behaviour as if an error occured.
--- Makefile.objs | 1 + block/nbd.c | 101 ++++++++++++++++++++++++++++++++------------------ 2 files changed, 65 insertions(+), 37 deletions(-)
Makefile.objs +1 -0
@@ -18,6 +18,7 @@ block-obj-y += block.o blockjob.o job.o block-obj-y += block/ scsi/ block-obj-y += qemu-io-cmds.o block-obj-$(CONFIG_REPLICATION) += replication.o +block-obj-y += yank.o block-obj-m = block/
block/nbd.c +64 -37
290 lines not on this page · raw message
--
2.20.1
PATCH v3 3/4 chardev/char-socket.c: Add yank feature
In reply to Lukas StraubRegister a yank function to shutdown the socket on yank.
--- Makefile.objs | 1 + chardev/char-socket.c | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+)
Makefile.objs +1 -0
@@ -10,6 +10,7 @@ qom-obj-y = qom/ ifeq ($(call lor,$(CONFIG_SOFTMMU),$(CONFIG_TOOLS)),y) chardev-obj-y = chardev/ +chardev-obj-y += yank.o authz-obj-y = authz/
chardev/char-socket.c +24 -0
@@ -34,6 +34,7 @@#include "qapi/error.h" #include "qapi/clone-visitor.h" #include "qapi/qapi-visit-sockets.h" +#include "yank.h" #include "chardev/char-io.h"
@@ -69,6 +70,7 @@ typedef struct {size_t read_msgfds_num;int *write_msgfds;size_t write_msgfds_num; +char *yank_name;SocketAddress *addr;bool is_listen;
@@ -409,6 +411,11 @@ static void tcp_chr_free_connection(Chardev *chr)tcp_set_msgfds (chr,NULL ,0 );remove_fd_in_watch (chr); +if (s->state ==TCP_CHARDEV_STATE_CONNECTING + || s->state ==TCP_CHARDEV_STATE_CONNECTED ) { +yank_unregister_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (s->sioc)); + }object_unref (OBJECT (s->sioc)); s->sioc =NULL ;object_unref (OBJECT (s->ioc));
@@ -912,6 +919,8 @@ static int tcp_chr_add_client(Chardev *chr, int fd) }tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING );tcp_chr_set_client_ioc_name (chr, sioc); +yank_register_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (sioc)); ret =tcp_chr_new_client (chr, sioc);object_unref (OBJECT (sioc));return ret;
@@ -926,6 +935,8 @@ static void tcp_chr_accept(QIONetListener *listener,tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING );tcp_chr_set_client_ioc_name (chr, cioc); +yank_register_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (cioc));tcp_chr_new_client (chr, cioc); }
@@ -941,6 +952,8 @@ static int tcp_chr_connect_client_sync(Chardev *chr, Error **errp)object_unref (OBJECT (sioc));return -1 ; } +yank_register_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (sioc));tcp_chr_new_client (chr, sioc);object_unref (OBJECT (sioc));return 0 ;
@@ -956,6 +969,8 @@ static void tcp_chr_accept_server_sync(Chardev *chr)tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING ); sioc =qio_net_listener_wait_client (s->listener);tcp_chr_set_client_ioc_name (chr, sioc); +yank_register_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (sioc));tcp_chr_new_client (chr, sioc);object_unref (OBJECT (sioc)); }
@@ -1066,6 +1081,8 @@ static void char_socket_finalize(Object *obj)object_unref (OBJECT (s->tls_creds)); }g_free (s->tls_authz); +yank_unregister_instance (s->yank_name); +g_free (s->yank_name);qemu_chr_be_event (chr,CHR_EVENT_CLOSED ); }
@@ -1081,6 +1098,8 @@ static void qemu_chr_socket_connected(QIOTask *task, void *opaque)if (qio_task_propagate_error (task, &err)) {tcp_chr_change_state (s,TCP_CHARDEV_STATE_DISCONNECTED ); +yank_unregister_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (sioc));check_report_connect_error (chr, err);error_free (err); goto cleanup;
@@ -1115,6 +1134,8 @@ static void tcp_chr_connect_client_async(Chardev *chr)tcp_chr_change_state (s,TCP_CHARDEV_STATE_CONNECTING ); sioc =qio_channel_socket_new ();tcp_chr_set_client_ioc_name (chr, sioc); +yank_register_function (s->yank_name, yank_generic_iochannel, +QIO_CHANNEL (sioc));/* * Normally code would use the qio_channel_socket_connect_async * method which uses a QIOTask + qio_task_set_error internally
@@ -1356,6 +1377,9 @@ static void qmp_chardev_open_socket(Chardev *chr,qemu_chr_set_feature (chr,QEMU_CHAR_FEATURE_FD_PASS ); } + s->yank_name =g_strconcat ("chardev:" , chr->label,NULL ); +yank_register_instance (s->yank_name); +/* be isn't opened until we get a connection */ *be_opened = false;
--
2.20.1
PATCH v3 4/4 migration: Add yank feature
In reply to Lukas StraubRegister yank functions on sockets to shut them down.
--- Makefile.objs | 1 + migration/channel.c | 12 ++++++++++++ migration/migration.c | 18 +++++++++++++++++- migration/multifd.c | 10 ++++++++++ migration/qemu-file-channel.c | 6 ++++++ 5 files changed, 46 insertions(+), 1 deletion(-)
Makefile.objs +1 -0
@@ -58,6 +58,7 @@ common-obj-$(CONFIG_LINUX) += fsdev/ common-obj-y += accel/ common-obj-y += migration/ +common-obj-y += yank.o common-obj-y += audio/ common-obj-m += audio/
migration/channel.c +12 -0
@@ -18,6 +18,8 @@#include "trace.h" #include "qapi/error.h" #include "io/channel-tls.h" +#include "io/channel-socket.h" +#include "yank.h" /** * @migration_channel_process_incoming - Create new incoming migration channel
@@ -35,6 +37,11 @@ void migration_channel_process_incoming(QIOChannel *ioc)trace_migration_set_incoming_channel ( ioc,object_get_typename (OBJECT (ioc))); +if (object_dynamic_cast (OBJECT (ioc),TYPE_QIO_CHANNEL_SOCKET )) { +yank_register_function ((char *)"migration" , yank_generic_iochannel, +QIO_CHANNEL (ioc)); + } +if (s->parameters.tls_creds && *s->parameters.tls_creds && !object_dynamic_cast(OBJECT(ioc),
@@ -67,6 +74,11 @@ void migration_channel_connect(MigrationState *s, ioc, object_get_typename(OBJECT(ioc)), hostname, error); if (!error) { + if (object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_SOCKET)) { + yank_register_function((char *) "migration", yank_generic_iochannel, + QIO_CHANNEL(ioc)); + } + if (s->parameters.tls_creds && *s->parameters.tls_creds && !object_dynamic_cast(OBJECT(ioc),
migration/migration.c +17 -1
@@ -54,6 +54,7 @@#include "net/announce.h" #include "qemu/queue.h" #include "multifd.h" +#include "yank.h" #define MAX_THROTTLE (32 << 20)/* Migration transfer speed throttling */
@@ -231,6 +232,8 @@ void migration_incoming_state_destroy(void)qapi_free_SocketAddressList (mis->socket_address_list); mis->socket_address_list =NULL ; } + +yank_unregister_instance ((char *)"migration" ); }static void migrate_generate_event (int new_state)
@@ -362,7 +365,9 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)const char *p;qapi_event_send_migration (MIGRATION_STATUS_SETUP ); +yank_register_instance ((char *)"migration" );if (!strcmp (uri,"defer" )) { +yank_unregister_instance ((char *)"migration" );deferred_incoming_migration (errp); }else if (strstart (uri,"tcp:" , &p)) {tcp_start_incoming_migration (p, errp);
@@ -377,6 +382,7 @@ void qemu_start_incoming_migration(const char *uri, Error **errp) }else if (strstart (uri,"fd:" , &p)) {fd_start_incoming_migration (p, errp); }else { +yank_unregister_instance ((char *)"migration" );error_setg (errp,"unknown migration protocol: %s" , uri); } }
@@ -1632,6 +1638,7 @@ static void migrate_fd_cleanup(MigrationState *s) }notifier_list_notify (&migration_state_notifiers, s);block_cleanup_parameters (s); +yank_unregister_instance ((char *)"migration" ); }static void migrate_fd_cleanup_schedule (MigrationState *s)
@@ -1899,6 +1906,7 @@ void qmp_migrate_recover(const char *uri, Error **errp)* only re-setup the migration stream and poke existing migration * to continue using that newly established channel. */ +yank_unregister_instance ((char *)"migration" );qemu_start_incoming_migration (uri, errp); }
@@ -2035,7 +2043,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,/* Error detected, put into errp */ return ; } - +if (!(has_resume && resume)) { +yank_register_instance ((char *)"migration" ); + }if (strstart (uri,"tcp:" , &p)) {tcp_start_outgoing_migration (s, p, &local_err);#ifdef CONFIG_RDMA
@@ -2049,6 +2059,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, }else if (strstart (uri,"fd:" , &p)) {fd_start_outgoing_migration (s, p, &local_err); }else { +if (!(has_resume && resume)) { +yank_unregister_instance ((char *)"migration" ); + }error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"uri" ,"a valid migration protocol" ); migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
@@ -2058,6 +2071,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, }if (local_err) { +if (!(has_resume && resume)) { +yank_unregister_instance ((char *)"migration" ); + }migrate_fd_error (s, local_err);error_propagate (errp, local_err);return ;
migration/multifd.c +10 -0
@@ -24,6 +24,9 @@#include "trace.h" #include "multifd.h" +#include "yank.h" +#include "io/channel-socket.h" +/* Multiple fd's */ #define MULTIFD_MAGIC 0x11223344U
@@ -856,6 +859,13 @@ int multifd_load_cleanup(Error **errp)for (i =0 ; i <migrate_multifd_channels (); i++) {MultiFDRecvParams *p = &multifd_recv_state->params[i]; +if (object_dynamic_cast (OBJECT (p->c),TYPE_QIO_CHANNEL_SOCKET ) + &&OBJECT (p->c)->ref ==1 ) { +yank_unregister_function ((char *)"migration" , + yank_generic_iochannel, +QIO_CHANNEL (p->c)); + } +object_unref (OBJECT (p->c)); p->c =NULL ;qemu_mutex_destroy (&p->mutex);
migration/qemu-file-channel.c +6 -0
@@ -27,6 +27,7 @@#include "qemu-file.h" #include "io/channel-socket.h" #include "qemu/iov.h" +#include "yank.h" static ssize_t channel_writev_buffer(void *opaque,
@@ -104,6 +105,11 @@ static int channel_close(void *opaque, Error **errp)int ret;QIOChannel *ioc =QIO_CHANNEL (opaque); ret =qio_channel_close (ioc, errp); +if (object_dynamic_cast (OBJECT (ioc),TYPE_QIO_CHANNEL_SOCKET ) + &&OBJECT (ioc)->ref ==1 ) { +yank_unregister_function ((char *)"migration" , yank_generic_iochannel, +QIO_CHANNEL (ioc)); + }object_unref (OBJECT (ioc));return ret; }
--
2.20.1
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===
migration/qemu-file-channel.o: In function `channel_close':/tmp/qemu-test/src/migration/qemu-file-channel.c:110: undefined reference to `yank_generic_iochannel'/tmp/qemu-test/src/migration/qemu-file-channel.c:110: undefined reference to `yank_unregister_function'
collect2: error: ld returned 1 exit status
LINK tests/test-cutils
make: *** [tests/test-vmstate] Error 1
make: *** Waiting for unfinished jobs....
TEST iotest-qcow2: 001
TEST iotest-qcow2: 002
---
Not run: 259
Failures: 267
Failed 1 of 119 iotests
make: *** [check-tests/check-block.sh] Error 1
make: *** wait: No child processes. Stop.
Traceback (most recent call last):
File "./tests/docker/docker.py", line 664, in <module>
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=db483e1a7b374ed8a9cf3528ad2357b2', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-4gmncabb/src/docker-src.2020-05-24-17.04.07.24804:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=db483e1a7b374ed8a9cf3528ad2357b2
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-4gmncabb/src'
make: *** [docker-run-test-quick@centos7] Error 2
real 12m5.130s
user 0m8.082s
The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1 === TEST SCRIPT END ===
migration/qemu-file-channel.o: in function `channel_close':/tmp/qemu-test/src/migration/qemu-file-channel.c:111: undefined reference to `yank_generic_iochannel'/usr/bin/ld: /tmp/qemu-test/src/migration/qemu-file-channel.c:110: undefined reference to `yank_unregister_function'
clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [/tmp/qemu-test/src/rules.mak:124: tests/test-vmstate] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
File "./tests/docker/docker.py", line 664, in <module>
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=18e540c0fae14ac68e571023cc096e31', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=x86_64-softmmu', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-92q_wrsw/src/docker-src.2020-05-24-17.15.58.27154:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-debug']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=18e540c0fae14ac68e571023cc096e31
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-92q_wrsw/src'
make: *** [docker-run-test-debug@fedora] Error 2
real 5m14.699s
user 0m8.970s
The full log is available at
http://patchew.org/logs/[email protected]/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]