Skip to contentSkip to conversation

Introduce 'yank' oob qmp command to recover from hanging qemu

PATCH Superseded Lukas Straub 5 participants 14 messages May 20, 2020 – Jun 19, 2020

Subsystems (2)

v2 of 14 Superseded Latest is v14 →

lore
Share & tools
lore
#0 lore raw

v2 cover Introduce 'yank' oob qmp command to recover from hanging qemu

Hello Everyone,
In many cases, if qemu has a network connection (qmp, migration, chardev, etc.) to some other server and that server dies or hangs, qemu hangs too. These patches introduce the new 'yank' out-of-band qmp command to recover from these kinds of hangs. The different subsystems register callbacks which get executed with the yank command. For example the callback can shutdown() a socket. This is intended for the colo use-case, but it can be used for other things too of course.

Regards,
Lukas Straub

v2:
-don't touch io/ code anymore
-always register yank functions
-'yank' now takes a list of instances to yank -'query-yank' returns a list of yankable instances

Lukas Straub (4):
Introduce yank feature
block/nbd.c: Add yank feature
chardev/char-socket.c: Add yank feature
migration: Add yank feature

 Makefile.objs                 |   1 +
 block/nbd.c                   | 101 ++++++++++++--------
 chardev/char-socket.c         |  24 +++++
 migration/migration.c         |   9 ++
 migration/qemu-file-channel.c |   6 ++
 migration/socket.c            |  11 +++
 qapi/misc.json                |  45 +++++++++
 softmmu/vl.c                  |   2 +
 yank.c                        | 174 ++++++++++++++++++++++++++++++++++
 yank.h                        |  69 ++++++++++++++
 10 files changed, 405 insertions(+), 37 deletions(-)
 create mode 100644 yank.c
 create mode 100644 yank.h

--
2.20.1

#9 lore raw

PATCH v2 3/4 chardev/char-socket.c: Add yank feature

Register a yank function to shutdown the socket on yank.

---
 chardev/char-socket.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
chardev/char-socket.c +24 -0
index 185fe38dda..d5c6cd2153 100644
@@ -34,6 +34,7 @@
 #include "qapi/error.h"
 #include "qapi/clone-visitor.h"
 #include "qapi/qapi-visit-sockets.h"
+#include "yank.h"
 
 #include "chardev/char-io.h"
 
@@ -69,6 +70,7 @@ typedef struct {
     size_t read_msgfds_num;
     int *write_msgfds;
     size_t write_msgfds_num;
+    char *yank_name;
 
     SocketAddress *addr;
     bool is_listen;
@@ -409,6 +411,11 @@ static void tcp_chr_free_connection(Chardev *chr)
 
     tcp_set_msgfds(chr, NULL, 0);
     remove_fd_in_watch(chr);
+    if (s->state == TCP_CHARDEV_STATE_CONNECTING
+        || s->state == TCP_CHARDEV_STATE_CONNECTED) {
+        yank_unregister_function(s->yank_name, yank_generic_iochannel,
+                                 QIO_CHANNEL(s->sioc));
+    }
     object_unref(OBJECT(s->sioc));
     s->sioc = NULL;
     object_unref(OBJECT(s->ioc));
@@ -912,6 +919,8 @@ static int tcp_chr_add_client(Chardev *chr, int fd)
     }
     tcp_chr_change_state(s, TCP_CHARDEV_STATE_CONNECTING);
     tcp_chr_set_client_ioc_name(chr, sioc);
+    yank_register_function(s->yank_name, yank_generic_iochannel,
+                           QIO_CHANNEL(sioc));
     ret = tcp_chr_new_client(chr, sioc);
     object_unref(OBJECT(sioc));
     return ret;
@@ -926,6 +935,8 @@ static void tcp_chr_accept(QIONetListener *listener,
 
     tcp_chr_change_state(s, TCP_CHARDEV_STATE_CONNECTING);
     tcp_chr_set_client_ioc_name(chr, cioc);
+    yank_register_function(s->yank_name, yank_generic_iochannel,
+                           QIO_CHANNEL(cioc));
     tcp_chr_new_client(chr, cioc);
 }
 
@@ -941,6 +952,8 @@ static int tcp_chr_connect_client_sync(Chardev *chr, Error **errp)
         object_unref(OBJECT(sioc));
         return -1;
     }
+    yank_register_function(s->yank_name, yank_generic_iochannel,
+                           QIO_CHANNEL(sioc));
     tcp_chr_new_client(chr, sioc);
     object_unref(OBJECT(sioc));
     return 0;
@@ -956,6 +969,8 @@ static void tcp_chr_accept_server_sync(Chardev *chr)
     tcp_chr_change_state(s, TCP_CHARDEV_STATE_CONNECTING);
     sioc = qio_net_listener_wait_client(s->listener);
     tcp_chr_set_client_ioc_name(chr, sioc);
+    yank_register_function(s->yank_name, yank_generic_iochannel,
+                           QIO_CHANNEL(sioc));
     tcp_chr_new_client(chr, sioc);
     object_unref(OBJECT(sioc));
 }
@@ -1066,6 +1081,8 @@ static void char_socket_finalize(Object *obj)
         object_unref(OBJECT(s->tls_creds));
     }
     g_free(s->tls_authz);
+    yank_unregister_instance(s->yank_name);
+    g_free(s->yank_name);
 
     qemu_chr_be_event(chr, CHR_EVENT_CLOSED);
 }
@@ -1081,6 +1098,8 @@ static void qemu_chr_socket_connected(QIOTask *task, void *opaque)
 
     if (qio_task_propagate_error(task, &err)) {
         tcp_chr_change_state(s, TCP_CHARDEV_STATE_DISCONNECTED);
+        yank_unregister_function(s->yank_name, yank_generic_iochannel,
+                                 QIO_CHANNEL(sioc));
         check_report_connect_error(chr, err);
         error_free(err);
         goto cleanup;
@@ -1115,6 +1134,8 @@ static void tcp_chr_connect_client_async(Chardev *chr)
     tcp_chr_change_state(s, TCP_CHARDEV_STATE_CONNECTING);
     sioc = qio_channel_socket_new();
     tcp_chr_set_client_ioc_name(chr, sioc);
+    yank_register_function(s->yank_name, yank_generic_iochannel,
+                           QIO_CHANNEL(sioc));
     /*
      * Normally code would use the qio_channel_socket_connect_async
      * method which uses a QIOTask + qio_task_set_error internally
@@ -1356,6 +1377,9 @@ static void qmp_chardev_open_socket(Chardev *chr,
         qemu_chr_set_feature(chr, QEMU_CHAR_FEATURE_FD_PASS);
     }
 
+    s->yank_name = g_strconcat("chardev:", chr->label, NULL);
+    yank_register_instance(s->yank_name);
+
     /* be isn't opened until we get a connection */
     *be_opened = false;
 

--
2.20.1

#1 lore raw

PATCH v2 1/4 Introduce yank feature

In reply to Lukas Straub

The yank feature allows to recover from hanging qemu by "yanking" at various parts. Other qemu systems can register themselves and multiple yank functions. Then all yank functions for selected instances can be called by the 'yank' out-of-band qmp command. Available instances can be queried by a 'query-yank' oob command.

---
 qapi/misc.json |  45 +++++++++++++
 softmmu/vl.c   |   2 +
 yank.c         | 174 +++++++++++++++++++++++++++++++++++++++++++++++++
 yank.h         |  69 ++++++++++++++++++++
 4 files changed, 290 insertions(+)
 create mode 100644 yank.c
 create mode 100644 yank.h
qapi/misc.json +45 -0
index 99b90ac80b..f5228b2502 100644
@@ -1550,3 +1550,48 @@
 ##
 { 'command': 'query-vm-generation-id', 'returns': 'GuidInfo' }
 
+##
+# @YankInstances:
+#
+# @instances: List of yank instances.
+#
+# Yank instances are named after the following schema:
+# "blockdev:<node-name>", "chardev:<chardev-name>" and "migration"
+#
+# Since: 5.1
+##
+{ 'struct': 'YankInstances', 'data': {'instances': ['str'] } }
+
+##
+# @yank:
+#
+# Recover from hanging qemu by yanking the specified instances.
+#
+# Takes @YankInstances as argument.
+#
+# Returns: nothing.
+#
+# Example:
+#
+# -> { "execute": "yank", "arguments": { "instances": ["blockdev:nbd0"] } }
+# <- { "return": {} }
+#
+# Since: 5.1
+##
+{ 'command': 'yank', 'data': 'YankInstances', 'allow-oob': true }
+
+##
+# @query-yank:
+#
+# Query yank instances.
+#
+# Returns: @YankInstances
+#
+# Example:
+#
+# -> { "execute": "query-yank" }
+# <- { "return": { "instances": ["blockdev:nbd0"] } }
+#
+# Since: 5.1
+##
+{ 'command': 'query-yank', 'returns': 'YankInstances', 'allow-oob': true }
softmmu/vl.c +2 -0
index 32c0047889..5d99749d29 100644
@@ -112,6 +112,7 @@
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 #include "qemu/guest-random.h"
+#include "yank.h"
 
 #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char **envp)
     precopy_infrastructure_init();
     postcopy_infrastructure_init();
     monitor_init_globals();
+    yank_init();
 
     if (qcrypto_init(&err) < 0) {
         error_reportf_err(err, "cannot initialize crypto: ");
yank.c new file +174 -0

174 lines not on this page · raw message

yank.h new file +69 -0

69 lines not on this page · raw message

--
2.20.1

#8 lore raw

PATCH v2 2/4 block/nbd.c: Add yank feature

In reply to Lukas Straub

Register a yank function which shuts down the socket and sets s->state = NBD_CLIENT_QUIT. This is the same behaviour as if an error occured.

---
 Makefile.objs |   1 +
 block/nbd.c   | 101 ++++++++++++++++++++++++++++++++------------------
 2 files changed, 65 insertions(+), 37 deletions(-)
Makefile.objs +1 -0
index a7c967633a..8e403b81f3 100644
@@ -18,6 +18,7 @@ block-obj-y += block.o blockjob.o job.o
 block-obj-y += block/ scsi/
 block-obj-y += qemu-io-cmds.o
 block-obj-$(CONFIG_REPLICATION) += replication.o
+block-obj-y += yank.o
 
 block-obj-m = block/
 
block/nbd.c +64 -37

290 lines not on this page · raw message

--
2.20.1

#10 lore raw

PATCH v2 4/4 migration: Add yank feature

In reply to Lukas Straub

Register yank functions on sockets to shut them down.

---
 migration/migration.c         |  9 +++++++++
 migration/qemu-file-channel.c |  6 ++++++
 migration/socket.c            | 11 +++++++++++
 3 files changed, 26 insertions(+)
migration/migration.c +9 -0
index 187ac0410c..f89fcba198 100644
@@ -54,6 +54,7 @@
 #include "net/announce.h"
 #include "qemu/queue.h"
 #include "multifd.h"
+#include "yank.h"
 
 #define MAX_THROTTLE  (32 << 20)      /* Migration transfer speed throttling */
 
@@ -231,6 +232,8 @@ void migration_incoming_state_destroy(void)
         qapi_free_SocketAddressList(mis->socket_address_list);
         mis->socket_address_list = NULL;
     }
+
+    yank_unregister_instance((char *) "migration");
 }
 
 static void migrate_generate_event(int new_state)
@@ -362,6 +365,7 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)
     const char *p;
 
     qapi_event_send_migration(MIGRATION_STATUS_SETUP);
+    yank_register_instance((char *) "migration");
     if (!strcmp(uri, "defer")) {
         deferred_incoming_migration(errp);
     } else if (strstart(uri, "tcp:", &p)) {
@@ -377,6 +381,7 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)
     } else if (strstart(uri, "fd:", &p)) {
         fd_start_incoming_migration(p, errp);
     } else {
+        yank_unregister_instance((char *) "migration");
         error_setg(errp, "unknown migration protocol: %s", uri);
     }
 }
@@ -1632,6 +1637,7 @@ static void migrate_fd_cleanup(MigrationState *s)
     }
     notifier_list_notify(&migration_state_notifiers, s);
     block_cleanup_parameters(s);
+    yank_unregister_instance((char *) "migration");
 }
 
 static void migrate_fd_cleanup_schedule(MigrationState *s)
@@ -2036,6 +2042,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
         return;
     }
 
+    yank_register_instance((char *) "migration");
     if (strstart(uri, "tcp:", &p)) {
         tcp_start_outgoing_migration(s, p, &local_err);
 #ifdef CONFIG_RDMA
@@ -2049,6 +2056,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
     } else if (strstart(uri, "fd:", &p)) {
         fd_start_outgoing_migration(s, p, &local_err);
     } else {
+        yank_unregister_instance((char *) "migration");
         error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "uri",
                    "a valid migration protocol");
         migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
@@ -2058,6 +2066,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
     }
 
     if (local_err) {
+        yank_unregister_instance((char *) "migration");
         migrate_fd_error(s, local_err);
         error_propagate(errp, local_err);
         return;
migration/qemu-file-channel.c +6 -0
index d2ce32f4b9..6224bda029 100644
@@ -27,6 +27,7 @@
 #include "qemu-file.h"
 #include "io/channel-socket.h"
 #include "qemu/iov.h"
+#include "yank.h"
 
 
 static ssize_t channel_writev_buffer(void *opaque,
@@ -104,6 +105,11 @@ static int channel_close(void *opaque, Error **errp)
     int ret;
     QIOChannel *ioc = QIO_CHANNEL(opaque);
     ret = qio_channel_close(ioc, errp);
+    if (object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_SOCKET)
+        && OBJECT(ioc)->ref == 2) {
+        yank_unregister_function((char *) "migration", yank_generic_iochannel,
+                                 QIO_CHANNEL(ioc));
+    }
     object_unref(OBJECT(ioc));
     return ret;
 }
migration/socket.c +11 -0
index 97c9efde59..bbca53cc49 100644
@@ -26,6 +26,7 @@
 #include "io/channel-socket.h"
 #include "io/net-listener.h"
 #include "trace.h"
+#include "yank.h"
 
 
 struct SocketOutgoingArgs {
@@ -35,6 +36,8 @@ struct SocketOutgoingArgs {
 void socket_send_channel_create(QIOTaskFunc f, void *data)
 {
     QIOChannelSocket *sioc = qio_channel_socket_new();
+    yank_register_function((char *) "migration", yank_generic_iochannel,
+                           QIO_CHANNEL(sioc));
     qio_channel_socket_connect_async(sioc, outgoing_args.saddr,
                                      f, data, NULL, NULL);
 }
@@ -42,6 +45,8 @@ void socket_send_channel_create(QIOTaskFunc f, void *data)
 int socket_send_channel_destroy(QIOChannel *send)
 {
     /* Remove channel */
+    yank_unregister_function((char *) "migration", yank_generic_iochannel,
+                             QIO_CHANNEL(send));
     object_unref(OBJECT(send));
     if (outgoing_args.saddr) {
         qapi_free_SocketAddress(outgoing_args.saddr);
@@ -101,6 +106,8 @@ static void socket_outgoing_migration(QIOTask *task,
     Error *err = NULL;
 
     if (qio_task_propagate_error(task, &err)) {
+        yank_unregister_function((char *) "migration", yank_generic_iochannel,
+                                 QIO_CHANNEL(sioc));
         trace_migration_socket_outgoing_error(error_get_pretty(err));
     } else {
         trace_migration_socket_outgoing_connected(data->hostname);
@@ -127,6 +134,8 @@ static void socket_start_outgoing_migration(MigrationState *s,
     }
 
     qio_channel_set_name(QIO_CHANNEL(sioc), "migration-socket-outgoing");
+    yank_register_function((char *) "migration", yank_generic_iochannel,
+                           QIO_CHANNEL(sioc));
     qio_channel_socket_connect_async(sioc,
                                      saddr,
                                      socket_outgoing_migration,
@@ -163,6 +172,8 @@ static void socket_accept_incoming_migration(QIONetListener *listener,
     trace_migration_socket_incoming_accepted();
 
     qio_channel_set_name(QIO_CHANNEL(cioc), "migration-socket-incoming");
+    yank_register_function((char *) "migration", yank_generic_iochannel,
+                           QIO_CHANNEL(cioc));
     migration_channel_process_incoming(QIO_CHANNEL(cioc));
 
     if (migration_has_all_channels()) {

--
2.20.1

#2 lore raw

Re: [PATCH v2 1/4] Introduce yank feature

In reply to Lukas Straub

yank.c line 174

Quoted 5 lines · Lukas Straub
yank.c new file +5 -0
+
+void yank_init(void)
+{
+    qemu_mutex_init(&lock);
+}

You can use __constructor__ for this to avoid the call in vl.c. See job.c for an example.

Thanks,

Paolo

#12 lore raw
In reply to Lukas Straub
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===

/tmp/qemu-test/src/chardev/char-socket.c:1381: undefined reference to yank_register_instance' chardev/char-socket.o: In function char_socket_finalize':
/tmp/qemu-test/src/chardev/char-socket.c:1084: undefined reference to `yank_unregister_instance'
collect2: error: ld returned 1 exit status

  CC      s_normSubnormalF64Sig.o
  CC      s_roundPackToF64.o
  CC      s_normRoundPackToF64.o

make: *** [tests/test-char] Error 1
make: *** Waiting for unfinished jobs....
CC s_addMagsF64.o
CC s_subMagsF64.o
---
Not run: 259
Failures: 140 143 267
Failed 3 of 119 iotests
make: *** [check-tests/check-block.sh] Error 1
make: *** wait: No child processes. Stop.
Traceback (most recent call last):
File "./tests/docker/docker.py", line 664, in <module>
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=01104a80117541dfabf73e596a8c2328', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-1_i_19yu/src/docker-src.2020-05-20-19.03.34.15520:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=01104a80117541dfabf73e596a8c2328
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-1_i_19yu/src'
make: *** [docker-run-test-quick@centos7] Error 2

real 12m8.434s
user 0m8.504s

The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#13 lore raw
In reply to Lukas Straub
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1 === TEST SCRIPT END ===

/usr/bin/ld: /tmp/qemu-test/src/chardev/char-socket.c:1101: undefined reference to `yank_unregister_function'
clang++ -g -Wl,--warn-common -fsanitize=undefined -fsanitize=address -Wl,-z,relro -Wl,-z,now -pie -m64 -fstack-protector-strong -o tests/check-qnull tests/check-qnull.o libqemuutil.a -lm -lz -lgthread-2.0 -pthread -lglib-2.0 -lnettle -lgnutls -L/usr/lib -lzstd -lrt -lz -lutil -lcap-ng
clang -iquote /tmp/qemu-test/build/. -iquote . -iquote /tmp/qemu-test/src/tcg/i386 -isystem /tmp/qemu-test/src/linux-headers -isystem /tmp/qemu-test/build/linux-headers -iquote . -iquote /tmp/qemu-test/src -iquote /tmp/qemu-test/src/accel/tcg -iquote /tmp/qemu-test/src/include -iquote /tmp/qemu-test/src/disas/libvixl -I/tmp/qemu-test/src/tests/fp -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/include -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/8086-SSE -I/tmp/qemu-test/src/tests/fp/berkeley-testfloat-3/source -I/usr/include/pixman-1 -Werror -fsanitize=undefined -fsanitize=address -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wno-string-plus-int -Wno-typedef-redefinition -Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1 -DLEGACY_RDMA_REG_MR -DSTRUCT_IOVEC_DEFINED -I/usr/include/libpng16 -I/usr/include/spice-1 -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/nss3 -I/usr/include/nspr4 -pthread -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/pixman-1 -DHW_POISON_H -DTARGET_ARM -DSOFTFLOAT_ROUND_ODD -DINLINE_LEVEL=5 -DSOFTFLOAT_FAST_DIV32TO16 -DSOFTFLOAT_FAST_DIV64TO32 -DSOFTFLOAT_FAST_INT64 -DFLOAT16 -DFLOAT64 -DEXTFLOAT80 -DFLOAT128 -DFLOAT_ROUND_ODD -DLONG_DOUBLE_IS_EXTFLOAT80 -Wno-missing-prototypes -Wno-redundant-decls -Wno-return-type -Wno-error -MMD -MP -MT s_addMagsF16.o -MF ./s_addMagsF16.d -g -c -o s_addMagsF16.o /tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/s_addMagsF16.c
clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
clang -iquote /tmp/qemu-test/build/. -iquote . -iquote /tmp/qemu-test/src/tcg/i386 -isystem /tmp/qemu-test/src/linux-headers -isystem /tmp/qemu-test/build/linux-headers -iquote . -iquote /tmp/qemu-test/src -iquote /tmp/qemu-test/src/accel/tcg -iquote /tmp/qemu-test/src/include -iquote /tmp/qemu-test/src/disas/libvixl -I/tmp/qemu-test/src/tests/fp -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/include -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/8086-SSE -I/tmp/qemu-test/src/tests/fp/berkeley-testfloat-3/source -I/usr/include/pixman-1 -Werror -fsanitize=undefined -fsanitize=address -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wno-string-plus-int -Wno-typedef-redefinition -Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1 -DLEGACY_RDMA_REG_MR -DSTRUCT_IOVEC_DEFINED -I/usr/include/libpng16 -I/usr/include/spice-1 -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/nss3 -I/usr/include/nspr4 -pthread -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/pixman-1 -DHW_POISON_H -DTARGET_ARM -DSOFTFLOAT_ROUND_ODD -DINLINE_LEVEL=5 -DSOFTFLOAT_FAST_DIV32TO16 -DSOFTFLOAT_FAST_DIV64TO32 -DSOFTFLOAT_FAST_INT64 -DFLOAT16 -DFLOAT64 -DEXTFLOAT80 -DFLOAT128 -DFLOAT_ROUND_ODD -DLONG_DOUBLE_IS_EXTFLOAT80 -Wno-missing-prototypes -Wno-redundant-decls -Wno-return-type -Wno-error -MMD -MP -MT s_subMagsF16.o -MF ./s_subMagsF16.d -g -c -o s_subMagsF16.o /tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/s_subMagsF16.c
clang -iquote /tmp/qemu-test/build/. -iquote . -iquote /tmp/qemu-test/src/tcg/i386 -isystem /tmp/qemu-test/src/linux-headers -isystem /tmp/qemu-test/build/linux-headers -iquote . -iquote /tmp/qemu-test/src -iquote /tmp/qemu-test/src/accel/tcg -iquote /tmp/qemu-test/src/include -iquote /tmp/qemu-test/src/disas/libvixl -I/tmp/qemu-test/src/tests/fp -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/include -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/8086-SSE -I/tmp/qemu-test/src/tests/fp/berkeley-testfloat-3/source -I/usr/include/pixman-1 -Werror -fsanitize=undefined -fsanitize=address -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wno-string-plus-int -Wno-typedef-redefinition -Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1 -DLEGACY_RDMA_REG_MR -DSTRUCT_IOVEC_DEFINED -I/usr/include/libpng16 -I/usr/include/spice-1 -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/nss3 -I/usr/include/nspr4 -pthread -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/pixman-1 -DHW_POISON_H -DTARGET_ARM -DSOFTFLOAT_ROUND_ODD -DINLINE_LEVEL=5 -DSOFTFLOAT_FAST_DIV32TO16 -DSOFTFLOAT_FAST_DIV64TO32 -DSOFTFLOAT_FAST_INT64 -DFLOAT16 -DFLOAT64 -DEXTFLOAT80 -DFLOAT128 -DFLOAT_ROUND_ODD -DLONG_DOUBLE_IS_EXTFLOAT80 -Wno-missing-prototypes -Wno-redundant-decls -Wno-return-type -Wno-error -MMD -MP -MT s_mulAddF16.o -MF ./s_mulAddF16.d -g -c -o s_mulAddF16.o /tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/s_mulAddF16.c
clang -iquote /tmp/qemu-test/build/. -iquote . -iquote /tmp/qemu-test/src/tcg/i386 -isystem /tmp/qemu-test/src/linux-headers -isystem /tmp/qemu-test/build/linux-headers -iquote . -iquote /tmp/qemu-test/src -iquote /tmp/qemu-test/src/accel/tcg -iquote /tmp/qemu-test/src/include -iquote /tmp/qemu-test/src/disas/libvixl -I/tmp/qemu-test/src/tests/fp -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/include -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/8086-SSE -I/tmp/qemu-test/src/tests/fp/berkeley-testfloat-3/source -I/usr/include/pixman-1 -Werror -fsanitize=undefined -fsanitize=address -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wno-string-plus-int -Wno-typedef-redefinition -Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1 -DLEGACY_RDMA_REG_MR -DSTRUCT_IOVEC_DEFINED -I/usr/include/libpng16 -I/usr/include/spice-1 -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/nss3 -I/usr/include/nspr4 -pthread -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/pixman-1 -DHW_POISON_H -DTARGET_ARM -DSOFTFLOAT_ROUND_ODD -DINLINE_LEVEL=5 -DSOFTFLOAT_FAST_DIV32TO16 -DSOFTFLOAT_FAST_DIV64TO32 -DSOFTFLOAT_FAST_INT64 -DFLOAT16 -DFLOAT64 -DEXTFLOAT80 -DFLOAT128 -DFLOAT_ROUND_ODD -DLONG_DOUBLE_IS_EXTFLOAT80 -Wno-missing-prototypes -Wno-redundant-decls -Wno-return-type -Wno-error -MMD -MP -MT s_normSubnormalF32Sig.o -MF ./s_normSubnormalF32Sig.d -g -c -o s_normSubnormalF32Sig.o /tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/s_normSubnormalF32Sig.c
---
clang -iquote /tmp/qemu-test/build/. -iquote . -iquote /tmp/qemu-test/src/tcg/i386 -isystem /tmp/qemu-test/src/linux-headers -isystem /tmp/qemu-test/build/linux-headers -iquote . -iquote /tmp/qemu-test/src -iquote /tmp/qemu-test/src/accel/tcg -iquote /tmp/qemu-test/src/include -iquote /tmp/qemu-test/src/disas/libvixl -I/tmp/qemu-test/src/tests/fp -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/include -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/8086-SSE -I/tmp/qemu-test/src/tests/fp/berkeley-testfloat-3/source -I/usr/include/pixman-1 -Werror -fsanitize=undefined -fsanitize=address -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wno-string-plus-int -Wno-typedef-redefinition -Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1 -DLEGACY_RDMA_REG_MR -DSTRUCT_IOVEC_DEFINED -I/usr/include/libpng16 -I/usr/include/spice-1 -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/nss3 -I/usr/include/nspr4 -pthread -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/pixman-1 -DHW_POISON_H -DTARGET_ARM -DSOFTFLOAT_ROUND_ODD -DINLINE_LEVEL=5 -DSOFTFLOAT_FAST_DIV32TO16 -DSOFTFLOAT_FAST_DIV64TO32 -DSOFTFLOAT_FAST_INT64 -DFLOAT16 -DFLOAT64 -DEXTFLOAT80 -DFLOAT128 -DFLOAT_ROUND_ODD -DLONG_DOUBLE_IS_EXTFLOAT80 -Wno-missing-prototypes -Wno-redundant-decls -Wno-return-type -Wno-error -MMD -MP -MT s_roundPackToF64.o -MF ./s_roundPackToF64.d -g -c -o s_roundPackToF64.o /tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/s_roundPackToF64.c
clang -iquote /tmp/qemu-test/build/. -iquote . -iquote /tmp/qemu-test/src/tcg/i386 -isystem /tmp/qemu-test/src/linux-headers -isystem /tmp/qemu-test/build/linux-headers -iquote . -iquote /tmp/qemu-test/src -iquote /tmp/qemu-test/src/accel/tcg -iquote /tmp/qemu-test/src/include -iquote /tmp/qemu-test/src/disas/libvixl -I/tmp/qemu-test/src/tests/fp -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/include -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/8086-SSE -I/tmp/qemu-test/src/tests/fp/berkeley-testfloat-3/source -I/usr/include/pixman-1 -Werror -fsanitize=undefined -fsanitize=address -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wno-string-plus-int -Wno-typedef-redefinition -Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1 -DLEGACY_RDMA_REG_MR -DSTRUCT_IOVEC_DEFINED -I/usr/include/libpng16 -I/usr/include/spice-1 -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/nss3 -I/usr/include/nspr4 -pthread -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/pixman-1 -DHW_POISON_H -DTARGET_ARM -DSOFTFLOAT_ROUND_ODD -DINLINE_LEVEL=5 -DSOFTFLOAT_FAST_DIV32TO16 -DSOFTFLOAT_FAST_DIV64TO32 -DSOFTFLOAT_FAST_INT64 -DFLOAT16 -DFLOAT64 -DEXTFLOAT80 -DFLOAT128 -DFLOAT_ROUND_ODD -DLONG_DOUBLE_IS_EXTFLOAT80 -Wno-missing-prototypes -Wno-redundant-decls -Wno-return-type -Wno-error -MMD -MP -MT s_normRoundPackToF64.o -MF ./s_normRoundPackToF64.d -g -c -o s_normRoundPackToF64.o /tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/s_normRoundPackToF64.c
clang -iquote /tmp/qemu-test/build/. -iquote . -iquote /tmp/qemu-test/src/tcg/i386 -isystem /tmp/qemu-test/src/linux-headers -isystem /tmp/qemu-test/build/linux-headers -iquote . -iquote /tmp/qemu-test/src -iquote /tmp/qemu-test/src/accel/tcg -iquote /tmp/qemu-test/src/include -iquote /tmp/qemu-test/src/disas/libvixl -I/tmp/qemu-test/src/tests/fp -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/include -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/8086-SSE -I/tmp/qemu-test/src/tests/fp/berkeley-testfloat-3/source -I/usr/include/pixman-1 -Werror -fsanitize=undefined -fsanitize=address -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wno-string-plus-int -Wno-typedef-redefinition -Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1 -DLEGACY_RDMA_REG_MR -DSTRUCT_IOVEC_DEFINED -I/usr/include/libpng16 -I/usr/include/spice-1 -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/nss3 -I/usr/include/nspr4 -pthread -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/pixman-1 -DHW_POISON_H -DTARGET_ARM -DSOFTFLOAT_ROUND_ODD -DINLINE_LEVEL=5 -DSOFTFLOAT_FAST_DIV32TO16 -DSOFTFLOAT_FAST_DIV64TO32 -DSOFTFLOAT_FAST_INT64 -DFLOAT16 -DFLOAT64 -DEXTFLOAT80 -DFLOAT128 -DFLOAT_ROUND_ODD -DLONG_DOUBLE_IS_EXTFLOAT80 -Wno-missing-prototypes -Wno-redundant-decls -Wno-return-type -Wno-error -MMD -MP -MT s_addMagsF64.o -MF ./s_addMagsF64.d -g -c -o s_addMagsF64.o /tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/s_addMagsF64.c
make: *** [/tmp/qemu-test/src/rules.mak:124: tests/test-char] Error 1
make: *** Waiting for unfinished jobs....
clang -iquote /tmp/qemu-test/build/. -iquote . -iquote /tmp/qemu-test/src/tcg/i386 -isystem /tmp/qemu-test/src/linux-headers -isystem /tmp/qemu-test/build/linux-headers -iquote . -iquote /tmp/qemu-test/src -iquote /tmp/qemu-test/src/accel/tcg -iquote /tmp/qemu-test/src/include -iquote /tmp/qemu-test/src/disas/libvixl -I/tmp/qemu-test/src/tests/fp -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/include -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/8086-SSE -I/tmp/qemu-test/src/tests/fp/berkeley-testfloat-3/source -I/usr/include/pixman-1 -Werror -fsanitize=undefined -fsanitize=address -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wno-string-plus-int -Wno-typedef-redefinition -Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1 -DLEGACY_RDMA_REG_MR -DSTRUCT_IOVEC_DEFINED -I/usr/include/libpng16 -I/usr/include/spice-1 -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/nss3 -I/usr/include/nspr4 -pthread -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/pixman-1 -DHW_POISON_H -DTARGET_ARM -DSOFTFLOAT_ROUND_ODD -DINLINE_LEVEL=5 -DSOFTFLOAT_FAST_DIV32TO16 -DSOFTFLOAT_FAST_DIV64TO32 -DSOFTFLOAT_FAST_INT64 -DFLOAT16 -DFLOAT64 -DEXTFLOAT80 -DFLOAT128 -DFLOAT_ROUND_ODD -DLONG_DOUBLE_IS_EXTFLOAT80 -Wno-missing-prototypes -Wno-redundant-decls -Wno-return-type -Wno-error -MMD -MP -MT s_subMagsF64.o -MF ./s_subMagsF64.d -g -c -o s_subMagsF64.o /tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/s_subMagsF64.c
clang -iquote /tmp/qemu-test/build/. -iquote . -iquote /tmp/qemu-test/src/tcg/i386 -isystem /tmp/qemu-test/src/linux-headers -isystem /tmp/qemu-test/build/linux-headers -iquote . -iquote /tmp/qemu-test/src -iquote /tmp/qemu-test/src/accel/tcg -iquote /tmp/qemu-test/src/include -iquote /tmp/qemu-test/src/disas/libvixl -I/tmp/qemu-test/src/tests/fp -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/include -I/tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/8086-SSE -I/tmp/qemu-test/src/tests/fp/berkeley-testfloat-3/source -I/usr/include/pixman-1 -Werror -fsanitize=undefined -fsanitize=address -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wno-string-plus-int -Wno-typedef-redefinition -Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1 -DLEGACY_RDMA_REG_MR -DSTRUCT_IOVEC_DEFINED -I/usr/include/libpng16 -I/usr/include/spice-1 -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/nss3 -I/usr/include/nspr4 -pthread -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/pixman-1 -DHW_POISON_H -DTARGET_ARM -DSOFTFLOAT_ROUND_ODD -DINLINE_LEVEL=5 -DSOFTFLOAT_FAST_DIV32TO16 -DSOFTFLOAT_FAST_DIV64TO32 -DSOFTFLOAT_FAST_INT64 -DFLOAT16 -DFLOAT64 -DEXTFLOAT80 -DFLOAT128 -DFLOAT_ROUND_ODD -DLONG_DOUBLE_IS_EXTFLOAT80 -Wno-missing-prototypes -Wno-redundant-decls -Wno-return-type -Wno-error -MMD -MP -MT s_mulAddF64.o -MF ./s_mulAddF64.d -g -c -o s_mulAddF64.o /tmp/qemu-test/src/tests/fp/berkeley-softfloat-3/source/s_mulAddF64.c
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=2789d6e2430144ccaaa2afa8f60d2912', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=x86_64-softmmu', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-4srx9vwp/src/docker-src.2020-05-20-19.16.23.17821:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-debug']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=2789d6e2430144ccaaa2afa8f60d2912
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-4srx9vwp/src'
make: *** [docker-run-test-debug@fedora] Error 2

real 4m52.381s
user 0m8.277s

The full log is available at
http://patchew.org/logs/[email protected]/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#3 lore raw

Re: [PATCH v2 1/4] Introduce yank feature

In reply to Lukas Straub
Quoted 24 lines · Lukas Straub
yank.c new file +24 -0
+void yank_generic_iochannel(void *opaque)
+{
+    QIOChannel *ioc = QIO_CHANNEL(opaque);
+
+    qio_channel_shutdown(ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
+}
+
+void qmp_yank(strList *instances, Error **errp)
+{
+    strList *tmp;
+    struct YankInstance *instance;
+    struct YankFuncAndParam *entry;
+
+    qemu_mutex_lock(&lock);
+    tmp = instances;
+    for (; tmp; tmp = tmp->next) {
+        instance = yank_find_instance(tmp->value);
+        if (!instance) {
+            error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
+                      "Instance '%s' not found", tmp->value);
+            qemu_mutex_unlock(&lock);
+            return;
+        }
+    }

yank.c line 148

Quoted 10 lines
yank.c new file +10 -0
+    tmp = instances;
+    for (; tmp; tmp = tmp->next) {
+        instance = yank_find_instance(tmp->value);
+        assert(instance);
+        QLIST_FOREACH(entry, &instance->yankfns, next) {
+            entry->func(entry->opaque);
+        }
+    }
+    qemu_mutex_unlock(&lock);
+}

From docs/devel/qapi-code-gen.txt:

An OOB-capable command handler must satisfy the following conditions:

- It terminates quickly.
- It does not invoke system calls that may block.
- It does not access guest RAM that may block when userfaultfd is enabled for postcopy live migration.
- It takes only "fast" locks, i.e. all critical sections protected by any lock it takes also satisfy the conditions for OOB command handler code.

This patch series violates these rules and calls existing functions that were not designed for OOB execution.

Please explain why it is safe to do this.

Stefan

#4 lore raw

Re: [PATCH v2 1/4] Introduce yank feature

Quoted 38 lines · On Thu, 21 May 2020 16:03:35 +0100 Stefan Hajnoczi <[email protected]> wrote:
Quoted 24 lines · Lukas Straub
yank.c new file +24 -0
+void yank_generic_iochannel(void *opaque)
+{
+    QIOChannel *ioc = QIO_CHANNEL(opaque);
+
+    qio_channel_shutdown(ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
+}
+
+void qmp_yank(strList *instances, Error **errp)
+{
+    strList *tmp;
+    struct YankInstance *instance;
+    struct YankFuncAndParam *entry;
+
+    qemu_mutex_lock(&lock);
+    tmp = instances;
+    for (; tmp; tmp = tmp->next) {
+        instance = yank_find_instance(tmp->value);
+        if (!instance) {
+            error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
+                      "Instance '%s' not found", tmp->value);
+            qemu_mutex_unlock(&lock);
+            return;
+        }
+    }

yank.c line 148

Quoted 10 lines
yank.c new file +10 -0
+    tmp = instances;
+    for (; tmp; tmp = tmp->next) {
+        instance = yank_find_instance(tmp->value);
+        assert(instance);
+        QLIST_FOREACH(entry, &instance->yankfns, next) {
+            entry->func(entry->opaque);
+        }
+    }
+    qemu_mutex_unlock(&lock);
+}  

From docs/devel/qapi-code-gen.txt:

Quoted 3 lines

An OOB-capable command handler must satisfy the following conditions:

- It terminates quickly.

Check.

Quoted 1 line

- It does not invoke system calls that may block.

brk/sbrk (malloc and friends):
The manpage doesn't say anything about blocking, but malloc is already used while handling the qmp command.

shutdown():
The manpage doesn't say anything about blocking, but this is already used in migration oob qmp commands.

There are no other syscalls involved to my knowledge.

Quoted 2 lines

- It does not access guest RAM that may block when userfaultfd is enabled for postcopy live migration.

Check.

Quoted 3 lines

- It takes only "fast" locks, i.e. all critical sections protected by any lock it takes also satisfy the conditions for OOB command handler code.

The lock in yank.c satisfies this requirement.

qio_channel_shutdown doesn't take any locks.

Regards,
Lukas Straub

Quoted 6 lines

This patch series violates these rules and calls existing functions that were not designed for OOB execution.

Please explain why it is safe to do this.

Stefan

#5 lore raw

Re: [PATCH v2 1/4] Introduce yank feature

In reply to Lukas Straub
Quoted 44 lines · Lukas Straub
Quoted 38 lines · On Thu, 21 May 2020 16:03:35 +0100 Stefan Hajnoczi <[email protected]> wrote:
Quoted 24 lines · Lukas Straub
yank.c new file +24 -0
+void yank_generic_iochannel(void *opaque)
+{
+    QIOChannel *ioc = QIO_CHANNEL(opaque);
+
+    qio_channel_shutdown(ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
+}
+
+void qmp_yank(strList *instances, Error **errp)
+{
+    strList *tmp;
+    struct YankInstance *instance;
+    struct YankFuncAndParam *entry;
+
+    qemu_mutex_lock(&lock);
+    tmp = instances;
+    for (; tmp; tmp = tmp->next) {
+        instance = yank_find_instance(tmp->value);
+        if (!instance) {
+            error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
+                      "Instance '%s' not found", tmp->value);
+            qemu_mutex_unlock(&lock);
+            return;
+        }
+    }

yank.c line 148

Quoted 10 lines
yank.c new file +10 -0
+    tmp = instances;
+    for (; tmp; tmp = tmp->next) {
+        instance = yank_find_instance(tmp->value);
+        assert(instance);
+        QLIST_FOREACH(entry, &instance->yankfns, next) {
+            entry->func(entry->opaque);
+        }
+    }
+    qemu_mutex_unlock(&lock);
+}  

From docs/devel/qapi-code-gen.txt:

Quoted 3 lines

An OOB-capable command handler must satisfy the following conditions:

- It terminates quickly.

Quoted 8 lines

Check.

Quoted 1 line

- It does not invoke system calls that may block.

brk/sbrk (malloc and friends):
The manpage doesn't say anything about blocking, but malloc is already used while handling the qmp command.

shutdown():
The manpage doesn't say anything about blocking, but this is already used in migration oob qmp commands.

It just marks the socket state in local kernel side. It doesn't involve any blocking roundtrips over the wire, so this is fine.

Quoted 7 lines

There are no other syscalls involved to my knowledge.

Quoted 2 lines

- It does not access guest RAM that may block when userfaultfd is enabled for postcopy live migration.

Check.

Quoted 7 lines
Quoted 3 lines

- It takes only "fast" locks, i.e. all critical sections protected by any lock it takes also satisfy the conditions for OOB command handler code.

The lock in yank.c satisfies this requirement.

qio_channel_shutdown doesn't take any locks.

Agreed, I think the yank code is compliant with all the points listed above.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw

Re: [PATCH v2 1/4] Introduce yank feature

Quoted 68 lines · Daniel P. Berrangé
Quoted 44 lines · Lukas Straub
Quoted 38 lines · On Thu, 21 May 2020 16:03:35 +0100 Stefan Hajnoczi <[email protected]> wrote:
Quoted 24 lines · Lukas Straub
yank.c new file +24 -0
+void yank_generic_iochannel(void *opaque)
+{
+    QIOChannel *ioc = QIO_CHANNEL(opaque);
+
+    qio_channel_shutdown(ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
+}
+
+void qmp_yank(strList *instances, Error **errp)
+{
+    strList *tmp;
+    struct YankInstance *instance;
+    struct YankFuncAndParam *entry;
+
+    qemu_mutex_lock(&lock);
+    tmp = instances;
+    for (; tmp; tmp = tmp->next) {
+        instance = yank_find_instance(tmp->value);
+        if (!instance) {
+            error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
+                      "Instance '%s' not found", tmp->value);
+            qemu_mutex_unlock(&lock);
+            return;
+        }
+    }

yank.c line 148

Quoted 10 lines
yank.c new file +10 -0
+    tmp = instances;
+    for (; tmp; tmp = tmp->next) {
+        instance = yank_find_instance(tmp->value);
+        assert(instance);
+        QLIST_FOREACH(entry, &instance->yankfns, next) {
+            entry->func(entry->opaque);
+        }
+    }
+    qemu_mutex_unlock(&lock);
+}  

From docs/devel/qapi-code-gen.txt:

Quoted 3 lines

An OOB-capable command handler must satisfy the following conditions:

- It terminates quickly.

Quoted 8 lines

Check.

Quoted 1 line

- It does not invoke system calls that may block.

brk/sbrk (malloc and friends):
The manpage doesn't say anything about blocking, but malloc is already used while handling the qmp command.

shutdown():
The manpage doesn't say anything about blocking, but this is already used in migration oob qmp commands.

It just marks the socket state in local kernel side. It doesn't involve any blocking roundtrips over the wire, so this is fine.

Quoted 11 lines

There are no other syscalls involved to my knowledge.

Quoted 2 lines

- It does not access guest RAM that may block when userfaultfd is enabled for postcopy live migration.

Check.

Quoted 3 lines

- It takes only "fast" locks, i.e. all critical sections protected by any lock it takes also satisfy the conditions for OOB command handler code.

Quoted 6 lines
Quoted 3 lines

The lock in yank.c satisfies this requirement.

qio_channel_shutdown doesn't take any locks.

Agreed, I think the yank code is compliant with all the points listed above.

The code does not document this in any way. In fact, it's an interface for registering arbitrary callback functions which execute in this special environment.

If you don't document this explicitly it will break when someone changes the code, even if it works right now.

Please document this in the yank APIs and also in the implementation.

For example, QemuMutex yank has the priority inversion problem: no other function may violate the oob rules while holding the mutex, otherwise the oob function will hang while waiting for the lock when the other function is blocked.

Stefan

#7 lore raw

Re: [PATCH v2 1/4] Introduce yank feature

Quoted 76 lines · On Wed, 17 Jun 2020 15:39:36 +0100 Stefan Hajnoczi <[email protected]> wrote:
Quoted 68 lines · Daniel P. Berrangé
Quoted 44 lines · Lukas Straub
Quoted 38 lines · On Thu, 21 May 2020 16:03:35 +0100 Stefan Hajnoczi <[email protected]> wrote:
Quoted 24 lines · Lukas Straub
yank.c new file +24 -0
+void yank_generic_iochannel(void *opaque)
+{
+    QIOChannel *ioc = QIO_CHANNEL(opaque);
+
+    qio_channel_shutdown(ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
+}
+
+void qmp_yank(strList *instances, Error **errp)
+{
+    strList *tmp;
+    struct YankInstance *instance;
+    struct YankFuncAndParam *entry;
+
+    qemu_mutex_lock(&lock);
+    tmp = instances;
+    for (; tmp; tmp = tmp->next) {
+        instance = yank_find_instance(tmp->value);
+        if (!instance) {
+            error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
+                      "Instance '%s' not found", tmp->value);
+            qemu_mutex_unlock(&lock);
+            return;
+        }
+    }

yank.c line 148

Quoted 10 lines
yank.c new file +10 -0
+    tmp = instances;
+    for (; tmp; tmp = tmp->next) {
+        instance = yank_find_instance(tmp->value);
+        assert(instance);
+        QLIST_FOREACH(entry, &instance->yankfns, next) {
+            entry->func(entry->opaque);
+        }
+    }
+    qemu_mutex_unlock(&lock);
+}    

From docs/devel/qapi-code-gen.txt:

Quoted 3 lines

An OOB-capable command handler must satisfy the following conditions:

- It terminates quickly.

Quoted 8 lines

Check.

Quoted 1 line

- It does not invoke system calls that may block.

brk/sbrk (malloc and friends):
The manpage doesn't say anything about blocking, but malloc is already used while handling the qmp command.

shutdown():
The manpage doesn't say anything about blocking, but this is already used in migration oob qmp commands.

It just marks the socket state in local kernel side. It doesn't involve any blocking roundtrips over the wire, so this is fine.

Quoted 11 lines

There are no other syscalls involved to my knowledge.

Quoted 2 lines

- It does not access guest RAM that may block when userfaultfd is enabled for postcopy live migration.

Check.

Quoted 3 lines

- It takes only "fast" locks, i.e. all critical sections protected by any lock it takes also satisfy the conditions for OOB command handler code.

Quoted 6 lines
Quoted 3 lines

The lock in yank.c satisfies this requirement.

qio_channel_shutdown doesn't take any locks.

Agreed, I think the yank code is compliant with all the points listed above.

Quoted 8 lines

The code does not document this in any way. In fact, it's an interface for registering arbitrary callback functions which execute in this special environment.

If you don't document this explicitly it will break when someone changes the code, even if it works right now.

Please document this in the yank APIs and also in the implementation.

Hi,
It is documented in yank.h:

/**
* yank_register_function: Register a yank function
*
* This registers a yank function. All limitations of qmp oob commands apply
* to the yank function as well.
*
* This function is thread-safe.
*
* @instance_name: The name of the instance
* @func: The yank function
* @opaque: Will be passed to the yank function
*/

Thanks,
Lukas Straub

Quoted 6 lines

For example, QemuMutex yank has the priority inversion problem: no other function may violate the oob rules while holding the mutex, otherwise the oob function will hang while waiting for the lock when the other function is blocked.

Stefan