Skip to contentSkip to conversation

Allow storing the qcow2 L2 cache in disk

RFC PATCH In review Alberto Garcia 6 participants 16 messages Dec 9, 2016 – Dec 13, 2016 Block layer

lore
Share & tools
lore
#0 lore raw

cover Allow storing the qcow2 L2 cache in disk

Hi all,

as we all know, one of the main things that can make the qcow2 format slow is the need to load entries from the L2 table in order to map a guest offset (on the virtual disk) to a host offset (on the qcow2 image).

We have an L2 cache to deal with this, and as long as the cache is big enough then the peformance is comparable to that of a raw image.

For large qcow2 images the amount of RAM we need in order to cache all L2 tables can be big (128MB per TB of disk image if we're using the default cluster size of 64KB). In order to solve this problem we have a setting that allows the user to clean unused cache entries after a certain interval of time. This works fine most of the time, although we can still have peaks of RAM usage if there's a lot of I/O going on in one or more VMs.

In some scenarios, however, there's a different alternative: if the qcow2 image is stored in a slow backend (eg. HDD), we could save memory by putting the L2 cache in a faster one (SSD) instead of in RAM.

I have been making some tests with exactly that scenario and the results look good: storing the cache in disk gives roughly the same performance as storing it in memory.

|---------------------+------------+------+------------+--------|
|                     | Random 4k reads   | Sequential 4k reads |
|                     | Throughput | IOPS | Throughput |  IOPS  |
|---------------------+------------+------+------------+--------|
| Cache in memory/SSD | 406 KB/s   |   99 | 84 MB/s    |  21000 |
| Default cache (1MB) | 200 KB/s   |   60 | 83 MB/s    |  21000 |
| No cache            | 200 KB/s   |   49 | 56 MB/s    |  14000 |
|---------------------+------------+------+------------+--------|

I'm including the patch that I used to get these results. This is the simplest approach that I could think of.

Opinions, questions?

Thanks,

Berto

Alberto Garcia (1):
qcow2: Allow storing the qcow2 L2 cache in disk

 block/qcow2-cache.c | 56 +++++++++++++++++++++++++++++++++++++++++++++--------
 block/qcow2.c       | 11 +++++++++--
 block/qcow2.h       |  3 ++-
 3 files changed, 59 insertions(+), 11 deletions(-)
Signature · 1 line
2.10.2
#1 lore raw

RFC PATCH 1/1 qcow2: Allow storing the qcow2 L2 cache in disk

In scenarios where we have a large qcow2 file stored in a slow storage backend, we can save memory by storing its L2 cache in a faster drive rather than keeping it in RAM.

---
 block/qcow2-cache.c | 56 +++++++++++++++++++++++++++++++++++++++++++++--------
 block/qcow2.c       | 11 +++++++++--
 block/qcow2.h       |  3 ++-
 3 files changed, 59 insertions(+), 11 deletions(-)
block/qcow2-cache.c +48 -8
index 1d25147..8ba8ed9 100644
@@ -23,6 +23,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/mmap-alloc.h"
 #include "block/block_int.h"
 #include "qemu-common.h"
 #include "qcow2.h"
@@ -36,6 +37,7 @@ typedef struct Qcow2CachedTable {
 } Qcow2CachedTable;
 
 struct Qcow2Cache {
+    int                     fd;
     Qcow2CachedTable       *entries;
     struct Qcow2Cache      *depends;
     int                     size;
@@ -63,7 +65,7 @@ static inline int qcow2_cache_get_table_idx(BlockDriverState *bs,
 }
 
 static void qcow2_cache_table_release(BlockDriverState *bs, Qcow2Cache *c,
-                                      int i, int num_tables)
+                                      int i, int num_tables, bool flush)
 {
 /* Using MADV_DONTNEED to discard memory is a Linux-specific feature */
 #ifdef CONFIG_LINUX
@@ -74,6 +76,10 @@ static void qcow2_cache_table_release(BlockDriverState *bs, Qcow2Cache *c,
     size_t offset = QEMU_ALIGN_UP((uintptr_t) t, align) - (uintptr_t) t;
     size_t length = QEMU_ALIGN_DOWN(mem_size - offset, align);
     if (length > 0) {
+        if (flush) {
+            assert(c->fd > 0);
+            msync((uint8_t *) t + offset, length, MS_SYNC);
+        }
         madvise((uint8_t *) t + offset, length, MADV_DONTNEED);
     }
 #endif
@@ -106,26 +112,50 @@ void qcow2_cache_clean_unused(BlockDriverState *bs, Qcow2Cache *c)
         }
 
         if (to_clean > 0) {
-            qcow2_cache_table_release(bs, c, i - to_clean, to_clean);
+            qcow2_cache_table_release(bs, c, i - to_clean, to_clean, false);
         }
     }
 
     c->cache_clean_lru_counter = c->lru_counter;
 }
 
-Qcow2Cache *qcow2_cache_create(BlockDriverState *bs, int num_tables)
+Qcow2Cache *qcow2_cache_create(BlockDriverState *bs, int num_tables,
+                               const char *cache_file_path)
 {
     BDRVQcow2State *s = bs->opaque;
     Qcow2Cache *c;
+    size_t total_size = (size_t) num_tables * s->cluster_size;
 
     c = g_new0(Qcow2Cache, 1);
     c->size = num_tables;
     c->entries = g_try_new0(Qcow2CachedTable, num_tables);
-    c->table_array = qemu_try_blockalign(bs->file->bs,
-                                         (size_t) num_tables * s->cluster_size);
+    if (cache_file_path) {
+        size_t align = MAX(bdrv_opt_mem_align(bs->file->bs), getpagesize());
+        char *filename = g_strdup_printf("%s/qemu-qcow2-cache.XXXXXX",
+                                         cache_file_path);
+        c->fd = mkstemp(filename);
+        if (c->fd > 0) {
+            unlink(filename);
+        }
+        g_free(filename);
+        if (c->fd == -1 || ftruncate(c->fd, total_size)) {
+            goto out;
+        }
+        c->table_array = qemu_ram_mmap(c->fd, total_size, align, true);
+    } else {
+        c->table_array = qemu_try_blockalign(bs->file->bs, total_size);
+    }
 
+out:
     if (!c->entries || !c->table_array) {
-        qemu_vfree(c->table_array);
+        if (cache_file_path) {
+            qemu_ram_munmap(c->table_array, total_size);
+            if (c->fd > 0) {
+                close(c->fd);
+            }
+        } else {
+            qemu_vfree(c->table_array);
+        }
         g_free(c->entries);
         g_free(c);
         c = NULL;
@@ -142,7 +172,14 @@ int qcow2_cache_destroy(BlockDriverState *bs, Qcow2Cache *c)
         assert(c->entries[i].ref == 0);
     }
 
-    qemu_vfree(c->table_array);
+    if (c->fd) {
+        BDRVQcow2State *s = bs->opaque;
+        size_t total_size = (size_t) c->size * s->cluster_size;
+        qemu_ram_munmap(c->table_array, total_size);
+        close(c->fd);
+    } else {
+        qemu_vfree(c->table_array);
+    }
     g_free(c->entries);
     g_free(c);
 
@@ -297,7 +334,7 @@ int qcow2_cache_empty(BlockDriverState *bs, Qcow2Cache *c)
         c->entries[i].lru_counter = 0;
     }
 
-    qcow2_cache_table_release(bs, c, 0, c->size);
+    qcow2_cache_table_release(bs, c, 0, c->size, false);
 
     c->lru_counter = 0;
 
@@ -399,6 +436,9 @@ void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table)
 
     if (c->entries[i].ref == 0) {
         c->entries[i].lru_counter = ++c->lru_counter;
+        if (c->fd) {
+            qcow2_cache_table_release(bs, c, i, 1, true);
+        }
     }
 
     assert(c->entries[i].ref >= 0);
block/qcow2.c +9 -2
index 96fb8a8..4d5d092 100644
@@ -461,6 +461,11 @@ static QemuOptsList qcow2_runtime_opts = {
             .type = QEMU_OPT_NUMBER,
             .help = "Clean unused cache entries after this time (in seconds)",
         },
+        {
+            .name = "l2-cache-path",
+            .type = QEMU_OPT_STRING,
+            .help = "Directory where to store the L2 cache",
+        },
         { /* end of list */ }
     },
 };
@@ -656,8 +661,10 @@ static int qcow2_update_options_prepare(BlockDriverState *bs,
         }
     }
 
-    r->l2_table_cache = qcow2_cache_create(bs, l2_cache_size);
-    r->refcount_block_cache = qcow2_cache_create(bs, refcount_cache_size);
+    const char *path = qemu_opt_get(opts, "l2-cache-path");
+
+    r->l2_table_cache = qcow2_cache_create(bs, l2_cache_size, path);
+    r->refcount_block_cache = qcow2_cache_create(bs, refcount_cache_size, NULL);
     if (r->l2_table_cache == NULL || r->refcount_block_cache == NULL) {
         error_setg(errp, "Could not allocate metadata caches");
         ret = -ENOMEM;
block/qcow2.h +2 -1
index 1823414..494f1ed 100644
@@ -569,7 +569,8 @@ void qcow2_free_snapshots(BlockDriverState *bs);
 int qcow2_read_snapshots(BlockDriverState *bs);
 
 /* qcow2-cache.c functions */
-Qcow2Cache *qcow2_cache_create(BlockDriverState *bs, int num_tables);
+Qcow2Cache *qcow2_cache_create(BlockDriverState *bs, int num_tables,
+                               const char *cache_file_path);
 int qcow2_cache_destroy(BlockDriverState* bs, Qcow2Cache *c);
 
 void qcow2_cache_entry_mark_dirty(BlockDriverState *bs, Qcow2Cache *c,
Signature · 1 line
2.10.2
#2 lore raw
In reply to Alberto Garcia
Your series failed automatic build test. Please find the testing commands and their output below. If you have docker installed, you can probably reproduce it locally.

Hi,

Your series failed automatic build test. Please find the testing commands and their output below. If you have docker installed, you can probably reproduce it locally.

Subject: [Qemu-devel] [PATCH RFC 0/1] Allow storing the qcow2 L2 cache in disk
Type: series
Message-id: [email protected]

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=16
make docker-test-quick@centos6
make docker-test-mingw@fedora
make docker-test-build@min-glib
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384 From https://github.com/patchew-project/qemu
* [new tag] patchew/[email protected] -> patchew/[email protected]
Switched to a new branch 'test'
e1b19c9 qcow2: Allow storing the qcow2 L2 cache in disk

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc' Cloning into 'dtc'...
Submodule path 'dtc': checked out '65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf'
BUILD centos6
make[1]: Entering directory `/var/tmp/patchew-tester-tmp-9l5w0aja/src'
ARCHIVE qemu.tgz
ARCHIVE dtc.tgz
COPY RUNNER
RUN test-quick in qemu:centos6
Packages installed:
SDL-devel-1.2.14-7.el6_7.1.x86_64
ccache-3.1.6-2.el6.x86_64
epel-release-6-8.noarch
gcc-4.4.7-17.el6.x86_64
git-1.7.1-4.el6_7.1.x86_64
glib2-devel-2.28.8-5.el6.x86_64
libfdt-devel-1.4.0-1.el6.x86_64
make-3.81-23.el6.x86_64
package g++ is not installed
pixman-devel-0.32.8-1.el6.x86_64
tar-1.23-15.el6_8.x86_64
zlib-devel-1.2.3-29.el6.x86_64

Environment variables:
PACKAGES=libfdt-devel ccache tar git make gcc g++ zlib-devel glib2-devel SDL-devel pixman-devel epel-release
HOSTNAME=ab913e537085
TERM=xterm
MAKEFLAGS= -j16
HISTSIZE=1000
J=16
USER=root
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
MAIL=/var/spool/mail/root PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PWD=/
LANG=en_US.UTF-8
TARGET_LIST=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
FEATURES= dtc
DEBUG=
G_BROKEN_FILENAMES=1
CCACHE_HASHDIR=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/var/tmp/qemu-build/install No C++ compiler available; disabling C++ specific optional code

Install prefix    /var/tmp/qemu-build/install
BIOS directory    /var/tmp/qemu-build/install/share/qemu
binary directory  /var/tmp/qemu-build/install/bin

library directory /var/tmp/qemu-build/install/lib module directory /var/tmp/qemu-build/install/lib/qemu libexec directory /var/tmp/qemu-build/install/libexec include directory /var/tmp/qemu-build/install/include config directory /var/tmp/qemu-build/install/etc
local state directory /var/tmp/qemu-build/install/var
Manual directory /var/tmp/qemu-build/install/share/man ELF interp prefix /usr/gnemul/qemu-%M

Source path       /tmp/qemu-test/src
C compiler        cc
Host C compiler   cc
C++ compiler      
Objective-C compiler cc
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/include/pixman-1    -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all
LDFLAGS           -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
tcg debug enabled no
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
pixman            system
SDL support       yes (1.2.14)
GTK support       no 
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    no
GNUTLS rnd        no
libgcrypt         no
libgcrypt kdf     no
nettle            no 
nettle kdf        no
libtasn1          no
curses support    no
virgl support     no
curl support      no
mingw32 support   no
Audio drivers     oss
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
VNC support       yes
VNC SASL support  no
VNC JPEG support  no
VNC PNG support   no
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               yes
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
COLO support      yes
RDMA support      no
TCG interpreter   no
fdt support       yes
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes

libcap-ng support no
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes

Trace backends    log
spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            no
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no

build guest agent yes
QGA VSS support no
QGA w32 disk info no
QGA MSI support no
seccomp support no
coroutine backend ucontext
coroutine pool yes
debug stack usage no
GlusterFS support no
Archipelago support no

gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   no
TPM passthrough   yes
QOM debugging     yes
lzo support       no
snappy support    no
bzip2 support     no

NUMA host support no
tcmalloc support no
jemalloc support no
avx2 optimization no
replication support yes

  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     config-host.h
  GEN     qemu-options.def
  GEN     qmp-commands.h
  GEN     qapi-visit.h
  GEN     qapi-types.h
  GEN     qapi-event.h
  GEN     qmp-introspect.h
  GEN     x86_64-softmmu/config-devices.mak
  GEN     module_block.h
  GEN     aarch64-softmmu/config-devices.mak
  GEN     tests/test-qapi-types.h
  GEN     tests/test-qapi-visit.h
  GEN     tests/test-qmp-commands.h
  GEN     tests/test-qapi-event.h
  GEN     tests/test-qmp-introspect.h
  GEN     config-all-devices.mak
  GEN     trace/generated-tracers.h
  GEN     trace/generated-tcg-tracers.h
  GEN     trace/generated-helpers-wrappers.h
  GEN     trace/generated-helpers.h
  CC      tests/qemu-iotests/socket_scm_helper.o
  GEN     qga/qapi-generated/qga-qapi-types.h
  GEN     qga/qapi-generated/qga-qapi-visit.h
  GEN     qga/qapi-generated/qga-qmp-commands.h
  GEN     qga/qapi-generated/qga-qapi-types.c
  GEN     qga/qapi-generated/qga-qapi-visit.c
  GEN     qga/qapi-generated/qga-qmp-marshal.c
  GEN     qmp-introspect.c
  GEN     qapi-types.c
  GEN     qapi-visit.c
  GEN     qapi-event.c
  CC      qapi/qapi-visit-core.o
  CC      qapi/qapi-dealloc-visitor.o
  CC      qapi/qobject-input-visitor.o
  CC      qapi/qobject-output-visitor.o
  CC      qapi/qmp-registry.o
  CC      qapi/qmp-dispatch.o
  CC      qapi/string-input-visitor.o
  CC      qapi/string-output-visitor.o
  CC      qapi/opts-visitor.o
  CC      qapi/qapi-clone-visitor.o
  CC      qapi/qmp-event.o
  CC      qapi/qapi-util.o
  CC      qobject/qnull.o
  CC      qobject/qint.o
  CC      qobject/qstring.o
  CC      qobject/qdict.o
  CC      qobject/qlist.o
  CC      qobject/qfloat.o
  CC      qobject/qbool.o
  CC      qobject/qjson.o
  CC      qobject/qobject.o
  CC      qobject/json-lexer.o
  CC      qobject/json-streamer.o
  CC      qobject/json-parser.o
  GEN     trace/generated-tracers.c
  CC      trace/control.o
  CC      trace/qmp.o
  CC      util/osdep.o
  CC      util/cutils.o
  CC      util/unicode.o
  CC      util/qemu-timer-common.o
  CC      util/bufferiszero.o
  CC      util/compatfd.o
  CC      util/event_notifier-posix.o
  CC      util/mmap-alloc.o
  CC      util/oslib-posix.o
  CC      util/qemu-openpty.o
  CC      util/qemu-thread-posix.o
  CC      util/memfd.o
  CC      util/envlist.o
  CC      util/path.o
  CC      util/module.o
  CC      util/bitmap.o
  CC      util/bitops.o
  CC      util/fifo8.o
  CC      util/acl.o
  CC      util/hbitmap.o
  CC      util/error.o
  CC      util/id.o
  CC      util/qemu-error.o
  CC      util/iov.o
  CC      util/qemu-config.o
  CC      util/qemu-sockets.o
  CC      util/uri.o
  CC      util/notify.o
  CC      util/qemu-option.o
  CC      util/qemu-progress.o
  CC      util/hexdump.o
  CC      util/crc32c.o
  CC      util/getauxval.o
  CC      util/uuid.o
  CC      util/readline.o
  CC      util/throttle.o
  CC      util/rcu.o
  CC      util/qemu-coroutine.o
  CC      util/qemu-coroutine-lock.o
  CC      util/qemu-coroutine-io.o
  CC      util/qemu-coroutine-sleep.o
  CC      util/coroutine-ucontext.o
  CC      util/buffer.o
  CC      util/timed-average.o
  CC      util/base64.o
  CC      util/log.o
  CC      util/qdist.o
  CC      util/qht.o
  CC      util/range.o
  CC      crypto/pbkdf-stub.o
  CC      stubs/arch-query-cpu-def.o
  CC      stubs/arch-query-cpu-model-expansion.o
  CC      stubs/bdrv-next-monitor-owned.o
  CC      stubs/arch-query-cpu-model-comparison.o
  CC      stubs/arch-query-cpu-model-baseline.o
  CC      stubs/blk-commit-all.o
  CC      stubs/blockdev-close-all-bdrv-states.o
  CC      stubs/cpu-get-clock.o
  CC      stubs/clock-warp.o
  CC      stubs/dump.o
  CC      stubs/cpu-get-icount.o
  CC      stubs/fdset-add-fd.o
  CC      stubs/error-printf.o
  CC      stubs/fdset-find-fd.o
  CC      stubs/fdset-get-fd.o
  CC      stubs/fdset-remove-fd.o
  CC      stubs/gdbstub.o
  CC      stubs/get-fd.o
  CC      stubs/get-next-serial.o
  CC      stubs/get-vm-name.o
  CC      stubs/iothread.o
  CC      stubs/iothread-lock.o
  CC      stubs/is-daemonized.o
  CC      stubs/machine-init-done.o
  CC      stubs/migr-blocker.o
  CC      stubs/mon-is-qmp.o
  CC      stubs/monitor-init.o
  CC      stubs/notify-event.o
  CC      stubs/qtest.o
  CC      stubs/replay.o
  CC      stubs/replay-user.o
  CC      stubs/reset.o
  CC      stubs/runstate-check.o
  CC      stubs/set-fd-handler.o
  CC      stubs/slirp.o
  CC      stubs/sysbus.o
  CC      stubs/trace-control.o
  CC      stubs/uuid.o
  CC      stubs/vm-stop.o
  CC      stubs/vmstate.o
  CC      stubs/cpus.o
  CC      stubs/kvm.o
  CC      stubs/qmp_pc_dimm_device_list.o
  CC      stubs/target-monitor-defs.o
  CC      stubs/target-get-monitor-def.o
  CC      stubs/vhost.o
  CC      stubs/iohandler.o
  CC      stubs/smbios_type_38.o
  CC      stubs/ipmi.o
  CC      stubs/pc_madt_cpu_entry.o
  CC      stubs/migration-colo.o
  CC      contrib/ivshmem-client/ivshmem-client.o
  CC      contrib/ivshmem-client/main.o
  CC      contrib/ivshmem-server/ivshmem-server.o
  CC      qemu-nbd.o
  CC      contrib/ivshmem-server/main.o
  CC      async.o
  CC      thread-pool.o
  CC      block.o
  CC      blockjob.o
  CC      iohandler.o
  CC      main-loop.o
  CC      qemu-timer.o
  CC      aio-posix.o
  CC      qemu-io-cmds.o
  CC      replication.o
  CC      block/raw_bsd.o
  CC      block/qcow.o
  CC      block/vdi.o
  CC      block/vmdk.o
  CC      block/cloop.o
  CC      block/bochs.o
  CC      block/vpc.o
  CC      block/vvfat.o
  CC      block/qcow2.o
  CC      block/dmg.o
  CC      block/qcow2-refcount.o
  CC      block/qcow2-cluster.o
  CC      block/qcow2-snapshot.o
  CC      block/qcow2-cache.o
  CC      block/qed.o
  CC      block/qed-gencb.o
  CC      block/qed-l2-cache.o
  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
  CC      block/vhdx.o
  CC      block/vhdx-endian.o
  CC      block/vhdx-log.o
  CC      block/quorum.o
  CC      block/parallels.o
  CC      block/blkdebug.o
  CC      block/blkverify.o
  CC      block/blkreplay.o
  CC      block/block-backend.o
  CC      block/qapi.o
  CC      block/snapshot.o
  CC      block/null.o
  CC      block/raw-posix.o
  CC      block/mirror.o
  CC      block/commit.o
  CC      block/io.o
  CC      block/throttle-groups.o
  CC      block/nbd.o
  CC      block/nbd-client.o
  CC      block/sheepdog.o
  CC      block/accounting.o
  CC      block/dirty-bitmap.o
  CC      block/write-threshold.o
  CC      block/backup.o
  CC      block/replication.o
  CC      block/crypto.o
  CC      nbd/server.o
  CC      nbd/client.o
  CC      crypto/init.o
  CC      nbd/common.o
  CC      crypto/hash.o
  CC      crypto/hash-glib.o
  CC      crypto/aes.o
  CC      crypto/desrfb.o
  CC      crypto/tlscreds.o
  CC      crypto/cipher.o
  CC      crypto/tlscredsanon.o
  CC      crypto/tlscredsx509.o
  CC      crypto/tlssession.o
  CC      crypto/random-platform.o
  CC      crypto/secret.o
  CC      crypto/ivgen.o
  CC      crypto/pbkdf.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain.o
  CC      crypto/ivgen-plain64.o
  CC      crypto/afsplit.o
  CC      crypto/xts.o
  CC      crypto/block.o
  CC      crypto/block-qcow.o
  CC      crypto/block-luks.o
  CC      io/channel.o
  CC      io/channel-buffer.o
  CC      io/channel-command.o
  CC      io/channel-file.o
  CC      io/channel-socket.o
  CC      io/channel-tls.o
  CC      io/channel-watch.o
  CC      io/channel-websock.o
  CC      io/channel-util.o
  CC      io/task.o
  CC      qom/object.o
  CC      qom/container.o
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  GEN     qemu-img-cmds.h
  CC      qemu-io.o
  CC      qemu-bridge-helper.o
  CC      blockdev.o
  CC      blockdev-nbd.o
  CC      iothread.o
  CC      device-hotplug.o
  CC      qdev-monitor.o
  CC      os-posix.o
  CC      qemu-char.o
  CC      accel.o
  CC      bt-host.o
  CC      page_cache.o
  CC      bt-vhci.o
  CC      dma-helpers.o
  CC      vl.o
  CC      tpm.o
  CC      device_tree.o
  GEN     qmp-marshal.c
  CC      qmp.o
  CC      cpus-common.o
  CC      hmp.o
  CC      audio/audio.o
  CC      audio/noaudio.o
  CC      audio/wavaudio.o
  CC      audio/mixeng.o
  CC      audio/sdlaudio.o
  CC      audio/ossaudio.o
  CC      audio/wavcapture.o
  CC      backends/rng.o
  CC      backends/rng-egd.o
  CC      backends/rng-random.o
  CC      backends/msmouse.o
  CC      backends/testdev.o
  CC      backends/tpm.o
  CC      backends/hostmem.o
  CC      backends/hostmem-ram.o
  CC      backends/hostmem-file.o
  CC      backends/cryptodev.o
  CC      block/stream.o
  CC      backends/cryptodev-builtin.o
  CC      disas/arm.o
  CC      disas/i386.o
  CC      fsdev/qemu-fsdev-dummy.o
  CC      fsdev/qemu-fsdev-opts.o
  CC      hw/acpi/core.o
  CC      hw/acpi/pcihp.o
  CC      hw/acpi/piix4.o
  CC      hw/acpi/ich9.o
  CC      hw/acpi/tco.o
  CC      hw/acpi/cpu_hotplug.o
  CC      hw/acpi/memory_hotplug.o
  CC      hw/acpi/memory_hotplug_acpi_table.o
  CC      hw/acpi/cpu.o
  CC      hw/acpi/nvdimm.o
  CC      hw/acpi/acpi_interface.o
  CC      hw/acpi/bios-linker-loader.o
  CC      hw/acpi/aml-build.o
  CC      hw/acpi/ipmi.o
  CC      hw/audio/sb16.o
  CC      hw/audio/es1370.o
  CC      hw/audio/ac97.o
  CC      hw/audio/fmopl.o
  CC      hw/audio/adlib.o
  CC      hw/audio/gus.o
  CC      hw/audio/gusemu_hal.o
  CC      hw/audio/gusemu_mixer.o
  CC      hw/audio/cs4231a.o
  CC      hw/audio/intel-hda.o
  CC      hw/audio/hda-codec.o
  CC      hw/audio/pcspk.o
  CC      hw/audio/wm8750.o
  CC      hw/audio/pl041.o
  CC      hw/audio/lm4549.o
  CC      hw/audio/marvell_88w8618.o
  CC      hw/block/block.o
  CC      hw/block/cdrom.o
  CC      hw/block/hd-geometry.o
  CC      hw/block/fdc.o
  CC      hw/block/m25p80.o
  CC      hw/block/nand.o
  CC      hw/block/pflash_cfi01.o
  CC      hw/block/pflash_cfi02.o
  CC      hw/block/ecc.o
  CC      hw/block/onenand.o
  CC      hw/block/nvme.o
  CC      hw/bt/core.o
  CC      hw/bt/l2cap.o
  CC      hw/bt/sdp.o
  CC      hw/bt/hci.o
  CC      hw/bt/hid.o
  CC      hw/bt/hci-csr.o
  CC      hw/char/pl011.o
  CC      hw/char/ipoctal232.o
  CC      hw/char/parallel.o
  CC      hw/char/serial.o
  CC      hw/char/serial-isa.o
  CC      hw/char/serial-pci.o
  CC      hw/char/virtio-console.o
  CC      hw/char/cadence_uart.o
  CC      hw/char/debugcon.o
  CC      hw/char/imx_serial.o
  CC      hw/core/qdev.o
  CC      hw/core/qdev-properties.o
  CC      hw/core/bus.o
  CC      hw/core/fw-path-provider.o
  CC      hw/core/irq.o
  CC      hw/core/hotplug.o
  CC      hw/core/ptimer.o
  CC      hw/core/sysbus.o
  CC      hw/core/machine.o
  CC      hw/core/null-machine.o
  CC      hw/core/loader.o
  CC      hw/core/qdev-properties-system.o
  CC      hw/core/register.o
  CC      hw/core/or-irq.o
  CC      hw/core/platform-bus.o
  CC      hw/display/cirrus_vga.o
  CC      hw/display/ads7846.o
  CC      hw/display/pl110.o
  CC      hw/display/ssd0303.o
  CC      hw/display/ssd0323.o
  CC      hw/display/vga-pci.o
  CC      hw/display/vga-isa.o
  CC      hw/display/vmware_vga.o
  CC      hw/display/blizzard.o
  CC      hw/display/exynos4210_fimd.o
  CC      hw/display/framebuffer.o
  CC      hw/display/tc6393xb.o
  CC      hw/dma/pl080.o
  CC      hw/dma/pl330.o
  CC      hw/dma/i8257.o
  CC      hw/dma/xlnx-zynq-devcfg.o
  CC      hw/gpio/max7310.o
  CC      hw/gpio/pl061.o
  CC      hw/gpio/zaurus.o
  CC      hw/gpio/gpio_key.o
  CC      hw/i2c/core.o
  CC      hw/i2c/smbus.o
  CC      hw/i2c/smbus_eeprom.o
  CC      hw/i2c/i2c-ddc.o
  CC      hw/i2c/versatile_i2c.o
  CC      hw/i2c/pm_smbus.o
  CC      hw/i2c/smbus_ich9.o
  CC      hw/i2c/bitbang_i2c.o
  CC      hw/i2c/exynos4210_i2c.o
  CC      hw/i2c/imx_i2c.o
  CC      hw/i2c/aspeed_i2c.o
  CC      hw/ide/core.o
  CC      hw/ide/atapi.o
  CC      hw/ide/qdev.o
  CC      hw/ide/pci.o
  CC      hw/ide/piix.o
  CC      hw/ide/isa.o
  CC      hw/ide/microdrive.o
  CC      hw/ide/ahci.o
  CC      hw/ide/ich.o
  CC      hw/input/hid.o
  CC      hw/input/lm832x.o
  CC      hw/input/pckbd.o
  CC      hw/input/pl050.o
  CC      hw/input/ps2.o
  CC      hw/input/stellaris_input.o
  CC      hw/input/tsc2005.o
  CC      hw/input/vmmouse.o
  CC      hw/input/virtio-input.o
  CC      hw/input/virtio-input-hid.o
  CC      hw/intc/i8259_common.o
  CC      hw/input/virtio-input-host.o
  CC      hw/intc/i8259.o
  CC      hw/intc/pl190.o
  CC      hw/intc/imx_avic.o
  CC      hw/intc/realview_gic.o
  CC      hw/intc/ioapic_common.o
  CC      hw/intc/arm_gic_common.o
  CC      hw/intc/arm_gic.o
  CC      hw/intc/arm_gicv2m.o
  CC      hw/intc/arm_gicv3_common.o
  CC      hw/intc/arm_gicv3.o
  CC      hw/intc/arm_gicv3_dist.o
  CC      hw/intc/arm_gicv3_redist.o
  CC      hw/intc/arm_gicv3_its_common.o
  CC      hw/intc/intc.o
  CC      hw/ipack/ipack.o
  CC      hw/ipack/tpci200.o
  CC      hw/ipmi/ipmi.o
  CC      hw/ipmi/ipmi_bmc_sim.o
  CC      hw/ipmi/ipmi_bmc_extern.o
  CC      hw/ipmi/isa_ipmi_kcs.o
  CC      hw/ipmi/isa_ipmi_bt.o
  CC      hw/isa/isa-bus.o
  CC      hw/isa/apm.o
  CC      hw/mem/pc-dimm.o
  CC      hw/mem/nvdimm.o
  CC      hw/misc/applesmc.o
  CC      hw/misc/max111x.o
  CC      hw/misc/tmp105.o
  CC      hw/misc/debugexit.o
  CC      hw/misc/sga.o
  CC      hw/misc/pc-testdev.o
  CC      hw/misc/pci-testdev.o
  CC      hw/misc/arm_l2x0.o
  CC      hw/misc/arm_integrator_debug.o
  CC      hw/misc/a9scu.o
  CC      hw/net/ne2000.o
  CC      hw/misc/arm11scu.o
  CC      hw/net/eepro100.o
  CC      hw/net/pcnet-pci.o
  CC      hw/net/pcnet.o
  CC      hw/net/e1000.o
  CC      hw/net/e1000x_common.o
  CC      hw/net/net_tx_pkt.o
  CC      hw/net/net_rx_pkt.o
  CC      hw/net/e1000e.o
  CC      hw/net/e1000e_core.o
  CC      hw/net/rtl8139.o
  CC      hw/net/smc91c111.o
  CC      hw/net/vmxnet3.o
  CC      hw/net/lan9118.o
  CC      hw/net/ne2000-isa.o
  CC      hw/net/xgmac.o
  CC      hw/net/allwinner_emac.o
  CC      hw/net/imx_fec.o
  CC      hw/net/cadence_gem.o
  CC      hw/net/rocker/rocker_fp.o
  CC      hw/net/stellaris_enet.o
  CC      hw/net/rocker/rocker.o
  CC      hw/net/rocker/rocker_desc.o
  CC      hw/net/rocker/rocker_world.o
  CC      hw/net/rocker/rocker_of_dpa.o
  CC      hw/nvram/eeprom93xx.o
  CC      hw/nvram/fw_cfg.o
  CC      hw/nvram/chrp_nvram.o
  CC      hw/pci-bridge/pci_expander_bridge.o
  CC      hw/pci-bridge/pci_bridge_dev.o
  CC      hw/pci-bridge/xio3130_upstream.o
  CC      hw/pci-bridge/xio3130_downstream.o
  CC      hw/pci-bridge/ioh3420.o
  CC      hw/pci-bridge/i82801b11.o
  CC      hw/pci-host/pam.o
  CC      hw/pci-host/versatile.o
  CC      hw/pci-host/q35.o
  CC      hw/pci-host/piix.o
  CC      hw/pci-host/gpex.o
  CC      hw/pci/pci.o
  CC      hw/pci/msix.o
  CC      hw/pci/pci_bridge.o
  CC      hw/pci/msi.o
  CC      hw/pci/shpc.o
/tmp/qemu-test/src/hw/nvram/fw_cfg.c: In function ‘fw_cfg_dma_transfer’:
/tmp/qemu-test/src/hw/nvram/fw_cfg.c:329: warning: ‘read’ may be used uninitialized in this function
  CC      hw/pci/slotid_cap.o
  CC      hw/pci/pcie.o
  CC      hw/pci/pci_host.o
  CC      hw/pci/pcie_aer.o
  CC      hw/pci/pcie_host.o
  CC      hw/pci/pcie_port.o
  CC      hw/pci/pci-stub.o
  CC      hw/pcmcia/pcmcia.o
  CC      hw/scsi/scsi-disk.o
  CC      hw/scsi/scsi-generic.o
  CC      hw/scsi/scsi-bus.o
  CC      hw/scsi/lsi53c895a.o
  CC      hw/scsi/mptsas.o
  CC      hw/scsi/mptconfig.o
  CC      hw/scsi/megasas.o
  CC      hw/scsi/mptendian.o
  CC      hw/scsi/vmw_pvscsi.o
  CC      hw/scsi/esp.o
  CC      hw/scsi/esp-pci.o
  CC      hw/sd/pl181.o
  CC      hw/sd/ssi-sd.o
  CC      hw/sd/core.o
  CC      hw/sd/sd.o
  CC      hw/sd/sdhci.o
  CC      hw/smbios/smbios.o
  CC      hw/smbios/smbios_type_38.o
  CC      hw/ssi/pl022.o
  CC      hw/ssi/ssi.o
  CC      hw/ssi/xilinx_spips.o
  CC      hw/ssi/aspeed_smc.o
  CC      hw/ssi/stm32f2xx_spi.o
  CC      hw/timer/arm_timer.o
  CC      hw/timer/arm_mptimer.o
  CC      hw/timer/a9gtimer.o
  CC      hw/timer/ds1338.o
  CC      hw/timer/hpet.o
  CC      hw/timer/cadence_ttc.o
  CC      hw/timer/i8254_common.o
  CC      hw/timer/i8254.o
  CC      hw/timer/pl031.o
  CC      hw/timer/twl92230.o
  CC      hw/timer/imx_epit.o
  CC      hw/timer/imx_gpt.o
  CC      hw/timer/stm32f2xx_timer.o
  CC      hw/timer/aspeed_timer.o
  CC      hw/tpm/tpm_tis.o
  CC      hw/tpm/tpm_passthrough.o
  CC      hw/tpm/tpm_util.o
  CC      hw/usb/core.o
  CC      hw/usb/combined-packet.o
  CC      hw/usb/bus.o
  CC      hw/usb/libhw.o
  CC      hw/usb/desc.o
  CC      hw/usb/desc-msos.o
  CC      hw/usb/hcd-uhci.o
  CC      hw/usb/hcd-ohci.o
  CC      hw/usb/hcd-ehci.o
  CC      hw/usb/hcd-ehci-pci.o
  CC      hw/usb/hcd-ehci-sysbus.o
  CC      hw/usb/hcd-xhci.o
  CC      hw/usb/hcd-musb.o
  CC      hw/usb/dev-hub.o
  CC      hw/usb/dev-hid.o
  CC      hw/usb/dev-wacom.o
  CC      hw/usb/dev-storage.o
  CC      hw/usb/dev-uas.o
  CC      hw/usb/dev-audio.o
  CC      hw/usb/dev-serial.o
  CC      hw/usb/dev-network.o
  CC      hw/usb/dev-bluetooth.o
  CC      hw/usb/host-stub.o
  CC      hw/usb/dev-smartcard-reader.o
  CC      hw/virtio/virtio-rng.o
  CC      hw/usb/dev-mtp.o
  CC      hw/virtio/virtio-pci.o
  CC      hw/virtio/virtio-bus.o
  CC      hw/virtio/virtio-mmio.o
  CC      hw/watchdog/watchdog.o
  CC      hw/watchdog/wdt_ib700.o
  CC      migration/migration.o
  CC      hw/watchdog/wdt_i6300esb.o
  CC      migration/socket.o
  CC      migration/fd.o
  CC      migration/colo-comm.o
  CC      migration/exec.o
  CC      migration/tls.o
  CC      migration/colo.o
  CC      migration/colo-failover.o
  CC      migration/vmstate.o
  CC      migration/qemu-file.o
  CC      migration/qemu-file-channel.o
  CC      migration/postcopy-ram.o
  CC      migration/xbzrle.o
  CC      migration/qjson.o
  CC      migration/block.o
  CC      net/net.o
  CC      net/queue.o
  CC      net/checksum.o
  CC      net/util.o
  CC      net/hub.o
  CC      net/dump.o
  CC      net/socket.o
  CC      net/eth.o
  CC      net/l2tpv3.o
  CC      net/tap.o
  CC      net/vhost-user.o
  CC      net/slirp.o
  CC      net/tap-linux.o
  CC      net/filter.o
  CC      net/filter-buffer.o
  CC      net/filter-mirror.o
  CC      net/colo-compare.o
  CC      net/colo.o
  CC      net/filter-rewriter.o
  CC      qom/cpu.o
  CC      replay/replay.o
  CC      replay/replay-internal.o
  CC      replay/replay-time.o
  CC      replay/replay-events.o
  CC      replay/replay-input.o
  CC      replay/replay-char.o
  CC      replay/replay-snapshot.o
  CC      slirp/cksum.o
  CC      slirp/if.o
  CC      slirp/ip_icmp.o
  CC      slirp/ip6_icmp.o
/tmp/qemu-test/src/replay/replay-internal.c: In function ‘replay_put_array’:
/tmp/qemu-test/src/replay/replay-internal.c:65: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result
  CC      slirp/ip6_input.o
  CC      slirp/ip6_output.o
  CC      slirp/ip_input.o
  CC      slirp/ip_output.o
  CC      slirp/dnssearch.o
  CC      slirp/dhcpv6.o
  CC      slirp/slirp.o
  CC      slirp/mbuf.o
  CC      slirp/misc.o
  CC      slirp/socket.o
  CC      slirp/sbuf.o
  CC      slirp/tcp_subr.o
  CC      slirp/tcp_input.o
  CC      slirp/tcp_output.o
  CC      slirp/tcp_timer.o
  CC      slirp/udp.o
  CC      slirp/udp6.o
  CC      slirp/bootp.o
  CC      slirp/arp_table.o
  CC      slirp/tftp.o
  CC      slirp/ndp_table.o
  CC      ui/keymaps.o
  CC      ui/console.o
  CC      ui/cursor.o
  CC      ui/qemu-pixman.o
  CC      ui/input.o

/tmp/qemu-test/src/slirp/tcp_input.c: In function ‘tcp_input’:
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_p’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_len’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_tos’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_id’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_off’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_ttl’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_sum’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_src.s_addr’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_dst.s_addr’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:220: warning: ‘save_ip6.ip_nh’ may be used uninitialized in this function

  CC      ui/input-keymap.o
  CC      ui/input-legacy.o
  CC      ui/input-linux.o
  CC      ui/sdl.o
  CC      ui/sdl_zoom.o
  CC      ui/x_keymap.o
  CC      ui/vnc-enc-hextile.o
  CC      ui/vnc.o
  CC      ui/vnc-enc-tight.o
  CC      ui/vnc-enc-zlib.o
  CC      ui/vnc-palette.o
  CC      ui/vnc-enc-zrle.o
  CC      ui/vnc-auth-vencrypt.o
  CC      ui/vnc-ws.o
  CC      ui/vnc-jobs.o
  LINK    tests/qemu-iotests/socket_scm_helper
  CC      qga/commands.o
  CC      qga/guest-agent-command-state.o
  CC      qga/main.o
  CC      qga/commands-posix.o
  CC      qga/qapi-generated/qga-qapi-types.o
  CC      qga/channel-posix.o
  CC      qga/qapi-generated/qga-qapi-visit.o
  CC      qmp-introspect.o
  CC      qapi-event.o
  CC      qga/qapi-generated/qga-qmp-marshal.o
  CC      qapi-types.o
  CC      qapi-visit.o
  AR      libqemustub.a
  CC      qmp-marshal.o
  CC      qemu-img.o
  CC      trace/generated-tracers.o
  AR      libqemuutil.a
  AS      optionrom/multiboot.o
  AS      optionrom/linuxboot.o
  CC      optionrom/linuxboot_dma.o
  AS      optionrom/kvmvapic.o
cc unrecognized option '-no-integrated-as'
cc unrecognized option '-no-integrated-as'
  BUILD   optionrom/linuxboot_dma.img
  BUILD   optionrom/linuxboot_dma.raw
  BUILD   optionrom/multiboot.img
  BUILD   optionrom/linuxboot.img
  BUILD   optionrom/kvmvapic.img
  SIGN    optionrom/linuxboot_dma.bin
  BUILD   optionrom/multiboot.raw
  BUILD   optionrom/linuxboot.raw
  BUILD   optionrom/kvmvapic.raw
  SIGN    optionrom/multiboot.bin
  SIGN    optionrom/linuxboot.bin
  SIGN    optionrom/kvmvapic.bin
  LINK    qemu-ga
  LINK    ivshmem-client
  LINK    ivshmem-server
  LINK    qemu-nbd
  LINK    qemu-img
  LINK    qemu-io
  LINK    qemu-bridge-helper
  GEN     aarch64-softmmu/config-target.h
  GEN     aarch64-softmmu/hmp-commands.h
  GEN     aarch64-softmmu/hmp-commands-info.h
  GEN     x86_64-softmmu/hmp-commands.h
  GEN     x86_64-softmmu/hmp-commands-info.h
  GEN     x86_64-softmmu/config-target.h
  CC      aarch64-softmmu/translate-all.o
  CC      aarch64-softmmu/exec.o
  CC      aarch64-softmmu/cpu-exec.o
  CC      aarch64-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/translate-common.o
  CC      aarch64-softmmu/cpu-exec-common.o
  CC      aarch64-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/tcg/optimize.o
  CC      aarch64-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/disas.o
  CC      aarch64-softmmu/tcg-runtime.o
  GEN     aarch64-softmmu/gdbstub-xml.c
  CC      aarch64-softmmu/kvm-stub.o
  CC      aarch64-softmmu/arch_init.o
  CC      aarch64-softmmu/cpus.o
  CC      aarch64-softmmu/monitor.o
  CC      aarch64-softmmu/gdbstub.o
  CC      aarch64-softmmu/balloon.o
  CC      aarch64-softmmu/ioport.o
  CC      aarch64-softmmu/numa.o
  CC      aarch64-softmmu/qtest.o
  CC      aarch64-softmmu/bootdevice.o
  CC      aarch64-softmmu/memory.o
  CC      aarch64-softmmu/cputlb.o
  CC      aarch64-softmmu/dump.o
  CC      aarch64-softmmu/memory_mapping.o
  CC      aarch64-softmmu/migration/ram.o
  CC      aarch64-softmmu/migration/savevm.o
  CC      aarch64-softmmu/xen-common-stub.o
  CC      aarch64-softmmu/xen-hvm-stub.o
  CC      aarch64-softmmu/hw/adc/stm32f2xx_adc.o
  CC      aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      aarch64-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/hw/char/exynos4210_uart.o
  CC      aarch64-softmmu/hw/char/omap_uart.o
  CC      aarch64-softmmu/hw/char/digic-uart.o
  CC      aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC      aarch64-softmmu/hw/char/bcm2835_aux.o
  CC      aarch64-softmmu/hw/char/virtio-serial-bus.o
  CC      aarch64-softmmu/hw/core/nmi.o
  CC      aarch64-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC      aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC      aarch64-softmmu/hw/cpu/a9mpcore.o
  CC      aarch64-softmmu/hw/cpu/a15mpcore.o
  CC      aarch64-softmmu/hw/cpu/core.o
  CC      aarch64-softmmu/hw/display/omap_dss.o
  CC      aarch64-softmmu/hw/display/omap_lcdc.o
  CC      aarch64-softmmu/hw/display/pxa2xx_lcd.o
  CC      aarch64-softmmu/hw/display/bcm2835_fb.o
  CC      aarch64-softmmu/hw/display/vga.o
  CC      aarch64-softmmu/hw/display/virtio-gpu.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-softmmu/hw/display/dpcd.o
  CC      aarch64-softmmu/hw/display/xlnx_dp.o
  CC      aarch64-softmmu/hw/dma/omap_dma.o
  CC      aarch64-softmmu/hw/dma/xlnx_dpdma.o
  CC      x86_64-softmmu/exec.o
  CC      x86_64-softmmu/translate-all.o
  CC      x86_64-softmmu/cpu-exec.o
  CC      aarch64-softmmu/hw/dma/soc_dma.o
  CC      aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC      aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC      aarch64-softmmu/hw/gpio/omap_gpio.o
  CC      aarch64-softmmu/hw/gpio/imx_gpio.o
  CC      aarch64-softmmu/hw/i2c/omap_i2c.o
  CC      aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC      x86_64-softmmu/translate-common.o
  CC      x86_64-softmmu/cpu-exec-common.o
  CC      x86_64-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/hw/input/tsc210x.o
  CC      x86_64-softmmu/tcg/tcg-op.o
  CC      x86_64-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/tcg/optimize.o
  CC      x86_64-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC      x86_64-softmmu/disas.o
  CC      aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC      x86_64-softmmu/tcg-runtime.o
  CC      aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC      x86_64-softmmu/arch_init.o
  CC      aarch64-softmmu/hw/intc/omap_intc.o
  CC      x86_64-softmmu/cpus.o
  CC      aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC      x86_64-softmmu/monitor.o
  CC      aarch64-softmmu/hw/intc/bcm2836_control.o
  CC      aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC      aarch64-softmmu/hw/intc/aspeed_vic.o
  CC      aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      aarch64-softmmu/hw/misc/ivshmem.o
  CC      aarch64-softmmu/hw/misc/arm_sysctl.o
  CC      x86_64-softmmu/gdbstub.o
  CC      x86_64-softmmu/ioport.o
  CC      x86_64-softmmu/balloon.o
  CC      aarch64-softmmu/hw/misc/cbus.o
  CC      x86_64-softmmu/numa.o
  CC      x86_64-softmmu/qtest.o
  CC      x86_64-softmmu/bootdevice.o
  CC      x86_64-softmmu/kvm-all.o
  CC      x86_64-softmmu/memory.o
  CC      x86_64-softmmu/cputlb.o
  CC      x86_64-softmmu/memory_mapping.o
  CC      aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC      x86_64-softmmu/dump.o
  CC      x86_64-softmmu/migration/ram.o
  CC      x86_64-softmmu/migration/savevm.o
  CC      x86_64-softmmu/xen-common-stub.o
  CC      x86_64-softmmu/xen-hvm-stub.o
  CC      x86_64-softmmu/hw/block/virtio-blk.o
  CC      x86_64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      x86_64-softmmu/hw/char/virtio-serial-bus.o
  CC      x86_64-softmmu/hw/core/nmi.o
  CC      x86_64-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/hw/misc/imx_ccm.o
  CC      aarch64-softmmu/hw/misc/imx31_ccm.o
  CC      x86_64-softmmu/hw/cpu/core.o
  CC      aarch64-softmmu/hw/misc/imx25_ccm.o
  CC      aarch64-softmmu/hw/misc/imx6_ccm.o
  CC      x86_64-softmmu/hw/display/vga.o
  CC      aarch64-softmmu/hw/misc/imx6_src.o
  CC      aarch64-softmmu/hw/misc/mst_fpga.o
  CC      x86_64-softmmu/hw/display/virtio-gpu.o
  CC      aarch64-softmmu/hw/misc/omap_gpmc.o
  CC      aarch64-softmmu/hw/misc/omap_clk.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/misc/omap_l4.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-softmmu/hw/misc/omap_sdrc.o
  CC      x86_64-softmmu/hw/display/virtio-vga.o
  CC      x86_64-softmmu/hw/intc/apic.o
  CC      aarch64-softmmu/hw/misc/omap_tap.o
  CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC      aarch64-softmmu/hw/misc/bcm2835_property.o
  CC      x86_64-softmmu/hw/intc/apic_common.o
  CC      aarch64-softmmu/hw/misc/zynq_slcr.o
  CC      aarch64-softmmu/hw/misc/zynq-xadc.o
  CC      x86_64-softmmu/hw/intc/ioapic.o
  CC      aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      x86_64-softmmu/hw/isa/lpc_ich9.o
  CC      x86_64-softmmu/hw/misc/vmport.o
  CC      x86_64-softmmu/hw/misc/ivshmem.o
  CC      x86_64-softmmu/hw/misc/edu.o
  CC      x86_64-softmmu/hw/misc/pvpanic.o
  CC      x86_64-softmmu/hw/misc/hyperv_testdev.o
  CC      aarch64-softmmu/hw/misc/edu.o
  CC      aarch64-softmmu/hw/misc/auxbus.o
  CC      aarch64-softmmu/hw/misc/aspeed_scu.o
  CC      aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC      x86_64-softmmu/hw/net/virtio-net.o
  CC      x86_64-softmmu/hw/net/vhost_net.o
  CC      aarch64-softmmu/hw/net/virtio-net.o
  CC      aarch64-softmmu/hw/net/vhost_net.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/pcmcia/pxa2xx.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC      x86_64-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi.o
  CC      x86_64-softmmu/hw/timer/mc146818rtc.o
  CC      aarch64-softmmu/hw/sd/omap_mmc.o
  CC      x86_64-softmmu/hw/vfio/common.o
  CC      x86_64-softmmu/hw/vfio/pci.o
  CC      aarch64-softmmu/hw/sd/pxa2xx_mmci.o
  CC      x86_64-softmmu/hw/vfio/pci-quirks.o
  CC      aarch64-softmmu/hw/ssi/omap_spi.o
  CC      x86_64-softmmu/hw/vfio/platform.o
  CC      x86_64-softmmu/hw/vfio/calxeda-xgmac.o
  CC      aarch64-softmmu/hw/ssi/imx_spi.o
  CC      x86_64-softmmu/hw/vfio/amd-xgbe.o
  CC      aarch64-softmmu/hw/timer/exynos4210_mct.o
  CC      aarch64-softmmu/hw/timer/exynos4210_pwm.o
  CC      aarch64-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/timer/omap_gptimer.o
  CC      aarch64-softmmu/hw/timer/omap_synctimer.o
  CC      aarch64-softmmu/hw/timer/pxa2xx_timer.o
  CC      x86_64-softmmu/hw/vfio/spapr.o
  CC      x86_64-softmmu/hw/virtio/virtio.o
  CC      aarch64-softmmu/hw/timer/digic-timer.o
  CC      x86_64-softmmu/hw/virtio/virtio-balloon.o
  CC      x86_64-softmmu/hw/virtio/vhost.o
  CC      x86_64-softmmu/hw/virtio/vhost-backend.o
  CC      x86_64-softmmu/hw/virtio/vhost-user.o
  CC      aarch64-softmmu/hw/timer/allwinner-a10-pit.o
  CC      aarch64-softmmu/hw/usb/tusb6010.o
  CC      aarch64-softmmu/hw/vfio/common.o
  CC      aarch64-softmmu/hw/vfio/pci.o
  CC      x86_64-softmmu/hw/virtio/vhost-vsock.o
  CC      aarch64-softmmu/hw/vfio/pci-quirks.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      aarch64-softmmu/hw/vfio/platform.o
  CC      x86_64-softmmu/hw/i386/multiboot.o
  CC      x86_64-softmmu/hw/i386/pc.o
  CC      aarch64-softmmu/hw/vfio/calxeda-xgmac.o
  CC      x86_64-softmmu/hw/i386/pc_piix.o
  CC      x86_64-softmmu/hw/i386/pc_q35.o
  CC      aarch64-softmmu/hw/vfio/amd-xgbe.o
  CC      x86_64-softmmu/hw/i386/pc_sysfw.o
  CC      aarch64-softmmu/hw/vfio/spapr.o
  CC      x86_64-softmmu/hw/i386/x86-iommu.o
  CC      x86_64-softmmu/hw/i386/intel_iommu.o
  CC      aarch64-softmmu/hw/virtio/virtio.o
  CC      aarch64-softmmu/hw/virtio/virtio-balloon.o
  CC      x86_64-softmmu/hw/i386/amd_iommu.o
  CC      x86_64-softmmu/hw/i386/kvmvapic.o
/tmp/qemu-test/src/hw/i386/pc_piix.c: In function ‘igd_passthrough_isa_bridge_create’:
/tmp/qemu-test/src/hw/i386/pc_piix.c:1046: warning: ‘pch_rev_id’ may be used uninitialized in this function
  CC      x86_64-softmmu/hw/i386/acpi-build.o
  CC      aarch64-softmmu/hw/virtio/vhost.o
  CC      aarch64-softmmu/hw/virtio/vhost-backend.o
  CC      aarch64-softmmu/hw/virtio/vhost-user.o
  CC      x86_64-softmmu/hw/i386/pci-assign-load-rom.o
  CC      x86_64-softmmu/hw/i386/kvm/clock.o
  CC      x86_64-softmmu/hw/i386/kvm/apic.o
  CC      aarch64-softmmu/hw/virtio/vhost-vsock.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      x86_64-softmmu/hw/i386/kvm/i8259.o
  CC      x86_64-softmmu/hw/i386/kvm/ioapic.o
  CC      x86_64-softmmu/hw/i386/kvm/i8254.o
  CC      aarch64-softmmu/hw/arm/boot.o
  CC      aarch64-softmmu/hw/arm/collie.o
  CC      x86_64-softmmu/hw/i386/kvm/pci-assign.o
  CC      aarch64-softmmu/hw/arm/exynos4_boards.o
/tmp/qemu-test/src/hw/i386/acpi-build.c: In function ‘build_append_pci_bus_devices’:
/tmp/qemu-test/src/hw/i386/acpi-build.c:501: warning: ‘notify_method’ may be used uninitialized in this function
  CC      x86_64-softmmu/target-i386/translate.o
  CC      aarch64-softmmu/hw/arm/gumstix.o
  CC      aarch64-softmmu/hw/arm/highbank.o
  CC      x86_64-softmmu/target-i386/helper.o
  CC      x86_64-softmmu/target-i386/cpu.o
  CC      aarch64-softmmu/hw/arm/integratorcp.o
  CC      aarch64-softmmu/hw/arm/digic_boards.o
  CC      x86_64-softmmu/target-i386/bpt_helper.o
  CC      x86_64-softmmu/target-i386/excp_helper.o
  CC      x86_64-softmmu/target-i386/fpu_helper.o
  CC      x86_64-softmmu/target-i386/cc_helper.o
  CC      x86_64-softmmu/target-i386/int_helper.o
  CC      x86_64-softmmu/target-i386/svm_helper.o
  CC      aarch64-softmmu/hw/arm/mainstone.o
  CC      x86_64-softmmu/target-i386/smm_helper.o
  CC      aarch64-softmmu/hw/arm/musicpal.o
  CC      aarch64-softmmu/hw/arm/nseries.o
  CC      aarch64-softmmu/hw/arm/omap_sx1.o
  CC      aarch64-softmmu/hw/arm/palm.o
  CC      x86_64-softmmu/target-i386/misc_helper.o
  CC      aarch64-softmmu/hw/arm/realview.o
  CC      x86_64-softmmu/target-i386/mem_helper.o
  CC      aarch64-softmmu/hw/arm/spitz.o
  CC      aarch64-softmmu/hw/arm/stellaris.o
  CC      aarch64-softmmu/hw/arm/tosa.o
  CC      aarch64-softmmu/hw/arm/versatilepb.o
  CC      x86_64-softmmu/target-i386/seg_helper.o
  CC      x86_64-softmmu/target-i386/mpx_helper.o
  CC      x86_64-softmmu/target-i386/gdbstub.o
  CC      aarch64-softmmu/hw/arm/vexpress.o
  CC      x86_64-softmmu/target-i386/machine.o
  CC      x86_64-softmmu/target-i386/arch_memory_mapping.o
  CC      aarch64-softmmu/hw/arm/virt.o
  CC      x86_64-softmmu/target-i386/arch_dump.o
  CC      x86_64-softmmu/target-i386/monitor.o
  CC      x86_64-softmmu/target-i386/kvm.o
  CC      x86_64-softmmu/target-i386/hyperv.o
  CC      aarch64-softmmu/hw/arm/xilinx_zynq.o
  CC      aarch64-softmmu/hw/arm/z2.o
  CC      aarch64-softmmu/hw/arm/virt-acpi-build.o
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/hw/arm/netduino2.o
  CC      x86_64-softmmu/trace/control-target.o
  CC      aarch64-softmmu/hw/arm/armv7m.o
  CC      aarch64-softmmu/hw/arm/sysbus-fdt.o
  CC      aarch64-softmmu/hw/arm/exynos4210.o
  CC      aarch64-softmmu/hw/arm/pxa2xx.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_gpio.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_pic.o
  CC      aarch64-softmmu/hw/arm/digic.o
  CC      aarch64-softmmu/hw/arm/omap1.o
  CC      aarch64-softmmu/hw/arm/omap2.o
  CC      aarch64-softmmu/hw/arm/strongarm.o
  CC      aarch64-softmmu/hw/arm/allwinner-a10.o
  CC      aarch64-softmmu/hw/arm/cubieboard.o
  CC      aarch64-softmmu/hw/arm/bcm2835_peripherals.o
  CC      aarch64-softmmu/hw/arm/bcm2836.o
  CC      aarch64-softmmu/hw/arm/raspi.o
  CC      aarch64-softmmu/hw/arm/stm32f205_soc.o
  CC      aarch64-softmmu/hw/arm/xlnx-zynqmp.o
  CC      aarch64-softmmu/hw/arm/xlnx-ep108.o
  CC      aarch64-softmmu/hw/arm/fsl-imx25.o
  CC      aarch64-softmmu/hw/arm/imx25_pdk.o
  CC      aarch64-softmmu/hw/arm/fsl-imx31.o
  CC      aarch64-softmmu/hw/arm/kzm.o
  CC      aarch64-softmmu/hw/arm/fsl-imx6.o
  CC      aarch64-softmmu/hw/arm/sabrelite.o
  CC      aarch64-softmmu/hw/arm/aspeed_soc.o
  CC      aarch64-softmmu/hw/arm/aspeed.o
  CC      aarch64-softmmu/target-arm/arm-semi.o
  CC      aarch64-softmmu/target-arm/machine.o
  CC      aarch64-softmmu/target-arm/psci.o
  CC      aarch64-softmmu/target-arm/arch_dump.o
  CC      aarch64-softmmu/target-arm/monitor.o
  CC      aarch64-softmmu/target-arm/kvm-stub.o
  CC      aarch64-softmmu/target-arm/translate.o
  CC      aarch64-softmmu/target-arm/op_helper.o
  CC      aarch64-softmmu/target-arm/helper.o
  CC      aarch64-softmmu/target-arm/cpu.o
  CC      aarch64-softmmu/target-arm/neon_helper.o
  CC      aarch64-softmmu/target-arm/iwmmxt_helper.o
  CC      aarch64-softmmu/target-arm/gdbstub.o
  CC      aarch64-softmmu/target-arm/cpu64.o
  CC      aarch64-softmmu/target-arm/translate-a64.o
  CC      aarch64-softmmu/target-arm/helper-a64.o
  CC      aarch64-softmmu/target-arm/gdbstub64.o
  CC      aarch64-softmmu/target-arm/crypto_helper.o
  CC      aarch64-softmmu/target-arm/arm-powerctl.o

/tmp/qemu-test/src/target-arm/translate-a64.c: In function ‘handle_shri_with_rndacc’:
/tmp/qemu-test/src/target-arm/translate-a64.c:6391: warning: ‘tcg_src_hi’ may be used uninitialized in this function /tmp/qemu-test/src/target-arm/translate-a64.c: In function ‘disas_simd_scalar_two_reg_misc’:
/tmp/qemu-test/src/target-arm/translate-a64.c:8118: warning: ‘rmode’ may be used uninitialized in this function

  CC      x86_64-softmmu/trace/generated-helpers.o
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/trace/control-target.o
  CC      aarch64-softmmu/gdbstub-xml.o
  LINK    x86_64-softmmu/qemu-system-x86_64
  CC      aarch64-softmmu/trace/generated-helpers.o
  LINK    aarch64-softmmu/qemu-system-aarch64
  TEST    tests/qapi-schema/alternate-any.out
  TEST    tests/qapi-schema/alternate-array.out
  TEST    tests/qapi-schema/alternate-base.out
  TEST    tests/qapi-schema/alternate-conflict-string.out
  TEST    tests/qapi-schema/alternate-clash.out
  TEST    tests/qapi-schema/alternate-conflict-dict.out
  TEST    tests/qapi-schema/alternate-empty.out
  TEST    tests/qapi-schema/alternate-nested.out
  TEST    tests/qapi-schema/alternate-unknown.out
  TEST    tests/qapi-schema/args-alternate.out
  TEST    tests/qapi-schema/args-any.out
  TEST    tests/qapi-schema/args-array-empty.out
  TEST    tests/qapi-schema/args-array-unknown.out
  TEST    tests/qapi-schema/args-bad-boxed.out
  TEST    tests/qapi-schema/args-boxed-anon.out
  TEST    tests/qapi-schema/args-boxed-empty.out
  TEST    tests/qapi-schema/args-boxed-string.out
  TEST    tests/qapi-schema/args-int.out
  TEST    tests/qapi-schema/args-invalid.out
  TEST    tests/qapi-schema/args-member-array-bad.out
  TEST    tests/qapi-schema/args-member-case.out
  TEST    tests/qapi-schema/args-member-unknown.out
  TEST    tests/qapi-schema/args-name-clash.out
  TEST    tests/qapi-schema/args-union.out
  TEST    tests/qapi-schema/args-unknown.out
  TEST    tests/qapi-schema/bad-base.out
  TEST    tests/qapi-schema/bad-data.out
  TEST    tests/qapi-schema/bad-ident.out
  TEST    tests/qapi-schema/bad-type-bool.out
  TEST    tests/qapi-schema/bad-type-dict.out
  TEST    tests/qapi-schema/bad-type-int.out
  TEST    tests/qapi-schema/base-cycle-direct.out
  TEST    tests/qapi-schema/base-cycle-indirect.out
  TEST    tests/qapi-schema/command-int.out
  TEST    tests/qapi-schema/comments.out
  TEST    tests/qapi-schema/double-data.out
  TEST    tests/qapi-schema/double-type.out
  TEST    tests/qapi-schema/duplicate-key.out
  TEST    tests/qapi-schema/empty.out
  TEST    tests/qapi-schema/enum-bad-name.out
  TEST    tests/qapi-schema/enum-bad-prefix.out
  TEST    tests/qapi-schema/enum-clash-member.out
  TEST    tests/qapi-schema/enum-dict-member.out
  TEST    tests/qapi-schema/enum-int-member.out
  TEST    tests/qapi-schema/enum-member-case.out
  TEST    tests/qapi-schema/enum-wrong-data.out
  TEST    tests/qapi-schema/enum-missing-data.out
  TEST    tests/qapi-schema/escape-outside-string.out
  TEST    tests/qapi-schema/escape-too-big.out
  TEST    tests/qapi-schema/escape-too-short.out
  TEST    tests/qapi-schema/event-boxed-empty.out
  TEST    tests/qapi-schema/event-case.out
  TEST    tests/qapi-schema/event-nest-struct.out
  TEST    tests/qapi-schema/flat-union-array-branch.out
  TEST    tests/qapi-schema/flat-union-bad-base.out
  TEST    tests/qapi-schema/flat-union-bad-discriminator.out
  TEST    tests/qapi-schema/flat-union-base-any.out
  TEST    tests/qapi-schema/flat-union-base-union.out
  TEST    tests/qapi-schema/flat-union-clash-member.out
  TEST    tests/qapi-schema/flat-union-empty.out
  TEST    tests/qapi-schema/flat-union-incomplete-branch.out
  TEST    tests/qapi-schema/flat-union-inline.out
  TEST    tests/qapi-schema/flat-union-int-branch.out
  TEST    tests/qapi-schema/flat-union-invalid-branch-key.out
  TEST    tests/qapi-schema/flat-union-invalid-discriminator.out
  TEST    tests/qapi-schema/flat-union-no-base.out
  TEST    tests/qapi-schema/flat-union-optional-discriminator.out
  TEST    tests/qapi-schema/flat-union-string-discriminator.out
  TEST    tests/qapi-schema/funny-char.out
  TEST    tests/qapi-schema/ident-with-escape.out
  TEST    tests/qapi-schema/include-before-err.out
  TEST    tests/qapi-schema/include-cycle.out
  TEST    tests/qapi-schema/include-format-err.out
  TEST    tests/qapi-schema/include-nested-err.out
  TEST    tests/qapi-schema/include-no-file.out
  TEST    tests/qapi-schema/include-non-file.out
  TEST    tests/qapi-schema/include-relpath.out
  TEST    tests/qapi-schema/include-repetition.out
  TEST    tests/qapi-schema/include-self-cycle.out
  TEST    tests/qapi-schema/indented-expr.out
  TEST    tests/qapi-schema/include-simple.out
  TEST    tests/qapi-schema/leading-comma-list.out
  TEST    tests/qapi-schema/leading-comma-object.out
  TEST    tests/qapi-schema/missing-colon.out
  TEST    tests/qapi-schema/missing-comma-list.out
  TEST    tests/qapi-schema/missing-comma-object.out
  TEST    tests/qapi-schema/missing-type.out
  TEST    tests/qapi-schema/nested-struct-data.out
  TEST    tests/qapi-schema/non-objects.out
  TEST    tests/qapi-schema/qapi-schema-test.out
  TEST    tests/qapi-schema/quoted-structural-chars.out
  TEST    tests/qapi-schema/redefined-builtin.out
  TEST    tests/qapi-schema/redefined-command.out
  TEST    tests/qapi-schema/redefined-event.out
  TEST    tests/qapi-schema/redefined-type.out
  TEST    tests/qapi-schema/reserved-command-q.out
  TEST    tests/qapi-schema/reserved-enum-q.out
  TEST    tests/qapi-schema/reserved-member-q.out
  TEST    tests/qapi-schema/reserved-member-has.out
  TEST    tests/qapi-schema/reserved-member-u.out
  TEST    tests/qapi-schema/reserved-member-underscore.out
  TEST    tests/qapi-schema/reserved-type-kind.out
  TEST    tests/qapi-schema/reserved-type-list.out
  TEST    tests/qapi-schema/returns-alternate.out
  TEST    tests/qapi-schema/returns-dict.out
  TEST    tests/qapi-schema/returns-array-bad.out
  TEST    tests/qapi-schema/returns-unknown.out
  TEST    tests/qapi-schema/returns-whitelist.out
  TEST    tests/qapi-schema/struct-base-clash-deep.out
  TEST    tests/qapi-schema/struct-base-clash.out
  TEST    tests/qapi-schema/struct-data-invalid.out
  TEST    tests/qapi-schema/struct-member-invalid.out
  TEST    tests/qapi-schema/trailing-comma-object.out
  TEST    tests/qapi-schema/trailing-comma-list.out
  TEST    tests/qapi-schema/type-bypass-bad-gen.out
  TEST    tests/qapi-schema/unclosed-list.out
  TEST    tests/qapi-schema/unclosed-object.out
  TEST    tests/qapi-schema/unclosed-string.out
  TEST    tests/qapi-schema/unicode-str.out
  TEST    tests/qapi-schema/union-base-no-discriminator.out
  TEST    tests/qapi-schema/union-branch-case.out
  TEST    tests/qapi-schema/union-clash-branches.out
  TEST    tests/qapi-schema/union-empty.out
  TEST    tests/qapi-schema/union-invalid-base.out
  TEST    tests/qapi-schema/union-optional-branch.out
  TEST    tests/qapi-schema/union-unknown.out
  TEST    tests/qapi-schema/unknown-expr-key.out
  TEST    tests/qapi-schema/unknown-escape.out
  CC      tests/check-qdict.o
  CC      tests/test-char.o
  CC      tests/check-qfloat.o
  CC      tests/check-qint.o
  CC      tests/check-qstring.o
  CC      tests/check-qnull.o
  CC      tests/check-qlist.o
  CC      tests/check-qjson.o
  CC      tests/test-qobject-output-visitor.o
  GEN     tests/test-qapi-types.c
  GEN     tests/test-qapi-visit.c
  GEN     tests/test-qapi-event.c
  CC      tests/test-clone-visitor.o
  GEN     tests/test-qmp-introspect.c
  CC      tests/test-qobject-input-strict.o
  CC      tests/test-qobject-input-visitor.o
  GEN     tests/test-qmp-marshal.c
  CC      tests/test-qmp-commands.o
  CC      tests/test-string-input-visitor.o
  CC      tests/test-string-output-visitor.o
  CC      tests/test-qmp-event.o
  CC      tests/test-opts-visitor.o
  CC      tests/test-coroutine.o
  CC      tests/test-visitor-serialization.o
  CC      tests/test-iov.o
  CC      tests/test-aio.o
  CC      tests/test-throttle.o
  CC      tests/test-thread-pool.o
  CC      tests/test-hbitmap.o
  CC      tests/test-blockjob.o
  CC      tests/test-blockjob-txn.o
  CC      tests/test-x86-cpuid.o
  CC      tests/test-xbzrle.o
  CC      tests/test-vmstate.o
  CC      tests/test-cutils.o
  CC      tests/test-mul64.o
  CC      tests/test-int128.o
  CC      tests/rcutorture.o
  CC      tests/test-rcu-list.o
  CC      tests/test-qdist.o
  CC      tests/test-qht.o
  CC      tests/test-qht-par.o
  CC      tests/qht-bench.o
  CC      tests/test-bitops.o
  CC      tests/check-qom-interface.o
/tmp/qemu-test/src/tests/test-int128.c:180: warning: ‘__noclone__’ attribute directive ignored
  CC      tests/check-qom-proplist.o
  CC      tests/test-qemu-opts.o
  CC      tests/test-write-threshold.o
  CC      tests/test-crypto-hash.o
  CC      tests/test-crypto-cipher.o
  CC      tests/test-crypto-secret.o
  CC      tests/test-qga.o
  CC      tests/libqtest.o
  CC      tests/test-timed-average.o
  CC      tests/test-io-task.o
  CC      tests/test-io-channel-socket.o
  CC      tests/io-channel-helpers.o
  CC      tests/test-io-channel-command.o
  CC      tests/test-io-channel-file.o
  CC      tests/test-io-channel-buffer.o
  CC      tests/test-base64.o
  CC      tests/test-crypto-ivgen.o
  CC      tests/test-crypto-afsplit.o
  CC      tests/test-crypto-xts.o
  CC      tests/test-crypto-block.o
  CC      tests/test-logging.o
  CC      tests/test-bufferiszero.o
  CC      tests/test-uuid.o
  CC      tests/test-replication.o
  CC      tests/ptimer-test-stubs.o
  CC      tests/ptimer-test.o
  CC      tests/vhost-user-test.o
  CC      tests/libqos/pci.o
  CC      tests/libqos/fw_cfg.o
  CC      tests/libqos/malloc.o
  CC      tests/libqos/i2c.o
  CC      tests/libqos/libqos.o
  CC      tests/libqos/malloc-spapr.o
  CC      tests/libqos/libqos-spapr.o
  CC      tests/libqos/rtas.o
  CC      tests/libqos/pci-spapr.o
  CC      tests/libqos/malloc-pc.o
  CC      tests/libqos/pci-pc.o
  CC      tests/libqos/libqos-pc.o
  CC      tests/libqos/ahci.o
  CC      tests/libqos/virtio.o
  CC      tests/libqos/virtio-pci.o
  CC      tests/libqos/virtio-mmio.o
  CC      tests/libqos/malloc-generic.o
  CC      tests/endianness-test.o
  CC      tests/fdc-test.o
  CC      tests/ide-test.o
  CC      tests/ahci-test.o
  CC      tests/hd-geo-test.o
  CC      tests/bios-tables-test.o
  CC      tests/boot-order-test.o
  CC      tests/boot-sector.o
  CC      tests/boot-serial-test.o
  CC      tests/pxe-test.o

/tmp/qemu-test/src/tests/ide-test.c: In function ‘cdrom_pio_impl’:
/tmp/qemu-test/src/tests/ide-test.c:791: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result /tmp/qemu-test/src/tests/ide-test.c: In function ‘test_cdrom_dma’:
/tmp/qemu-test/src/tests/ide-test.c:886: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result

  CC      tests/ipmi-kcs-test.o
  CC      tests/rtc-test.o
  CC      tests/ipmi-bt-test.o
  CC      tests/i440fx-test.o
  CC      tests/fw_cfg-test.o
  CC      tests/wdt_ib700-test.o
  CC      tests/drive_del-test.o
  CC      tests/tco-test.o
  CC      tests/e1000-test.o
  CC      tests/e1000e-test.o
  CC      tests/rtl8139-test.o
  CC      tests/eepro100-test.o
  CC      tests/pcnet-test.o
  CC      tests/ne2000-test.o
  CC      tests/nvme-test.o
  CC      tests/ac97-test.o
  CC      tests/es1370-test.o
  CC      tests/virtio-balloon-test.o
  CC      tests/virtio-net-test.o
  CC      tests/virtio-blk-test.o
  CC      tests/virtio-scsi-test.o
  CC      tests/virtio-rng-test.o
  CC      tests/virtio-serial-test.o
  CC      tests/virtio-console-test.o
  CC      tests/ipoctal232-test.o
  CC      tests/tpci200-test.o
  CC      tests/display-vga-test.o
  CC      tests/intel-hda-test.o
  CC      tests/ivshmem-test.o
  CC      tests/vmxnet3-test.o
  CC      tests/pvpanic-test.o
  CC      tests/i82801b11-test.o
  CC      tests/usb-hcd-ohci-test.o
  CC      tests/ioh3420-test.o
  CC      tests/libqos/usb.o
  CC      tests/usb-hcd-uhci-test.o
  CC      tests/usb-hcd-ehci-test.o
  CC      tests/usb-hcd-xhci-test.o
  CC      tests/pc-cpu-test.o
  CC      tests/q35-test.o
  CC      tests/test-netfilter.o
  CC      tests/test-filter-mirror.o
  CC      tests/postcopy-test.o
  CC      tests/test-filter-redirector.o
  CC      tests/test-x86-cpuid-compat.o
  CC      tests/device-introspect-test.o
  CC      tests/qom-test.o
  LINK    tests/check-qdict
  LINK    tests/test-char
  LINK    tests/check-qfloat
  LINK    tests/check-qint
  LINK    tests/check-qstring
  LINK    tests/check-qlist
  LINK    tests/check-qnull
  LINK    tests/check-qjson
  CC      tests/test-qapi-visit.o
  CC      tests/test-qapi-types.o
  CC      tests/test-qapi-event.o
  CC      tests/test-qmp-introspect.o
  CC      tests/test-qmp-marshal.o
  LINK    tests/test-coroutine
  LINK    tests/test-visitor-serialization
  LINK    tests/test-iov
  LINK    tests/test-aio
  LINK    tests/test-throttle
  LINK    tests/test-thread-pool
  LINK    tests/test-hbitmap
  LINK    tests/test-blockjob
  LINK    tests/test-blockjob-txn
  LINK    tests/test-x86-cpuid
  LINK    tests/test-xbzrle
  LINK    tests/test-vmstate
  LINK    tests/test-cutils
  LINK    tests/test-mul64
  LINK    tests/test-int128
  LINK    tests/rcutorture
  LINK    tests/test-rcu-list
  LINK    tests/test-qdist
  LINK    tests/test-qht
  LINK    tests/qht-bench
  LINK    tests/test-bitops
  LINK    tests/check-qom-interface
  LINK    tests/check-qom-proplist
  LINK    tests/test-qemu-opts
  LINK    tests/test-write-threshold
  LINK    tests/test-crypto-hash
  LINK    tests/test-crypto-cipher
  LINK    tests/test-crypto-secret
  LINK    tests/test-qga
  LINK    tests/test-timed-average
  LINK    tests/test-io-task
  LINK    tests/test-io-channel-socket
  LINK    tests/test-io-channel-file
  LINK    tests/test-io-channel-command
  LINK    tests/test-io-channel-buffer
  LINK    tests/test-base64
  LINK    tests/test-crypto-ivgen
  LINK    tests/test-crypto-afsplit
  LINK    tests/test-crypto-xts
  LINK    tests/test-crypto-block
  LINK    tests/test-logging
  LINK    tests/test-replication
  LINK    tests/test-bufferiszero
  LINK    tests/test-uuid
  LINK    tests/ptimer-test
  LINK    tests/vhost-user-test
  LINK    tests/endianness-test
  LINK    tests/fdc-test
  LINK    tests/ide-test
  LINK    tests/ahci-test
  LINK    tests/hd-geo-test
  LINK    tests/boot-order-test
  LINK    tests/bios-tables-test
  LINK    tests/boot-serial-test
  LINK    tests/pxe-test
  LINK    tests/rtc-test
  LINK    tests/ipmi-kcs-test
  LINK    tests/ipmi-bt-test
  LINK    tests/i440fx-test
  LINK    tests/fw_cfg-test
  LINK    tests/drive_del-test
  LINK    tests/wdt_ib700-test
  LINK    tests/tco-test
  LINK    tests/e1000-test
  LINK    tests/e1000e-test
  LINK    tests/rtl8139-test
  LINK    tests/pcnet-test
  LINK    tests/eepro100-test
  LINK    tests/ne2000-test
  LINK    tests/nvme-test
  LINK    tests/ac97-test
  LINK    tests/es1370-test
  LINK    tests/virtio-net-test
  LINK    tests/virtio-balloon-test
  LINK    tests/virtio-blk-test
  LINK    tests/virtio-rng-test
  LINK    tests/virtio-scsi-test
  LINK    tests/virtio-serial-test
  LINK    tests/virtio-console-test
  LINK    tests/tpci200-test
  LINK    tests/ipoctal232-test
  LINK    tests/display-vga-test
  LINK    tests/intel-hda-test
  LINK    tests/ivshmem-test
  LINK    tests/vmxnet3-test
  LINK    tests/pvpanic-test
  LINK    tests/i82801b11-test
  LINK    tests/ioh3420-test
  LINK    tests/usb-hcd-ohci-test
  LINK    tests/usb-hcd-uhci-test
  LINK    tests/usb-hcd-ehci-test
  LINK    tests/usb-hcd-xhci-test
  LINK    tests/pc-cpu-test
  LINK    tests/q35-test
  LINK    tests/test-netfilter
  LINK    tests/test-filter-mirror
  LINK    tests/test-filter-redirector
  LINK    tests/postcopy-test
  LINK    tests/test-x86-cpuid-compat
  LINK    tests/device-introspect-test
  LINK    tests/qom-test
  GTESTER tests/test-char
  GTESTER tests/check-qdict
  GTESTER tests/check-qfloat
  GTESTER tests/check-qint
  GTESTER tests/check-qstring
  GTESTER tests/check-qlist
  GTESTER tests/check-qnull
  GTESTER tests/check-qjson
  LINK    tests/test-qobject-output-visitor
  LINK    tests/test-clone-visitor
  LINK    tests/test-qobject-input-visitor
  LINK    tests/test-qobject-input-strict
  LINK    tests/test-qmp-commands
  LINK    tests/test-string-input-visitor
  LINK    tests/test-string-output-visitor
  LINK    tests/test-qmp-event
  LINK    tests/test-opts-visitor

GTESTER tests/test-coroutine
GTESTER tests/test-visitor-serialization
GTESTER tests/test-iov
GTESTER tests/test-aio
GTESTER tests/test-throttle
GTESTER tests/test-thread-pool
GTESTER tests/test-hbitmap
GTESTER tests/test-blockjob
GTESTER tests/test-blockjob-txn
GTESTER tests/test-x86-cpuid
GTESTER tests/test-xbzrle
GTESTER tests/test-vmstate
GTESTER tests/test-cutils
GTESTER tests/test-mul64
GTESTER tests/test-int128
GTESTER tests/rcutorture
GTESTER tests/test-rcu-list
GTESTER tests/test-qdist
Failed to load simple/primitive:b_1
Failed to load simple/primitive:i64_2
Failed to load simple/primitive:i32_1
Failed to load simple/primitive:i32_1
GTESTER tests/test-qht
LINK tests/test-qht-par
GTESTER tests/test-bitops
GTESTER tests/check-qom-interface
GTESTER tests/check-qom-proplist
GTESTER tests/test-qemu-opts
GTESTER tests/test-write-threshold
GTESTER tests/test-crypto-hash
GTESTER tests/test-crypto-cipher
GTESTER tests/test-crypto-secret
GTESTER tests/test-qga
GTESTER tests/test-timed-average
GTESTER tests/test-io-task
GTESTER tests/test-io-channel-socket
GTESTER tests/test-io-channel-file
GTESTER tests/test-io-channel-command
GTESTER tests/test-io-channel-buffer
GTESTER tests/test-base64
GTESTER tests/test-crypto-ivgen
GTESTER tests/test-crypto-afsplit
GTESTER tests/test-crypto-xts
GTESTER tests/test-crypto-block
GTESTER tests/test-logging
GTESTER tests/test-replication
GTESTER tests/test-bufferiszero
GTESTER tests/test-uuid
GTESTER tests/ptimer-test
GTESTER check-qtest-x86_64
GTESTER check-qtest-aarch64
GTESTER tests/test-qobject-output-visitor
GTESTER tests/test-clone-visitor
GTESTER tests/test-qobject-input-visitor
GTESTER tests/test-qobject-input-strict
GTESTER tests/test-qmp-commands
GTESTER tests/test-string-input-visitor
ftruncate: Permission denied
GTESTER tests/test-string-output-visitor
GTESTER tests/test-qmp-event
GTESTER tests/test-opts-visitor
GTESTER tests/test-qht-par
ftruncate: Permission denied
ftruncate: Permission denied
**
ERROR:/tmp/qemu-test/src/tests/vhost-user-test.c:668:test_migrate: assertion failed: (qdict_haskey(rsp, "return"))
GTester: last random seed: R02S2f5900cdd8e985a5338c1da425c28112
ftruncate: Permission denied
ftruncate: Permission denied
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-9l5w0aja/src'
BUILD fedora
make[1]: Entering directory `/var/tmp/patchew-tester-tmp-9l5w0aja/src'
ARCHIVE qemu.tgz
ARCHIVE dtc.tgz
COPY RUNNER
RUN test-mingw in qemu:fedora
Packages installed:
PyYAML-3.11-12.fc24.x86_64
SDL-devel-1.2.15-21.fc24.x86_64
bc-1.06.95-16.fc24.x86_64
bison-3.0.4-4.fc24.x86_64
ccache-3.3.2-1.fc24.x86_64
clang-3.8.0-2.fc24.x86_64
findutils-4.6.0-7.fc24.x86_64
flex-2.6.0-2.fc24.x86_64
gcc-6.2.1-2.fc24.x86_64
gcc-c++-6.2.1-2.fc24.x86_64
git-2.7.4-3.fc24.x86_64
glib2-devel-2.48.2-1.fc24.x86_64
libfdt-devel-1.4.2-1.fc24.x86_64
make-4.1-5.fc24.x86_64
mingw32-SDL-1.2.15-7.fc24.noarch
mingw32-bzip2-1.0.6-7.fc24.noarch
mingw32-curl-7.47.0-1.fc24.noarch
mingw32-glib2-2.48.2-1.fc24.noarch
mingw32-gmp-6.1.0-1.fc24.noarch
mingw32-gnutls-3.4.14-1.fc24.noarch
mingw32-gtk2-2.24.31-1.fc24.noarch
mingw32-gtk3-3.20.9-1.fc24.noarch
mingw32-libjpeg-turbo-1.5.0-1.fc24.noarch
mingw32-libpng-1.6.23-1.fc24.noarch
mingw32-libssh2-1.4.3-5.fc24.noarch
mingw32-libtasn1-4.5-2.fc24.noarch
mingw32-nettle-3.2-1.fc24.noarch
mingw32-pixman-0.34.0-1.fc24.noarch
mingw32-pkg-config-0.28-6.fc24.x86_64
mingw64-SDL-1.2.15-7.fc24.noarch
mingw64-bzip2-1.0.6-7.fc24.noarch
mingw64-curl-7.47.0-1.fc24.noarch
mingw64-glib2-2.48.2-1.fc24.noarch
mingw64-gmp-6.1.0-1.fc24.noarch
mingw64-gnutls-3.4.14-1.fc24.noarch
mingw64-gtk2-2.24.31-1.fc24.noarch
mingw64-gtk3-3.20.9-1.fc24.noarch
mingw64-libjpeg-turbo-1.5.0-1.fc24.noarch
mingw64-libpng-1.6.23-1.fc24.noarch
mingw64-libssh2-1.4.3-5.fc24.noarch
mingw64-libtasn1-4.5-2.fc24.noarch
mingw64-nettle-3.2-1.fc24.noarch
mingw64-pixman-0.34.0-1.fc24.noarch
mingw64-pkg-config-0.28-6.fc24.x86_64
perl-5.22.2-362.fc24.x86_64
pixman-devel-0.34.0-2.fc24.x86_64
sparse-0.5.0-7.fc24.x86_64
tar-1.28-7.fc24.x86_64
which-2.20-13.fc24.x86_64
zlib-devel-1.2.8-10.fc24.x86_64

Environment variables:
PACKAGES=ccache git tar PyYAML sparse flex bison glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel gcc gcc-c++ clang make perl which bc findutils mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1 mingw32-libjpeg-turbo mingw32-libpng mingw32-curl mingw32-libssh2 mingw32-bzip2 mingw64-pixman mingw64-glib2 mingw64-gmp mingw64-SDL mingw64-pkg-config mingw64-gtk2 mingw64-gtk3 mingw64-gnutls mingw64-nettle mingw64-libtasn1 mingw64-libjpeg-turbo mingw64-libpng mingw64-curl mingw64-libssh2 mingw64-bzip2
HOSTNAME=
TERM=xterm
MAKEFLAGS= -j16
HISTSIZE=1000
J=16
USER=root
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
MAIL=/var/spool/mail/root PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PWD=/
TARGET_LIST=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
FEATURES=mingw clang pyyaml dtc
DEBUG=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/var/tmp/qemu-build/install --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-debug --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent --with-sdlabi=1.2 --with-gtkabi=2.0

Install prefix    /var/tmp/qemu-build/install
BIOS directory    /var/tmp/qemu-build/install
binary directory  /var/tmp/qemu-build/install

library directory /var/tmp/qemu-build/install/lib module directory /var/tmp/qemu-build/install/lib libexec directory /var/tmp/qemu-build/install/libexec include directory /var/tmp/qemu-build/install/include config directory /var/tmp/qemu-build/install
local state directory queried at runtime

Windows SDK       no
Source path       /tmp/qemu-test/src
C compiler        x86_64-w64-mingw32-gcc
Host C compiler   cc
C++ compiler      x86_64-w64-mingw32-g++
Objective-C compiler clang
ARFLAGS           rv
CFLAGS            -g 
QEMU_CFLAGS       -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/pixman-1  -I$(SRC_PATH)/dtc/libfdt -Werror -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -m64 -mcx16 -mthreads -D__USE_MINGW_ANSI_STDIO=1 -DWIN32_LEAN_AND_MEAN -DWINVER=0x501 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wno-shift-negative-value -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -I/usr/x86_64-w64-mingw32/sys-root/mingw/include   -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 
LDFLAGS           -Wl,--nxcompat -Wl,--no-seh -Wl,--dynamicbase -Wl,--warn-common -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
tcg debug enabled yes
gprof enabled     no
sparse enabled    no
strip binaries    no
profiler          no
static build      no
pixman            system
SDL support       yes (1.2.15)
GTK support       yes (2.24.31)
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (3.2)
nettle kdf        yes
libtasn1          yes
curses support    no
virgl support     no
curl support      yes
mingw32 support   yes
Audio drivers     dsound
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
VNC support       yes
VNC SASL support  no
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               no
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support no
Install blobs     yes
KVM support       no
COLO support      yes
RDMA support      no
TCG interpreter   no
fdt support       yes
preadv support    no
fdatasync         no
madvise           no
posix_madvise     no

libcap-ng support no
vhost-net support no
vhost-scsi support no
vhost-vsock support no
Trace backends simple
Trace output file trace-<pid>

spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            no
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no

build guest agent yes
QGA VSS support no
QGA w32 disk info yes
QGA MSI support no
seccomp support no
coroutine backend win32
coroutine pool yes
debug stack usage no
GlusterFS support no
Archipelago support no

gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   yes
TPM passthrough   no
QOM debugging     yes
lzo support       no
snappy support    no
bzip2 support     yes

NUMA host support no
tcmalloc support no
jemalloc support no
avx2 optimization yes
replication support yes
mkdir -p dtc/libfdt
GEN x86_64-softmmu/config-devices.mak.tmp
mkdir -p dtc/tests

  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     config-host.h
  GEN     qemu-options.def
  GEN     qmp-commands.h
  GEN     qapi-types.h
  GEN     qapi-visit.h
  GEN     qapi-event.h
  GEN     qmp-introspect.h
  GEN     module_block.h
  GEN     tests/test-qapi-types.h
  GEN     tests/test-qapi-visit.h
  GEN     tests/test-qmp-commands.h
  GEN     tests/test-qapi-event.h
  GEN     tests/test-qmp-introspect.h
  GEN     x86_64-softmmu/config-devices.mak
  GEN     aarch64-softmmu/config-devices.mak
  GEN     trace/generated-tracers.h
  GEN     trace/generated-tcg-tracers.h

DEP /tmp/qemu-test/src/dtc/tests/dumptrees.c
DEP /tmp/qemu-test/src/dtc/tests/trees.S
GEN trace/generated-helpers-wrappers.h
GEN trace/generated-helpers.h
DEP /tmp/qemu-test/src/dtc/tests/testutils.c
DEP /tmp/qemu-test/src/dtc/tests/value-labels.c DEP /tmp/qemu-test/src/dtc/tests/asm_tree_dump.c DEP /tmp/qemu-test/src/dtc/tests/truncated_property.c
GEN config-all-devices.mak
DEP /tmp/qemu-test/src/dtc/tests/subnode_iterate.c DEP /tmp/qemu-test/src/dtc/tests/integer-expressions.c DEP /tmp/qemu-test/src/dtc/tests/utilfdt_test.c DEP /tmp/qemu-test/src/dtc/tests/path_offset_aliases.c DEP /tmp/qemu-test/src/dtc/tests/add_subnode_with_nops.c DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_unordered.c DEP /tmp/qemu-test/src/dtc/tests/dtb_reverse.c DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_ordered.c DEP /tmp/qemu-test/src/dtc/tests/extra-terminating-null.c DEP /tmp/qemu-test/src/dtc/tests/incbin.c
DEP /tmp/qemu-test/src/dtc/tests/boot-cpuid.c
DEP /tmp/qemu-test/src/dtc/tests/phandle_format.c DEP /tmp/qemu-test/src/dtc/tests/path-references.c DEP /tmp/qemu-test/src/dtc/tests/references.c
DEP /tmp/qemu-test/src/dtc/tests/string_escapes.c DEP /tmp/qemu-test/src/dtc/tests/propname_escapes.c DEP /tmp/qemu-test/src/dtc/tests/appendprop1.c DEP /tmp/qemu-test/src/dtc/tests/appendprop2.c DEP /tmp/qemu-test/src/dtc/tests/del_node.c
DEP /tmp/qemu-test/src/dtc/tests/del_property.c DEP /tmp/qemu-test/src/dtc/tests/setprop.c
DEP /tmp/qemu-test/src/dtc/tests/set_name.c
DEP /tmp/qemu-test/src/dtc/tests/rw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/open_pack.c
DEP /tmp/qemu-test/src/dtc/tests/nopulate.c
DEP /tmp/qemu-test/src/dtc/tests/mangle-layout.c DEP /tmp/qemu-test/src/dtc/tests/move_and_save.c DEP /tmp/qemu-test/src/dtc/tests/sw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/nop_node.c
DEP /tmp/qemu-test/src/dtc/tests/nop_property.c DEP /tmp/qemu-test/src/dtc/tests/setprop_inplace.c DEP /tmp/qemu-test/src/dtc/tests/notfound.c
DEP /tmp/qemu-test/src/dtc/tests/sized_cells.c DEP /tmp/qemu-test/src/dtc/tests/char_literal.c DEP /tmp/qemu-test/src/dtc/tests/get_alias.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_compatible.c DEP /tmp/qemu-test/src/dtc/tests/node_check_compatible.c DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_phandle.c DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_prop_value.c DEP /tmp/qemu-test/src/dtc/tests/parent_offset.c DEP /tmp/qemu-test/src/dtc/tests/supernode_atdepth_offset.c DEP /tmp/qemu-test/src/dtc/tests/get_path.c
DEP /tmp/qemu-test/src/dtc/tests/get_phandle.c DEP /tmp/qemu-test/src/dtc/tests/getprop.c
DEP /tmp/qemu-test/src/dtc/tests/get_name.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset.c DEP /tmp/qemu-test/src/dtc/tests/find_property.c DEP /tmp/qemu-test/src/dtc/tests/subnode_offset.c DEP /tmp/qemu-test/src/dtc/tests/root_node.c
DEP /tmp/qemu-test/src/dtc/tests/get_mem_rsv.c DEP /tmp/qemu-test/src/dtc/libfdt/fdt_empty_tree.c DEP /tmp/qemu-test/src/dtc/libfdt/fdt_strerror.c DEP /tmp/qemu-test/src/dtc/libfdt/fdt_rw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_sw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_wip.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_ro.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt.c
DEP /tmp/qemu-test/src/dtc/util.c
DEP /tmp/qemu-test/src/dtc/fdtput.c
DEP /tmp/qemu-test/src/dtc/fdtget.c
DEP /tmp/qemu-test/src/dtc/fdtdump.c
LEX convert-dtsv0-lexer.lex.c
DEP /tmp/qemu-test/src/dtc/srcpos.c
BISON dtc-parser.tab.c
LEX dtc-lexer.lex.c
DEP /tmp/qemu-test/src/dtc/treesource.c
DEP /tmp/qemu-test/src/dtc/livetree.c
DEP /tmp/qemu-test/src/dtc/fstree.c
DEP /tmp/qemu-test/src/dtc/flattree.c
DEP /tmp/qemu-test/src/dtc/dtc.c
DEP /tmp/qemu-test/src/dtc/data.c
DEP /tmp/qemu-test/src/dtc/checks.c
DEP convert-dtsv0-lexer.lex.c
DEP dtc-parser.tab.c
DEP dtc-lexer.lex.c
CHK version_gen.h
UPD version_gen.h
DEP /tmp/qemu-test/src/dtc/util.c
CC libfdt/fdt.o
CC libfdt/fdt_ro.o
CC libfdt/fdt_wip.o
CC libfdt/fdt_sw.o
CC libfdt/fdt_rw.o
CC libfdt/fdt_strerror.o
CC libfdt/fdt_empty_tree.o
AR libfdt/libfdt.a
x86_64-w64-mingw32-ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o

  RC      version.lo
  RC      version.o
  GEN     qga/qapi-generated/qga-qapi-types.h
  GEN     qga/qapi-generated/qga-qapi-visit.h
  GEN     qga/qapi-generated/qga-qmp-commands.h
  GEN     qga/qapi-generated/qga-qapi-types.c
  GEN     qga/qapi-generated/qga-qapi-visit.c
  GEN     qga/qapi-generated/qga-qmp-marshal.c
  GEN     qmp-introspect.c
  GEN     qapi-types.c
  GEN     qapi-visit.c
  GEN     qapi-event.c
  CC      qapi/qapi-visit-core.o
  CC      qapi/qapi-dealloc-visitor.o
  CC      qapi/qobject-input-visitor.o
  CC      qapi/qobject-output-visitor.o
  CC      qapi/qmp-registry.o
  CC      qapi/qmp-dispatch.o
  CC      qapi/string-input-visitor.o
  CC      qapi/string-output-visitor.o
  CC      qapi/opts-visitor.o
  CC      qapi/qapi-clone-visitor.o
  CC      qapi/qmp-event.o
  CC      qapi/qapi-util.o
  CC      qobject/qnull.o
  CC      qobject/qint.o
  CC      qobject/qstring.o
  CC      qobject/qdict.o
  CC      qobject/qlist.o
  CC      qobject/qfloat.o
  CC      qobject/qbool.o
  CC      qobject/qjson.o
  CC      qobject/qobject.o
  CC      qobject/json-lexer.o
  CC      qobject/json-streamer.o
  CC      qobject/json-parser.o
  GEN     trace/generated-tracers.c
  CC      trace/simple.o
  CC      trace/control.o
  CC      trace/qmp.o
  CC      util/osdep.o
  CC      util/cutils.o
  CC      util/unicode.o
  CC      util/qemu-timer-common.o
  CC      util/event_notifier-win32.o
  CC      util/bufferiszero.o
  CC      util/qemu-thread-win32.o
  CC      util/oslib-win32.o
  CC      util/envlist.o
  CC      util/path.o
  CC      util/module.o
  CC      util/bitmap.o
  CC      util/bitops.o
  CC      util/hbitmap.o
  CC      util/fifo8.o
  CC      util/acl.o
  CC      util/error.o
  CC      util/qemu-error.o
  CC      util/id.o
  CC      util/iov.o
  CC      util/qemu-config.o
  CC      util/qemu-sockets.o
  CC      util/uri.o
  CC      util/notify.o
  CC      util/qemu-option.o
  CC      util/hexdump.o
  CC      util/qemu-progress.o
  CC      util/crc32c.o
  CC      util/uuid.o
  CC      util/throttle.o
  CC      util/getauxval.o
  CC      util/readline.o
  CC      util/rcu.o
  CC      util/qemu-coroutine.o
  CC      util/qemu-coroutine-io.o
  CC      util/qemu-coroutine-lock.o
  CC      util/qemu-coroutine-sleep.o
  CC      util/coroutine-win32.o
  CC      util/timed-average.o
  CC      util/buffer.o
  CC      util/base64.o
  CC      util/log.o
  CC      util/qdist.o
  CC      util/qht.o
  CC      util/range.o
  CC      crypto/pbkdf-stub.o
  CC      stubs/arch-query-cpu-def.o
  CC      stubs/arch-query-cpu-model-expansion.o
  CC      stubs/arch-query-cpu-model-comparison.o
  CC      stubs/arch-query-cpu-model-baseline.o
  CC      stubs/bdrv-next-monitor-owned.o
  CC      stubs/blk-commit-all.o
  CC      stubs/blockdev-close-all-bdrv-states.o
  CC      stubs/clock-warp.o
  CC      stubs/cpu-get-clock.o
  CC      stubs/cpu-get-icount.o
  CC      stubs/dump.o
  CC      stubs/error-printf.o
  CC      stubs/fdset-add-fd.o
  CC      stubs/fdset-find-fd.o
  CC      stubs/fdset-get-fd.o
  CC      stubs/fdset-remove-fd.o
  CC      stubs/gdbstub.o
  CC      stubs/get-fd.o
  CC      stubs/get-next-serial.o
  CC      stubs/get-vm-name.o
  CC      stubs/iothread.o
  CC      stubs/iothread-lock.o
  CC      stubs/is-daemonized.o
  CC      stubs/machine-init-done.o
  CC      stubs/migr-blocker.o
  CC      stubs/mon-is-qmp.o
  CC      stubs/monitor-init.o
  CC      stubs/notify-event.o
  CC      stubs/qtest.o
  CC      stubs/replay.o
  CC      stubs/replay-user.o
  CC      stubs/reset.o
  CC      stubs/runstate-check.o
  CC      stubs/set-fd-handler.o
  CC      stubs/slirp.o
  CC      stubs/sysbus.o
  CC      stubs/trace-control.o
  CC      stubs/uuid.o
  CC      stubs/vm-stop.o
  CC      stubs/vmstate.o
  CC      stubs/fd-register.o
  CC      stubs/cpus.o
  CC      stubs/kvm.o
  CC      stubs/qmp_pc_dimm_device_list.o
  CC      stubs/target-monitor-defs.o
  CC      stubs/target-get-monitor-def.o
  CC      stubs/vhost.o
  CC      stubs/iohandler.o
  CC      stubs/smbios_type_38.o
  CC      stubs/ipmi.o
  CC      stubs/pc_madt_cpu_entry.o
  CC      stubs/migration-colo.o
  GEN     qemu-img-cmds.h
  CC      async.o
  CC      thread-pool.o
  CC      block.o
  CC      blockjob.o
  CC      main-loop.o
  CC      iohandler.o
  CC      qemu-timer.o
  CC      aio-win32.o
  CC      qemu-io-cmds.o
  CC      replication.o
  CC      block/raw_bsd.o
  CC      block/qcow.o
  CC      block/vdi.o
  CC      block/vmdk.o
  CC      block/cloop.o
  CC      block/bochs.o
  CC      block/vpc.o
  CC      block/vvfat.o
  CC      block/qcow2.o
  CC      block/dmg.o
  CC      block/qcow2-refcount.o
  CC      block/qcow2-cluster.o
  CC      block/qcow2-snapshot.o
  CC      block/qcow2-cache.o
  CC      block/qed.o
  CC      block/qed-gencb.o
  CC      block/qed-l2-cache.o
  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
  CC      block/vhdx.o
  CC      block/vhdx-endian.o
  CC      block/vhdx-log.o
  CC      block/quorum.o
  CC      block/parallels.o
  CC      block/blkdebug.o
  CC      block/blkreplay.o
  CC      block/blkverify.o
  CC      block/block-backend.o
  CC      block/snapshot.o
  CC      block/qapi.o
  CC      block/raw-win32.o
  CC      block/win32-aio.o
  CC      block/null.o
  CC      block/commit.o
  CC      block/mirror.o
  CC      block/io.o
  CC      block/throttle-groups.o
  CC      block/nbd.o
  CC      block/nbd-client.o
  CC      block/sheepdog.o
  CC      block/accounting.o
  CC      block/dirty-bitmap.o
  CC      block/write-threshold.o
  CC      block/backup.o
  CC      block/replication.o
  CC      block/crypto.o
  CC      nbd/server.o
  CC      nbd/client.o
  CC      nbd/common.o
  CC      block/curl.o
  CC      block/ssh.o
  CC      block/dmg-bz2.o
  CC      crypto/init.o
  CC      crypto/hash.o
  CC      crypto/hash-nettle.o
  CC      crypto/aes.o
  CC      crypto/desrfb.o
  CC      crypto/cipher.o
  CC      crypto/tlscreds.o
  CC      crypto/tlscredsanon.o
  CC      crypto/tlscredsx509.o
  CC      crypto/tlssession.o
  CC      crypto/secret.o
  CC      crypto/random-gnutls.o
  CC      crypto/pbkdf.o
  CC      crypto/pbkdf-nettle.o
  CC      crypto/ivgen.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain.o
  CC      crypto/ivgen-plain64.o
  CC      crypto/afsplit.o
  CC      crypto/xts.o
  CC      crypto/block.o
  CC      crypto/block-qcow.o
  CC      crypto/block-luks.o
  CC      io/channel.o
  CC      io/channel-buffer.o
  CC      io/channel-command.o
  CC      io/channel-file.o
  CC      io/channel-socket.o
  CC      io/channel-tls.o
  CC      io/channel-watch.o
  CC      io/channel-websock.o
  CC      io/channel-util.o
  CC      io/task.o
  CC      qom/object.o
  CC      qom/container.o
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  CC      qemu-io.o
  CC      blockdev.o
  CC      blockdev-nbd.o
  CC      iothread.o
  CC      qdev-monitor.o
  CC      device-hotplug.o
  CC      os-win32.o
  CC      qemu-char.o
  CC      page_cache.o
  CC      accel.o
  CC      bt-host.o
  CC      bt-vhci.o
  CC      dma-helpers.o
  CC      vl.o
  CC      tpm.o
  CC      device_tree.o
  GEN     qmp-marshal.c
  CC      qmp.o
  CC      hmp.o
  CC      cpus-common.o
  CC      audio/audio.o
  CC      audio/noaudio.o
  CC      audio/wavaudio.o
  CC      audio/mixeng.o
  CC      audio/sdlaudio.o
  CC      audio/dsoundaudio.o
  CC      audio/audio_win_int.o
  CC      audio/wavcapture.o
  CC      backends/rng.o
  CC      backends/rng-egd.o
  CC      backends/msmouse.o
  CC      backends/testdev.o
  CC      backends/tpm.o
  CC      backends/hostmem.o
  CC      backends/hostmem-ram.o
  CC      backends/cryptodev.o
  CC      backends/cryptodev-builtin.o
  CC      block/stream.o
  CC      disas/arm.o
  CXX     disas/arm-a64.o
  CC      disas/i386.o
  CXX     disas/libvixl/vixl/utils.o
  CXX     disas/libvixl/vixl/compiler-intrinsics.o
  CXX     disas/libvixl/vixl/a64/instructions-a64.o
  CXX     disas/libvixl/vixl/a64/disasm-a64.o
  CXX     disas/libvixl/vixl/a64/decoder-a64.o
  CC      hw/acpi/core.o
  CC      hw/acpi/piix4.o
  CC      hw/acpi/pcihp.o
  CC      hw/acpi/ich9.o
  CC      hw/acpi/tco.o
  CC      hw/acpi/cpu_hotplug.o
  CC      hw/acpi/memory_hotplug.o
  CC      hw/acpi/memory_hotplug_acpi_table.o
  CC      hw/acpi/cpu.o
  CC      hw/acpi/nvdimm.o
  CC      hw/acpi/acpi_interface.o
  CC      hw/acpi/bios-linker-loader.o
  CC      hw/acpi/aml-build.o
  CC      hw/acpi/ipmi.o
  CC      hw/audio/sb16.o
  CC      hw/audio/es1370.o
  CC      hw/audio/ac97.o
  CC      hw/audio/fmopl.o
  CC      hw/audio/adlib.o
  CC      hw/audio/gus.o
  CC      hw/audio/gusemu_hal.o
  CC      hw/audio/gusemu_mixer.o
  CC      hw/audio/cs4231a.o
  CC      hw/audio/intel-hda.o
  CC      hw/audio/hda-codec.o
  CC      hw/audio/pcspk.o
  CC      hw/audio/wm8750.o
  CC      hw/audio/pl041.o
  CC      hw/audio/lm4549.o
  CC      hw/audio/marvell_88w8618.o
  CC      hw/block/block.o
  CC      hw/block/cdrom.o
  CC      hw/block/hd-geometry.o
  CC      hw/block/fdc.o
  CC      hw/block/m25p80.o
  CC      hw/block/nand.o
  CC      hw/block/pflash_cfi01.o
  CC      hw/block/pflash_cfi02.o
  CC      hw/block/ecc.o
  CC      hw/block/onenand.o
  CC      hw/block/nvme.o
  CC      hw/bt/core.o
  CC      hw/bt/l2cap.o
  CC      hw/bt/sdp.o
  CC      hw/bt/hci.o
  CC      hw/bt/hid.o
  CC      hw/bt/hci-csr.o
  CC      hw/char/ipoctal232.o
  CC      hw/char/parallel.o
  CC      hw/char/pl011.o
  CC      hw/char/serial.o
  CC      hw/char/serial-isa.o
  CC      hw/char/serial-pci.o
  CC      hw/char/virtio-console.o
  CC      hw/char/cadence_uart.o
  CC      hw/char/debugcon.o
  CC      hw/char/imx_serial.o
  CC      hw/core/qdev.o
  CC      hw/core/qdev-properties.o
  CC      hw/core/bus.o
  CC      hw/core/fw-path-provider.o
  CC      hw/core/irq.o
  CC      hw/core/hotplug.o
  CC      hw/core/ptimer.o
  CC      hw/core/sysbus.o
  CC      hw/core/machine.o
  CC      hw/core/null-machine.o
  CC      hw/core/loader.o
  CC      hw/core/qdev-properties-system.o
  CC      hw/core/register.o
  CC      hw/core/or-irq.o
  CC      hw/core/platform-bus.o
  CC      hw/display/ads7846.o
  CC      hw/display/cirrus_vga.o
  CC      hw/display/pl110.o
  CC      hw/display/ssd0303.o
  CC      hw/display/ssd0323.o
  CC      hw/display/vga-pci.o
  CC      hw/display/vga-isa.o
  CC      hw/display/vmware_vga.o
  CC      hw/display/blizzard.o
  CC      hw/display/exynos4210_fimd.o
  CC      hw/display/framebuffer.o
  CC      hw/display/tc6393xb.o
  CC      hw/dma/pl080.o
  CC      hw/dma/pl330.o
  CC      hw/dma/i8257.o
  CC      hw/dma/xlnx-zynq-devcfg.o
  CC      hw/gpio/max7310.o
  CC      hw/gpio/pl061.o
  CC      hw/gpio/zaurus.o
  CC      hw/gpio/gpio_key.o
  CC      hw/i2c/core.o
  CC      hw/i2c/smbus.o
  CC      hw/i2c/smbus_eeprom.o
  CC      hw/i2c/i2c-ddc.o
  CC      hw/i2c/versatile_i2c.o
  CC      hw/i2c/smbus_ich9.o
  CC      hw/i2c/pm_smbus.o
  CC      hw/i2c/bitbang_i2c.o
  CC      hw/i2c/exynos4210_i2c.o
  CC      hw/i2c/imx_i2c.o
  CC      hw/i2c/aspeed_i2c.o
  CC      hw/ide/core.o
  CC      hw/ide/atapi.o
  CC      hw/ide/qdev.o
  CC      hw/ide/pci.o
  CC      hw/ide/isa.o
  CC      hw/ide/piix.o
  CC      hw/ide/microdrive.o
  CC      hw/ide/ahci.o
  CC      hw/ide/ich.o
  CC      hw/input/hid.o
  CC      hw/input/lm832x.o
  CC      hw/input/pckbd.o
  CC      hw/input/pl050.o
  CC      hw/input/ps2.o
  CC      hw/input/stellaris_input.o
  CC      hw/input/tsc2005.o
  CC      hw/input/vmmouse.o
  CC      hw/input/virtio-input.o
  CC      hw/input/virtio-input-hid.o
  CC      hw/intc/i8259_common.o
  CC      hw/intc/i8259.o
  CC      hw/intc/pl190.o
  CC      hw/intc/imx_avic.o
  CC      hw/intc/realview_gic.o
  CC      hw/intc/ioapic_common.o
  CC      hw/intc/arm_gic_common.o
  CC      hw/intc/arm_gic.o
  CC      hw/intc/arm_gicv2m.o
  CC      hw/intc/arm_gicv3_common.o
  CC      hw/intc/arm_gicv3.o
  CC      hw/intc/arm_gicv3_dist.o
  CC      hw/intc/arm_gicv3_redist.o
  CC      hw/intc/arm_gicv3_its_common.o
  CC      hw/intc/intc.o
  CC      hw/ipack/ipack.o
  CC      hw/ipack/tpci200.o
  CC      hw/ipmi/ipmi.o
  CC      hw/ipmi/ipmi_bmc_extern.o
  CC      hw/ipmi/ipmi_bmc_sim.o
  CC      hw/ipmi/isa_ipmi_kcs.o
  CC      hw/ipmi/isa_ipmi_bt.o
  CC      hw/isa/isa-bus.o
  CC      hw/isa/apm.o
  CC      hw/mem/pc-dimm.o
  CC      hw/mem/nvdimm.o
  CC      hw/misc/applesmc.o
  CC      hw/misc/max111x.o
  CC      hw/misc/tmp105.o
  CC      hw/misc/debugexit.o
  CC      hw/misc/sga.o
  CC      hw/misc/pc-testdev.o
  CC      hw/misc/pci-testdev.o
  CC      hw/misc/arm_l2x0.o
  CC      hw/misc/arm_integrator_debug.o
  CC      hw/misc/a9scu.o
  CC      hw/misc/arm11scu.o
  CC      hw/net/ne2000.o
  CC      hw/net/eepro100.o
  CC      hw/net/pcnet-pci.o
  CC      hw/net/pcnet.o
  CC      hw/net/e1000.o
  CC      hw/net/e1000x_common.o
  CC      hw/net/net_tx_pkt.o
  CC      hw/net/net_rx_pkt.o
  CC      hw/net/e1000e.o
  CC      hw/net/rtl8139.o
  CC      hw/net/e1000e_core.o
  CC      hw/net/vmxnet3.o
  CC      hw/net/smc91c111.o
  CC      hw/net/lan9118.o
  CC      hw/net/ne2000-isa.o
  CC      hw/net/xgmac.o
  CC      hw/net/allwinner_emac.o
  CC      hw/net/imx_fec.o
  CC      hw/net/cadence_gem.o
  CC      hw/net/stellaris_enet.o
  CC      hw/net/rocker/rocker_fp.o
  CC      hw/net/rocker/rocker.o
  CC      hw/net/rocker/rocker_desc.o
  CC      hw/net/rocker/rocker_of_dpa.o
  CC      hw/net/rocker/rocker_world.o
  CC      hw/nvram/eeprom93xx.o
  CC      hw/nvram/fw_cfg.o
  CC      hw/nvram/chrp_nvram.o
  CC      hw/pci-bridge/pci_bridge_dev.o
  CC      hw/pci-bridge/pci_expander_bridge.o
  CC      hw/pci-bridge/xio3130_upstream.o
  CC      hw/pci-bridge/xio3130_downstream.o
  CC      hw/pci-bridge/ioh3420.o
  CC      hw/pci-bridge/i82801b11.o
  CC      hw/pci-host/versatile.o
  CC      hw/pci-host/pam.o
  CC      hw/pci-host/piix.o
  CC      hw/pci-host/q35.o
  CC      hw/pci-host/gpex.o
  CC      hw/pci/pci.o
  CC      hw/pci/pci_bridge.o
  CC      hw/pci/msix.o
  CC      hw/pci/msi.o
  CC      hw/pci/shpc.o
  CC      hw/pci/slotid_cap.o
  CC      hw/pci/pci_host.o
  CC      hw/pci/pcie_host.o
  CC      hw/pci/pcie.o
  CC      hw/pci/pcie_aer.o
  CC      hw/pci/pcie_port.o
  CC      hw/pci/pci-stub.o
  CC      hw/pcmcia/pcmcia.o
  CC      hw/scsi/scsi-disk.o
  CC      hw/scsi/scsi-generic.o
  CC      hw/scsi/scsi-bus.o
  CC      hw/scsi/lsi53c895a.o
  CC      hw/scsi/mptsas.o
  CC      hw/scsi/mptconfig.o
  CC      hw/scsi/mptendian.o
  CC      hw/scsi/megasas.o
  CC      hw/scsi/vmw_pvscsi.o
  CC      hw/scsi/esp.o
  CC      hw/scsi/esp-pci.o
  CC      hw/sd/pl181.o
  CC      hw/sd/ssi-sd.o
  CC      hw/sd/sd.o
  CC      hw/sd/core.o
  CC      hw/sd/sdhci.o
  CC      hw/smbios/smbios.o
  CC      hw/smbios/smbios_type_38.o
  CC      hw/ssi/pl022.o
  CC      hw/ssi/ssi.o
  CC      hw/ssi/xilinx_spips.o
  CC      hw/ssi/aspeed_smc.o
  CC      hw/ssi/stm32f2xx_spi.o
  CC      hw/timer/arm_timer.o
  CC      hw/timer/arm_mptimer.o
  CC      hw/timer/a9gtimer.o
  CC      hw/timer/cadence_ttc.o
  CC      hw/timer/ds1338.o
  CC      hw/timer/hpet.o
  CC      hw/timer/i8254_common.o
  CC      hw/timer/i8254.o
  CC      hw/timer/pl031.o
  CC      hw/timer/twl92230.o
  CC      hw/timer/imx_epit.o
  CC      hw/timer/imx_gpt.o
  CC      hw/timer/stm32f2xx_timer.o
  CC      hw/timer/aspeed_timer.o
  CC      hw/tpm/tpm_tis.o
  CC      hw/usb/core.o
  CC      hw/usb/combined-packet.o
  CC      hw/usb/bus.o
  CC      hw/usb/libhw.o
  CC      hw/usb/desc.o
  CC      hw/usb/desc-msos.o
  CC      hw/usb/hcd-uhci.o
  CC      hw/usb/hcd-ohci.o
  CC      hw/usb/hcd-ehci.o
  CC      hw/usb/hcd-ehci-pci.o
  CC      hw/usb/hcd-ehci-sysbus.o
  CC      hw/usb/hcd-xhci.o
  CC      hw/usb/hcd-musb.o
  CC      hw/usb/dev-hub.o
  CC      hw/usb/dev-hid.o
  CC      hw/usb/dev-wacom.o
  CC      hw/usb/dev-storage.o
  CC      hw/usb/dev-uas.o
  CC      hw/usb/dev-audio.o
  CC      hw/usb/dev-serial.o
  CC      hw/usb/dev-network.o
  CC      hw/usb/dev-bluetooth.o
  CC      hw/usb/dev-smartcard-reader.o
  CC      hw/usb/host-stub.o
  CC      hw/virtio/virtio-rng.o
  CC      hw/virtio/virtio-bus.o
  CC      hw/virtio/virtio-pci.o
  CC      hw/virtio/virtio-mmio.o
  CC      hw/watchdog/watchdog.o
  CC      hw/watchdog/wdt_i6300esb.o
  CC      hw/watchdog/wdt_ib700.o
  CC      migration/migration.o
  CC      migration/socket.o
  CC      migration/fd.o
  CC      migration/exec.o
  CC      migration/tls.o
  CC      migration/colo-comm.o
  CC      migration/colo.o
  CC      migration/colo-failover.o
  CC      migration/vmstate.o
  CC      migration/qemu-file.o
  CC      migration/qemu-file-channel.o
  CC      migration/xbzrle.o
  CC      migration/postcopy-ram.o
  CC      migration/qjson.o
  CC      migration/block.o
  CC      net/net.o
  CC      net/queue.o
  CC      net/checksum.o
  CC      net/util.o
  CC      net/hub.o
  CC      net/socket.o
  CC      net/dump.o
  CC      net/eth.o
  CC      net/tap-win32.o
  CC      net/slirp.o
  CC      net/filter.o
  CC      net/filter-buffer.o
  CC      net/filter-mirror.o
  CC      net/colo-compare.o
  CC      net/colo.o
  CC      net/filter-rewriter.o
  CC      qom/cpu.o
  CC      replay/replay.o
  CC      replay/replay-internal.o
  CC      replay/replay-events.o
  CC      replay/replay-time.o
  CC      replay/replay-input.o
  CC      replay/replay-char.o
  CC      replay/replay-snapshot.o
  CC      slirp/cksum.o
  CC      slirp/if.o
  CC      slirp/ip_icmp.o
  CC      slirp/ip6_icmp.o
  CC      slirp/ip6_input.o
  CC      slirp/ip6_output.o
  CC      slirp/ip_input.o
  CC      slirp/ip_output.o
  CC      slirp/dnssearch.o
  CC      slirp/dhcpv6.o
  CC      slirp/slirp.o
  CC      slirp/mbuf.o
  CC      slirp/misc.o
  CC      slirp/sbuf.o
  CC      slirp/socket.o
  CC      slirp/tcp_input.o
  CC      slirp/tcp_output.o
  CC      slirp/tcp_subr.o
  CC      slirp/tcp_timer.o
  CC      slirp/udp.o
  CC      slirp/udp6.o
  CC      slirp/bootp.o
  CC      slirp/tftp.o
  CC      slirp/arp_table.o
  CC      slirp/ndp_table.o
  CC      ui/keymaps.o
  CC      ui/console.o
  CC      ui/cursor.o
  CC      ui/qemu-pixman.o
  CC      ui/input.o
  CC      ui/input-keymap.o
  CC      ui/input-legacy.o
  CC      ui/sdl.o
  CC      ui/sdl_zoom.o
  CC      ui/x_keymap.o
  CC      ui/vnc.o
  CC      ui/vnc-enc-zlib.o
  CC      ui/vnc-enc-hextile.o
  CC      ui/vnc-enc-tight.o
  CC      ui/vnc-palette.o
  CC      ui/vnc-enc-zrle.o
  CC      ui/vnc-auth-vencrypt.o
  CC      ui/vnc-jobs.o
  CC      ui/vnc-ws.o
  CC      ui/gtk.o
  CC      qga/commands.o
  CC      qga/guest-agent-command-state.o
  CC      qga/main.o
  CC      qga/commands-win32.o
  CC      qga/channel-win32.o
  CC      qga/service-win32.o
  CC      qga/vss-win32.o
  CC      qga/qapi-generated/qga-qapi-types.o
  CC      qga/qapi-generated/qga-qapi-visit.o
  CC      qga/qapi-generated/qga-qmp-marshal.o
  CC      qmp-introspect.o
  CC      qapi-types.o
  CC      qapi-visit.o
  CC      qapi-event.o
  AR      libqemustub.a
  CC      qemu-img.o
  CC      qmp-marshal.o
  CC      trace/generated-tracers.o
  AS      optionrom/multiboot.o
  AS      optionrom/linuxboot.o
  CC      optionrom/linuxboot_dma.o
  AS      optionrom/kvmvapic.o
  BUILD   optionrom/linuxboot_dma.img
  AR      libqemuutil.a
  BUILD   optionrom/linuxboot_dma.raw
  SIGN    optionrom/linuxboot_dma.bin
  BUILD   optionrom/multiboot.img
  BUILD   optionrom/linuxboot.img
  BUILD   optionrom/kvmvapic.img
  BUILD   optionrom/multiboot.raw
  BUILD   optionrom/linuxboot.raw
  BUILD   optionrom/kvmvapic.raw
  SIGN    optionrom/multiboot.bin
  SIGN    optionrom/linuxboot.bin
  SIGN    optionrom/kvmvapic.bin
  LINK    qemu-ga.exe
  LINK    qemu-img.exe
  LINK    qemu-io.exe

block/qcow2-cache.o: In function `qcow2_cache_create':
/tmp/qemu-test/src/block/qcow2-cache.c:144: undefined reference to qemu_ram_mmap' /tmp/qemu-test/src/block/qcow2-cache.c:152: undefined reference to qemu_ram_munmap' block/qcow2-cache.o: In function `qcow2_cache_destroy':
/tmp/qemu-test/src/block/qcow2-cache.c:178: undefined reference to `qemu_ram_munmap'
collect2: error: ld returned 1 exit status /tmp/qemu-test/src/rules.mak:105: recipe for target 'qemu-img.exe' failed
make: *** [qemu-img.exe] Error 1
make: *** Waiting for unfinished jobs....
block/qcow2-cache.o: In function `qcow2_cache_create':
/tmp/qemu-test/src/block/qcow2-cache.c:144: undefined reference to qemu_ram_mmap' /tmp/qemu-test/src/block/qcow2-cache.c:152: undefined reference to qemu_ram_munmap' block/qcow2-cache.o: In function `qcow2_cache_destroy':
/tmp/qemu-test/src/block/qcow2-cache.c:178: undefined reference to `qemu_ram_munmap'
collect2: error: ld returned 1 exit status /tmp/qemu-test/src/rules.mak:105: recipe for target 'qemu-io.exe' failed
make: *** [qemu-io.exe] Error 1

  GEN     x86_64-softmmu/hmp-commands.h
  GEN     x86_64-softmmu/hmp-commands-info.h
  GEN     x86_64-softmmu/config-target.h
  CC      x86_64-softmmu/exec.o
  CC      x86_64-softmmu/translate-all.o
  CC      x86_64-softmmu/cpu-exec.o
  CC      x86_64-softmmu/translate-common.o
  CC      x86_64-softmmu/cpu-exec-common.o
  CC      x86_64-softmmu/tcg/tcg.o
  CC      x86_64-softmmu/tcg/tcg-op.o
  CC      x86_64-softmmu/tcg/optimize.o
  CC      x86_64-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/fpu/softfloat.o
  CC      x86_64-softmmu/disas.o
  CC      x86_64-softmmu/tcg-runtime.o
  CC      x86_64-softmmu/kvm-stub.o
  CC      x86_64-softmmu/arch_init.o
  CC      x86_64-softmmu/cpus.o
  CC      x86_64-softmmu/monitor.o
  CC      x86_64-softmmu/gdbstub.o
  CC      x86_64-softmmu/balloon.o
  CC      x86_64-softmmu/ioport.o
  CC      x86_64-softmmu/numa.o
  CC      x86_64-softmmu/qtest.o
  CC      x86_64-softmmu/bootdevice.o
  CC      x86_64-softmmu/memory.o
  GEN     aarch64-softmmu/hmp-commands.h
  GEN     aarch64-softmmu/hmp-commands-info.h
  GEN     aarch64-softmmu/config-target.h
  CC      x86_64-softmmu/cputlb.o
  CC      x86_64-softmmu/memory_mapping.o
  CC      x86_64-softmmu/dump.o
  CC      x86_64-softmmu/migration/savevm.o
  CC      x86_64-softmmu/migration/ram.o
  CC      x86_64-softmmu/xen-common-stub.o
  CC      x86_64-softmmu/hw/block/virtio-blk.o
  CC      x86_64-softmmu/xen-hvm-stub.o
  CC      x86_64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      x86_64-softmmu/hw/char/virtio-serial-bus.o
  CC      x86_64-softmmu/hw/core/nmi.o
  CC      x86_64-softmmu/hw/core/generic-loader.o
  CC      x86_64-softmmu/hw/cpu/core.o
  CC      x86_64-softmmu/hw/display/virtio-gpu.o
  CC      x86_64-softmmu/hw/display/vga.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-3d.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-pci.o
  CC      x86_64-softmmu/hw/display/virtio-vga.o
  CC      x86_64-softmmu/hw/intc/apic.o
  CC      x86_64-softmmu/hw/intc/apic_common.o
  CC      x86_64-softmmu/hw/intc/ioapic.o
  CC      x86_64-softmmu/hw/isa/lpc_ich9.o
  CC      x86_64-softmmu/hw/misc/vmport.o
  CC      x86_64-softmmu/hw/misc/pvpanic.o
  CC      x86_64-softmmu/hw/misc/edu.o
  CC      x86_64-softmmu/hw/net/virtio-net.o
  CC      x86_64-softmmu/hw/net/vhost_net.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      x86_64-softmmu/hw/timer/mc146818rtc.o
  CC      x86_64-softmmu/hw/virtio/virtio.o
  CC      x86_64-softmmu/hw/virtio/virtio-balloon.o
  CC      aarch64-softmmu/exec.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto.o
  CC      aarch64-softmmu/translate-all.o
  CC      aarch64-softmmu/cpu-exec.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      aarch64-softmmu/translate-common.o
  CC      x86_64-softmmu/hw/i386/multiboot.o
  CC      x86_64-softmmu/hw/i386/pc.o
  CC      aarch64-softmmu/cpu-exec-common.o
  CC      aarch64-softmmu/tcg/tcg.o
  CC      x86_64-softmmu/hw/i386/pc_piix.o
  CC      aarch64-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/tcg/optimize.o
  CC      x86_64-softmmu/hw/i386/pc_q35.o
  CC      aarch64-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/disas.o
  CC      aarch64-softmmu/tcg-runtime.o
  CC      x86_64-softmmu/hw/i386/pc_sysfw.o
  GEN     aarch64-softmmu/gdbstub-xml.c
  CC      x86_64-softmmu/hw/i386/x86-iommu.o
  CC      x86_64-softmmu/hw/i386/intel_iommu.o
  CC      aarch64-softmmu/kvm-stub.o
  CC      aarch64-softmmu/arch_init.o
  CC      x86_64-softmmu/hw/i386/amd_iommu.o
  CC      aarch64-softmmu/cpus.o
  CC      x86_64-softmmu/hw/i386/kvmvapic.o
  CC      x86_64-softmmu/hw/i386/acpi-build.o
  CC      aarch64-softmmu/monitor.o
  CC      x86_64-softmmu/hw/i386/pci-assign-load-rom.o
  CC      x86_64-softmmu/target-i386/translate.o
  CC      x86_64-softmmu/target-i386/helper.o
  CC      aarch64-softmmu/gdbstub.o
  CC      aarch64-softmmu/balloon.o
  CC      aarch64-softmmu/ioport.o
  CC      x86_64-softmmu/target-i386/cpu.o
  CC      x86_64-softmmu/target-i386/bpt_helper.o
  CC      aarch64-softmmu/numa.o
  CC      aarch64-softmmu/qtest.o
  CC      x86_64-softmmu/target-i386/excp_helper.o
  CC      aarch64-softmmu/bootdevice.o
  CC      x86_64-softmmu/target-i386/fpu_helper.o
  CC      aarch64-softmmu/memory.o
  CC      aarch64-softmmu/cputlb.o
  CC      x86_64-softmmu/target-i386/cc_helper.o
  CC      aarch64-softmmu/memory_mapping.o
  CC      x86_64-softmmu/target-i386/int_helper.o
  CC      aarch64-softmmu/dump.o
  CC      x86_64-softmmu/target-i386/svm_helper.o
  CC      x86_64-softmmu/target-i386/smm_helper.o
  CC      x86_64-softmmu/target-i386/misc_helper.o
  CC      aarch64-softmmu/migration/ram.o
  CC      x86_64-softmmu/target-i386/mem_helper.o
  CC      aarch64-softmmu/migration/savevm.o
  CC      x86_64-softmmu/target-i386/seg_helper.o
  CC      aarch64-softmmu/xen-common-stub.o
  CC      aarch64-softmmu/xen-hvm-stub.o
  CC      aarch64-softmmu/hw/adc/stm32f2xx_adc.o
  CC      x86_64-softmmu/target-i386/mpx_helper.o
  CC      aarch64-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      x86_64-softmmu/target-i386/gdbstub.o
  CC      aarch64-softmmu/hw/char/exynos4210_uart.o
  CC      x86_64-softmmu/target-i386/machine.o
  CC      aarch64-softmmu/hw/char/omap_uart.o
  CC      x86_64-softmmu/target-i386/arch_memory_mapping.o
  CC      x86_64-softmmu/target-i386/arch_dump.o
  CC      aarch64-softmmu/hw/char/digic-uart.o
  CC      aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC      x86_64-softmmu/target-i386/monitor.o
  CC      aarch64-softmmu/hw/char/bcm2835_aux.o
  CC      aarch64-softmmu/hw/char/virtio-serial-bus.o
  CC      x86_64-softmmu/target-i386/kvm-stub.o
  CC      aarch64-softmmu/hw/core/nmi.o
  CC      aarch64-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC      aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC      aarch64-softmmu/hw/cpu/a9mpcore.o
  CC      aarch64-softmmu/hw/cpu/a15mpcore.o
  CC      aarch64-softmmu/hw/cpu/core.o
  CC      aarch64-softmmu/hw/display/omap_dss.o
  CC      aarch64-softmmu/hw/display/omap_lcdc.o
  CC      aarch64-softmmu/hw/display/pxa2xx_lcd.o
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/hw/display/bcm2835_fb.o
  CC      aarch64-softmmu/hw/display/vga.o
  CC      x86_64-softmmu/trace/control-target.o
  CC      aarch64-softmmu/hw/display/virtio-gpu.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-softmmu/hw/display/dpcd.o
  CC      aarch64-softmmu/hw/display/xlnx_dp.o
  CC      aarch64-softmmu/hw/dma/xlnx_dpdma.o
  CC      aarch64-softmmu/hw/dma/omap_dma.o
  CC      aarch64-softmmu/hw/dma/soc_dma.o
  CC      aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC      aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC      aarch64-softmmu/hw/gpio/omap_gpio.o
  CC      aarch64-softmmu/hw/gpio/imx_gpio.o
  CC      aarch64-softmmu/hw/i2c/omap_i2c.o
  CC      aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC      aarch64-softmmu/hw/input/tsc210x.o
  CC      aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC      aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC      aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC      aarch64-softmmu/hw/intc/omap_intc.o
  CC      aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC      aarch64-softmmu/hw/intc/bcm2836_control.o
  CC      aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC      aarch64-softmmu/hw/intc/aspeed_vic.o
  CC      aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      aarch64-softmmu/hw/misc/arm_sysctl.o
  CC      aarch64-softmmu/hw/misc/cbus.o
  CC      aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC      aarch64-softmmu/hw/misc/imx_ccm.o
  CC      aarch64-softmmu/hw/misc/imx31_ccm.o
  CC      aarch64-softmmu/hw/misc/imx25_ccm.o
  CC      aarch64-softmmu/hw/misc/imx6_ccm.o
  CC      aarch64-softmmu/hw/misc/imx6_src.o
  CC      aarch64-softmmu/hw/misc/mst_fpga.o
  CC      aarch64-softmmu/hw/misc/omap_clk.o
  CC      aarch64-softmmu/hw/misc/omap_gpmc.o
  CC      aarch64-softmmu/hw/misc/omap_sdrc.o
  CC      aarch64-softmmu/hw/misc/omap_l4.o
  CC      aarch64-softmmu/hw/misc/omap_tap.o
  CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC      aarch64-softmmu/hw/misc/bcm2835_property.o
  CC      aarch64-softmmu/hw/misc/zynq_slcr.o
  CC      aarch64-softmmu/hw/misc/zynq-xadc.o
  CC      aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      aarch64-softmmu/hw/misc/edu.o
  CC      aarch64-softmmu/hw/misc/auxbus.o
  CC      aarch64-softmmu/hw/misc/aspeed_scu.o
  CC      aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC      x86_64-softmmu/trace/generated-helpers.o
  CC      aarch64-softmmu/hw/net/virtio-net.o
  CC      aarch64-softmmu/hw/net/vhost_net.o
  CC      aarch64-softmmu/hw/pcmcia/pxa2xx.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/sd/omap_mmc.o
  CC      aarch64-softmmu/hw/sd/pxa2xx_mmci.o
  CC      aarch64-softmmu/hw/ssi/omap_spi.o
  CC      aarch64-softmmu/hw/ssi/imx_spi.o
  CC      aarch64-softmmu/hw/timer/exynos4210_mct.o
  CC      aarch64-softmmu/hw/timer/exynos4210_pwm.o
  CC      aarch64-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/timer/omap_gptimer.o
  CC      aarch64-softmmu/hw/timer/omap_synctimer.o
  CC      aarch64-softmmu/hw/timer/pxa2xx_timer.o
  CC      aarch64-softmmu/hw/timer/digic-timer.o
  CC      aarch64-softmmu/hw/timer/allwinner-a10-pit.o
  CC      aarch64-softmmu/hw/usb/tusb6010.o
  CC      aarch64-softmmu/hw/virtio/virtio.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto.o
  CC      aarch64-softmmu/hw/virtio/virtio-balloon.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      aarch64-softmmu/hw/arm/boot.o
  CC      aarch64-softmmu/hw/arm/collie.o
  CC      aarch64-softmmu/hw/arm/exynos4_boards.o
  CC      aarch64-softmmu/hw/arm/gumstix.o
  CC      aarch64-softmmu/hw/arm/highbank.o
  CC      aarch64-softmmu/hw/arm/digic_boards.o
  CC      aarch64-softmmu/hw/arm/integratorcp.o
  CC      aarch64-softmmu/hw/arm/mainstone.o
  CC      aarch64-softmmu/hw/arm/musicpal.o
  CC      aarch64-softmmu/hw/arm/nseries.o
  CC      aarch64-softmmu/hw/arm/omap_sx1.o
  CC      aarch64-softmmu/hw/arm/palm.o
  CC      aarch64-softmmu/hw/arm/realview.o
  CC      aarch64-softmmu/hw/arm/spitz.o
  CC      aarch64-softmmu/hw/arm/stellaris.o
  CC      aarch64-softmmu/hw/arm/tosa.o
  CC      aarch64-softmmu/hw/arm/versatilepb.o
  CC      aarch64-softmmu/hw/arm/vexpress.o
  CC      aarch64-softmmu/hw/arm/virt.o
  CC      aarch64-softmmu/hw/arm/xilinx_zynq.o
  CC      aarch64-softmmu/hw/arm/z2.o
  CC      aarch64-softmmu/hw/arm/virt-acpi-build.o
  CC      aarch64-softmmu/hw/arm/netduino2.o
  LINK    x86_64-softmmu/qemu-system-x86_64w.exe
  CC      aarch64-softmmu/hw/arm/sysbus-fdt.o
  CC      aarch64-softmmu/hw/arm/armv7m.o
  CC      aarch64-softmmu/hw/arm/exynos4210.o
  CC      aarch64-softmmu/hw/arm/pxa2xx.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_gpio.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_pic.o
  CC      aarch64-softmmu/hw/arm/digic.o
  CC      aarch64-softmmu/hw/arm/omap1.o
  CC      aarch64-softmmu/hw/arm/omap2.o
  CC      aarch64-softmmu/hw/arm/strongarm.o
  CC      aarch64-softmmu/hw/arm/allwinner-a10.o
  CC      aarch64-softmmu/hw/arm/cubieboard.o
  CC      aarch64-softmmu/hw/arm/bcm2835_peripherals.o
  CC      aarch64-softmmu/hw/arm/bcm2836.o
  CC      aarch64-softmmu/hw/arm/raspi.o
  CC      aarch64-softmmu/hw/arm/stm32f205_soc.o
  CC      aarch64-softmmu/hw/arm/xlnx-zynqmp.o
  CC      aarch64-softmmu/hw/arm/xlnx-ep108.o
  CC      aarch64-softmmu/hw/arm/fsl-imx25.o
  CC      aarch64-softmmu/hw/arm/imx25_pdk.o
  CC      aarch64-softmmu/hw/arm/fsl-imx31.o
  CC      aarch64-softmmu/hw/arm/kzm.o
  CC      aarch64-softmmu/hw/arm/fsl-imx6.o
  CC      aarch64-softmmu/hw/arm/aspeed_soc.o
  CC      aarch64-softmmu/hw/arm/sabrelite.o
  CC      aarch64-softmmu/hw/arm/aspeed.o
  CC      aarch64-softmmu/target-arm/arm-semi.o
  CC      aarch64-softmmu/target-arm/machine.o
  CC      aarch64-softmmu/target-arm/psci.o
  CC      aarch64-softmmu/target-arm/arch_dump.o
  CC      aarch64-softmmu/target-arm/monitor.o
  CC      aarch64-softmmu/target-arm/kvm-stub.o
  CC      aarch64-softmmu/target-arm/translate.o
  CC      aarch64-softmmu/target-arm/op_helper.o
  CC      aarch64-softmmu/target-arm/helper.o
  CC      aarch64-softmmu/target-arm/cpu.o
  CC      aarch64-softmmu/target-arm/neon_helper.o
  CC      aarch64-softmmu/target-arm/iwmmxt_helper.o
  CC      aarch64-softmmu/target-arm/gdbstub.o
  CC      aarch64-softmmu/target-arm/cpu64.o
  CC      aarch64-softmmu/target-arm/translate-a64.o
  CC      aarch64-softmmu/target-arm/helper-a64.o
  CC      aarch64-softmmu/target-arm/gdbstub64.o
  CC      aarch64-softmmu/target-arm/crypto_helper.o
  CC      aarch64-softmmu/target-arm/arm-powerctl.o
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/trace/control-target.o
  CC      aarch64-softmmu/gdbstub-xml.o
  CC      aarch64-softmmu/trace/generated-helpers.o
  LINK    aarch64-softmmu/qemu-system-aarch64w.exe

../block/qcow2-cache.o: In function `qcow2_cache_create':
/tmp/qemu-test/src/block/qcow2-cache.c:144: undefined reference to qemu_ram_mmap' /tmp/qemu-test/src/block/qcow2-cache.c:152: undefined reference to qemu_ram_munmap' ../block/qcow2-cache.o: In function `qcow2_cache_destroy':
/tmp/qemu-test/src/block/qcow2-cache.c:178: undefined reference to `qemu_ram_munmap'
collect2: error: ld returned 1 exit status
Makefile:198: recipe for target 'qemu-system-x86_64w.exe' failed make[1]: *** [qemu-system-x86_64w.exe] Error 1
Makefile:202: recipe for target 'subdir-x86_64-softmmu' failed
make: *** [subdir-x86_64-softmmu] Error 2
../block/qcow2-cache.o: In function `qcow2_cache_create':
/tmp/qemu-test/src/block/qcow2-cache.c:144: undefined reference to qemu_ram_mmap' /tmp/qemu-test/src/block/qcow2-cache.c:152: undefined reference to qemu_ram_munmap' ../block/qcow2-cache.o: In function `qcow2_cache_destroy':
/tmp/qemu-test/src/block/qcow2-cache.c:178: undefined reference to `qemu_ram_munmap'
collect2: error: ld returned 1 exit status
Makefile:198: recipe for target 'qemu-system-aarch64w.exe' failed make[1]: *** [qemu-system-aarch64w.exe] Error 1
Makefile:202: recipe for target 'subdir-aarch64-softmmu' failed
make: *** [subdir-aarch64-softmmu] Error 2
make[1]: *** [docker-run] Error 2
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-9l5w0aja/src'
make: *** [docker-run-test-mingw@fedora] Error 2
=== OUTPUT END ===

Test command exited with code: 2

---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to [email protected]

#3 lore raw
In reply to Alberto Garcia

Am 09.12.2016 um 14:47 hat Alberto Garcia geschrieben:

Quoted 36 lines

Hi all,

as we all know, one of the main things that can make the qcow2 format slow is the need to load entries from the L2 table in order to map a guest offset (on the virtual disk) to a host offset (on the qcow2 image).

We have an L2 cache to deal with this, and as long as the cache is big enough then the peformance is comparable to that of a raw image.

For large qcow2 images the amount of RAM we need in order to cache all L2 tables can be big (128MB per TB of disk image if we're using the default cluster size of 64KB). In order to solve this problem we have a setting that allows the user to clean unused cache entries after a certain interval of time. This works fine most of the time, although we can still have peaks of RAM usage if there's a lot of I/O going on in one or more VMs.

In some scenarios, however, there's a different alternative: if the qcow2 image is stored in a slow backend (eg. HDD), we could save memory by putting the L2 cache in a faster one (SSD) instead of in RAM.

I have been making some tests with exactly that scenario and the results look good: storing the cache in disk gives roughly the same performance as storing it in memory.

|---------------------+------------+------+------------+--------|
|                     | Random 4k reads   | Sequential 4k reads |
|                     | Throughput | IOPS | Throughput |  IOPS  |
|---------------------+------------+------+------------+--------|
| Cache in memory/SSD | 406 KB/s   |   99 | 84 MB/s    |  21000 |
| Default cache (1MB) | 200 KB/s   |   60 | 83 MB/s    |  21000 |
| No cache            | 200 KB/s   |   49 | 56 MB/s    |  14000 |
|---------------------+------------+------+------------+--------|
Quoted 4 lines

I'm including the patch that I used to get these results. This is the simplest approach that I could think of.

Opinions, questions?

I suppose you used the fact that the cache is now on disk to increase the cache size so that it covers the whole image?

If so, are you sure that you aren't just testing that accessing memory in the kernel page cache is just as fast as accessing memory in qemu's own cache? It seems this would just bypass the cache size limit given to qemu by instead leaving things cached in the kernel where the limit doesn't apply.

Kevin

#6 lore raw
In reply to Alberto Garcia
Quoted 36 lines · Alberto Garcia

Hi all,

as we all know, one of the main things that can make the qcow2 format slow is the need to load entries from the L2 table in order to map a guest offset (on the virtual disk) to a host offset (on the qcow2 image).

We have an L2 cache to deal with this, and as long as the cache is big enough then the peformance is comparable to that of a raw image.

For large qcow2 images the amount of RAM we need in order to cache all L2 tables can be big (128MB per TB of disk image if we're using the default cluster size of 64KB). In order to solve this problem we have a setting that allows the user to clean unused cache entries after a certain interval of time. This works fine most of the time, although we can still have peaks of RAM usage if there's a lot of I/O going on in one or more VMs.

In some scenarios, however, there's a different alternative: if the qcow2 image is stored in a slow backend (eg. HDD), we could save memory by putting the L2 cache in a faster one (SSD) instead of in RAM.

I have been making some tests with exactly that scenario and the results look good: storing the cache in disk gives roughly the same performance as storing it in memory.

|---------------------+------------+------+------------+--------|
|                     | Random 4k reads   | Sequential 4k reads |
|                     | Throughput | IOPS | Throughput |  IOPS  |
|---------------------+------------+------+------------+--------|
| Cache in memory/SSD | 406 KB/s   |   99 | 84 MB/s    |  21000 |
| Default cache (1MB) | 200 KB/s   |   60 | 83 MB/s    |  21000 |
| No cache            | 200 KB/s   |   49 | 56 MB/s    |  14000 |
|---------------------+------------+------+------------+--------|
Quoted 4 lines

I'm including the patch that I used to get these results. This is the simplest approach that I could think of.

Opinions, questions?

Well, from a full design standpoint, it doesn't make a lot of sense to me:

We have a two-level on-disk structure for cluster mapping so as to not waste memory for unused areas and so that we don't need to keep one large continuous chunk of metadata. Accessing the disk is slow, so we also have an in-memory cache which is just a single level fully associative cache replicating the same data (but just a part of it).

Now you want to replicate all of it and store it on disk. My mind tells me that is duplicate data: We already have all of the metadata elsewhere on disk, namely in the qcow2 file, and even better, it is not stored in a fully associative structure there but directly mapped, making finding the correct entry much quicker.

Therefore, this is not a good idea. The existing structures already exist and are just better.

However, the thing is that the existing structures also only exist in the original qcow2 file and cannot be just placed anywhere else, as opposed to our cache. In order to solve this, we would need to (incompatibly) modify the qcow2 format to allow storing data independently from metadata. I think this would be certainly doable, but the question is whether it is worth the effort.

I'm not sure, maybe it actually is worth the effort. Your patch is nice and simple and certainly improves things now. But the question is whether we can't do better than to look up all cluster mappings in a fully associative table.

Maybe we can at least make the cache directly mapped if it is supposed to cover the whole image? That is, we would basically just load all of the L2 tables into memory and bypass the existing cache.

Max

#4 lore raw
In reply to Kevin Wolf
Quoted 13 lines · Kevin Wolf
Quoted 13 lines

I have been making some tests with exactly that scenario and the results look good: storing the cache in disk gives roughly the same performance as storing it in memory.

|---------------------+------------+------+------------+--------|
|                     | Random 4k reads   | Sequential 4k reads |
|                     | Throughput | IOPS | Throughput |  IOPS  |
|---------------------+------------+------+------------+--------|
| Cache in memory/SSD | 406 KB/s   |   99 | 84 MB/s    |  21000 |
| Default cache (1MB) | 200 KB/s   |   60 | 83 MB/s    |  21000 |
| No cache            | 200 KB/s   |   49 | 56 MB/s    |  14000 |
|---------------------+------------+------+------------+--------|
Quoted 7 lines
Quoted 4 lines

I'm including the patch that I used to get these results. This is the simplest approach that I could think of.

Opinions, questions?

I suppose you used the fact that the cache is now on disk to increase the cache size so that it covers the whole image?

Right, the wording on the table is not clear, but that's what I did. I also don't think this makes much sense if the cache is not big enough to cover the whole image.

Quoted 5 lines

If so, are you sure that you aren't just testing that accessing memory in the kernel page cache is just as fast as accessing memory in qemu's own cache? It seems this would just bypass the cache size limit given to qemu by instead leaving things cached in the kernel where the limit doesn't apply.

Fair question: what I checked is that the PSS/RSS values match the expected values (i.e, they don't grow as you read from the disk image). If the kernel is caching those pages so accessing them after MADV_DONTNEED does not require going to disk again is a possibility that I haven't ruled out.

Berto

#5 lore raw

Am 09.12.2016 um 16:00 hat Alberto Garcia geschrieben:

Quoted 32 lines · Kevin Wolf
Quoted 13 lines
Quoted 13 lines

I have been making some tests with exactly that scenario and the results look good: storing the cache in disk gives roughly the same performance as storing it in memory.

|---------------------+------------+------+------------+--------|
|                     | Random 4k reads   | Sequential 4k reads |
|                     | Throughput | IOPS | Throughput |  IOPS  |
|---------------------+------------+------+------------+--------|
| Cache in memory/SSD | 406 KB/s   |   99 | 84 MB/s    |  21000 |
| Default cache (1MB) | 200 KB/s   |   60 | 83 MB/s    |  21000 |
| No cache            | 200 KB/s   |   49 | 56 MB/s    |  14000 |
|---------------------+------------+------+------------+--------|
Quoted 7 lines
Quoted 4 lines

I'm including the patch that I used to get these results. This is the simplest approach that I could think of.

Opinions, questions?

I suppose you used the fact that the cache is now on disk to increase the cache size so that it covers the whole image?

Right, the wording on the table is not clear, but that's what I did. I also don't think this makes much sense if the cache is not big enough to cover the whole image.

Quoted 5 lines

If so, are you sure that you aren't just testing that accessing memory in the kernel page cache is just as fast as accessing memory in qemu's own cache? It seems this would just bypass the cache size limit given to qemu by instead leaving things cached in the kernel where the limit doesn't apply.

Quoted 5 lines

Fair question: what I checked is that the PSS/RSS values match the expected values (i.e, they don't grow as you read from the disk image). If the kernel is caching those pages so accessing them after MADV_DONTNEED does not require going to disk again is a possibility that I haven't ruled out.

This is what the man page says:

Note that, when applied to shared mappings, MADV_DONTNEED might not lead to immediate freeing of the pages in the range. The kernel is free to delay freeing the pages until an appropriate moment. The resident set size (RSS) of the calling process will be immediately reduced however.

I think it makes sense that this is what is happening in your case. Your patch uses MADV_DONTNEED after every single access, so even if you're using an SSD, accessing it should be slower than accessing memory. This means that at least for sequential I/O where cached tables are reused all the time, your patch would have to be considerably slower than the default cache.

Maybe try putting the image on some storage where you definitely notice whether the kernel accesses it or not. Using a floppy should be fun, but an NBD device connected to qemu-nbd with enabled throttling or tracing could do the job as well.

Kevin

#7 lore raw
In reply to Max Reitz
Quoted 14 lines · Max Reitz
Quoted 4 lines

In some scenarios, however, there's a different alternative: if the qcow2 image is stored in a slow backend (eg. HDD), we could save memory by putting the L2 cache in a faster one (SSD) instead of in RAM.

Well, from a full design standpoint, it doesn't make a lot of sense to me:

We have a two-level on-disk structure for cluster mapping so as to not waste memory for unused areas and so that we don't need to keep one large continuous chunk of metadata. Accessing the disk is slow, so we also have an in-memory cache which is just a single level fully associative cache replicating the same data (but just a part of it).

Quoted 5 lines

Now you want to replicate all of it and store it on disk. My mind tells me that is duplicate data: We already have all of the metadata elsewhere on disk, namely in the qcow2 file, and even better, it is not stored in a fully associative structure there but directly mapped, making finding the correct entry much quicker.

Yes but the use case is that the qcow2 image is stored in a slow disk, so things will be faster if we avoid having to read it too often.

But the data is there and it needs to be read, so we have three options:

1) Read it everytime we need it. It slows things down.
2) Keep (part of) it in memory. It can use a lot of memory.
3) Keep it in a faster disk.

We're talking about 3) here, and this it not about creating new structures, but about changing the storage backend of the existing L2 cache (disk rather than RAM).

Quoted 6 lines

However, the thing is that the existing structures also only exist in the original qcow2 file and cannot be just placed anywhere else, as opposed to our cache. In order to solve this, we would need to (incompatibly) modify the qcow2 format to allow storing data independently from metadata. I think this would be certainly doable, but the question is whether it is worth the effort.

You mean split the qcow2 file in two: data and metadata? I don't think it's worth the effort.

Quoted 3 lines

Maybe we can at least make the cache directly mapped if it is supposed to cover the whole image? That is, we would basically just load all of the L2 tables into memory and bypass the existing cache.

I don't see how this addresses the original use case that I described.

But leaving that aside, would that improve anything? I don't think the cache itself adds any significant overhead here, IIRC even in your presentation at KVM Forum 2015 qcow2 was comparable to raw as long as all L2 tables were cached in memory.

Berto

#15 lore raw
In reply to Alberto Garcia
Quoted 34 lines · Alberto Garcia

as we all know, one of the main things that can make the qcow2 format slow is the need to load entries from the L2 table in order to map a guest offset (on the virtual disk) to a host offset (on the qcow2 image).

We have an L2 cache to deal with this, and as long as the cache is big enough then the peformance is comparable to that of a raw image.

For large qcow2 images the amount of RAM we need in order to cache all L2 tables can be big (128MB per TB of disk image if we're using the default cluster size of 64KB). In order to solve this problem we have a setting that allows the user to clean unused cache entries after a certain interval of time. This works fine most of the time, although we can still have peaks of RAM usage if there's a lot of I/O going on in one or more VMs.

In some scenarios, however, there's a different alternative: if the qcow2 image is stored in a slow backend (eg. HDD), we could save memory by putting the L2 cache in a faster one (SSD) instead of in RAM.

I have been making some tests with exactly that scenario and the results look good: storing the cache in disk gives roughly the same performance as storing it in memory.

|---------------------+------------+------+------------+--------|
|                     | Random 4k reads   | Sequential 4k reads |
|                     | Throughput | IOPS | Throughput |  IOPS  |
|---------------------+------------+------+------------+--------|
| Cache in memory/SSD | 406 KB/s   |   99 | 84 MB/s    |  21000 |
| Default cache (1MB) | 200 KB/s   |   60 | 83 MB/s    |  21000 |
| No cache            | 200 KB/s   |   49 | 56 MB/s    |  14000 |
|---------------------+------------+------+------------+--------|
Quoted 4 lines

I'm including the patch that I used to get these results. This is the simplest approach that I could think of.

Opinions, questions?

The root of the performance problem is the L2 table on-disk format, which also happens to be used as the in-memory L2 table format. It does not scale to large disk images.

The simplest tweak is to use larger cluster sizes. 64 KB has been the default for a long time and it may be time to evaluate performance effects of increasing it. I suspect this doesn't solve the problem, instead we need to decouple metadata scalability from the cluster size...

Is it time for a new on-disk representation? Modern file systems seem to use extent trees instead of offset tables. That brings a lot of complication because a good B-tree implementation would require quite a bit of code changes.

Maybe a more modest change to the on-disk representation could solve most of the performance problem. In a very sparsely allocated L2 table something like run-length encoding is more space-efficient than an offset table. In a very densely allocated L2 table it may be possible to choose a "base offset" and then use much smaller offset entries relative to the base. For example:

typedef struct {
    uint64_t base_offset;
    uint16_t rel_offset[];  /* covers 4 GB with 64 KB cluster size */
} L2TableRelative;

uint64_t offset = l2->base_offset + l2->rel_offset[i] * cluster_size;

A final option is to leave the on-disk representation alone but convert to an efficient in-memory representation when loading from disk.

Stefan

#8 lore raw
In reply to Alberto Garcia
Quoted 27 lines · Alberto Garcia
Quoted 14 lines · Max Reitz
Quoted 4 lines

In some scenarios, however, there's a different alternative: if the qcow2 image is stored in a slow backend (eg. HDD), we could save memory by putting the L2 cache in a faster one (SSD) instead of in RAM.

Well, from a full design standpoint, it doesn't make a lot of sense to me:

We have a two-level on-disk structure for cluster mapping so as to not waste memory for unused areas and so that we don't need to keep one large continuous chunk of metadata. Accessing the disk is slow, so we also have an in-memory cache which is just a single level fully associative cache replicating the same data (but just a part of it).

Quoted 5 lines

Now you want to replicate all of it and store it on disk. My mind tells me that is duplicate data: We already have all of the metadata elsewhere on disk, namely in the qcow2 file, and even better, it is not stored in a fully associative structure there but directly mapped, making finding the correct entry much quicker.

Yes but the use case is that the qcow2 image is stored in a slow disk, so things will be faster if we avoid having to read it too often.

But the data is there and it needs to be read, so we have three options:

Quoted 7 lines

1) Read it everytime we need it. It slows things down.
2) Keep (part of) it in memory. It can use a lot of memory.
3) Keep it in a faster disk.

We're talking about 3) here, and this it not about creating new structures, but about changing the storage backend of the existing L2 cache (disk rather than RAM).

I'm arguing that we already have an on-disk L2 structure and that is called simply the L1-L2 structure in the qcow2 file. The cache only makes sense because it is in RAM.

Quoted 9 lines
Quoted 6 lines

However, the thing is that the existing structures also only exist in the original qcow2 file and cannot be just placed anywhere else, as opposed to our cache. In order to solve this, we would need to (incompatibly) modify the qcow2 format to allow storing data independently from metadata. I think this would be certainly doable, but the question is whether it is worth the effort.

You mean split the qcow2 file in two: data and metadata? I don't think it's worth the effort.

That's the thing. I don't know.

I definitely like how simple your approach is, but from a design standpoint it is not exactly optimal, because O(n) for a cluster lookup is simply worse than O(1).

Quoted 5 lines
Quoted 3 lines

Maybe we can at least make the cache directly mapped if it is supposed to cover the whole image? That is, we would basically just load all of the L2 tables into memory and bypass the existing cache.

I don't see how this addresses the original use case that I described.

It just fixes the issue that the cache is fully associative and then the only issue I would have with your approach is that we are keeping duplicate data.

Quoted 4 lines

But leaving that aside, would that improve anything? I don't think the cache itself adds any significant overhead here, IIRC even in your presentation at KVM Forum 2015 qcow2 was comparable to raw as long as all L2 tables were cached in memory.

I haven't compared CPU usage, though. That may have gone up quite a bit, I don't know. For large enough images, it may even become a bottleneck.

Max

#9 lore raw
Quoted 9 lines · Max Reitz
Quoted 9 lines

Yes but the use case is that the qcow2 image is stored in a slow disk, so things will be faster if we avoid having to read it too often.

But the data is there and it needs to be read, so we have three options:

1) Read it everytime we need it. It slows things down.
2) Keep (part of) it in memory. It can use a lot of memory.
3) Keep it in a faster disk.

Quoted 7 lines
Quoted 3 lines

We're talking about 3) here, and this it not about creating new structures, but about changing the storage backend of the existing L2 cache (disk rather than RAM).

I'm arguing that we already have an on-disk L2 structure and that is called simply the L1-L2 structure in the qcow2 file. The cache only makes sense because it is in RAM.

I am looking at this problem from a different angle:

Assume a qcow2 image created by a known version of QEMU with the option preallocation=falloc, it is theoretically possible to do a hacky optimization in qcow2 I/O path that, instead of looking up in L1/L2, we can calculate file offset for any given virtual address, by adding the file offset of LBA 0.

If we make this optimization a header extention and add it to the spec, it is no longer a hack, and it should offer nearly the same performance as raw in all cases - if we say raw is doing "identical mapping" from LBA to file offset, this new operation mode of qcow2 file is basically doing a "shift by constant" mapping.

Fam

#10 lore raw

Am 13.12.2016 um 11:16 hat Fam Zheng geschrieben:

Quoted 25 lines · Max Reitz
Quoted 9 lines
Quoted 9 lines

Yes but the use case is that the qcow2 image is stored in a slow disk, so things will be faster if we avoid having to read it too often.

But the data is there and it needs to be read, so we have three options:

1) Read it everytime we need it. It slows things down.
2) Keep (part of) it in memory. It can use a lot of memory.
3) Keep it in a faster disk.

Quoted 7 lines
Quoted 3 lines

We're talking about 3) here, and this it not about creating new structures, but about changing the storage backend of the existing L2 cache (disk rather than RAM).

I'm arguing that we already have an on-disk L2 structure and that is called simply the L1-L2 structure in the qcow2 file. The cache only makes sense because it is in RAM.

I am looking at this problem from a different angle:

Assume a qcow2 image created by a known version of QEMU with the option preallocation=falloc, it is theoretically possible to do a hacky optimization in qcow2 I/O path that, instead of looking up in L1/L2, we can calculate file offset for any given virtual address, by adding the file offset of LBA 0.

Quoted 4 lines

If we make this optimization a header extention and add it to the spec, it is no longer a hack, and it should offer nearly the same performance as raw in all cases - if we say raw is doing "identical mapping" from LBA to file offset, this new operation mode of qcow2 file is basically doing a "shift by constant" mapping.

Is this a scenario that matters in practice, though? If you want performance and you don't need COW functionality, just use raw images to begin with.

Kevin

#12 lore raw
In reply to Max Reitz
Quoted 3 lines · Max Reitz

I definitely like how simple your approach is, but from a design standpoint it is not exactly optimal, because O(n) for a cluster lookup is simply worse than O(1).

It's actually not O(n) anymore since I rewrote that code last year.

When I changed the algorithm to use LRU for cache eviction I also optimized the lookup:

1) qcow2_cache_put() simply divides the table offset by the cluster size and it immediately obtains the index.

2) qcow2_cache_get() hashes the offset to do the lookup. In my tests doing random I/O and a 2048 entry cache the average number of comparisons was 2.5 (430 before the rewrite).

The cache was ~13% faster after the rewrite, but the results in terms of actual disk I/O improvements were negligible, because that's not where the bottleneck is. We even discussed how the hash function that I was using was weak, but we decided not to touch it because it was not worth the effort.

Quoted 8 lines
Quoted 4 lines

But leaving that aside, would that improve anything? I don't think the cache itself adds any significant overhead here, IIRC even in your presentation at KVM Forum 2015 qcow2 was comparable to raw as long as all L2 tables were cached in memory.

I haven't compared CPU usage, though. That may have gone up quite a bit, I don't know. For large enough images, it may even become a bottleneck.

I don't know, I don't have any data to suggest that there's a problem there, particularly after the rewrite from last year I think the cache is reasonably fast (as long as the cache is big enough; and if it's not big enough the problem is going to be in the extra I/O).

Berto

#11 lore raw
In reply to Kevin Wolf
Quoted 26 lines · Kevin Wolf

Am 13.12.2016 um 11:16 hat Fam Zheng geschrieben:

Quoted 25 lines · Max Reitz
Quoted 9 lines
Quoted 9 lines

Yes but the use case is that the qcow2 image is stored in a slow disk, so things will be faster if we avoid having to read it too often.

But the data is there and it needs to be read, so we have three options:

1) Read it everytime we need it. It slows things down.
2) Keep (part of) it in memory. It can use a lot of memory.
3) Keep it in a faster disk.

Quoted 7 lines
Quoted 3 lines

We're talking about 3) here, and this it not about creating new structures, but about changing the storage backend of the existing L2 cache (disk rather than RAM).

I'm arguing that we already have an on-disk L2 structure and that is called simply the L1-L2 structure in the qcow2 file. The cache only makes sense because it is in RAM.

I am looking at this problem from a different angle:

Assume a qcow2 image created by a known version of QEMU with the option preallocation=falloc, it is theoretically possible to do a hacky optimization in qcow2 I/O path that, instead of looking up in L1/L2, we can calculate file offset for any given virtual address, by adding the file offset of LBA 0.

Quoted 8 lines
Quoted 4 lines

If we make this optimization a header extention and add it to the spec, it is no longer a hack, and it should offer nearly the same performance as raw in all cases - if we say raw is doing "identical mapping" from LBA to file offset, this new operation mode of qcow2 file is basically doing a "shift by constant" mapping.

Is this a scenario that matters in practice, though? If you want performance and you don't need COW functionality, just use raw images to begin with.

The intention is actually to combine the advantages of both: external snapshot can be supported by an allocation status bitmap generated from scanning the whole L1/L2 table, or even from loading a new type of qcow2 bitmap specifically for making that efficient.

I share the same doubt as Stefan regarding the on disk format. Modern file systems support sparse file quite well, so I wonder what advantages are still there in using L1/L2 tables instead of a simple bitmap, if the user doesn't use internal snapshot in the first place?

Fam

#13 lore raw
In reply to Alberto Garcia
Quoted 18 lines · Alberto Garcia
Quoted 3 lines · Max Reitz

I definitely like how simple your approach is, but from a design standpoint it is not exactly optimal, because O(n) for a cluster lookup is simply worse than O(1).

It's actually not O(n) anymore since I rewrote that code last year.

When I changed the algorithm to use LRU for cache eviction I also optimized the lookup:

1) qcow2_cache_put() simply divides the table offset by the cluster size and it immediately obtains the index.

2) qcow2_cache_get() hashes the offset to do the lookup. In my tests doing random I/O and a 2048 entry cache the average number of comparisons was 2.5 (430 before the rewrite).

Quoted 5 lines

The cache was ~13% faster after the rewrite, but the results in terms of actual disk I/O improvements were negligible, because that's not where the bottleneck is. We even discussed how the hash function that I was using was weak, but we decided not to touch it because it was not worth the effort.

13% is not really what O(1) vs. O(n) is about, but you are completely right, For some reason I haven't seen that. OK, so the code is indeed O(1) in the best case (and probably in the average case, too), so that's good.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

But leaving that aside, would that improve anything? I don't think the cache itself adds any significant overhead here, IIRC even in your presentation at KVM Forum 2015 qcow2 was comparable to raw as long as all L2 tables were cached in memory.

Quoted 8 lines
Quoted 3 lines

I haven't compared CPU usage, though. That may have gone up quite a bit, I don't know. For large enough images, it may even become a bottleneck.

I don't know, I don't have any data to suggest that there's a problem there, particularly after the rewrite from last year I think the cache is reasonably fast (as long as the cache is big enough; and if it's not big enough the problem is going to be in the extra I/O).

So the remaining question is whether we want a cache for the cache. If the cache is stored directly on an SSD, any access to it has to go there and back, and that seems like we may still want to have the option of having an in-RAM cache on top of it.

If we don't want to split the qcow2 file into separate metadata and data files, then we could still create a sparse temporary file at runtime which contains all L2 tables. Reading an L2 table then wouldn't go through the qcow2 file but through that temporary file and our existing metadata cache would work on top of it.

Max

#14 lore raw
Quoted 14 lines · Max Reitz
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

But leaving that aside, would that improve anything? I don't think the cache itself adds any significant overhead here, IIRC even in your presentation at KVM Forum 2015 qcow2 was comparable to raw as long as all L2 tables were cached in memory.

Quoted 8 lines
Quoted 3 lines

I haven't compared CPU usage, though. That may have gone up quite a bit, I don't know. For large enough images, it may even become a bottleneck.

I don't know, I don't have any data to suggest that there's a problem there, particularly after the rewrite from last year I think the cache is reasonably fast (as long as the cache is big enough; and if it's not big enough the problem is going to be in the extra I/O).

Quoted 4 lines

So the remaining question is whether we want a cache for the cache. If the cache is stored directly on an SSD, any access to it has to go there and back, and that seems like we may still want to have the option of having an in-RAM cache on top of it.

That would be indeed possible... but I wouldn't have any without data showing that it's worth it.

Quoted 5 lines

If we don't want to split the qcow2 file into separate metadata and data files, then we could still create a sparse temporary file at runtime which contains all L2 tables. Reading an L2 table then wouldn't go through the qcow2 file but through that temporary file and our existing metadata cache would work on top of it.

And if you modify a table you have to do it in both places.

The basic idea is the same, anyway. I just started with the assumption that the L2 cache structures don't add significant overhead and went for the simplest solution (that's why it's an RFC after all).

But implementation details aside, the main question is whether the best way to go is to keep the existing metadata and store it in a faster disk or, as Stefan suggested, think about replacing the current structures with something else that scales better. I think that's at least worth exploring.

Berto