Patch Round-up for stable 4.0.1, freeze on 2019-10-10
Subsystems (3)
PATCH 11/97 kbd-state: fix autorepeat handling
In reply to Michael RothFrom: Gerd Hoffmann <[email protected]>
When allowing multiple down-events in a row (key autorepeat) we can't use change_bit() any more to update the state, because autorepeat events don't change the key state. We have to explicitly use set_bit() and clear_bit() instead.
Message-id: [email protected]
--- ui/kbd-state.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
ui/kbd-state.c +5 -1
@@ -59,7 +59,11 @@ void qkbd_state_key_event(QKbdState *kbd, QKeyCode qcode, bool down) }/* update key and modifier state */ -change_bit (qcode, kbd->keys); +if (down) { +set_bit (qcode, kbd->keys); + }else { +clear_bit (qcode, kbd->keys); + }switch (qcode) {case Q_KEY_CODE_SHIFT :case Q_KEY_CODE_SHIFT_R :
Signature · 1 line
2.17.1
PATCH 1/97 qcow2: Avoid COW during metadata preallocation
In reply to Michael RothFrom: Kevin Wolf <[email protected]>
Limiting the allocation to INT_MAX bytes isn't particularly clever because it means that the final cluster will be a partial cluster which will be completed through a COW operation. This results in unnecessary data read and write requests which lead to an unwanted non-sparse filesystem block for metadata preallocation.
Align the maximum allocation size down to the cluster size to avoid this situation.
--- block/qcow2.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/qcow2.c +1 -1
@@ -2734,7 +2734,7 @@ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset, bytes = new_length - offset;while (bytes) { - cur_bytes =MIN (bytes,INT_MAX ); + cur_bytes =MIN (bytes,QEMU_ALIGN_DOWN (INT_MAX , s->cluster_size)); ret =qcow2_alloc_cluster_offset (bs, offset, &cur_bytes, &host_offset, &meta);if (ret <0 ) {
Signature · 1 line
2.17.1
PATCH 22/97 iotests.py: rewrite run_job to be pickier
In reply to Michael RothFrom: John Snow <[email protected]>
Don't pull events out of the queue that don't belong to us; be choosier so that we can use this method to drive jobs that were launched by transactions that may have more jobs.
Message-id: [email protected]
--- tests/qemu-iotests/iotests.py | 48 +++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 19 deletions(-)
tests/qemu-iotests/iotests.py +29 -19
@@ -540,27 +540,37 @@ class VM(qtest.QEMUQtestMachine): # Returns None on success, and an error string on failure def run_job(self, job, auto_finalize=True, auto_dismiss=False, pre_finalize=None, wait=60.0): + match_device = {'data': {'device': job}} + match_id = {'data': {'id': job}} + events = [ + ('BLOCK_JOB_COMPLETED', match_device), + ('BLOCK_JOB_CANCELLED', match_device), + ('BLOCK_JOB_ERROR', match_device), + ('BLOCK_JOB_READY', match_device), + ('BLOCK_JOB_PENDING', match_id), + ('JOB_STATUS_CHANGE', match_id) + ] error = None while True: - for ev in self.get_qmp_events_filtered(wait=wait): - if ev['event'] == 'JOB_STATUS_CHANGE': - status = ev['data']['status'] - if status == 'aborting': - result = self.qmp('query-jobs') - for j in result['return']: - if j['id'] == job: - error = j['error'] - log('Job failed: %s' % (j['error'])) - elif status == 'pending' and not auto_finalize: - if pre_finalize: - pre_finalize() - self.qmp_log('job-finalize', id=job) - elif status == 'concluded' and not auto_dismiss: - self.qmp_log('job-dismiss', id=job) - elif status == 'null': - return error - else: - log(ev) + ev = filter_qmp_event(self.events_wait(events)) + if ev['event'] != 'JOB_STATUS_CHANGE': + log(ev) + continue + status = ev['data']['status'] + if status == 'aborting': + result = self.qmp('query-jobs') + for j in result['return']: + if j['id'] == job: + error = j['error'] + log('Job failed: %s' % (j['error'])) + elif status == 'pending' and not auto_finalize: + if pre_finalize: + pre_finalize() + self.qmp_log('job-finalize', id=job) + elif status == 'concluded' and not auto_dismiss: + self.qmp_log('job-dismiss', id=job) + elif status == 'null': + return error def node_info(self, node_name): nodes = self.qmp('query-named-block-nodes')
Signature · 1 line
2.17.1
PATCH 14/97 iotests: Test unaligned raw images with O_DIRECT
In reply to Michael RothFrom: Max Reitz <[email protected]>
We already have 221 for accesses through the page cache, but it is better to create a new file for O_DIRECT instead of integrating those test cases into 221. This way, we can make use of _supported_cache_modes (and _default_cache_mode) so the test is automatically skipped on filesystems that do not support O_DIRECT.
As part of the split, add _supported_cache_modes to 221. With that, it no longer fails when run with -c none or -c directsync.
Conflicts:
tests/qemu-iotests/group
*fix context deps on test groups not in 4.0
--- tests/qemu-iotests/221 | 4 ++ tests/qemu-iotests/253 | 84 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/253.out | 14 +++++++ tests/qemu-iotests/group | 1 + 4 files changed, 103 insertions(+) create mode 100755 tests/qemu-iotests/253 create mode 100644 tests/qemu-iotests/253.out
tests/qemu-iotests/221 +4 -0
@@ -1,6 +1,7 @@ #!/usr/bin/env bash # # Test qemu-img vs. unaligned images +# (See also 253, which is the O_DIRECT version) # # Copyright (C) 2018 Red Hat, Inc. #
@@ -37,6 +38,9 @@ _supported_fmt raw _supported_proto file _supported_os Linux +_default_cache_mode writeback +_supported_cache_modes writeback writethrough unsafe + echo echo "=== Check mapping of unaligned raw image ===" echo
tests/qemu-iotests/253 new file +84 -0
84 lines not on this page · raw message
tests/qemu-iotests/253.out new file +14 -0
@@ -0,0 +1,14 @@ +QA output created by 253 + +=== Check mapping of unaligned raw image === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575 +[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] +[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] +wrote 65535/65535 bytes at offset 983040 +63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, +{ "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] +[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET}, +{ "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] +*** done
tests/qemu-iotests/group +1 -0
@@ -248,3 +248,4 @@ 246 rw auto quick 247 rw auto quick 248 rw auto quick +253 rw auto quick
Signature · 1 line
2.17.1
PATCH 24/97 migration/dirty-bitmaps: change bitmap enumeration method
In reply to Michael RothFrom: John Snow <[email protected]>
Shift from looking at every root BDS to *every* BDS. This will migrate bitmaps that are attached to blockdev created nodes instead of just ones attached to emulated storage devices.
Note that this will not migrate anonymous or internal-use bitmaps, as those are defined as having no name.
This will also fix the Coverity issues Peter Maydell has been asking about for the past several releases, as well as fixing a real bug.
Message-id: [email protected]
--- migration/block-dirty-bitmap.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-)
migration/block-dirty-bitmap.c +4 -10
@@ -273,7 +273,6 @@ static int init_dirty_bitmap_migration(void)BlockDriverState *bs;BdrvDirtyBitmap *bitmap;DirtyBitmapMigBitmapState *dbms; -BdrvNextIterator it;Error *local_err =NULL ; dirty_bitmap_mig_state.bulk_completed = false;
@@ -281,13 +280,8 @@ static int init_dirty_bitmap_migration(void) dirty_bitmap_mig_state.prev_bitmap =NULL ; dirty_bitmap_mig_state.no_bitmaps = false; -for (bs =bdrv_first (&it); bs; bs =bdrv_next (&it)) { -const char *drive_name =bdrv_get_device_or_node_name (bs); - -/* skip automatically inserted nodes */ -while (bs && bs->drv && bs->implicit) { - bs =backing_bs (bs); - } +for (bs =bdrv_next_all_states (NULL ); bs; bs =bdrv_next_all_states (bs)) { +const char *name =bdrv_get_device_or_node_name (bs);for (bitmap =bdrv_dirty_bitmap_next (bs,NULL ); bitmap; bitmap =bdrv_dirty_bitmap_next (bs, bitmap))
@@ -296,7 +290,7 @@ static int init_dirty_bitmap_migration(void)continue ; } -if (drive_name ==NULL ) { +if (!name ||strcmp (name,"" ) ==0 ) {error_report ("Found bitmap '%s' in unnamed node %p. It can't " "be migrated" ,bdrv_dirty_bitmap_name (bitmap), bs); goto fail;
@@ -313,7 +307,7 @@ static int init_dirty_bitmap_migration(void) dbms =g_new0 (DirtyBitmapMigBitmapState,1 ); dbms->bs = bs; - dbms->node_name = drive_name; + dbms->node_name = name; dbms->bitmap = bitmap; dbms->total_sectors =bdrv_nb_sectors (bs); dbms->sectors_per_chunk = CHUNK_SIZE * 8 *
Signature · 1 line
2.17.1
PATCH 15/97 s390x/cpumodel: ignore csske for expansion
In reply to Michael RothFrom: Christian Borntraeger <[email protected]>
csske will be removed in a future machine. Ignore it for expanding the cpu model. Otherwise qemu falls back to z9.
Message-Id: <[email protected]>
--- target/s390x/cpu_models.c | 2 ++ 1 file changed, 2 insertions(+)
target/s390x/cpu_models.c +2 -0
@@ -1327,6 +1327,8 @@ static void init_ignored_base_feat(void)S390_FEAT_KM_TDEA_192 ,S390_FEAT_KIMD_SHA_1 ,S390_FEAT_KLMD_SHA_1 , +/* CSSKE is deprecated on newer generations */ + S390_FEAT_CONDITIONAL_SSKE, };int i;
Signature · 1 line
2.17.1
PATCH 21/97 iotests.py: Fix VM.run_job
In reply to Michael RothFrom: Max Reitz <[email protected]>
log() is in the current module, there is no need to prefix it. In fact, doing so may make VM.run_job() unusable in tests that never use iotests.log() themselves.
*prereq for d6a79af0e6
--- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +1 -1
@@ -560,7 +560,7 @@ class VM(qtest.QEMUQtestMachine): elif status == 'null': return error else: - iotests.log(ev) + log(ev) def node_info(self, node_name): nodes = self.qmp('query-named-block-nodes')
Signature · 1 line
2.17.1
PATCH 34/97 target/i386: define md-clear bit
In reply to Michael RothFrom: Paolo Bonzini <[email protected]>
md-clear is a new CPUID bit which is set when microcode provides the mechanism to invoke a flush of various exploitable CPU buffers by invoking the VERW instruction.
Message-Id: <[email protected]>
--- target/i386/cpu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
target/i386/cpu.c +1 -1
@@ -1076,7 +1076,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = { .feat_names = {NULL ,NULL ,"avx512-4vnniw" ,"avx512-4fmaps" ,NULL ,NULL ,NULL ,NULL , -NULL ,NULL ,NULL ,NULL , +NULL ,NULL ,"md-clear" ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , NULL, NULL, NULL, NULL,
Signature · 1 line
2.17.1
PATCH 12/97 usb-tablet: fix serial compat property
In reply to Michael RothFrom: Gerd Hoffmann <[email protected]>
s/kbd/tablet/, fixes cut+paste bug.
Message-id: [email protected]
--- hw/core/machine.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
hw/core/machine.c +1 -1
@@ -33,7 +33,7 @@ GlobalProperty hw_compat_3_1[] = { { "tpm-tis", "ppi", "false" }, { "usb-kbd", "serial", "42" }, { "usb-mouse", "serial", "42" }, - { "usb-kbd", "serial", "42" }, + { "usb-tablet", "serial", "42" }, { "virtio-blk-device", "discard", "false" }, { "virtio-blk-device", "write-zeroes", "false" }, };
Signature · 1 line
2.17.1
PATCH 16/97 blockdev-backup: don't check aio_context too early
In reply to Michael RothFrom: John Snow <[email protected]>
in blockdev_backup_prepare, we check to make sure that the target is associated with a compatible aio context. However, do_blockdev_backup is called later and has some logic to move the target to a compatible aio_context. The transaction version will fail certain commands needlessly early as a result.
Allow blockdev_backup_prepare to simply call do_blockdev_backup, which will ultimately decide if the contexts are compatible or not.
Note: the transaction version has always disallowed this operation since its initial commit bd8baecd (2014), whereas the version of qmp_blockdev_backup at the time, from commit c29c1dd312f, tried to enforce the aio_context switch instead. It's not clear, and I can't see from the mailing list archives at the time, why the two functions take a different approach. It wasn't until later in efd7556708b (2016) that the standalone version tried to determine if it could set the context or not.
Message-id: [email protected]
--- blockdev.c | 4 ---- 1 file changed, 4 deletions(-)
blockdev.c +0 -4
@@ -1871,10 +1871,6 @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp) } aio_context =bdrv_get_aio_context (bs); -if (aio_context !=bdrv_get_aio_context (target)) { -error_setg (errp,"Backup between two IO threads is not implemented" ); -return ; - }aio_context_acquire (aio_context); state->bs = bs;
Signature · 1 line
2.17.1
PATCH 18/97 iotests: Test commit job start with concurrent I/O
In reply to Michael RothFrom: Kevin Wolf <[email protected]>
This tests that concurrent requests are correctly drained before making graph modifications instead of running into assertions in bdrv_replace_node().
Conflicts:
tests/qemu-iotests/group
*prereq for d81e1efb tests
--- tests/qemu-iotests/255 | 83 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/255.out | 16 +++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 10 ++++- 4 files changed, 109 insertions(+), 1 deletion(-) create mode 100755 tests/qemu-iotests/255 create mode 100644 tests/qemu-iotests/255.out
tests/qemu-iotests/255 new file +83 -0
@@ -0,0 +1,83 @@ +#!/usr/bin/env python +# +# Test commit job graph modifications while requests are active +# +# Copyright (C) 2019 Red Hat, Inc. +# +# Creator/Owner: Kevin Wolf <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import iotests +from iotests import imgfmt + +iotests.verify_image_format(supported_fmts=['qcow2']) + +def blockdev_create(vm, options): + result = vm.qmp_log('blockdev-create', + filters=[iotests.filter_qmp_testfiles], + job_id='job0', options=options) + + if 'return' in result: + assert result['return'] == {} + vm.run_job('job0') + iotests.log("") + +with iotests.FilePath('t.qcow2') as disk_path, \ + iotests.FilePath('t.qcow2.mid') as mid_path, \ + iotests.FilePath('t.qcow2.base') as base_path, \ + iotests.VM() as vm: + + iotests.log("=== Create backing chain and start VM ===") + iotests.log("") + + size = 128 * 1024 * 1024 + size_str = str(size) + + iotests.create_image(base_path, size) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, mid_path, size_str) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, disk_path, size_str) + + # Create a backing chain like this: + # base <- [throttled: bps-read=4096] <- mid <- overlay + + vm.add_object('throttle-group,x-bps-read=4096,id=throttle0') + vm.add_blockdev('file,filename=%s,node-name=base' % (base_path)) + vm.add_blockdev('throttle,throttle-group=throttle0,file=base,node-name=throttled') + vm.add_blockdev('file,filename=%s,node-name=mid-file' % (mid_path)) + vm.add_blockdev('qcow2,file=mid-file,node-name=mid,backing=throttled') + vm.add_drive_raw('if=none,id=overlay,driver=qcow2,file=%s,backing=mid' % (disk_path)) + + vm.launch() + + iotests.log("=== Start background read requests ===") + iotests.log("") + + def start_requests(): + vm.hmp_qemu_io('overlay', 'aio_read 0 4k') + vm.hmp_qemu_io('overlay', 'aio_read 0 4k') + + start_requests() + + iotests.log("=== Run a commit job ===") + iotests.log("") + + result = vm.qmp_log('block-commit', job_id='job0', auto_finalize=False, + device='overlay', top_node='mid') + + vm.run_job('job0', auto_finalize=False, pre_finalize=start_requests, + auto_dismiss=True) + + vm.shutdown()
tests/qemu-iotests/255.out new file +16 -0
@@ -0,0 +1,16 @@ +=== Create backing chain and start VM === + +Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +=== Start background read requests === + +=== Run a commit job === + +{"execute": "block-commit", "arguments": {"auto-finalize": false, "device": "overlay", "job-id": "job0", "top-node": "mid"}} +{"return": {}} +{"execute": "job-finalize", "arguments": {"id": "job0"}} +{"return": {}} +{"data": {"id": "job0", "type": "commit"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +{"data": {"device": "job0", "len": 134217728, "offset": 134217728, "speed": 0, "type": "commit"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
tests/qemu-iotests/group +1 -0
@@ -249,3 +249,4 @@ 247 rw auto quick 248 rw auto quick 253 rw auto quick +255 rw auto quick
tests/qemu-iotests/iotests.py +9 -1
@@ -126,6 +126,11 @@ def qemu_img_pipe(*args): sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args)))) return subp.communicate()[0] +def qemu_img_log(*args): + result = qemu_img_pipe(*args) + log(result, filters=[filter_testfiles]) + return result + def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]): args = [ 'info' ] if imgopts:
@@ -533,7 +538,8 @@ class VM(qtest.QEMUQtestMachine): return result # Returns None on success, and an error string on failure - def run_job(self, job, auto_finalize=True, auto_dismiss=False): + def run_job(self, job, auto_finalize=True, auto_dismiss=False, + pre_finalize=None): error = None while True: for ev in self.get_qmp_events_filtered(wait=True):
@@ -546,6 +552,8 @@ class VM(qtest.QEMUQtestMachine): error = j['error'] log('Job failed: %s' % (j['error'])) elif status == 'pending' and not auto_finalize: + if pre_finalize: + pre_finalize() self.qmp_log('job-finalize', id=job) elif status == 'concluded' and not auto_dismiss: self.qmp_log('job-dismiss', id=job)
Signature · 1 line
2.17.1
PATCH 31/97 q35: Revert to kernel irqchip
In reply to Michael RothFrom: Alex Williamson <[email protected]>
Backport of QEMU v4.1 commit for stable v4.0.1 release
commit c87759ce876a7a0b17c2bf4f0b964bd51f0ee871
Author: Alex Williamson <[email protected]> Date: Tue May 14 14:14:41 2019 -0600
q35: Revert to kernel irqchip
Commit b2fc91db8447 ("q35: set split kernel irqchip as default") changed the default for the pc-q35-4.0 machine type to use split irqchip, which turned out to have disasterous effects on vfio-pci INTx support. KVM resampling irqfds are registered for handling these interrupts, but these are non-functional in split irqchip mode. We can't simply test for split irqchip in QEMU as userspace handling of this interrupt is a significant performance regression versus KVM handling (GeForce GPUs assigned to Windows VMs are non-functional without forcing MSI mode or re-enabling kernel irqchip).
The resolution is to revert the change in default irqchip mode in the pc-q35-4.1 machine and create a pc-q35-4.0.1 machine for the 4.0-stable branch. The qemu-q35-4.0 machine type should not be used in vfio-pci configurations for devices requiring legacy INTx support without explicitly modifying the VM configuration to use kernel irqchip.
(upstream commit c87759ce876a7a0b17c2bf4f0b964bd51f0ee871)
*add comments regarding AML mismatch warnings from
tests/bios-tables-test.c
--- hw/core/machine.c | 3 +++ hw/i386/pc.c | 3 +++ hw/i386/pc_q35.c | 16 ++++++++++++++-- include/hw/boards.h | 3 +++ include/hw/i386/pc.h | 3 +++ tests/bios-tables-test.c | 9 +++++++++ 6 files changed, 35 insertions(+), 2 deletions(-)
hw/core/machine.c +3 -0
@@ -24,6 +24,9 @@#include "hw/pci/pci.h" #include "hw/mem/nvdimm.h" +GlobalProperty hw_compat_4_0[] = {}; +const size_t hw_compat_4_0_len =G_N_ELEMENTS (hw_compat_4_0); +GlobalProperty hw_compat_3_1[] = { {"pcie-root-port" ,"x-speed" ,"2_5" }, {"pcie-root-port" ,"x-width" ,"1" },
hw/i386/pc.c +3 -0
@@ -115,6 +115,9 @@ struct hpet_fw_config hpet_cfg = {.count = UINT8_MAX};/* Physical Address of PVH entry point read from kernel ELF NOTE */ static size_t pvh_start_addr; +GlobalProperty pc_compat_4_0[] = {}; +const size_t pc_compat_4_0_len =G_N_ELEMENTS (pc_compat_4_0); +GlobalProperty pc_compat_3_1[] = { {"intel-iommu" ,"dma-drain" ,"off" }, {"Opteron_G3" "-" TYPE_X86_CPU ,"rdtscp" ,"off" },
hw/i386/pc_q35.c +14 -2
@@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m) m->units_per_default_bus =1 ; m->default_machine_opts ="firmware=bios-256k.bin" ; m->default_display ="std" ; - m->default_kernel_irqchip_split = true; + m->default_kernel_irqchip_split = false; m->no_floppy =1 ;machine_class_allow_dynamic_sysbus_dev (m,TYPE_AMD_IOMMU_DEVICE );machine_class_allow_dynamic_sysbus_dev (m,TYPE_INTEL_IOMMU_DEVICE );
@@ -365,12 +365,24 @@ static void pc_q35_machine_options(MachineClass *m) m->max_cpus =288 ; } -static void pc_q35_4_0_machine_options (MachineClass *m) +static void pc_q35_4_0_1_machine_options (MachineClass *m) {pc_q35_machine_options (m); m->alias ="q35" ; } +DEFINE_Q35_MACHINE (v4_0_1,"pc-q35-4.0.1" ,NULL , + pc_q35_4_0_1_machine_options); + +static void pc_q35_4_0_machine_options (MachineClass *m) +{ +pc_q35_4_0_1_machine_options (m); + m->default_kernel_irqchip_split = true; + m->alias =NULL ; +compat_props_add (m->compat_props, hw_compat_4_0, hw_compat_4_0_len); +compat_props_add (m->compat_props, pc_compat_4_0, pc_compat_4_0_len); +} +DEFINE_Q35_MACHINE (v4_0,"pc-q35-4.0" ,NULL , pc_q35_4_0_machine_options);
include/hw/boards.h +3 -0
@@ -293,6 +293,9 @@ struct MachineState { } \ type_init(machine_initfn##_register_types) +extern GlobalProperty hw_compat_4_0[]; +extern const size_t hw_compat_4_0_len; +extern GlobalProperty hw_compat_3_1[];extern const size_t hw_compat_3_1_len;
include/hw/i386/pc.h +3 -0
@@ -293,6 +293,9 @@ int e820_add_entry(uint64_t, uint64_t, uint32_t);int e820_get_num_entries (void );bool e820_get_entry (int ,uint32_t ,uint64_t *,uint64_t *); +extern GlobalProperty pc_compat_4_0[]; +extern const size_t pc_compat_4_0_len; +extern GlobalProperty pc_compat_3_1[];extern const size_t pc_compat_3_1_len;
tests/bios-tables-test.c +9 -0
@@ -354,6 +354,15 @@ static void test_acpi_asl(test_data *data) }else { sdt->tmp_files_retain = true; exp_sdt->tmp_files_retain = true; +/* NOTE: In QEMU 4.0.1, q35 aliases "pc-q35-4.0.1" instead of +* "pc-q35-4.0". This gets encoded into the smbios tables and +* due to the extra characters changes the offset of ACPI +* pointer values allocated by SeaBIOS, causing some tests to +* report AML mismatches. Since ACPI expected file paths rely +* on the "q35" alias being used we can't easily change the +* tests to use pc-q35-4.0, so for now hopefully this friendly +* note will suffice. +*/ fprintf(stderr, "acpi-test: Warning! %.4s mismatch. " "Actual [asl:%s, aml:%s], Expected [asl:%s, aml:%s].\n",
Signature · 1 line
2.17.1
PATCH 13/97 block/file-posix: Unaligned O_DIRECT block-status
In reply to Michael RothFrom: Max Reitz <[email protected]>
Currently, qemu crashes whenever someone queries the block status of an unaligned image tail of an O_DIRECT image:
$ echo > foo $ qemu-img map --image-opts driver=file,filename=foo,cache.direct=on Offset Length Mapped to File qemu-img: block/io.c:2093: bdrv_co_block_status: Assertion `*pnum && QEMU_IS_ALIGNED(*pnum, align) && align > offset - aligned_offset' failed.
This is because bdrv_co_block_status() checks that the result returned by the driver's implementation is aligned to the request_alignment, but file-posix can fail to do so, which is actually mentioned in a comment
there: "[...] possibly including a partial sector at EOF".
Fix this by rounding up those partial sectors.
There are two possible alternative fixes:
(1) We could refuse to open unaligned image files with O_DIRECT altogether. That sounds reasonable until you realize that qcow2 does necessarily not fill up its metadata clusters, and that nobody runs qemu-img create with O_DIRECT. Therefore, unpreallocated qcow2 files usually have an unaligned image tail.
(2) bdrv_co_block_status() could ignore unaligned tails. It actually throws away everything past the EOF already, so that sounds reasonable.
Unfortunately, the block layer knows file lengths only with a granularity of BDRV_SECTOR_SIZE, so bdrv_co_block_status() usually would have to guess whether its file length information is inexact or whether the driver is broken.
Fixing what raw_co_block_status() returns is the safest thing to do.
There seems to be no other block driver that sets request_alignment and does not make sure that it always returns aligned values.
--- block/file-posix.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
block/file-posix.c +16 -0
@@ -2475,6 +2475,8 @@ static int coroutine_fn raw_co_block_status(BlockDriverState *bs,off_t data =0 , hole =0 ;int ret; +assert (QEMU_IS_ALIGNED (offset | bytes, bs->bl.request_alignment)); + ret =fd_open (bs);if (ret <0 ) {return ret;
@@ -2500,6 +2502,20 @@ static int coroutine_fn raw_co_block_status(BlockDriverState *bs,/* On a data extent, compute bytes to the end of the extent, * possibly including a partial sector at EOF. */ *pnum =MIN (bytes, hole - offset); + +/* +* We are not allowed to return partial sectors, though, so +* round up if necessary. +*/ +if (!QEMU_IS_ALIGNED (*pnum, bs->bl.request_alignment)) { +int64_t file_length =raw_getlength (bs); +if (file_length >0 ) { +/* Ignore errors, this is just a safeguard */ +assert (hole == file_length); + } + *pnum =ROUND_UP (*pnum, bs->bl.request_alignment); + } + ret =BDRV_BLOCK_DATA ; }else {/* On a hole, compute bytes to the beginning of the next extent. */
Signature · 1 line
2.17.1
PATCH 20/97 QEMUMachine: add events_wait method
In reply to Michael RothFrom: John Snow <[email protected]>
Instead of event_wait which looks for a single event, add an events_wait which can look for any number of events simultaneously. However, it will still only return one at a time, whichever happens first.
Message-id: [email protected]
--- python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------ 1 file changed, 49 insertions(+), 20 deletions(-)
python/qemu/__init__.py +49 -20
@@ -408,42 +408,71 @@ class QEMUMachine(object): self._qmp.clear_events() return events - def event_wait(self, name, timeout=60.0, match=None): + @staticmethod + def event_match(event, match=None): """ - Wait for specified timeout on named event in QMP; optionally filter - results by match. + Check if an event matches optional match criteria. - The 'match' is checked to be a recursive subset of the 'event'; skips - branch processing on match's value None - {"foo": {"bar": 1}} matches {"foo": None} - {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}} + The match criteria takes the form of a matching subdict. The event is + checked to be a superset of the subdict, recursively, with matching + values whenever those values are not None. + + Examples, with the subdict queries on the left: + - None matches any object. + - {"foo": None} matches {"foo": {"bar": 1}} + - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}} + - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}} """ - def event_match(event, match=None): - if match is None: - return True + if match is None: + return True - for key in match: - if key in event: - if isinstance(event[key], dict): - if not event_match(event[key], match[key]): - return False - elif event[key] != match[key]: + for key in match: + if key in event: + if isinstance(event[key], dict): + if not QEMUMachine.event_match(event[key], match[key]): return False - else: + elif event[key] != match[key]: return False + else: + return False + return True - return True + def event_wait(self, name, timeout=60.0, match=None): + """ + event_wait waits for and returns a named event from QMP with a timeout. + + name: The event to wait for. + timeout: QEMUMonitorProtocol.pull_event timeout parameter. + match: Optional match criteria. See event_match for details. + """ + return self.events_wait([(name, match)], timeout) + + def events_wait(self, events, timeout=60.0): + """ + events_wait waits for and returns a named event from QMP with a timeout. + + events: a sequence of (name, match_criteria) tuples. + The match criteria are optional and may be None. + See event_match for details. + timeout: QEMUMonitorProtocol.pull_event timeout parameter. + """ + def _match(event): + for name, match in events: + if (event['event'] == name and + self.event_match(event, match)): + return True + return False # Search cached events for event in self._events: - if (event['event'] == name) and event_match(event, match): + if _match(event): self._events.remove(event) return event # Poll for new events while True: event = self._qmp.pull_event(wait=timeout) - if (event['event'] == name) and event_match(event, match): + if _match(event): return event self._events.append(event)
Signature · 1 line
2.17.1
PATCH 30/97 target/ppc: Fix lxvw4x, lxvh8x and lxvb16x
In reply to Michael RothFrom: Anton Blanchard <[email protected]>
During the conversion these instructions were incorrectly treated as stores. We need to use set_cpu_vsr* and not get_cpu_vsr*.
Message-Id: <[email protected]>
(upstream commit 2a1224359008e23b051b7b45be4789afa0269f8c)
--- target/ppc/translate/vsx-impl.inc.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
target/ppc/translate/vsx-impl.inc.c +7 -6
@@ -102,8 +102,7 @@ static void gen_lxvw4x(DisasContext *ctx) } xth =tcg_temp_new_i64 (); xtl =tcg_temp_new_i64 (); -get_cpu_vsrh (xth,xT (ctx->opcode)); -get_cpu_vsrl (xtl,xT (ctx->opcode)); +gen_set_access_type (ctx,ACCESS_INT );EA =tcg_temp_new ();
@@ -126,6 +125,8 @@ static void gen_lxvw4x(DisasContext *ctx)tcg_gen_addi_tl (EA ,EA ,8 );tcg_gen_qemu_ld_i64 (xtl,EA , ctx->mem_idx,MO_BEQ ); } +set_cpu_vsrh (xT (ctx->opcode), xth); +set_cpu_vsrl (xT (ctx->opcode), xtl);tcg_temp_free (EA );tcg_temp_free_i64 (xth);tcg_temp_free_i64 (xtl);
@@ -185,8 +186,6 @@ static void gen_lxvh8x(DisasContext *ctx) } xth =tcg_temp_new_i64 (); xtl =tcg_temp_new_i64 (); -get_cpu_vsrh (xth,xT (ctx->opcode)); -get_cpu_vsrl (xtl,xT (ctx->opcode));gen_set_access_type (ctx,ACCESS_INT );EA =tcg_temp_new ();
@@ -197,6 +196,8 @@ static void gen_lxvh8x(DisasContext *ctx)if (ctx->le_mode) {gen_bswap16x8 (xth, xtl, xth, xtl); } +set_cpu_vsrh (xT (ctx->opcode), xth); +set_cpu_vsrl (xT (ctx->opcode), xtl);tcg_temp_free (EA );tcg_temp_free_i64 (xth);tcg_temp_free_i64 (xtl);
@@ -214,14 +215,14 @@ static void gen_lxvb16x(DisasContext *ctx) } xth =tcg_temp_new_i64 (); xtl =tcg_temp_new_i64 (); -get_cpu_vsrh (xth,xT (ctx->opcode)); -get_cpu_vsrl (xtl,xT (ctx->opcode));gen_set_access_type (ctx,ACCESS_INT );EA =tcg_temp_new ();gen_addr_reg_index (ctx,EA );tcg_gen_qemu_ld_i64 (xth,EA , ctx->mem_idx,MO_BEQ );tcg_gen_addi_tl (EA ,EA ,8 );tcg_gen_qemu_ld_i64 (xtl,EA , ctx->mem_idx,MO_BEQ ); +set_cpu_vsrh (xT (ctx->opcode), xth); +set_cpu_vsrl (xT (ctx->opcode), xtl);tcg_temp_free (EA );tcg_temp_free_i64 (xth);tcg_temp_free_i64 (xtl);
Signature · 1 line
2.17.1
PATCH 40/97 sphinx: add qmp_lexer
In reply to Michael RothFrom: John Snow <[email protected]>
Sphinx, through Pygments, does not like annotated json examples very much. In some versions of Sphinx (1.7), it will render the non-json portions of code blocks in red, but in newer versions (2.0) it will throw an exception and not highlight the block at all. Though we can suppress this warning, it doesn't bring back highlighting on non-strict json blocks.
We can alleviate this by creating a custom lexer for QMP examples that allows us to properly highlight these examples in a robust way, keeping our directionality and elision notations.
Message-id: [email protected]
--- docs/conf.py | 4 ++-- docs/sphinx/qmp_lexer.py | 43 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 2 deletions(-) create mode 100644 docs/sphinx/qmp_lexer.py
docs/conf.py +2 -2
@@ -41,7 +41,7 @@ except NameError: # add these directories to sys.path here. If the directory is relative to the # documentation root, use an absolute path starting from qemu_docdir. # -# sys.path.insert(0, os.path.join(qemu_docdir, "my_subdir")) +sys.path.insert(0, os.path.join(qemu_docdir, "sphinx")) # -- General configuration ------------------------------------------------
@@ -54,7 +54,7 @@ needs_sphinx = '1.3' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = [] +extensions = ['qmp_lexer'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates']
docs/sphinx/qmp_lexer.py new file +43 -0
43 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH 25/97 vhost: fix vhost_log size overflow during migration
In reply to Michael RothFrom: Li Hangjing <[email protected]>
When a guest which doesn't support multiqueue is migrated with a multi queues vhost-user-blk deivce, a crash will occur like:
0 qemu_memfd_alloc (name=<value optimized out>, size=562949953421312, seals=<value optimized out>, fd=0x7f87171fe8b4, errp=0x7f87171fe8a8) at util/memfd.c:153 1 0x00007f883559d7cf in vhost_log_alloc (size=70368744177664, share=true) at hw/virtio/vhost.c:186
2 0x00007f88355a0758 in vhost_log_get (listener=0x7f8838bd7940, enable=1) at qemu-2-12/hw/virtio/vhost.c:211
3 vhost_dev_log_resize (listener=0x7f8838bd7940, enable=1) at hw/virtio/vhost.c:263
4 vhost_migration_log (listener=0x7f8838bd7940, enable=1) at hw/virtio/vhost.c:787
5 0x00007f88355463d6 in memory_global_dirty_log_start () at memory.c:2503
6 0x00007f8835550577 in ram_init_bitmaps (f=0x7f88384ce600, opaque=0x7f8836024098) at migration/ram.c:2173
7 ram_init_all (f=0x7f88384ce600, opaque=0x7f8836024098) at migration/ram.c:2192
8 ram_save_setup (f=0x7f88384ce600, opaque=0x7f8836024098) at migration/ram.c:2219
9 0x00007f88357a419d in qemu_savevm_state_setup (f=0x7f88384ce600) at migration/savevm.c:1002
10 0x00007f883579fc3e in migration_thread (opaque=0x7f8837530400) at migration/migration.c:2382
11 0x00007f8832447893 in start_thread () from /lib64/libpthread.so.0
12 0x00007f8832178bfd in clone () from /lib64/libc.so.6
This is because vhost_get_log_size() returns a overflowed vhost-log size. In this function, it uses the uninitialized variable vqs->used_phys and vqs->used_size to get the vhost-log size.
Message-Id: <[email protected]>
--- hw/virtio/vhost.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
hw/virtio/vhost.c +10 -0
@@ -131,6 +131,11 @@ static int vhost_sync_dirty_bitmap(struct vhost_dev *dev, }for (i =0 ; i < dev->nvqs; ++i) {struct vhost_virtqueue *vq = dev->vqs + i; + +if (!vq->used_phys && !vq->used_size) { +continue ; + } +vhost_dev_sync_region (dev, section, start_addr, end_addr, vq->used_phys,range_get_last (vq->used_phys, vq->used_size)); }
@@ -168,6 +173,11 @@ static uint64_t vhost_get_log_size(struct vhost_dev *dev) }for (i =0 ; i < dev->nvqs; ++i) {struct vhost_virtqueue *vq = dev->vqs + i; + +if (!vq->used_phys && !vq->used_size) { +continue ; + } +uint64_t last = vq->used_phys + vq->used_size -1 ; log_size =MAX (log_size, last /VHOST_LOG_CHUNK +1 ); }
Signature · 1 line
2.17.1
PATCH 33/97 target/i386: add MDS-NO feature
In reply to Michael RothFrom: Paolo Bonzini <[email protected]>
Microarchitectural Data Sampling is a hardware vulnerability which allows unprivileged speculative access to data which is available in various CPU internal buffers.
Some Intel processors use the ARCH_CAP_MDS_NO bit in the IA32_ARCH_CAPABILITIES
MSR to report that they are not vulnerable, make it available to guests.
Message-Id: <[email protected]>
--- target/i386/cpu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
target/i386/cpu.c +1 -1
@@ -1183,7 +1183,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = { .type = MSR_FEATURE_WORD, .feat_names = {"rdctl-no" ,"ibrs-all" ,"rsba" ,"skip-l1dfl-vmentry" , -"ssb-no" ,NULL ,NULL ,NULL , +"ssb-no" ,"mds-no" ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , NULL, NULL, NULL, NULL,
Signature · 1 line
2.17.1
PATCH 29/97 target/ppc: Fix vsum2sws
In reply to Michael RothFrom: Anton Blanchard <[email protected]>
A recent cleanup changed the pre zeroing of the result from 64 bit to 32 bit operations:
- result.u64[i] = 0;
+ result.VsrW(i) = 0;
This corrupts the result.
Message-Id: <[email protected]>
(upstream commit 7fa0ddc1d63806769d1b6246a62708d3bde39037)
--- target/ppc/int_helper.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
target/ppc/int_helper.c +1 -1
@@ -2030,7 +2030,7 @@ void helper_vsum2sws(CPUPPCState *env, ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b)for (i =0 ; i <ARRAY_SIZE (r->u64); i++) {int64_t t = (int64_t )b->VsrSW (upper + i *2 ); - result.VsrW (i) =0 ; + result.VsrD (i) =0 ;for (j =0 ; j <ARRAY_SIZE (r->u64); j++) { t += a->VsrSW (2 * i + j); }
Signature · 1 line
2.17.1
cover Patch Round-up for stable 4.0.1, freeze on 2019-10-10
Hi everyone,
The following new patches are queued for QEMU stable v4.0.1:
https://github.com/mdroth/qemu/commits/stable-4.0-staging
The release is planned for 2019-10-17:
https://wiki.qemu.org/Planning/4.0
Please respond here or CC [email protected] on any patches you think should be included in the release.
Note that this update falls outside the normal stable release support window (~1 development cycle), but is being released now since it was delayed from its intended release date.
Thanks!
----------------------------------------------------------------
Alberto Garcia (1):
qcow2: Fix the calculation of the maximum L2 cache size
Alex Williamson (1):
q35: Revert to kernel irqchip
Anthony PERARD (1):
xen-bus: Fix backend state transition on device reset
Anton Blanchard (5):
target/ppc: Fix xvabs[sd]p, xvnabs[sd]p, xvneg[sd]p, xvcpsgn[sd]p target/ppc: Fix xvxsigdp
target/ppc: Fix xxbrq, xxbrw
target/ppc: Fix vsum2sws
target/ppc: Fix lxvw4x, lxvh8x and lxvb16x
Christian Borntraeger (1):
s390x/cpumodel: ignore csske for expansion
Cédric Le Goater (1):
spapr/xive: fix EQ page addresses above 64GB
Daniel P. Berrangé (1):
docs: recommend use of md-clear feature on all Intel CPUs
David Hildenbrand (7):
virtio-balloon: Fix wrong sign extension of PFNs
virtio-balloon: Fix QEMU crashes on pagesize > BALLOON_PAGE_SIZE
virtio-balloon: Simplify deflate with pbp
virtio-balloon: Better names for offset variables in inflate/deflate code
virtio-balloon: Rework pbp tracking data
virtio-balloon: Use temporary PBP only
virtio-balloon: don't track subpages for the PBP
Eric Blake (1):
cutils: Fix size_to_str() on 32-bit platforms
Evgeny Yakovlev (2):
i386/acpi: show PCI Express bus on pxb-pcie expanders i386/acpi: fix gint overflow in crs_range_compare
Gerd Hoffmann (2):
kbd-state: fix autorepeat handling
usb-tablet: fix serial compat property
Jan Kiszka (1):
ioapic: kvm: Skip route updates for masked pins
Johannes Berg (1):
libvhost-user: fix SLAVE_SEND_FD handling
John Snow (13):
Makefile: add nit-picky mode to sphinx-build docs/interop/bitmaps: rewrite and modernize doc
blockdev-backup: don't check aio_context too early
iotests.py: do not use infinite waits
QEMUMachine: add events_wait method
iotests.py: rewrite run_job to be pickier
iotests: add iotest 256 for testing blockdev-backup across iothread contexts migration/dirty-bitmaps: change bitmap enumeration method docs/interop/bitmaps.rst: Fix typos
sphinx: add qmp_lexer
docs/bitmaps: use QMP lexer instead of json
Revert "ide/ahci: Check for -ECANCELED in aio callbacks"
iotests: add testing shim for script-style python tests
Kevin Wolf (7):
qcow2: Avoid COW during metadata preallocation
qcow2: Add errp to preallocate_co()
qcow2: Fix full preallocation with external data file
qcow2: Fix qcow2_make_empty() with external data file
block: Fix AioContext switch for bs->drv == NULL
block: Drain source node in bdrv_replace_node()
iotests: Test commit job start with concurrent I/O
Li Hangjing (1):
vhost: fix vhost_log size overflow during migration
Marc-André Lureau (2):
virtio-pci: fix missing device properties
usbredir: fix buffer-overflow on vmload
Markus Armbruster (2):
vl: Fix -drive / -blockdev persistent reservation management
pr-manager: Fix invalid g_free() crash bug
Matthew Rosato (1):
s390: PCI: fix IOMMU region init
Max Reitz (20):
block/file-posix: Unaligned O_DIRECT block-status
iotests: Test unaligned raw images with O_DIRECT iotests.py: Fix VM.run_job
backup: Copy only dirty areas
iotests: Test backup job with two guest writes
iotests: Test incremental backup after truncation
mirror: Only mirror granularity-aligned chunks
iotests: Test unaligned blocking mirror write
vpc: Return 0 from vpc_co_create() on success
iotests: Add supported protocols to execute_test()
iotests: Restrict file Python tests to file
iotests: Restrict nbd Python tests to nbd
iotests: Test blockdev-create for vpc
curl: Keep pointer to the CURLState in CURLSocket
curl: Keep *socket until the end of curl_sock_cb()
curl: Check completion in curl_multi_do()
curl: Pass CURLSocket to curl_multi_do()
curl: Report only ready sockets
curl: Handle success in multi_check_completion
curl: Check curl_multi_add_handle()'s return code
Michael Roth (2):
slirp: Fix heap overflow in ip_reass on big packet input
slirp: ip_reass: Fix use after free
Michael S. Tsirkin (1):
virtio-balloon: free pbp more aggressively
Paolo Bonzini (4):
target/i386: add MDS-NO feature
target/i386: define md-clear bit
dma-helpers: ensure AIO callback is invoked after cancellation
scsi: lsi: exit infinite loop while executing script (CVE-2019-12068)
Peter Lieven (2):
megasas: fix mapped frame size
block/nfs: tear down aio before nfs_close
Peter Maydell (1):
target/arm: Don't abort on M-profile exception return in linux-user mode
Philippe Mathieu-Daudé (4):
hw/ssi/xilinx_spips: Convert lqspi_read() to read_with_attrs hw/ssi/xilinx_spips: Avoid AXI writes to the LQSPI linear memory hw/ssi/xilinx_spips: Avoid out-of-bound access to lqspi_buf[] block/create: Do not abort if a block driver is not available
Sergio Lopez (1):
blockjob: update nodes head while removing all bdrv
Stefan Berger (2):
tpm: Exit in reset when backend indicates failure
tpm_emulator: Translate TPM error codes to strings
Stefan Hajnoczi (1):
virtio-balloon: fix QEMU 4.0 config size migration incompatibility
Thomas Huth (1):
hw/core/loader: Fix possible crash in rom_copy()
Vladimir Sementsov-Ogievskiy (7):
block/backup: simplify backup_incremental_init_copy_bitmap block/backup: move to copy_bitmap with granularity block/backup: refactor and tolerate unallocated cluster skipping block/backup: unify different modes code path block/backup: refactor: split out backup_calculate_cluster_size util/hbitmap: update orig_size on truncate block/backup: disable copy_range for compressed backup
Makefile | 2 +- block.c | 19 +- block/backup.c | 258 +++--- block/create.c | 6 +- block/curl.c | 133 ++- block/file-posix.c | 16 + block/mirror.c | 29 + block/nfs.c | 6 +- block/qcow2.c | 59 +- block/vpc.c | 3 +- blockdev.c | 4 - blockjob.c | 17 +- contrib/libvhost-user/libvhost-user.c | 3 +- dma-helpers.c | 13 +- docs/conf.py | 4 +- docs/interop/bitmaps.rst | 1585 ++++++++++++++++++++++++++------- docs/qemu-cpu-models.texi | 12 + docs/sphinx/qmp_lexer.py | 43 + hw/core/loader.c | 2 +- hw/core/machine.c | 8 +- hw/i386/acpi-build.c | 17 +- hw/i386/pc.c | 3 + hw/i386/pc_q35.c | 16 +- hw/ide/ahci.c | 3 - hw/ide/core.c | 14 - hw/intc/ioapic.c | 8 +- hw/intc/spapr_xive.c | 3 +- hw/intc/xive.c | 9 +- hw/s390x/s390-pci-bus.c | 7 +- hw/scsi/lsi53c895a.c | 41 +- hw/scsi/megasas.c | 2 +- hw/ssi/xilinx_spips.c | 43 +- hw/tpm/tpm_crb.c | 4 +- hw/tpm/tpm_emulator.c | 60 +- hw/tpm/tpm_int.h | 13 + hw/tpm/tpm_tis.c | 4 +- hw/usb/redirect.c | 5 + hw/virtio/vhost.c | 10 + hw/virtio/virtio-balloon.c | 143 +-- hw/virtio/virtio-pci.c | 28 +- hw/xen/xen-bus.c | 23 +- include/hw/boards.h | 3 + include/hw/i386/pc.h | 3 + include/hw/ppc/xive_regs.h | 6 + include/hw/virtio/virtio-balloon.h | 5 +- migration/block-dirty-bitmap.c | 14 +- python/qemu/__init__.py | 69 +- scsi/pr-manager.c | 1 - slirp/src/ip_input.c | 19 +- target/arm/translate.c | 21 +- target/i386/cpu.c | 4 +- target/ppc/int_helper.c | 2 +- target/ppc/translate/vsx-impl.inc.c | 23 +- target/s390x/cpu_models.c | 2 + tests/bios-tables-test.c | 9 + tests/qemu-iotests/030 | 3 +- tests/qemu-iotests/040 | 3 +- tests/qemu-iotests/041 | 3 +- tests/qemu-iotests/044 | 3 +- tests/qemu-iotests/045 | 3 +- tests/qemu-iotests/055 | 3 +- tests/qemu-iotests/056 | 42 +- tests/qemu-iotests/056.out | 4 +- tests/qemu-iotests/057 | 3 +- tests/qemu-iotests/065 | 3 +- tests/qemu-iotests/096 | 3 +- tests/qemu-iotests/118 | 3 +- tests/qemu-iotests/124 | 41 +- tests/qemu-iotests/124.out | 4 +- tests/qemu-iotests/129 | 3 +- tests/qemu-iotests/132 | 3 +- tests/qemu-iotests/139 | 3 +- tests/qemu-iotests/147 | 5 +- tests/qemu-iotests/148 | 3 +- tests/qemu-iotests/151 | 28 +- tests/qemu-iotests/151.out | 4 +- tests/qemu-iotests/152 | 3 +- tests/qemu-iotests/155 | 3 +- tests/qemu-iotests/163 | 3 +- tests/qemu-iotests/165 | 3 +- tests/qemu-iotests/169 | 3 +- tests/qemu-iotests/196 | 3 +- tests/qemu-iotests/199 | 3 +- tests/qemu-iotests/205 | 3 +- tests/qemu-iotests/221 | 4 + tests/qemu-iotests/245 | 3 +- tests/qemu-iotests/253 | 84 ++ tests/qemu-iotests/253.out | 14 + tests/qemu-iotests/255 | 83 ++ tests/qemu-iotests/255.out | 16 + tests/qemu-iotests/256 | 122 +++ tests/qemu-iotests/256.out | 119 +++ tests/qemu-iotests/266 | 153 ++++ tests/qemu-iotests/266.out | 137 +++ tests/qemu-iotests/group | 4 + tests/qemu-iotests/iotests.py | 106 ++- ui/kbd-state.c | 6 +- util/cutils.c | 2 +- util/hbitmap.c | 6 +- vl.c | 3 +- 100 files changed, 3022 insertions(+), 893 deletions(-) create mode 100644 docs/sphinx/qmp_lexer.py create mode 100755 tests/qemu-iotests/253 create mode 100644 tests/qemu-iotests/253.out create mode 100755 tests/qemu-iotests/255 create mode 100644 tests/qemu-iotests/255.out create mode 100755 tests/qemu-iotests/256 create mode 100644 tests/qemu-iotests/256.out create mode 100755 tests/qemu-iotests/266 create mode 100644 tests/qemu-iotests/266.out
PATCH 37/97 usbredir: fix buffer-overflow on vmload
From: Marc-André Lureau <[email protected]>
If interface_count is NO_INTERFACE_INFO, let's not access the arrays out-of-bounds.
==994==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x625000243930 at pc 0x5642068086a8 bp 0x7f0b6f9ffa50 sp 0x7f0b6f9ffa40 READ of size 1 at 0x625000243930 thread T0
#0 0x5642068086a7 in usbredir_check_bulk_receiving /home/elmarco/src/qemu/hw/usb/redirect.c:1503
#1 0x56420681301c in usbredir_post_load /home/elmarco/src/qemu/hw/usb/redirect.c:2154
#2 0x5642068a56c2 in vmstate_load_state /home/elmarco/src/qemu/migration/vmstate.c:168
#3 0x56420688e2ac in vmstate_load /home/elmarco/src/qemu/migration/savevm.c:829
#4 0x5642068980cb in qemu_loadvm_section_start_full /home/elmarco/src/qemu/migration/savevm.c:2211
#5 0x564206899645 in qemu_loadvm_state_main /home/elmarco/src/qemu/migration/savevm.c:2395
#6 0x5642068998cf in qemu_loadvm_state /home/elmarco/src/qemu/migration/savevm.c:2467
#7 0x56420685f3e9 in process_incoming_migration_co /home/elmarco/src/qemu/migration/migration.c:449
#8 0x564207106c47 in coroutine_trampoline /home/elmarco/src/qemu/util/coroutine-ucontext.c:115
#9 0x7f0c0604e37f (/lib64/libc.so.6+0x4d37f)Message-id: [email protected]
--- hw/usb/redirect.c | 5 +++++ 1 file changed, 5 insertions(+)
hw/usb/redirect.c +5 -0
@@ -1494,6 +1494,11 @@ static void usbredir_check_bulk_receiving(USBRedirDevice *dev)for (i =EP2I (USB_DIR_IN ); i <MAX_ENDPOINTS ; i++) { dev->endpoint[i].bulk_receiving_enabled =0 ; } + +if (dev->interface_info.interface_count ==NO_INTERFACE_INFO ) { +return ; + } +for (i =0 ; i < dev->interface_info.interface_count; i++) { quirks = usb_get_quirks(dev->device_info.vendor_id, dev->device_info.product_id,
Signature · 1 line
2.17.1
PATCH 2/97 qcow2: Add errp to preallocate_co()
In reply to Michael RothFrom: Kevin Wolf <[email protected]>
We'll add a bdrv_co_truncate() call in the next patch which can return an Error that we don't want to discard. So add an errp parameter to preallocate_co().
--- block/qcow2.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
block/qcow2.c +6 -5
@@ -2721,7 +2721,7 @@ static int qcow2_set_up_encryption(BlockDriverState *bs,* Returns: 0 on success, -errno on failure. */ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset, -uint64_t new_length) +uint64_t new_length,Error **errp) {BDRVQcow2State *s = bs->opaque;uint64_t bytes;
@@ -2738,6 +2738,7 @@ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset, ret =qcow2_alloc_cluster_offset (bs, offset, &cur_bytes, &host_offset, &meta);if (ret <0 ) { +error_setg_errno (errp, -ret,"Allocating clusters failed" );return ret; }
@@ -2746,6 +2747,7 @@ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset, ret =qcow2_alloc_cluster_link_l2 (bs, meta);if (ret <0 ) { +error_setg_errno (errp, -ret,"Mapping clusters failed" );qcow2_free_any_clusters (bs, meta->alloc_offset, meta->nb_clusters,QCOW2_DISCARD_NEVER );return ret;
@@ -2775,6 +2777,7 @@ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset, ret =bdrv_pwrite (s->data_file, (host_offset + cur_bytes) -1 , &data,1 );if (ret <0 ) { +error_setg_errno (errp, -ret,"Writing to EOF failed" );return ret; } }
@@ -3748,9 +3751,8 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,break ;case PREALLOC_MODE_METADATA : - ret =preallocate_co (bs, old_length, offset); + ret =preallocate_co (bs, old_length, offset, errp);if (ret <0 ) { -error_setg_errno (errp, -ret,"Preallocation failed" ); goto fail; }break ;
@@ -3766,9 +3768,8 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,/* With a data file, preallocation means just allocating the metadata * and forwarding the truncate request to the data file */ if (has_data_file (bs)) { - ret =preallocate_co (bs, old_length, offset); + ret =preallocate_co (bs, old_length, offset, errp);if (ret <0 ) { -error_setg_errno (errp, -ret,"Preallocation failed" ); goto fail; }break ;
Signature · 1 line
2.17.1
PATCH 32/97 vl: Fix -drive / -blockdev persistent reservation management
In reply to Michael RothFrom: Markus Armbruster <[email protected]>
qemu-system-FOO's main() acts on command line arguments in its own idiosyncratic order. There's not much method to its madness. Whenever we find a case where one kind of command line argument needs to refer to something created for another kind later, we rejigger the order.
Recent commit cda4aa9a5a "vl: Create block backends before setting machine properties" was such a rejigger. Block backends are now created before "delayed" objects. This broke persistent reservation management. Reproducer:
$ qemu-system-x86_64 -object pr-manager-helper,id=pr-helper0,path=/tmp/pr-helper0.sock-drive -drive file=/dev/mapper/crypt,file.pr-manager=pr-helper0,format=raw,if=none,id=drive-scsi0-0-0-2
qemu-system-x86_64: -drive file=/dev/mapper/crypt,file.pr-manager=pr-helper0,format=raw,if=none,id=drive-scsi0-0-0-2: No persistent reservation manager with id 'pr-helper0'The delayed pr-manager-helper object is created too late for use by -drive or -blockdev. Normal objects are still created in time.
pr-manager-helper has always been a delayed object (commit 7c9e527659 "scsi, file-posix: add support for persistent reservation management"). Turns out there's no real reason for that. Make it a normal object.
Message-Id: <[email protected]>
--- vl.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
vl.c +1 -2
@@ -2825,8 +2825,7 @@ static bool object_create_initial(const char *type, QemuOpts *opts)exit (0 ); } -if (g_str_equal (type,"rng-egd" ) || -g_str_has_prefix (type,"pr-manager-" )) { +if (g_str_equal (type,"rng-egd" )) {return false; }
Signature · 1 line
2.17.1
PATCH 28/97 target/ppc: Fix xxbrq, xxbrw
In reply to Michael RothFrom: Anton Blanchard <[email protected]>
Fix a typo in xxbrq and xxbrw where we put both results into the lower doubleword.
Message-Id: <[email protected]>
(upstream commit d47a751adab7833e9831408376077bc8dba41d5d)
--- target/ppc/translate/vsx-impl.inc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
target/ppc/translate/vsx-impl.inc.c +2 -2
@@ -1192,7 +1192,7 @@ static void gen_xxbrq(DisasContext *ctx)tcg_gen_bswap64_i64 (xtl, xbh);set_cpu_vsrl (xT (ctx->opcode), xtl);tcg_gen_mov_i64 (xth, t0); -set_cpu_vsrl (xT (ctx->opcode), xth); +set_cpu_vsrh (xT (ctx->opcode), xth);tcg_temp_free_i64 (t0);tcg_temp_free_i64 (xth);
@@ -1220,7 +1220,7 @@ static void gen_xxbrw(DisasContext *ctx)get_cpu_vsrl (xbl,xB (ctx->opcode));gen_bswap32x4 (xth, xtl, xbh, xbl); -set_cpu_vsrl (xT (ctx->opcode), xth); +set_cpu_vsrh (xT (ctx->opcode), xth);set_cpu_vsrl (xT (ctx->opcode), xtl);tcg_temp_free_i64 (xth);
Signature · 1 line
2.17.1
PATCH 43/97 hw/ssi/xilinx_spips: Avoid AXI writes to the LQSPI linear memory
In reply to Michael RothFrom: Philippe Mathieu-Daudé <[email protected]>
Lei Sun found while auditing the code that a CPU write would trigger a NULL pointer dereference.
Quoted 1 line
From UG1085 datasheet [*] AXI writes in this region are ignored
and generates an AXI Slave Error (SLVERR).
Fix by implementing the write_with_attrs() handler. Return MEMTX_ERROR when the region is accessed (this error maps to an AXI slave error).
[*] https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
--- hw/ssi/xilinx_spips.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
hw/ssi/xilinx_spips.c +16 -0
@@ -1217,8 +1217,24 @@ static MemTxResult lqspi_read(void *opaque, hwaddr addr, uint64_t *value,return lqspi_read (opaque, addr, value, size, attrs); } +static MemTxResult lqspi_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size,MemTxAttrs attrs) +{ +/* +* From UG1085, Chapter 24 (Quad-SPI controllers): +* - Writes are ignored +* - AXI writes generate an external AXI slave error (SLVERR) +*/ +qemu_log_mask (LOG_GUEST_ERROR ,"%s Unexpected %u-bit access to 0x%" PRIx64 +" (value: 0x%" PRIx64"\n" , + __func__, size <<3 , offset, value); + +return MEMTX_ERROR ; +} +static const MemoryRegionOps lqspi_ops = { .read_with_attrs = lqspi_read, + .write_with_attrs = lqspi_write, .endianness =DEVICE_NATIVE_ENDIAN , .valid = { .min_access_size =1 ,
Signature · 1 line
2.17.1
PATCH 39/97 docs/interop/bitmaps.rst: Fix typos
In reply to Michael RothFrom: John Snow <[email protected]>
Pygments and Sphinx get pickier all the time; Sphinx 2.1+ now catches these errors.
Message-id: [email protected]
--- docs/interop/bitmaps.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
docs/interop/bitmaps.rst +2 -2
@@ -399,7 +399,7 @@ in any one source bitmap, the target bitmap will mark that segment dirty. "arguments": { "node": "drive0", "target": "new_bitmap", - "bitmaps: [ "bitmap0" ] + "bitmaps": [ "bitmap0" ] } }
@@ -1437,7 +1437,7 @@ applied: .. code:: json <- { - "timestamp": {...} + "timestamp": {...}, "data": { "device": "drive0", "type": "backup",
Signature · 1 line
2.17.1
PATCH 49/97 virtio-balloon: Simplify deflate with pbp
In reply to Michael RothFrom: David Hildenbrand <[email protected]>
Let's simplify this - the case we are optimizing for is very hard to trigger and not worth the effort. If we're switching from inflation to deflation, let's reset the pbp.
Message-Id: <[email protected]>
--- hw/virtio/virtio-balloon.c | 26 +++++--------------------- 1 file changed, 5 insertions(+), 21 deletions(-)
hw/virtio/virtio-balloon.c +5 -21
@@ -117,7 +117,7 @@ static void balloon_deflate_page(VirtIOBalloon *balloon,void *addr =memory_region_get_ram_ptr (mr) + offset;RAMBlock *rb;size_t rb_page_size; -ram_addr_t ram_offset, host_page_base; +ram_addr_t ram_offset;void *host_addr;int ret;
@@ -125,27 +125,11 @@ static void balloon_deflate_page(VirtIOBalloon *balloon,* host address? */ rb =qemu_ram_block_from_host (addr, false, &ram_offset); rb_page_size =qemu_ram_pagesize (rb); - host_page_base = ram_offset & ~(rb_page_size -1 ); - -if (balloon->pbp - && rb == balloon->pbp->rb - && host_page_base == balloon->pbp->base) { -int subpages = rb_page_size /BALLOON_PAGE_SIZE ; -/* -* This means the guest has asked to discard some of the 4kiB -* subpages of a host page, but then changed its mind and -* asked to keep them after all. It's exceedingly unlikely -* for a guest to do this in practice, but handle it anyway, -* since getting it wrong could mean discarding memory the -* guest is still using. */ -clear_bit ((ram_offset - balloon->pbp->base) /BALLOON_PAGE_SIZE , - balloon->pbp->bitmap); - -if (bitmap_empty (balloon->pbp->bitmap, subpages)) { -g_free (balloon->pbp); - balloon->pbp =NULL ; - } +if (balloon->pbp) { +/* Let's play safe and always reset the pbp on deflation requests. */ +g_free (balloon->pbp); + balloon->pbp =NULL ; } host_addr = (void *)((uintptr_t )addr & ~(rb_page_size -1 ));
Signature · 1 line
2.17.1
PATCH 46/97 i386/acpi: show PCI Express bus on pxb-pcie expanders
In reply to Michael RothFrom: Evgeny Yakovlev <[email protected]>
Show PCIe host bridge PNP id with PCI host bridge as a compatible id when expanding a pcie bus.
Message-Id: <[email protected]>
--- hw/i386/acpi-build.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
hw/i386/acpi-build.c +4 -1
@@ -1895,10 +1895,13 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, scope =aml_scope ("\\_SB" ); dev =aml_device ("PC%.02X" , bus_num);aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num))); -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" )));aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num)));if (pci_bus_is_express (bus)) { +aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); +aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" )));aml_append (dev,build_q35_osc_method ()); + }else { +aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" ))); }if (numa_node !=NUMA_NODE_UNASSIGNED ) {
Signature · 1 line
2.17.1
PATCH 44/97 hw/ssi/xilinx_spips: Avoid out-of-bound access to lqspi_buf[]
In reply to Michael RothFrom: Philippe Mathieu-Daudé <[email protected]>
Both lqspi_read() and lqspi_load_cache() expect a 32-bit aligned address.
Quoted 1 line
From UG1085 datasheet [*] chapter on 'Quad-SPI Controller':
Transfer Size Limitations
Because of the 32-bit wide TX, RX, and generic FIFO, all APB/AXI transfers must be an integer multiple of 4-bytes. Shorter transfers are not possible.
Set MemoryRegionOps.impl values to force 32-bit accesses, this way we are sure we do not access the lqspi_buf[] array out of bound.
[*] https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
--- hw/ssi/xilinx_spips.c | 4 ++++ 1 file changed, 4 insertions(+)
hw/ssi/xilinx_spips.c +4 -0
@@ -1236,6 +1236,10 @@ static const MemoryRegionOps lqspi_ops = { .read_with_attrs = lqspi_read, .write_with_attrs = lqspi_write, .endianness = DEVICE_NATIVE_ENDIAN, + .impl = { + .min_access_size = 4, + .max_access_size = 4, + }, .valid = { .min_access_size =1 , .max_access_size = 4
Signature · 1 line
2.17.1
PATCH 3/97 qcow2: Fix full preallocation with external data file
In reply to Michael RothFrom: Kevin Wolf <[email protected]>
preallocate_co() already gave the data file the full size without forwarding the requested preallocation mode to the protocol. When bdrv_co_truncate() was called later with the preallocation mode, the file didn't actually grow any more, so the data file stayed unallocated even if full preallocation was requested.
Pass the right preallocation mode to preallocate_co() and remove the second bdrv_co_truncate() to fix this. As a side effect, the ugly one-byte write in preallocate_co() is replaced with a truncate call, now leaving the last block unallocated on the protocol level as it should be.
--- block/qcow2.c | 41 +++++++++++++++++++++++------------------ 1 file changed, 23 insertions(+), 18 deletions(-)
block/qcow2.c +23 -18
@@ -2721,11 +2721,13 @@ static int qcow2_set_up_encryption(BlockDriverState *bs,* Returns: 0 on success, -errno on failure. */ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset, -uint64_t new_length,Error **errp) +uint64_t new_length,PreallocMode mode, +Error **errp) {BDRVQcow2State *s = bs->opaque;uint64_t bytes;uint64_t host_offset =0 ; +int64_t file_length;unsigned int cur_bytes;int ret;QCowL2Meta *meta;
@@ -2772,12 +2774,19 @@ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset,* all of the allocated clusters (otherwise we get failing reads after * EOF). Extend the image to the last allocated sector. */ -if (host_offset !=0 ) { -uint8_t data =0 ; - ret =bdrv_pwrite (s->data_file, (host_offset + cur_bytes) -1 , - &data,1 ); + file_length =bdrv_getlength (s->data_file->bs); +if (file_length <0 ) { +error_setg_errno (errp, -file_length,"Could not get file size" ); +return file_length; + } + +if (host_offset + cur_bytes > file_length) { +if (mode ==PREALLOC_MODE_METADATA ) { + mode =PREALLOC_MODE_OFF ; + } + ret =bdrv_co_truncate (s->data_file, host_offset + cur_bytes, mode, + errp);if (ret <0 ) { -error_setg_errno (errp, -ret,"Writing to EOF failed" );return ret; } }
@@ -3748,10 +3757,16 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,switch (prealloc) {case PREALLOC_MODE_OFF : +if (has_data_file (bs)) { + ret =bdrv_co_truncate (s->data_file, offset, prealloc, errp); +if (ret <0 ) { + goto fail; + } + }break ;case PREALLOC_MODE_METADATA : - ret =preallocate_co (bs, old_length, offset, errp); + ret =preallocate_co (bs, old_length, offset, prealloc, errp);if (ret <0 ) { goto fail; }
@@ -3768,7 +3783,7 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,/* With a data file, preallocation means just allocating the metadata * and forwarding the truncate request to the data file */ if (has_data_file (bs)) { - ret =preallocate_co (bs, old_length, offset, errp); + ret =preallocate_co (bs, old_length, offset, prealloc, errp);if (ret <0 ) { goto fail; }
@@ -3883,16 +3898,6 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, bs->total_sectors = offset /BDRV_SECTOR_SIZE ; -if (has_data_file (bs)) { -if (prealloc ==PREALLOC_MODE_METADATA ) { - prealloc =PREALLOC_MODE_OFF ; - } - ret =bdrv_co_truncate (s->data_file, offset, prealloc, errp); -if (ret <0 ) { - goto fail; - } - } -/* write updated header.size */ offset =cpu_to_be64 (offset); ret = bdrv_pwrite_sync(bs->file, offsetof(QCowHeader, size),
Signature · 1 line
2.17.1
PATCH 36/97 virtio-pci: fix missing device properties
In reply to Michael RothFrom: Marc-André Lureau <[email protected]>
Since commit a4ee4c8baa37154 ("virtio: Helper for registering virtio device types"), virtio-gpu-pci, virtio-vga, and virtio-crypto-pci lost some properties: "ioeventfd" and "vectors". This may cause various issues, such as failing migration or invalid properties.
Since those VirtioPCI devices do not have a base name, their class are initialized with virtio_pci_generic_base_class_init(). However, if the VirtioPCIDeviceTypeInfo provided a class_init which sets dc->props, the properties were overwritten by virtio_pci_generic_class_init().
Instead, introduce an intermediary base-type to register the generic properties.
Message-Id: <[email protected]>
--- hw/virtio/virtio-pci.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
hw/virtio/virtio-pci.c +14 -14
@@ -1905,13 +1905,6 @@ static void virtio_pci_generic_class_init(ObjectClass *klass, void *data) dc->props = virtio_pci_generic_properties; } -/* Used when the generic type and the base type is the same */ -static void virtio_pci_generic_base_class_init (ObjectClass *klass,void *data) -{ -virtio_pci_base_class_init (klass, data); -virtio_pci_generic_class_init (klass,NULL ); -} -static void virtio_pci_transitional_instance_init (Object *obj) {VirtIOPCIProxy *proxy =VIRTIO_PCI (obj);
@@ -1930,14 +1923,13 @@ static void virtio_pci_non_transitional_instance_init(Object *obj)void virtio_pci_types_register (const VirtioPCIDeviceTypeInfo *t) { +char *base_name =NULL ;TypeInfo base_type_info = { .name = t->base_name, .parent = t->parent ? t->parent :TYPE_VIRTIO_PCI , .instance_size = t->instance_size, .instance_init = t->instance_init, .class_size = t->class_size, - .class_init = virtio_pci_base_class_init, - .class_data = (void *)t, .abstract = true, };TypeInfo generic_type_info = {
@@ -1953,13 +1945,20 @@ void virtio_pci_types_register(const VirtioPCIDeviceTypeInfo *t)if (!base_type_info.name) {/* No base type -> register a single generic device type */ - base_type_info.name = t->generic_name; - base_type_info.class_init = virtio_pci_generic_base_class_init; - base_type_info.interfaces = generic_type_info.interfaces; - base_type_info.abstract = false; - generic_type_info.name =NULL ; +/* use intermediate %s-base-type to add generic device props */ + base_name =g_strdup_printf ("%s-base-type" , t->generic_name); + base_type_info.name = base_name; + base_type_info.class_init = virtio_pci_generic_class_init; + + generic_type_info.parent = base_name; + generic_type_info.class_init = virtio_pci_base_class_init; + generic_type_info.class_data = (void *)t; +assert (!t->non_transitional_name);assert (!t->transitional_name); + }else { + base_type_info.class_init = virtio_pci_base_class_init; + base_type_info.class_data = (void *)t; }type_register (&base_type_info);
@@ -1997,6 +1996,7 @@ void virtio_pci_types_register(const VirtioPCIDeviceTypeInfo *t) };type_register (&transitional_type_info); } +g_free (base_name); }/* virtio-pci-bus */
Signature · 1 line
2.17.1
PATCH 35/97 docs: recommend use of md-clear feature on all Intel CPUs
In reply to Michael RothFrom: Daniel P. Berrangé <[email protected]>
Update x86 CPU model guidance to recommend that the md-clear feature is manually enabled with all Intel CPU models, when supported by the host microcode.
Message-Id: <[email protected]>
--- docs/qemu-cpu-models.texi | 12 ++++++++++++ 1 file changed, 12 insertions(+)
docs/qemu-cpu-models.texi +12 -0
@@ -200,6 +200,18 @@ Not included by default in any Intel CPU model. Should be explicitly turned on for all Intel CPU models. Note that not all CPU hardware will support this feature. + +@item @code{md-clear} + +Required to confirm the MDS (CVE-2018-12126, CVE-2018-12127, CVE-2018-12130, +CVE-2019-11091) fixes. + +Not included by default in any Intel CPU model. + +Must be explicitly turned on for all Intel CPU models. + +Requires the host CPU microcode to support this feature before it +can be used for guest CPUs. @end table
Signature · 1 line
2.17.1
PATCH 27/97 target/ppc: Fix xvxsigdp
In reply to Michael RothFrom: Anton Blanchard <[email protected]>
Fix a typo in xvxsigdp where we put both results into the lower doubleword.
Message-Id: <[email protected]>
(upstream commit cf4e9363f7fd889d8d804c8f78e8927782c2aa48)
--- target/ppc/translate/vsx-impl.inc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
target/ppc/translate/vsx-impl.inc.c +1 -1
@@ -1819,7 +1819,7 @@ static void gen_xvxsigdp(DisasContext *ctx)tcg_gen_movi_i64 (t0,0x0010000000000000 );tcg_gen_movcond_i64 (TCG_COND_EQ , t0, exp, zr, zr, t0);tcg_gen_movcond_i64 (TCG_COND_EQ , t0, exp, nan, zr, t0); -tcg_gen_deposit_i64 (xth, t0, xbl,0 ,52 ); +tcg_gen_deposit_i64 (xtl, t0, xbl,0 ,52 );set_cpu_vsrl (xT (ctx->opcode), xtl);tcg_temp_free_i64 (t0);
Signature · 1 line
2.17.1
PATCH 38/97 virtio-balloon: fix QEMU 4.0 config size migration incompatibility
In reply to Michael RothFrom: Stefan Hajnoczi <[email protected]>
The virtio-balloon config size changed in QEMU 4.0 even for existing machine types. Migration from QEMU 3.1 to 4.0 can fail in some circumstances with the following error:
qemu-system-x86_64: get_pci_config_device: Bad config data: i=0x10 read: a1 device: 1 cmask: ff wmask: c0 w1cmask:0
This happens because the virtio-balloon config size affects the VIRTIO Legacy I/O Memory PCI BAR size.
Introduce a qdev property called "qemu-4-0-config-size" and enable it only for the QEMU 4.0 machine types. This way <4.0 machine types use the old size, 4.0 uses the larger size, and >4.0 machine types use the appropriate size depending on enabled virtio-balloon features.
Live migration to and from old QEMUs to QEMU 4.1 works again as long as a versioned machine type is specified (do not use just "pc"!).
Message-Id: <[email protected]>
Conflicts:
hw/core/machine.c
* drop context dep. on 0a71966253c
--- hw/core/machine.c | 5 ++++- hw/virtio/virtio-balloon.c | 28 +++++++++++++++++++++++++--- include/hw/virtio/virtio-balloon.h | 2 ++ 3 files changed, 31 insertions(+), 4 deletions(-)
hw/core/machine.c +4 -1
@@ -24,7 +24,9 @@#include "hw/pci/pci.h" #include "hw/mem/nvdimm.h" -GlobalProperty hw_compat_4_0[] = {}; +GlobalProperty hw_compat_4_0[] = { + {"virtio-balloon-device" ,"qemu-4-0-config-size" ,"true" }, +};const size_t hw_compat_4_0_len =G_N_ELEMENTS (hw_compat_4_0);GlobalProperty hw_compat_3_1[] = {
@@ -39,6 +41,7 @@ GlobalProperty hw_compat_3_1[] = { { "usb-tablet", "serial", "42" }, { "virtio-blk-device", "discard", "false" }, { "virtio-blk-device", "write-zeroes", "false" }, + { "virtio-balloon-device", "qemu-4-0-config-size", "false" }, };const size_t hw_compat_3_1_len =G_N_ELEMENTS (hw_compat_3_1);
hw/virtio/virtio-balloon.c +25 -3
@@ -615,6 +615,22 @@ virtio_balloon_free_page_report_notify(NotifierWithReturn *n, void *data)return 0 ; } +static size_t virtio_balloon_config_size (VirtIOBalloon *s) +{ +uint64_t features = s->host_features; + +if (s->qemu_4_0_config_size) { +return sizeof (struct virtio_balloon_config ); + } +if (virtio_has_feature (features,VIRTIO_BALLOON_F_PAGE_POISON )) { +return sizeof (struct virtio_balloon_config ); + } +if (virtio_has_feature (features,VIRTIO_BALLOON_F_FREE_PAGE_HINT )) { +return offsetof(struct virtio_balloon_config , poison_val); + } +return offsetof(struct virtio_balloon_config , free_page_report_cmd_id); +} +static void virtio_balloon_get_config (VirtIODevice *vdev,uint8_t *config_data) {VirtIOBalloon *dev =VIRTIO_BALLOON (vdev);
@@ -635,7 +651,7 @@ static void virtio_balloon_get_config(VirtIODevice *vdev, uint8_t *config_data) }trace_virtio_balloon_get_config (config.num_pages, config.actual); -memcpy (config_data, &config,sizeof (struct virtio_balloon_config )); +memcpy (config_data, &config,virtio_balloon_config_size (dev)); }static int build_dimm_list (Object *obj,void *opaque)
@@ -679,7 +695,7 @@ static void virtio_balloon_set_config(VirtIODevice *vdev,uint32_t oldactual = dev->actual;ram_addr_t vm_ram_size =get_current_ram_size (); -memcpy (&config, config_data,sizeof (struct virtio_balloon_config )); +memcpy (&config, config_data,virtio_balloon_config_size (dev)); dev->actual =le32_to_cpu (config.actual);if (dev->actual != oldactual) { qapi_event_send_balloon_change(vm_ram_size -
@@ -766,7 +782,7 @@ static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)int ret;virtio_init (vdev,"virtio-balloon" ,VIRTIO_ID_BALLOON , -sizeof (struct virtio_balloon_config )); +virtio_balloon_config_size (s)); ret =qemu_add_balloon_handler (virtio_balloon_to_target, virtio_balloon_stat, s);
@@ -897,6 +913,12 @@ static Property virtio_balloon_properties[] = { VIRTIO_BALLOON_F_DEFLATE_ON_OOM, false),DEFINE_PROP_BIT ("free-page-hint" , VirtIOBalloon, host_features,VIRTIO_BALLOON_F_FREE_PAGE_HINT , false), +/* QEMU 4.0 accidentally changed the config size even when free-page-hint +* is disabled, resulting in QEMU 3.1 migration incompatibility. This +* property retains this quirk for QEMU 4.1 machine types. +*/ +DEFINE_PROP_BOOL ("qemu-4-0-config-size" , VirtIOBalloon, + qemu_4_0_config_size, false),DEFINE_PROP_LINK ("iothread" , VirtIOBalloon, iothread,TYPE_IOTHREAD , IOThread *), DEFINE_PROP_END_OF_LIST(),
include/hw/virtio/virtio-balloon.h +2 -0
@@ -71,6 +71,8 @@ typedef struct VirtIOBalloon {int64_t stats_poll_interval;uint32_t host_features;PartiallyBalloonedPage *pbp; + + bool qemu_4_0_config_size; } VirtIOBalloon;#endif
Signature · 1 line
2.17.1
PATCH 47/97 virtio-balloon: Fix wrong sign extension of PFNs
In reply to Michael RothFrom: David Hildenbrand <[email protected]>
If we directly cast from int to uint64_t, we will first sign-extend to an int64_t, which is wrong. We actually want to treat the PFNs like unsigned values.
As far as I can see, this dates back to the initial virtio-balloon commit, but wasn't triggered as fairly big guests would be required.
Message-Id: <[email protected]>
--- hw/virtio/virtio-balloon.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
hw/virtio/virtio-balloon.c +1 -1
@@ -343,8 +343,8 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq) }while (iov_to_buf (elem->out_sg, elem->out_num, offset, &pfn,4 ) ==4 ) { +unsigned int p =virtio_ldl_p (vdev, &pfn);hwaddr pa; -int p =virtio_ldl_p (vdev, &pfn); pa = (hwaddr ) p <<VIRTIO_BALLOON_PFN_SHIFT ; offset +=4 ;
Signature · 1 line
2.17.1
PATCH 42/97 hw/ssi/xilinx_spips: Convert lqspi_read() to read_with_attrs
In reply to Michael RothFrom: Philippe Mathieu-Daudé <[email protected]>
In the next commit we will implement the write_with_attrs() handler. To avoid using different APIs, convert the read() handler first.
--- hw/ssi/xilinx_spips.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-)
hw/ssi/xilinx_spips.c +11 -12
@@ -1199,27 +1199,26 @@ static void lqspi_load_cache(void *opaque, hwaddr addr) } } -static uint64_t -lqspi_read (void *opaque,hwaddr addr,unsigned int size) +static MemTxResult lqspi_read (void *opaque,hwaddr addr,uint64_t *value, +unsigned size,MemTxAttrs attrs) { -XilinxQSPIPS *q = opaque; -uint32_t ret; +XilinxQSPIPS *q =XILINX_QSPIPS (opaque);if (addr >= q->lqspi_cached_addr && addr <= q->lqspi_cached_addr +LQSPI_CACHE_SIZE -4 ) {uint8_t *retp = &q->lqspi_buf[addr - q->lqspi_cached_addr]; - ret =cpu_to_le32 (*(uint32_t *)retp); -DB_PRINT_L (1 ,"addr: %08x, data: %08x\n" , (unsigned )addr, - (unsigned )ret); -return ret; - }else { -lqspi_load_cache (opaque, addr); -return lqspi_read (opaque, addr, size); + *value =cpu_to_le32 (*(uint32_t *)retp); +DB_PRINT_L (1 ,"addr: %08" HWADDR_PRIx", data: %08" PRIx64"\n" , + addr, *value); +return MEMTX_OK ; } + +lqspi_load_cache (opaque, addr); +return lqspi_read (opaque, addr, value, size, attrs); }static const MemoryRegionOps lqspi_ops = { - .read = lqspi_read, + .read_with_attrs = lqspi_read, .endianness =DEVICE_NATIVE_ENDIAN , .valid = { .min_access_size =1 ,
Signature · 1 line
2.17.1
PATCH 26/97 target/ppc: Fix xvabs[sd]p, xvnabs[sd]p, xvneg[sd]p, xvcpsgn[sd]p
In reply to Michael RothFrom: Anton Blanchard <[email protected]>
We were using set_cpu_vsr*() when we should have used get_cpu_vsr*().
Message-Id: <20190509104912.6b754dff@kryten>
(upstream commit 77bd8937c03dd55e57cc257951ad07c185303c3e)
--- target/ppc/translate/vsx-impl.inc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
target/ppc/translate/vsx-impl.inc.c +2 -2
@@ -858,8 +858,8 @@ static void glue(gen_, name)(DisasContext * ctx) \ xbh =tcg_temp_new_i64 (); \ xbl =tcg_temp_new_i64 (); \ sgm =tcg_temp_new_i64 (); \ -set_cpu_vsrh (xB (ctx->opcode), xbh); \ -set_cpu_vsrl (xB (ctx->opcode), xbl); \ +get_cpu_vsrh (xbh,xB (ctx->opcode)); \ +get_cpu_vsrl (xbl,xB (ctx->opcode)); \tcg_gen_movi_i64 (sgm, sgn_mask); \switch (op) { \case OP_ABS : { \
Signature · 1 line
2.17.1
PATCH 45/97 ioapic: kvm: Skip route updates for masked pins
In reply to Michael RothFrom: Jan Kiszka <[email protected]>
Masked entries will not generate interrupt messages, thus do no need to be routed by KVM. This is a cosmetic cleanup, just avoiding warnings of the kind
qemu-system-x86_64: vtd_irte_get: detected non-present IRTE (index=0, high=0xff00, low=0x100)
if the masked entry happens to reference a non-present IRTE.
Message-Id: <[email protected]>
--- hw/intc/ioapic.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
hw/intc/ioapic.c +5 -3
@@ -188,9 +188,11 @@ static void ioapic_update_kvm_routes(IOAPICCommonState *s)MSIMessage msg;struct ioapic_entry_info info;ioapic_entry_parse (s->ioredtbl[i], &info); - msg.address = info.addr; - msg.data = info.data; -kvm_irqchip_update_msi_route (kvm_state, i, msg,NULL ); +if (!info.masked) { + msg.address = info.addr; + msg.data = info.data; +kvm_irqchip_update_msi_route (kvm_state, i, msg,NULL ); + } }kvm_irqchip_commit_routes (kvm_state); }
Signature · 1 line
2.17.1
PATCH 48/97 virtio-balloon: Fix QEMU crashes on pagesize > BALLOON_PAGE_SIZE
In reply to Michael RothFrom: David Hildenbrand <[email protected]>
We are using the wrong functions to set/clear bits, effectively touching multiple bits, writing out of range of the bitmap, resulting in memory corruptions. We have to use set_bit()/clear_bit() instead.
Can easily be reproduced by starting a qemu guest on hugetlbfs memory, inflating the balloon. QEMU crashes. This never could have worked properly - especially, also pages would have been discarded when the first sub-page would be inflated (the whole bitmap would be set).
While testing I realized, that on hugetlbfs it is pretty much impossible to discard a page - the guest just frees the 4k sub-pages in random order most of the time. I was only able to discard a hugepage a handful of times - so I hope that now works correctly.
Message-Id: <[email protected]>
--- hw/virtio/virtio-balloon.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
hw/virtio/virtio-balloon.c +4 -6
@@ -94,9 +94,8 @@ static void balloon_inflate_page(VirtIOBalloon *balloon, balloon->pbp->base = host_page_base; } -bitmap_set (balloon->pbp->bitmap, - (ram_offset - balloon->pbp->base) /BALLOON_PAGE_SIZE , - subpages); +set_bit ((ram_offset - balloon->pbp->base) /BALLOON_PAGE_SIZE , + balloon->pbp->bitmap);if (bitmap_full (balloon->pbp->bitmap, subpages)) {/* We've accumulated a full host page, we can actually discard
@@ -140,9 +139,8 @@ static void balloon_deflate_page(VirtIOBalloon *balloon,* for a guest to do this in practice, but handle it anyway, * since getting it wrong could mean discarding memory the * guest is still using. */ -bitmap_clear (balloon->pbp->bitmap, - (ram_offset - balloon->pbp->base) /BALLOON_PAGE_SIZE , - subpages); +clear_bit ((ram_offset - balloon->pbp->base) /BALLOON_PAGE_SIZE , + balloon->pbp->bitmap);if (bitmap_empty (balloon->pbp->bitmap, subpages)) {g_free (balloon->pbp);
Signature · 1 line
2.17.1
PATCH 53/97 virtio-balloon: don't track subpages for the PBP
In reply to Michael RothFrom: David Hildenbrand <[email protected]>
As ramblocks cannot get removed/readded while we are processing a bulk of inflation requests, there is no more need to track the page size in form of the number of subpages.
Message-Id: <[email protected]>
--- hw/virtio/virtio-balloon.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
hw/virtio/virtio-balloon.c +3 -5
@@ -36,7 +36,6 @@typedef struct PartiallyBalloonedPage {ram_addr_t base_gpa; -long subpages;unsigned long *bitmap; }PartiallyBalloonedPage ;
@@ -55,16 +54,15 @@ static PartiallyBalloonedPage *virtio_balloon_pbp_alloc(ram_addr_t base_gpa,PartiallyBalloonedPage *pbp =g_new0 (PartiallyBalloonedPage,1 ); pbp->base_gpa = base_gpa; - pbp->subpages = subpages; pbp->bitmap =bitmap_new (subpages);return pbp; }static bool virtio_balloon_pbp_matches (PartiallyBalloonedPage *pbp, -ram_addr_t base_gpa,long subpages) +ram_addr_t base_gpa) { -return pbp->subpages == subpages && pbp->base_gpa == base_gpa; +return pbp->base_gpa == base_gpa; } static void balloon_inflate_page(VirtIOBalloon *balloon,
@@ -106,7 +104,7 @@ static void balloon_inflate_page(VirtIOBalloon *balloon, base_gpa =memory_region_get_ram_addr (mr) + mr_offset - (rb_offset - rb_aligned_offset); -if (*pbp && !virtio_balloon_pbp_matches (*pbp, base_gpa, subpages)) { +if (*pbp && !virtio_balloon_pbp_matches (*pbp, base_gpa)) {/* We've partially ballooned part of a host page, but now * we're trying to balloon part of a different one. Too hard, * give up on the old partial page */
Signature · 1 line
2.17.1
PATCH 54/97 virtio-balloon: free pbp more aggressively
In reply to Michael RothFrom: "Michael S. Tsirkin" <[email protected]>
Previous patches switched to a temporary pbp but that does not go far
enough: after device uses a buffer, guest is free to reuse it, so tracking the page and freeing it later is wrong.
Free and reset the pbp after we push each element.
--- hw/virtio/virtio-balloon.c | 37 ++++++++++++++++--------------------- 1 file changed, 16 insertions(+), 21 deletions(-)
hw/virtio/virtio-balloon.c +16 -21
@@ -41,22 +41,19 @@ typedef struct PartiallyBalloonedPage {static void virtio_balloon_pbp_free (PartiallyBalloonedPage *pbp) { -if (!pbp) { +if (!pbp->bitmap) {return ; }g_free (pbp->bitmap); -g_free (pbp); + pbp->bitmap =NULL ; } -static PartiallyBalloonedPage *virtio_balloon_pbp_alloc (ram_addr_t base_gpa, -long subpages) +static void virtio_balloon_pbp_alloc (PartiallyBalloonedPage *pbp, +ram_addr_t base_gpa, +long subpages) { -PartiallyBalloonedPage *pbp =g_new0 (PartiallyBalloonedPage,1 ); - pbp->base_gpa = base_gpa; pbp->bitmap =bitmap_new (subpages); - -return pbp; } static bool virtio_balloon_pbp_matches(PartiallyBalloonedPage *pbp,
@@ -67,7 +64,7 @@ static bool virtio_balloon_pbp_matches(PartiallyBalloonedPage *pbp,static void balloon_inflate_page (VirtIOBalloon *balloon,MemoryRegion *mr,hwaddr mr_offset, -PartiallyBalloonedPage **pbp) +PartiallyBalloonedPage *pbp) {void *addr =memory_region_get_ram_ptr (mr) + mr_offset;ram_addr_t rb_offset, rb_aligned_offset, base_gpa;
@@ -104,22 +101,21 @@ static void balloon_inflate_page(VirtIOBalloon *balloon, base_gpa =memory_region_get_ram_addr (mr) + mr_offset - (rb_offset - rb_aligned_offset); -if (*pbp && !virtio_balloon_pbp_matches (*pbp, base_gpa)) { +if (pbp->bitmap && !virtio_balloon_pbp_matches (pbp, base_gpa)) {/* We've partially ballooned part of a host page, but now * we're trying to balloon part of a different one. Too hard, * give up on the old partial page */ -virtio_balloon_pbp_free (*pbp); - *pbp =NULL ; +virtio_balloon_pbp_free (pbp); } -if (!*pbp) { - *pbp =virtio_balloon_pbp_alloc (base_gpa, subpages); +if (!pbp->bitmap) { +virtio_balloon_pbp_alloc (pbp, base_gpa, subpages); }set_bit ((rb_offset - rb_aligned_offset) /BALLOON_PAGE_SIZE , - (*pbp)->bitmap); + pbp->bitmap); -if (bitmap_full ((*pbp)->bitmap, subpages)) { +if (bitmap_full (pbp->bitmap, subpages)) {/* We've accumulated a full host page, we can actually discard * it now */
@@ -127,8 +123,7 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,/* We ignore errors from ram_block_discard_range(), because it * has already reported them, and failing to discard a balloon * page is not fatal */ -virtio_balloon_pbp_free (*pbp); - *pbp =NULL ; +virtio_balloon_pbp_free (pbp); } }
@@ -328,13 +323,14 @@ static void balloon_stats_set_poll_interval(Object *obj, Visitor *v,static void virtio_balloon_handle_output (VirtIODevice *vdev,VirtQueue *vq) {VirtIOBalloon *s =VIRTIO_BALLOON (vdev); -PartiallyBalloonedPage *pbp =NULL ;VirtQueueElement *elem;MemoryRegionSection section;for (;;) { +PartiallyBalloonedPage pbp = {};size_t offset =0 ;uint32_t pfn; + elem =virtqueue_pop (vq,sizeof (VirtQueueElement));if (!elem) {break ;
@@ -379,9 +375,8 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)virtqueue_push (vq, elem, offset);virtio_notify (vdev, vq);g_free (elem); +virtio_balloon_pbp_free (&pbp); } - -virtio_balloon_pbp_free (pbp); }static void virtio_balloon_receive_stats (VirtIODevice *vdev,VirtQueue *vq)
Signature · 1 line
2.17.1
PATCH 41/97 docs/bitmaps: use QMP lexer instead of json
In reply to Michael RothFrom: John Snow <[email protected]>
The annotated style json we use in QMP documentation is not strict json and depending on the version of Sphinx (2.0+) or Pygments installed, might cause the build to fail.
Use the new QMP lexer.
Further, some versions of Sphinx can not apply custom lexers to "code" directives and require the use of "code-block" directives instead, so make that change at this time as well.
Tested under:
- Sphinx 1.3.6 and Pygments 2.4
- Sphinx 1.7.6 and Pygments 2.2 (Fedora 29 packages)
- Sphinx 2.0.1 and Pygments 2.4
- Sphinx 3.0.0+/f396b3a783 and Pygments 2.4 (From Sphinx git c4f44bdd)
Message-id: [email protected]
--- docs/interop/bitmaps.rst | 54 ++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 27 deletions(-)
docs/interop/bitmaps.rst +27 -27
@@ -199,7 +199,7 @@ persistence, and recording state can be adjusted at creation time. to create a new, actively recording persistent bitmap: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-add", "arguments": {
@@ -220,7 +220,7 @@ persistence, and recording state can be adjusted at creation time. To create a new, disabled (``-recording``), transient bitmap that tracks changes in 32KiB segments: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-add", "arguments": {
@@ -254,7 +254,7 @@ Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed. Remove a bitmap named ``bitmap0`` from node ``drive0``: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-remove", "arguments": {
@@ -280,7 +280,7 @@ Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared. Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-clear", "arguments": {
@@ -309,7 +309,7 @@ begin being recorded. ``+busy`` bitmaps cannot be enabled. To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-enable", "arguments": {
@@ -347,7 +347,7 @@ writes to begin being ignored. ``+busy`` bitmaps cannot be disabled. To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-disable", "arguments": {
@@ -393,7 +393,7 @@ in any one source bitmap, the target bitmap will mark that segment dirty. ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves a copy. - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-merge", "arguments": {
@@ -424,7 +424,7 @@ attached to nodes serving as the root for guest devices. API. This result highlights a bitmap ``bitmap0`` attached to the root node of device ``drive0``. - .. code:: json + .. code-block:: QMP -> { "execute": "query-block",
@@ -562,7 +562,7 @@ new, empty bitmap that records writes from this point in time forward. destination. These writes will be recorded in the bitmap accordingly. -.. code:: json +.. code-block:: QMP -> { "execute": "transaction",
@@ -650,7 +650,7 @@ Example: Resetting an Incremental Backup Anchor Point If we want to start a new backup chain with an existing bitmap, we can also use a transaction to reset the bitmap while making a new full backup: -.. code:: json +.. code-block:: QMP -> { "execute": "transaction",
@@ -730,7 +730,7 @@ Example: First Incremental Backup #. Issue an incremental backup command: - .. code:: json + .. code-block:: QMP -> { "execute": "drive-backup",
@@ -788,7 +788,7 @@ Example: Second Incremental Backup #. Issue a new incremental backup command. The only difference here is that we have changed the target image below. - .. code:: json + .. code-block:: QMP -> { "execute": "drive-backup",
@@ -869,7 +869,7 @@ image: #. Issue a new incremental backup command. Apart from the new destination image, there is no difference from the last two examples. - .. code:: json + .. code-block:: QMP -> { "execute": "drive-backup",
@@ -932,7 +932,7 @@ point in time. #. Create a full (anchor) backup for each drive, with accompanying bitmaps: - .. code:: json + .. code-block:: QMP -> { "execute": "transaction",
@@ -1018,7 +1018,7 @@ point in time. #. Issue a multi-drive incremental push backup transaction: - .. code:: json + .. code-block:: QMP -> { "execute": "transaction",
@@ -1121,7 +1121,7 @@ described above. This example demonstrates the single-job failure case: #. Attempt to create an incremental backup via QMP: - .. code:: json + .. code-block:: QMP -> { "execute": "drive-backup",
@@ -1139,7 +1139,7 @@ described above. This example demonstrates the single-job failure case: #. Receive a pair of events indicating failure: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
@@ -1175,7 +1175,7 @@ described above. This example demonstrates the single-job failure case: #. Retry the command after fixing the underlying problem, such as freeing up space on the backup volume: - .. code:: json + .. code-block:: QMP -> { "execute": "drive-backup",
@@ -1193,7 +1193,7 @@ described above. This example demonstrates the single-job failure case: #. Receive confirmation that the job completed successfully: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
@@ -1233,7 +1233,7 @@ and one succeeds: #. Issue the transaction to start a backup of both drives. - .. code:: json + .. code-block:: QMP -> { "execute": "transaction",
@@ -1267,13 +1267,13 @@ and one succeeds: #. Receive notice that the Transaction was accepted, and jobs were launched: - .. code:: json + .. code-block:: QMP <- { "return": {} } #. Receive notice that the first job has completed: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
@@ -1289,7 +1289,7 @@ and one succeeds: #. Receive notice that the second job has failed: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
@@ -1365,7 +1365,7 @@ applied: #. Issue the multi-drive incremental backup transaction: - .. code:: json + .. code-block:: QMP -> { "execute": "transaction",
@@ -1401,13 +1401,13 @@ applied: #. Receive notice that the Transaction was accepted, and jobs were launched: - .. code:: json + .. code-block:: QMP <- { "return": {} } #. Receive notification that the backup job for ``drive1`` has failed: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
@@ -1434,7 +1434,7 @@ applied: #. Receive notification that the job for ``drive0`` has been cancelled: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
Signature · 1 line
2.17.1
PATCH 65/97 util/hbitmap: update orig_size on truncate
In reply to Michael RothFrom: Vladimir Sementsov-Ogievskiy <[email protected]>
Without this, hbitmap_next_zero and hbitmap_next_dirty_area are broken after truncate. So, orig_size is broken since it's introduction in 76d570dc495c56bb.
Message-id: [email protected]
--- util/hbitmap.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
util/hbitmap.c +5 -1
@@ -53,7 +53,9 @@*/ struct HBitmap { -/* Size of the bitmap, as requested in hbitmap_alloc. */ +/* +* Size of the bitmap, as requested in hbitmap_alloc or in hbitmap_truncate. +*/ uint64_t orig_size;/* Number of total bits in the bottom level. */
@@ -732,6 +734,8 @@ void hbitmap_truncate(HBitmap *hb, uint64_t size)uint64_t num_elements = size;uint64_t old; + hb->orig_size = size; +/* Size comes in as logical elements, adjust for granularity. */ size = (size + (1ULL << hb->granularity) -1 ) >> hb->granularity;assert (size <= ((uint64_t )1 <<HBITMAP_LOG_MAX_SIZE ));
Signature · 1 line
2.17.1
PATCH 52/97 virtio-balloon: Use temporary PBP only
In reply to Michael RothFrom: David Hildenbrand <[email protected]>
We still have multiple issues in the current code
- The PBP is not freed during unrealize()
- The PBP is not reset on device resets: After a reset, the PBP is stale.
- We are not indicating VIRTIO_BALLOON_F_MUST_TELL_HOST, therefore guests (esp. legacy guests) will reuse pages without deflating, turning the PBP stale. Adding that would require compat handling.
Instead, let's use the PBP only temporarily, when processing one bulk of inflation requests. This will keep guest_page_size > 4k working (with Linux guests). There is nothing to do for deflation requests anymore. The pbp is only used for a limited amount of time.
Message-Id: <[email protected]>
--- hw/virtio/virtio-balloon.c | 21 +++++++++------------ include/hw/virtio/virtio-balloon.h | 3 --- 2 files changed, 9 insertions(+), 15 deletions(-)
hw/virtio/virtio-balloon.c +9 -12
@@ -34,11 +34,11 @@#define BALLOON_PAGE_SIZE (1 << VIRTIO_BALLOON_PFN_SHIFT) -struct PartiallyBalloonedPage { +typedef struct PartiallyBalloonedPage {ram_addr_t base_gpa;long subpages;unsigned long *bitmap; -}; +}PartiallyBalloonedPage ;static void virtio_balloon_pbp_free (PartiallyBalloonedPage *pbp) {
@@ -68,11 +68,11 @@ static bool virtio_balloon_pbp_matches(PartiallyBalloonedPage *pbp, }static void balloon_inflate_page (VirtIOBalloon *balloon, -MemoryRegion *mr,hwaddr mr_offset) +MemoryRegion *mr,hwaddr mr_offset, +PartiallyBalloonedPage **pbp) {void *addr =memory_region_get_ram_ptr (mr) + mr_offset;ram_addr_t rb_offset, rb_aligned_offset, base_gpa; -PartiallyBalloonedPage **pbp = &balloon->pbp;RAMBlock *rb;size_t rb_page_size;int subpages;
@@ -149,12 +149,6 @@ static void balloon_deflate_page(VirtIOBalloon *balloon, rb =qemu_ram_block_from_host (addr, false, &rb_offset); rb_page_size =qemu_ram_pagesize (rb); -if (balloon->pbp) { -/* Let's play safe and always reset the pbp on deflation requests. */ -virtio_balloon_pbp_free (balloon->pbp); - balloon->pbp =NULL ; - } - host_addr = (void *)((uintptr_t )addr & ~(rb_page_size -1 ));/* When a page is deflated, we hint the whole host page it lives
@@ -336,6 +330,7 @@ static void balloon_stats_set_poll_interval(Object *obj, Visitor *v,static void virtio_balloon_handle_output (VirtIODevice *vdev,VirtQueue *vq) {VirtIOBalloon *s =VIRTIO_BALLOON (vdev); +PartiallyBalloonedPage *pbp =NULL ;VirtQueueElement *elem;MemoryRegionSection section;
@@ -344,7 +339,7 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)uint32_t pfn; elem =virtqueue_pop (vq,sizeof (VirtQueueElement));if (!elem) { -return ; +break ; }while (iov_to_buf (elem->out_sg, elem->out_num, offset, &pfn,4 ) ==4 ) {
@@ -373,7 +368,7 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)if (!qemu_balloon_is_inhibited ()) {if (vq == s->ivq) {balloon_inflate_page (s, section.mr, - section.offset_within_region); + section.offset_within_region, &pbp); }else if (vq == s->dvq) {balloon_deflate_page (s, section.mr, section.offset_within_region); }else {
@@ -387,6 +382,8 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)virtio_notify (vdev, vq);g_free (elem); } + +virtio_balloon_pbp_free (pbp); }static void virtio_balloon_receive_stats (VirtIODevice *vdev,VirtQueue *vq)
include/hw/virtio/virtio-balloon.h +0 -3
@@ -33,8 +33,6 @@ typedef struct virtio_balloon_stat_modern {uint64_t val; } VirtIOBalloonStatModern; -typedef struct PartiallyBalloonedPage PartiallyBalloonedPage ; -enum virtio_balloon_free_page_report_status {FREE_PAGE_REPORT_S_STOP =0 ,FREE_PAGE_REPORT_S_REQUESTED =1 ,
@@ -70,7 +68,6 @@ typedef struct VirtIOBalloon {int64_t stats_last_update;int64_t stats_poll_interval;uint32_t host_features; -PartiallyBalloonedPage *pbp;bool qemu_4_0_config_size; } VirtIOBalloon;
Signature · 1 line
2.17.1
PATCH 50/97 virtio-balloon: Better names for offset variables in inflate/deflate code
In reply to Michael RothFrom: David Hildenbrand <[email protected]>
"host_page_base" is really confusing, let's make this clearer, also rename the other offsets to indicate to which base they apply.
offset -> mr_offsetram_offset -> rb_offsethost_page_base -> rb_aligned_offset
While at it, use QEMU_ALIGN_DOWN() instead of a handcrafted computation and move the computation to the place where it is needed.
Message-Id: <[email protected]>
--- hw/virtio/virtio-balloon.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
hw/virtio/virtio-balloon.c +13 -13
@@ -41,24 +41,23 @@ struct PartiallyBalloonedPage { };static void balloon_inflate_page (VirtIOBalloon *balloon, -MemoryRegion *mr,hwaddr offset) +MemoryRegion *mr,hwaddr mr_offset) { -void *addr =memory_region_get_ram_ptr (mr) + offset; +void *addr =memory_region_get_ram_ptr (mr) + mr_offset; +ram_addr_t rb_offset, rb_aligned_offset;RAMBlock *rb;size_t rb_page_size;int subpages; -ram_addr_t ram_offset, host_page_base;/* XXX is there a better way to get to the RAMBlock than via a * host address? */ - rb =qemu_ram_block_from_host (addr, false, &ram_offset); + rb =qemu_ram_block_from_host (addr, false, &rb_offset); rb_page_size =qemu_ram_pagesize (rb); - host_page_base = ram_offset & ~(rb_page_size -1 );if (rb_page_size ==BALLOON_PAGE_SIZE ) {/* Easy case */ -ram_block_discard_range (rb, ram_offset, rb_page_size); +ram_block_discard_range (rb, rb_offset, rb_page_size);/* We ignore errors from ram_block_discard_range(), because it * has already reported them, and failing to discard a balloon * page is not fatal */
@@ -74,11 +73,12 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,warn_report_once ("Balloon used with backing page size > 4kiB, this may not be reliable" ); + rb_aligned_offset =QEMU_ALIGN_DOWN (rb_offset, rb_page_size); subpages = rb_page_size /BALLOON_PAGE_SIZE ;if (balloon->pbp && (rb != balloon->pbp->rb - || host_page_base != balloon->pbp->base)) { + || rb_aligned_offset != balloon->pbp->base)) {/* We've partially ballooned part of a host page, but now * we're trying to balloon part of a different one. Too hard, * give up on the old partial page */
@@ -91,10 +91,10 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,size_t bitlen =BITS_TO_LONGS (subpages) *sizeof (unsigned long ); balloon->pbp =g_malloc0 (sizeof (PartiallyBalloonedPage) + bitlen); balloon->pbp->rb = rb; - balloon->pbp->base = host_page_base; + balloon->pbp->base = rb_aligned_offset; } -set_bit ((ram_offset - balloon->pbp->base) /BALLOON_PAGE_SIZE , +set_bit ((rb_offset - balloon->pbp->base) /BALLOON_PAGE_SIZE , balloon->pbp->bitmap);if (bitmap_full (balloon->pbp->bitmap, subpages)) {
@@ -112,18 +112,18 @@ static void balloon_inflate_page(VirtIOBalloon *balloon, }static void balloon_deflate_page (VirtIOBalloon *balloon, -MemoryRegion *mr,hwaddr offset) +MemoryRegion *mr,hwaddr mr_offset) { -void *addr =memory_region_get_ram_ptr (mr) + offset; +void *addr =memory_region_get_ram_ptr (mr) + mr_offset; +ram_addr_t rb_offset;RAMBlock *rb;size_t rb_page_size; -ram_addr_t ram_offset;void *host_addr;int ret;/* XXX is there a better way to get to the RAMBlock than via a * host address? */ - rb =qemu_ram_block_from_host (addr, false, &ram_offset); + rb =qemu_ram_block_from_host (addr, false, &rb_offset); rb_page_size =qemu_ram_pagesize (rb);if (balloon->pbp) {
Signature · 1 line
2.17.1
PATCH 55/97 i386/acpi: fix gint overflow in crs_range_compare
In reply to Michael RothFrom: Evgeny Yakovlev <[email protected]>
When very large regions (32GB sized in our case, PCI pass-through of GPUs) are compared substraction result does not fit into gint.
As a result crs_replace_with_free_ranges does not get sorted ranges and incorrectly computes PCI64 free space regions. Which then makes linux guest complain about device and PCI64 hole intersection and device becomes unusable.
Fix that by returning exactly fitting ranges.
Also fix indentation of an entire crs_replace_with_free_ranges to make checkpatch happy.
Message-Id: <[email protected]>
--- hw/i386/acpi-build.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
hw/i386/acpi-build.c +9 -3
@@ -743,10 +743,16 @@ static void crs_range_set_free(CrsRangeSet *range_set)static gint crs_range_compare (gconstpointer a,gconstpointer b) { -CrsRangeEntry *entry_a = *(CrsRangeEntry **)a; -CrsRangeEntry *entry_b = *(CrsRangeEntry **)b; +CrsRangeEntry *entry_a = *(CrsRangeEntry **)a; +CrsRangeEntry *entry_b = *(CrsRangeEntry **)b; -return (int64_t )entry_a->base - (int64_t )entry_b->base; +if (entry_a->base < entry_b->base) { +return -1 ; + }else if (entry_a->base > entry_b->base) { +return 1 ; + }else { +return 0 ; + } }/*
Signature · 1 line
2.17.1
PATCH 66/97 iotests: Test incremental backup after truncation
In reply to Michael RothFrom: Max Reitz <[email protected]>
Message-id: [email protected]
--- tests/qemu-iotests/124 | 38 ++++++++++++++++++++++++++++++++++---- tests/qemu-iotests/124.out | 4 ++-- 2 files changed, 36 insertions(+), 6 deletions(-)
tests/qemu-iotests/124 +34 -4
@@ -212,25 +212,28 @@ class TestIncrementalBackupBase(iotests.QMPTestCase): return bitmap - def prepare_backup(self, bitmap=None, parent=None): + def prepare_backup(self, bitmap=None, parent=None, **kwargs): if bitmap is None: bitmap = self.bitmaps[-1] if parent is None: parent, _ = bitmap.last_target() target, _ = bitmap.new_target() - self.img_create(target, bitmap.drive['fmt'], parent=parent) + self.img_create(target, bitmap.drive['fmt'], parent=parent, + **kwargs) return target def create_incremental(self, bitmap=None, parent=None, - parentFormat=None, validate=True): + parentFormat=None, validate=True, + target=None): if bitmap is None: bitmap = self.bitmaps[-1] if parent is None: parent, _ = bitmap.last_target() - target = self.prepare_backup(bitmap, parent) + if target is None: + target = self.prepare_backup(bitmap, parent) res = self.do_qmp_backup(job_id=bitmap.drive['id'], device=bitmap.drive['id'], sync='incremental', bitmap=bitmap.name,
@@ -572,6 +575,33 @@ class TestIncrementalBackup(TestIncrementalBackupBase): 'bitmap0', self.drives[0], granularity=64000) + def test_growing_before_backup(self): + ''' + Test: Add a bitmap, truncate the image, write past the old + end, do a backup. + + Incremental backup should not ignore dirty bits past the old + image end. + ''' + self.assert_no_active_block_jobs() + + self.create_anchor_backup() + + self.add_bitmap('bitmap0', self.drives[0]) + + res = self.vm.qmp('block_resize', device=self.drives[0]['id'], + size=(65 * 1048576)) + self.assert_qmp(res, 'return', {}) + + # Dirty the image past the old end + self.vm.hmp_qemu_io(self.drives[0]['id'], 'write 64M 64k') + + target = self.prepare_backup(size='65M') + self.create_incremental(target=target) + + self.vm.shutdown() + self.check_backups() + class TestIncrementalBackupBlkdebug(TestIncrementalBackupBase): '''Incremental backup tests that utilize a BlkDebug filter on drive0.'''
tests/qemu-iotests/124.out +2 -2
@@ -1,5 +1,5 @@ -............ +............. ---------------------------------------------------------------------- -Ran 12 tests +Ran 13 tests OK
Signature · 1 line
2.17.1
PATCH 5/97 qcow2: Fix qcow2_make_empty() with external data file
In reply to Michael RothFrom: Kevin Wolf <[email protected]>
make_completely_empty() is an optimisated path for bdrv_make_empty() where completely new metadata is created inside the image file instead of going through all clusters and discarding them. For an external data file, however, we actually need to do discard operations on the data file; just overwriting the qcow2 file doesn't get rid of the data.
The necessary slow path with an explicit discard operation already exists for other cases. Use it for external data files, too.
--- block/qcow2.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
block/qcow2.c +5 -2
@@ -4384,14 +4384,17 @@ static int qcow2_make_empty(BlockDriverState *bs)if (s->qcow_version >=3 && !s->snapshots && !s->nb_bitmaps &&3 + l1_clusters <= s->refcount_block_size && - s->crypt_method_header !=QCOW_CRYPT_LUKS ) { + s->crypt_method_header !=QCOW_CRYPT_LUKS && + !has_data_file (bs)) {/* The following function only works for qcow2 v3 images (it * requires the dirty flag) and only as long as there are no * features that reserve extra clusters (such as snapshots, * LUKS header, or persistent bitmaps), because it completely * empties the image. Furthermore, the L1 table and three * additional clusters (image header, refcount table, one -* refcount block) have to fit inside one refcount block. */ +* refcount block) have to fit inside one refcount block. It +* only resets the image file, i.e. does not work with an +* external data file. */ return make_completely_empty (bs); }
Signature · 1 line
2.17.1
PATCH 51/97 virtio-balloon: Rework pbp tracking data
In reply to Michael RothFrom: David Hildenbrand <[email protected]>
Using the address of a RAMBlock to test for a matching pbp is not really safe. Instead, let's use the guest physical address of the base page along with the page size (via the number of subpages).
Also, let's allocate the bitmap separately. This makes the code easier to read and maintain - we can reuse bitmap_new().
Prepare the code to move the PBP out of the device.
Message-Id: <[email protected]>
--- hw/virtio/virtio-balloon.c | 69 +++++++++++++++++++++++++------------- 1 file changed, 46 insertions(+), 23 deletions(-)
hw/virtio/virtio-balloon.c +46 -23
@@ -35,16 +35,44 @@#define BALLOON_PAGE_SIZE (1 << VIRTIO_BALLOON_PFN_SHIFT)struct PartiallyBalloonedPage { -RAMBlock *rb; -ram_addr_t base; -unsigned long bitmap[]; +ram_addr_t base_gpa; +long subpages; +unsigned long *bitmap; }; +static void virtio_balloon_pbp_free (PartiallyBalloonedPage *pbp) +{ +if (!pbp) { +return ; + } +g_free (pbp->bitmap); +g_free (pbp); +} + +static PartiallyBalloonedPage *virtio_balloon_pbp_alloc (ram_addr_t base_gpa, +long subpages) +{ +PartiallyBalloonedPage *pbp =g_new0 (PartiallyBalloonedPage,1 ); + + pbp->base_gpa = base_gpa; + pbp->subpages = subpages; + pbp->bitmap =bitmap_new (subpages); + +return pbp; +} + +static bool virtio_balloon_pbp_matches (PartiallyBalloonedPage *pbp, +ram_addr_t base_gpa,long subpages) +{ +return pbp->subpages == subpages && pbp->base_gpa == base_gpa; +} +static void balloon_inflate_page (VirtIOBalloon *balloon,MemoryRegion *mr,hwaddr mr_offset) {void *addr =memory_region_get_ram_ptr (mr) + mr_offset; -ram_addr_t rb_offset, rb_aligned_offset; +ram_addr_t rb_offset, rb_aligned_offset, base_gpa; +PartiallyBalloonedPage **pbp = &balloon->pbp;RAMBlock *rb;size_t rb_page_size;int subpages;
@@ -75,39 +103,34 @@ static void balloon_inflate_page(VirtIOBalloon *balloon, rb_aligned_offset =QEMU_ALIGN_DOWN (rb_offset, rb_page_size); subpages = rb_page_size /BALLOON_PAGE_SIZE ; + base_gpa =memory_region_get_ram_addr (mr) + mr_offset - + (rb_offset - rb_aligned_offset); -if (balloon->pbp - && (rb != balloon->pbp->rb - || rb_aligned_offset != balloon->pbp->base)) { +if (*pbp && !virtio_balloon_pbp_matches (*pbp, base_gpa, subpages)) {/* We've partially ballooned part of a host page, but now * we're trying to balloon part of a different one. Too hard, * give up on the old partial page */ -g_free (balloon->pbp); - balloon->pbp =NULL ; +virtio_balloon_pbp_free (*pbp); + *pbp =NULL ; } -if (!balloon->pbp) { -/* Starting on a new host page */ -size_t bitlen =BITS_TO_LONGS (subpages) *sizeof (unsigned long ); - balloon->pbp =g_malloc0 (sizeof (PartiallyBalloonedPage) + bitlen); - balloon->pbp->rb = rb; - balloon->pbp->base = rb_aligned_offset; +if (!*pbp) { + *pbp =virtio_balloon_pbp_alloc (base_gpa, subpages); } -set_bit ((rb_offset - balloon->pbp->base) /BALLOON_PAGE_SIZE , - balloon->pbp->bitmap); +set_bit ((rb_offset - rb_aligned_offset) /BALLOON_PAGE_SIZE , + (*pbp)->bitmap); -if (bitmap_full (balloon->pbp->bitmap, subpages)) { +if (bitmap_full ((*pbp)->bitmap, subpages)) {/* We've accumulated a full host page, we can actually discard * it now */ -ram_block_discard_range (rb, balloon->pbp->base, rb_page_size); +ram_block_discard_range (rb, rb_aligned_offset, rb_page_size);/* We ignore errors from ram_block_discard_range(), because it * has already reported them, and failing to discard a balloon * page is not fatal */ - -g_free (balloon->pbp); - balloon->pbp =NULL ; +virtio_balloon_pbp_free (*pbp); + *pbp =NULL ; } }
@@ -128,7 +151,7 @@ static void balloon_deflate_page(VirtIOBalloon *balloon,if (balloon->pbp) {/* Let's play safe and always reset the pbp on deflation requests. */ -g_free (balloon->pbp); +virtio_balloon_pbp_free (balloon->pbp); balloon->pbp =NULL ; }
Signature · 1 line
2.17.1
PATCH 62/97 block/backup: refactor: split out backup_calculate_cluster_size
In reply to Michael RothFrom: Vladimir Sementsov-Ogievskiy <[email protected]>
Split out cluster_size calculation. Move copy-bitmap creation above block-job creation, as we are going to share it with upcoming backup-top filter, which also should be created before actual block job creation.
Message-id: [email protected] [mreitz: Dropped a paragraph from the commit message that was left over
from a previous version]
*prereq for 110571be4e
--- block/backup.c | 82 ++++++++++++++++++++++++++++++++------------------ 1 file changed, 52 insertions(+), 30 deletions(-)
block/backup.c +52 -30
@@ -507,6 +507,42 @@ static const BlockJobDriver backup_job_driver = { .drain = backup_drain, }; +static int64_t backup_calculate_cluster_size (BlockDriverState *target, +Error **errp) +{ +int ret; +BlockDriverInfo bdi; + +/* +* If there is no backing file on the target, we cannot rely on COW if our +* backup cluster size is smaller than the target cluster size. Even for +* targets with a backing file, try to avoid COW if possible. +*/ + ret =bdrv_get_info (target, &bdi); +if (ret == -ENOTSUP && !target->backing) { +/* Cluster size is not defined */ +warn_report ("The target block device doesn't provide " +"information about the block size and it doesn't have a " +"backing file. The default block size of %u bytes is " +"used. If the actual block size of the target exceeds " +"this default, the backup may be unusable" , +BACKUP_CLUSTER_SIZE_DEFAULT ); +return BACKUP_CLUSTER_SIZE_DEFAULT ; + }else if (ret <0 && !target->backing) { +error_setg_errno (errp, -ret, +"Couldn't determine the cluster size of the target image, " +"which has no backing file" ); +error_append_hint (errp, +"Aborting, since this may create an unusable destination image\n" ); +return ret; + }else if (ret <0 && target->backing) { +/* Not fatal; just trudge on ahead. */ +return BACKUP_CLUSTER_SIZE_DEFAULT ; + } + +return MAX (BACKUP_CLUSTER_SIZE_DEFAULT , bdi.cluster_size); +} + BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, BlockDriverState *target, int64_t speed, MirrorSyncMode sync_mode, BdrvDirtyBitmap *sync_bitmap,
@@ -518,9 +554,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, JobTxn *txn, Error **errp) {int64_t len; -BlockDriverInfo bdi;BackupBlockJob *job =NULL ;int ret; +int64_t cluster_size; +HBitmap *copy_bitmap =NULL ;assert (bs);assert (target);
@@ -582,6 +619,13 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, goto error; } + cluster_size =backup_calculate_cluster_size (target, errp); +if (cluster_size <0 ) { + goto error; + } + + copy_bitmap =hbitmap_alloc (len,ctz32 (cluster_size)); +/* job->len is fixed, so we can't allow resize */ job = block_job_create(job_id, &backup_job_driver, txn, bs, BLK_PERM_CONSISTENT_READ,
@@ -610,35 +654,9 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,/* Detect image-fleecing (and similar) schemes */ job->serialize_target_writes =bdrv_chain_contains (target, bs); - -/* If there is no backing file on the target, we cannot rely on COW if our -* backup cluster size is smaller than the target cluster size. Even for -* targets with a backing file, try to avoid COW if possible. */ - ret =bdrv_get_info (target, &bdi); -if (ret == -ENOTSUP && !target->backing) { -/* Cluster size is not defined */ -warn_report ("The target block device doesn't provide " -"information about the block size and it doesn't have a " -"backing file. The default block size of %u bytes is " -"used. If the actual block size of the target exceeds " -"this default, the backup may be unusable" , -BACKUP_CLUSTER_SIZE_DEFAULT ); - job->cluster_size =BACKUP_CLUSTER_SIZE_DEFAULT ; - }else if (ret <0 && !target->backing) { -error_setg_errno (errp, -ret, -"Couldn't determine the cluster size of the target image, " -"which has no backing file" ); -error_append_hint (errp, -"Aborting, since this may create an unusable destination image\n" ); - goto error; - }else if (ret <0 && target->backing) { -/* Not fatal; just trudge on ahead. */ - job->cluster_size =BACKUP_CLUSTER_SIZE_DEFAULT ; - }else { - job->cluster_size =MAX (BACKUP_CLUSTER_SIZE_DEFAULT , bdi.cluster_size); - } - - job->copy_bitmap =hbitmap_alloc (len,ctz32 (job->cluster_size)); + job->cluster_size = cluster_size; + job->copy_bitmap = copy_bitmap; + copy_bitmap =NULL ; job->use_copy_range = true; job->copy_range_size =MIN_NON_ZERO (blk_get_max_transfer (job->common.blk),blk_get_max_transfer (job->target));
@@ -654,6 +672,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,return &job->common; error: +if (copy_bitmap) { +assert (!job || !job->copy_bitmap); +hbitmap_free (copy_bitmap); + }if (sync_bitmap) {bdrv_reclaim_dirty_bitmap (bs, sync_bitmap,NULL ); }
Signature · 1 line
2.17.1
PATCH 58/97 block/backup: simplify backup_incremental_init_copy_bitmap
In reply to Michael RothFrom: Vladimir Sementsov-Ogievskiy <[email protected]>
Simplify backup_incremental_init_copy_bitmap using the function bdrv_dirty_bitmap_next_dirty_area.
Note: move to job->len instead of bitmap size: it should not matter but less code.
Message-id: [email protected]
*prereq for 4a5b91ca
--- block/backup.c | 40 ++++++++++++---------------------------- 1 file changed, 12 insertions(+), 28 deletions(-)
block/backup.c +12 -28
@@ -403,43 +403,27 @@ static int coroutine_fn backup_run_incremental(BackupBlockJob *job)/* init copy_bitmap from sync_bitmap */ static void backup_incremental_init_copy_bitmap (BackupBlockJob *job) { -BdrvDirtyBitmapIter *dbi; -int64_t offset; -int64_t end =DIV_ROUND_UP (bdrv_dirty_bitmap_size (job->sync_bitmap), - job->cluster_size); - - dbi =bdrv_dirty_iter_new (job->sync_bitmap); -while ((offset =bdrv_dirty_iter_next (dbi)) !=-1 ) { -int64_t cluster = offset / job->cluster_size; -int64_t next_cluster; - - offset +=bdrv_dirty_bitmap_granularity (job->sync_bitmap); -if (offset >=bdrv_dirty_bitmap_size (job->sync_bitmap)) { -hbitmap_set (job->copy_bitmap, cluster, end - cluster); -break ; - } +uint64_t offset =0 ; +uint64_t bytes = job->len; - offset =bdrv_dirty_bitmap_next_zero (job->sync_bitmap, offset, -UINT64_MAX ); -if (offset ==-1 ) { -hbitmap_set (job->copy_bitmap, cluster, end - cluster); -break ; - } +while (bdrv_dirty_bitmap_next_dirty_area (job->sync_bitmap, + &offset, &bytes)) + { +uint64_t cluster = offset / job->cluster_size; +uint64_t end_cluster =DIV_ROUND_UP (offset + bytes, job->cluster_size); - next_cluster =DIV_ROUND_UP (offset, job->cluster_size); -hbitmap_set (job->copy_bitmap, cluster, next_cluster - cluster); -if (next_cluster >= end) { +hbitmap_set (job->copy_bitmap, cluster, end_cluster - cluster); + + offset = end_cluster * job->cluster_size; +if (offset >= job->len) {break ; } - -bdrv_set_dirty_iter (dbi, next_cluster * job->cluster_size); + bytes = job->len - offset; }/* TODO job_progress_set_remaining() would make more sense */ job_progress_update (&job->common.job, job->len -hbitmap_count (job->copy_bitmap) * job->cluster_size); - -bdrv_dirty_iter_free (dbi); } static int coroutine_fn backup_run(Job *job, Error **errp)
Signature · 1 line
2.17.1
PATCH 72/97 dma-helpers: ensure AIO callback is invoked after cancellation
In reply to Michael RothFrom: Paolo Bonzini <[email protected]>
dma_aio_cancel unschedules the BH if there is one, which corresponds to the reschedule_dma case of dma_blk_cb. This can stall the DMA permanently, because dma_complete will never get invoked and therefore nobody will ever invoke the original AIO callback in dbs->common.cb.
Fix this by invoking the callback (which is ensured to happen after a bdrv_aio_cancel_async, or done manually in the dbs->bh case), and add assertions to check that the DMA state machine is indeed waiting for dma_complete or reschedule_dma, but never both.
Message-id: [email protected]
--- dma-helpers.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
dma-helpers.c +9 -4
@@ -90,6 +90,7 @@ static void reschedule_dma(void *opaque) {DMAAIOCB *dbs = (DMAAIOCB *)opaque; +assert (!dbs->acb && dbs->bh);qemu_bh_delete (dbs->bh); dbs->bh =NULL ;dma_blk_cb (dbs,0 );
@@ -111,15 +112,12 @@ static void dma_complete(DMAAIOCB *dbs, int ret) {trace_dma_complete (dbs, ret, dbs->common.cb); +assert (!dbs->acb && !dbs->bh);dma_blk_unmap (dbs);if (dbs->common.cb) { dbs->common.cb (dbs->common.opaque, ret); }qemu_iovec_destroy (&dbs->iov); -if (dbs->bh) { -qemu_bh_delete (dbs->bh); - dbs->bh =NULL ; - }qemu_aio_unref (dbs); }
@@ -179,14 +177,21 @@ static void dma_aio_cancel(BlockAIOCB *acb)trace_dma_aio_cancel (dbs); +assert (!(dbs->acb && dbs->bh));if (dbs->acb) { +/* This will invoke dma_blk_cb. */ blk_aio_cancel_async (dbs->acb); +return ; } +if (dbs->bh) {cpu_unregister_map_client (dbs->bh);qemu_bh_delete (dbs->bh); dbs->bh =NULL ; } +if (dbs->common.cb) { + dbs->common.cb (dbs->common.opaque, -ECANCELED ); + } }static AioContext *dma_get_aio_context (BlockAIOCB *acb)
Signature · 1 line
2.17.1
PATCH 56/97 tpm: Exit in reset when backend indicates failure
In reply to Michael RothFrom: Stefan Berger <[email protected]>
Exit() in the frontend reset function when the backend indicates intialization failure.
--- hw/tpm/tpm_crb.c | 4 +++- hw/tpm/tpm_tis.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-)
hw/tpm/tpm_crb.c +3 -1
@@ -273,7 +273,9 @@ static void tpm_crb_reset(void *dev) s->be_buffer_size =MIN (tpm_backend_get_buffer_size (s->tpmbe),CRB_CTRL_CMD_SIZE ); -tpm_backend_startup_tpm (s->tpmbe, s->be_buffer_size); +if (tpm_backend_startup_tpm (s->tpmbe, s->be_buffer_size) <0 ) { +exit (1 ); + } }static void tpm_crb_realize (DeviceState *dev,Error **errp)
hw/tpm/tpm_tis.c +3 -1
@@ -909,7 +909,9 @@ static void tpm_tis_reset(DeviceState *dev) s->rw_offset =0 ; } -tpm_backend_startup_tpm (s->be_driver, s->be_buffer_size); +if (tpm_backend_startup_tpm (s->be_driver, s->be_buffer_size) <0 ) { +exit (1 ); + } }/* persistent state handling */
Signature · 1 line
2.17.1
PATCH 78/97 iotests: Add supported protocols to execute_test()
In reply to Michael RothFrom: Max Reitz <[email protected]>
--- tests/qemu-iotests/iotests.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +3 -1
@@ -830,7 +830,8 @@ def execute_unittest(output, verbosity, debug): def execute_test(test_function=None, supported_fmts=[], supported_oses=['linux'], - supported_cache_modes=[], unsupported_fmts=[]): + supported_cache_modes=[], unsupported_fmts=[], + supported_protocols=[], unsupported_protocols=[]): """Run either unittest or script-style tests.""" # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to
@@ -844,6 +845,7 @@ def execute_test(test_function=None, debug = '-d' in sys.argv verbosity = 1 verify_image_format(supported_fmts, unsupported_fmts) + verify_protocol(supported_protocols, unsupported_protocols) verify_platform(supported_oses) verify_cache_mode(supported_cache_modes)
Signature · 1 line
2.17.1
PATCH 64/97 iotests: Test backup job with two guest writes
In reply to Michael RothFrom: Max Reitz <[email protected]>
Perform two guest writes to not yet backed up areas of an image, where the former touches an inner area of the latter.
Before HEAD^, copy offloading broke this in two ways:
(1) The target image differs from the reference image (what the source was when the backup started).
(2) But you will not see that in the failing output, because the job offset is reported as being greater than the job length. This is because one cluster is copied twice, and thus accounted for twice, but of course the job length does not increase.
Message-id: [email protected]
--- tests/qemu-iotests/056 | 39 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++-- 2 files changed, 41 insertions(+), 2 deletions(-)
tests/qemu-iotests/056 +39 -0
@@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase): self.vm = iotests.VM() self.test_img = img_create('test') self.dest_img = img_create('dest') + self.ref_img = img_create('ref') self.vm.add_drive(self.test_img) self.vm.launch()
@@ -140,6 +141,7 @@ class BackupTest(iotests.QMPTestCase): self.vm.shutdown() try_remove(self.test_img) try_remove(self.dest_img) + try_remove(self.ref_img) def hmp_io_writes(self, drive, patterns): for pattern in patterns:
@@ -177,6 +179,43 @@ class BackupTest(iotests.QMPTestCase): self.assert_qmp(event, 'data/error', qerror) return False + def test_overlapping_writes(self): + # Write something to back up + self.hmp_io_writes('drive0', [('42', '0M', '2M')]) + + # Create a reference backup + self.qmp_backup_and_wait(device='drive0', format=iotests.imgfmt, + sync='full', target=self.ref_img, + auto_dismiss=False) + res = self.vm.qmp('block-job-dismiss', id='drive0') + self.assert_qmp(res, 'return', {}) + + # Now to the test backup: We simulate the following guest + # writes: + # (1) [1M + 64k, 1M + 128k): Afterwards, everything in that + # area should be in the target image, and we must not copy + # it again (because the source image has changed now) + # (64k is the job's cluster size) + # (2) [1M, 2M): The backup job must not get overeager. It + # must copy [1M, 1M + 64k) and [1M + 128k, 2M) separately, + # but not the area in between. + + self.qmp_backup(device='drive0', format=iotests.imgfmt, sync='full', + target=self.dest_img, speed=1, auto_dismiss=False) + + self.hmp_io_writes('drive0', [('23', '%ik' % (1024 + 64), '64k'), + ('66', '1M', '1M')]) + + # Let the job complete + res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) + self.assert_qmp(res, 'return', {}) + self.qmp_backup_wait('drive0') + res = self.vm.qmp('block-job-dismiss', id='drive0') + self.assert_qmp(res, 'return', {}) + + self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img), + 'target image does not match reference image') + def test_dismiss_false(self): res = self.vm.qmp('query-block-jobs') self.assert_qmp(res, 'return', [])
tests/qemu-iotests/056.out +2 -2
@@ -1,5 +1,5 @@ -......... +.......... ---------------------------------------------------------------------- -Ran 9 tests +Ran 10 tests OK
Signature · 1 line
2.17.1
PATCH 77/97 vpc: Return 0 from vpc_co_create() on success
In reply to Michael RothFrom: Max Reitz <[email protected]>
blockdev_create_run() directly uses .bdrv_co_create()'s return value as the job's return value. Jobs must return 0 on success, not just any nonnegative value. Therefore, using blockdev-create for VPC images may currently fail as the vpc driver may return a positive integer.
Because there is no point in returning a positive integer anywhere in the block layer (all non-negative integers are generally treated as complete success), we probably do not want to add more such cases. Therefore, fix this problem by making the vpc driver always return 0 in case of success.
--- block/vpc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
block/vpc.c +2 -1
@@ -881,6 +881,7 @@ static int create_dynamic_disk(BlockBackend *blk, uint8_t *buf, goto fail; } + ret =0 ; fail:return ret; }
@@ -904,7 +905,7 @@ static int create_fixed_disk(BlockBackend *blk, uint8_t *buf,return ret; } -return ret; +return 0 ; } static int calculate_rounded_image_size(BlockdevCreateOptionsVpc *vpc_opts,
Signature · 1 line
2.17.1
PATCH 83/97 block/create: Do not abort if a block driver is not available
In reply to Michael RothFrom: Philippe Mathieu-Daudé <[email protected]>
The 'blockdev-create' QMP command was introduced as experimental feature in commit b0292b851b8, using the assert() debug call. It got promoted to 'stable' command in 3fb588a0f2c, but the assert call was not removed.
Some block drivers are optional, and bdrv_find_format() might return a NULL value, triggering the assertion.
Stable code is not expected to abort, so return an error instead.
This is easily reproducible when libnfs is not installed:
./configure
[...]
module support no
Block whitelist (rw)
Block whitelist (ro)
libiscsi support yes
libnfs support no
[...]
Start QEMU:
$ qemu-system-x86_64 -S -qmp unix:/tmp/qemu.qmp,server,nowait
Send the 'blockdev-create' with the 'nfs' driver:
$ ( cat << 'EOF'
{'execute': 'qmp_capabilities'}
{'execute': 'blockdev-create', 'arguments': {'job-id': 'x', 'options': {'size': 0, 'driver': 'nfs', 'location': {'path': '/', 'server': {'host': '::1', 'type': 'inet'}}}}, 'id': 'x'}
EOF
) | socat STDIO UNIX:/tmp/qemu.qmp
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 4}, "package": "v4.1.0-733-g89ea03a7dc"}, "capabilities": ["oob"]}}
{"return": {}}QEMU crashes:
$ gdb qemu-system-x86_64 core Program received signal SIGSEGV, Segmentation fault. (gdb) bt #0 0x00007ffff510957f in raise () at /lib64/libc.so.6 #1 0x00007ffff50f3895 in abort () at /lib64/libc.so.6 #2 0x00007ffff50f3769 in _nl_load_domain.cold.0 () at /lib64/libc.so.6 #3 0x00007ffff5101a26 in .annobin_assert.c_end () at /lib64/libc.so.6 #4 0x0000555555d7e1f1 in qmp_blockdev_create (job_id=0x555556baee40 "x", options=0x555557666610, errp=0x7fffffffc770) at block/create.c:69 #5 0x0000555555c96b52 in qmp_marshal_blockdev_create (args=0x7fffdc003830, ret=0x7fffffffc7f8, errp=0x7fffffffc7f0) at qapi/qapi-commands-block-core.c:1314 #6 0x0000555555deb0a0 in do_qmp_dispatch (cmds=0x55555645de70 <qmp_commands>, request=0x7fffdc005c70, allow_oob=false, errp=0x7fffffffc898) at qapi/qmp-dispatch.c:131 #7 0x0000555555deb2a1 in qmp_dispatch (cmds=0x55555645de70 <qmp_commands>, request=0x7fffdc005c70, allow_oob=false) at qapi/qmp-dispatch.c:174
With this patch applied, QEMU returns a QMP error:
{'execute': 'blockdev-create', 'arguments': {'job-id': 'x', 'options': {'size': 0, 'driver': 'nfs', 'location': {'path': '/', 'server': {'host': '::1', 'type': 'inet'}}}}, 'id': 'x'} {"id": "x", "error": {"class": "GenericError", "desc": "Block driver 'nfs' not found or not supported"}}
--- block/create.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
block/create.c +5 -1
@@ -63,9 +63,13 @@ void qmp_blockdev_create(const char *job_id, BlockdevCreateOptions *options,const char *fmt =BlockdevDriver_str (options->driver);BlockDriver *drv =bdrv_find_format (fmt); +if (!drv) { +error_setg (errp,"Block driver '%s' not found or not supported" , fmt); +return ; + } +/* If the driver is in the schema, we know that it exists. But it may not * be whitelisted. */ -assert (drv);if (bdrv_uses_whitelist () && !bdrv_is_whitelisted (drv, false)) {error_setg (errp,"Driver is not whitelisted" );return ;
Signature · 1 line
2.17.1
PATCH 70/97 Revert "ide/ahci: Check for -ECANCELED in aio callbacks"
In reply to Michael RothFrom: John Snow <[email protected]>
This reverts commit 0d910cfeaf2076b116b4517166d5deb0fea76394.
It's not correct to just ignore an error code in a callback; we need to handle that error and possible report failure to the guest so that they don't wait indefinitely for an operation that will now never finish.
This ought to help cases reported by Nutanix where iSCSI returns a legitimate -ECANCELED for certain operations which should be propagated normally.
Message-id: [email protected]
--- hw/ide/ahci.c | 3 --- hw/ide/core.c | 14 -------------- 2 files changed, 17 deletions(-)
hw/ide/ahci.c +0 -3
@@ -1022,9 +1022,6 @@ static void ncq_cb(void *opaque, int ret)IDEState *ide_state = &ncq_tfs->drive->port.ifs[0 ]; ncq_tfs->aiocb =NULL ; -if (ret == -ECANCELED ) { -return ; - }if (ret <0 ) {bool is_read = ncq_tfs->cmd ==READ_FPDMA_QUEUED ;
hw/ide/core.c +0 -14
@@ -722,9 +722,6 @@ static void ide_sector_read_cb(void *opaque, int ret) s->pio_aiocb =NULL ; s->status &= ~BUSY_STAT ; -if (ret == -ECANCELED ) { -return ; - }if (ret !=0 ) {if (ide_handle_rw_error (s, -ret,IDE_RETRY_PIO |IDE_RETRY_READ )) {
@@ -840,10 +837,6 @@ static void ide_dma_cb(void *opaque, int ret)uint64_t offset;bool stay_active = false; -if (ret == -ECANCELED ) { -return ; - } -if (ret == -EINVAL ) {ide_dma_error (s);return ;
@@ -975,10 +968,6 @@ static void ide_sector_write_cb(void *opaque, int ret)IDEState *s = opaque;int n; -if (ret == -ECANCELED ) { -return ; - } - s->pio_aiocb =NULL ; s->status &= ~BUSY_STAT ;
@@ -1058,9 +1047,6 @@ static void ide_flush_cb(void *opaque, int ret) s->pio_aiocb =NULL ; -if (ret == -ECANCELED ) { -return ; - }if (ret <0 ) {/* XXX: What sector number to set here? */ if (ide_handle_rw_error (s, -ret,IDE_RETRY_FLUSH )) {
Signature · 1 line
2.17.1
PATCH 4/97 megasas: fix mapped frame size
In reply to Michael RothFrom: Peter Lieven <[email protected]>
the current value of 1024 bytes (16 * MFI_FRAME_SIZE) we map is not enough to hold the maximum number of scatter gather elements we advertise. We actually need a maximum of 2048 bytes. This is 128 max sg elements * 16 bytes (sizeof (union mfi_sgl)).
Message-Id: <[email protected]>
--- hw/scsi/megasas.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
hw/scsi/megasas.c +1 -1
@@ -477,7 +477,7 @@ static MegasasCmd *megasas_enqueue_frame(MegasasState *s, {PCIDevice *pcid =PCI_DEVICE (s);MegasasCmd *cmd =NULL ; -int frame_size =MFI_FRAME_SIZE *16 ; +int frame_size =MEGASAS_MAX_SGE *sizeof (union mfi_sgl );hwaddr frame_size_p = frame_size;unsigned long index;
Signature · 1 line
2.17.1
PATCH 80/97 iotests: Restrict nbd Python tests to nbd
In reply to Michael RothFrom: Max Reitz <[email protected]>
We have two Python unittest-style tests that test NBD. As such, they should specify supported_protocols=['nbd'] so they are skipped when the user wants to test some other protocol.
Furthermore, we should restrict their choice of formats to 'raw'. The idea of a protocol/format combination is to use some format over some protocol; but we always use the raw format over NBD. It does not really matter what the NBD server uses on its end, and it is not a useful test of the respective format driver anyway.
--- tests/qemu-iotests/147 | 5 ++--- tests/qemu-iotests/205 | 3 ++- 2 files changed, 4 insertions(+), 4 deletions(-)
tests/qemu-iotests/147 +2 -3
@@ -287,6 +287,5 @@ class BuiltinNBD(NBDBlockdevAddBase): if __name__ == '__main__': - # Need to support image creation - iotests.main(supported_fmts=['vpc', 'parallels', 'qcow', 'vdi', 'qcow2', - 'vmdk', 'raw', 'vhdx', 'qed']) + iotests.main(supported_fmts=['raw'], + supported_protocols=['nbd'])
tests/qemu-iotests/205 +2 -1
@@ -153,4 +153,5 @@ class TestNbdServerRemove(iotests.QMPTestCase): if __name__ == '__main__': - iotests.main(supported_fmts=['generic']) + iotests.main(supported_fmts=['raw'], + supported_protocols=['nbd'])
Signature · 1 line
2.17.1
PATCH 76/97 iotests: add testing shim for script-style python tests
In reply to Michael RothFrom: John Snow <[email protected]>
Because the new-style python tests don't use the iotests.main() test launcher, we don't turn on the debugger logging for these scripts when invoked via ./check -d.
Refactor the launcher shim into new and old style shims so that they share environmental configuration.
Two cleanup notes: debug was not actually used as a global, and there was no reason to create a class in an inner scope just to achieve default variables; we can simply create an instance of the runner with the values we want instead.
Message-id: [email protected]
*prereq for 1a37e31244/88d2aa533a
--- tests/qemu-iotests/iotests.py | 40 +++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 14 deletions(-)
tests/qemu-iotests/iotests.py +26 -14
@@ -61,7 +61,6 @@ cachemode = os.environ.get('CACHEMODE') qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE') socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper') -debug = False luks_default_secret_object = 'secret,id=keysec0,data=' + \ os.environ.get('IMGKEYSECRET', '')
@@ -817,11 +816,22 @@ def skip_if_unsupported(required_formats=[], read_only=False): return func_wrapper return skip_test_decorator -def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], - unsupported_fmts=[]): - '''Run tests''' +def execute_unittest(output, verbosity, debug): + runner = unittest.TextTestRunner(stream=output, descriptions=True, + verbosity=verbosity) + try: + # unittest.main() will use sys.exit(); so expect a SystemExit + # exception + unittest.main(testRunner=runner) + finally: + if not debug: + sys.stderr.write(re.sub(r'Ran (\d+) tests? in [\d.]+s', + r'Ran \1 tests', output.getvalue())) - global debug +def execute_test(test_function=None, + supported_fmts=[], supported_oses=['linux'], + supported_cache_modes=[], unsupported_fmts=[]): + """Run either unittest or script-style tests.""" # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to # indicate that we're not being run via "check". There may be
@@ -853,13 +863,15 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN)) - class MyTestRunner(unittest.TextTestRunner): - def __init__(self, stream=output, descriptions=True, verbosity=verbosity): - unittest.TextTestRunner.__init__(self, stream, descriptions, verbosity) + if not test_function: + execute_unittest(output, verbosity, debug) + else: + test_function() + +def script_main(test_function, *args, **kwargs): + """Run script-style tests outside of the unittest framework""" + execute_test(test_function, *args, **kwargs) - # unittest.main() will use sys.exit() so expect a SystemExit exception - try: - unittest.main(testRunner=MyTestRunner) - finally: - if not debug: - sys.stderr.write(re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1 tests', output.getvalue())) +def main(*args, **kwargs): + """Run tests using the unittest framework""" + execute_test(None, *args, **kwargs)
Signature · 1 line
2.17.1
PATCH 69/97 block/backup: disable copy_range for compressed backup
In reply to Michael RothFrom: Vladimir Sementsov-Ogievskiy <[email protected]>
Enabled by default copy_range ignores compress option. It's definitely unexpected for user.
It's broken since introduction of copy_range usage in backup in 9ded4a011496.
Message-id: [email protected]
--- block/backup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/backup.c +1 -1
@@ -666,7 +666,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, job->cluster_size = cluster_size; job->copy_bitmap = copy_bitmap; copy_bitmap =NULL ; - job->use_copy_range = true; + job->use_copy_range = !compress;/* compression isn't supported for it */ job->copy_range_size =MIN_NON_ZERO (blk_get_max_transfer (job->common.blk),blk_get_max_transfer (job->target)); job->copy_range_size = MAX(job->cluster_size,
Signature · 1 line
2.17.1
PATCH 68/97 iotests: Test unaligned blocking mirror write
In reply to Michael RothFrom: Max Reitz <[email protected]>
Message-id: [email protected]
--- tests/qemu-iotests/151 | 25 +++++++++++++++++++++++++ tests/qemu-iotests/151.out | 4 ++-- 2 files changed, 27 insertions(+), 2 deletions(-)
tests/qemu-iotests/151 +25 -0
@@ -114,6 +114,31 @@ class TestActiveMirror(iotests.QMPTestCase): def testActiveIOFlushed(self): self.doActiveIO(True) + def testUnalignedActiveIO(self): + # Fill the source image + result = self.vm.hmp_qemu_io('source', 'write -P 1 0 2M') + + # Start the block job (very slowly) + result = self.vm.qmp('blockdev-mirror', + job_id='mirror', + filter_node_name='mirror-node', + device='source-node', + target='target-node', + sync='full', + copy_mode='write-blocking', + buf_size=(1048576 // 4), + speed=1) + self.assert_qmp(result, 'return', {}) + + # Start an unaligned request to a dirty area + result = self.vm.hmp_qemu_io('source', 'write -P 2 %i 1' % (1048576 + 42)) + + # Let the job finish + result = self.vm.qmp('block-job-set-speed', device='mirror', speed=0) + self.assert_qmp(result, 'return', {}) + self.complete_and_wait(drive='mirror') + + self.potential_writes_in_flight = False if __name__ == '__main__':
tests/qemu-iotests/151.out +2 -2
@@ -1,5 +1,5 @@ -.. +... ---------------------------------------------------------------------- -Ran 2 tests +Ran 3 tests OK
Signature · 1 line
2.17.1
PATCH 88/97 curl: Check completion in curl_multi_do()
In reply to Michael RothFrom: Max Reitz <[email protected]>
While it is more likely that transfers complete after some file descriptor has data ready to read, we probably should not rely on it. Better be safe than sorry and call curl_multi_check_completion() in curl_multi_do(), too, just like it is done in curl_multi_read().
With this change, curl_multi_do() and curl_multi_read() are actually the same, so drop curl_multi_read() and use curl_multi_do() as the sole FD handler.
Message-id: [email protected]
--- block/curl.c | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-)
block/curl.c +2 -12
@@ -138,7 +138,6 @@ typedef struct BDRVCURLState {static void curl_clean_state (CURLState *s);static void curl_multi_do (void *arg); -static void curl_multi_read (void *arg);#ifdef NEED_CURL_TIMER_CALLBACK /* Called from curl_multi_do_locked, with s->mutex held. */
@@ -185,7 +184,7 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,switch (action) {case CURL_POLL_IN :aio_set_fd_handler (s->aio_context, fd, false, - curl_multi_read,NULL ,NULL , state); + curl_multi_do,NULL ,NULL , state);break ;case CURL_POLL_OUT : aio_set_fd_handler(s->aio_context, fd, false,
@@ -193,7 +192,7 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,break ;case CURL_POLL_INOUT :aio_set_fd_handler (s->aio_context, fd, false, - curl_multi_read, curl_multi_do,NULL , state); + curl_multi_do, curl_multi_do,NULL , state);break ;case CURL_POLL_REMOVE : aio_set_fd_handler(s->aio_context, fd, false,
@@ -415,15 +414,6 @@ static void curl_multi_do(void *arg) {CURLState *s = (CURLState *)arg; -qemu_mutex_lock (&s->s->mutex); -curl_multi_do_locked (s); -qemu_mutex_unlock (&s->s->mutex); -} - -static void curl_multi_read (void *arg) -{ -CURLState *s = (CURLState *)arg; -qemu_mutex_lock (&s->s->mutex);curl_multi_do_locked (s);curl_multi_check_completion (s->s);
Signature · 1 line
2.17.1
PATCH 67/97 mirror: Only mirror granularity-aligned chunks
In reply to Michael RothFrom: Max Reitz <[email protected]>
In write-blocking mode, all writes to the top node directly go to the target. We must only mirror chunks of data that are aligned to the job's granularity, because that is how the dirty bitmap works. Therefore, the request alignment for writes must be the job's granularity (in write-blocking mode).
Unfortunately, this forces all reads and writes to have the same granularity (we only need this alignment for writes to the target, not the source), but that is something to be fixed another time.
Message-id: [email protected]
--- block/mirror.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
block/mirror.c +29 -0
@@ -1477,6 +1477,15 @@ static void bdrv_mirror_top_child_perm(BlockDriverState *bs, BdrvChild *c, *nshared =BLK_PERM_ALL ; } +static void bdrv_mirror_top_refresh_limits (BlockDriverState *bs,Error **errp) +{ +MirrorBDSOpaque *s = bs->opaque; + +if (s && s->job && s->job->copy_mode ==MIRROR_COPY_MODE_WRITE_BLOCKING ) { + bs->bl.request_alignment = s->job->granularity; + } +} +/* Dummy node that provides consistent read to its users without requiring it * from its backing file and that allows writes on the backing file chain. */ static BlockDriver bdrv_mirror_top = {
@@ -1489,6 +1498,7 @@ static BlockDriver bdrv_mirror_top = { .bdrv_co_block_status = bdrv_co_block_status_from_backing, .bdrv_refresh_filename = bdrv_mirror_top_refresh_filename, .bdrv_child_perm = bdrv_mirror_top_child_perm, + .bdrv_refresh_limits = bdrv_mirror_top_refresh_limits, }; static void mirror_start_job(const char *job_id, BlockDriverState *bs,
@@ -1627,6 +1637,25 @@ static void mirror_start_job(const char *job_id, BlockDriverState *bs, s->should_complete = true; } +/* +* Must be called before we start tracking writes, but after +* +* ((MirrorBlockJob *) +* ((MirrorBDSOpaque *) +* mirror_top_bs->opaque +* )->job +* )->copy_mode +* +* has the correct value. +* (We start tracking writes as of the following +* bdrv_create_dirty_bitmap() call.) +*/ +bdrv_refresh_limits (mirror_top_bs, &local_err); +if (local_err) { +error_propagate (errp, local_err); + goto fail; + } + s->dirty_bitmap =bdrv_create_dirty_bitmap (bs, granularity,NULL , errp);if (!s->dirty_bitmap) { goto fail;
Signature · 1 line
2.17.1
PATCH 91/97 curl: Handle success in multi_check_completion
In reply to Michael RothFrom: Max Reitz <[email protected]>
Background: As of cURL 7.59.0, it verifies that several functions are not called from within a callback. Among these functions is curl_multi_add_handle().
curl_read_cb() is a callback from cURL and not a coroutine. Waking up acb->co will lead to entering it then and there, which means the current request will settle and the caller (if it runs in the same coroutine) may then issue the next request. In such a case, we will enter curl_setup_preadv() effectively from within curl_read_cb().
Calling curl_multi_add_handle() will then fail and the new request will not be processed.
Fix this by not letting curl_read_cb() wake up acb->co. Instead, leave the whole business of settling the AIOCB objects to curl_multi_check_completion() (which is called from our timer callback and our FD handler, so not from any cURL callbacks).
Message-id: [email protected]
--- block/curl.c | 69 ++++++++++++++++++++++------------------------------ 1 file changed, 29 insertions(+), 40 deletions(-)
block/curl.c +29 -40
@@ -228,7 +228,6 @@ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opaque) {CURLState *s = ((CURLState *)opaque);size_t realsize = size * nmemb; -int i;trace_curl_read_cb (realsize);
@@ -244,32 +243,6 @@ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opaque)memcpy (s->orig_buf + s->buf_off, ptr, realsize); s->buf_off += realsize; -for (i=0 ; i<CURL_NUM_ACB ; i++) { -CURLAIOCB *acb = s->acb[i]; - -if (!acb) -continue ; - -if ((s->buf_off >= acb->end)) { -size_t request_length = acb->bytes; - -qemu_iovec_from_buf (acb->qiov,0 , s->orig_buf + acb->start, - acb->end - acb->start); - -if (acb->end - acb->start < request_length) { -size_t offset = acb->end - acb->start; -qemu_iovec_memset (acb->qiov, offset,0 , - request_length - offset); - } - - acb->ret =0 ; - s->acb[i] =NULL ; -qemu_mutex_unlock (&s->s->mutex); -aio_co_wake (acb->co); -qemu_mutex_lock (&s->s->mutex); - } - } - read_end:/* curl will error out if we do not return this value */ return size * nmemb;
@@ -350,13 +323,14 @@ static void curl_multi_check_completion(BDRVCURLState *s)break ;if (msg->msg ==CURLMSG_DONE ) { +int i;CURLState *state =NULL ; +bool error = msg->data.result !=CURLE_OK ; +curl_easy_getinfo (msg->easy_handle,CURLINFO_PRIVATE , (char **)&state); -/* ACBs for successful messages get completed in curl_read_cb */ -if (msg->data.result !=CURLE_OK ) { -int i; +if (error) {static int errcount =100 ;/* Don't lose the original error message from curl, since
@@ -368,20 +342,35 @@ static void curl_multi_check_completion(BDRVCURLState *s)error_report ("curl: further errors suppressed" ); } } + } -for (i =0 ; i <CURL_NUM_ACB ; i++) { -CURLAIOCB *acb = state->acb[i]; +for (i =0 ; i <CURL_NUM_ACB ; i++) { +CURLAIOCB *acb = state->acb[i]; -if (acb ==NULL ) { -continue ; - } +if (acb ==NULL ) { +continue ; + } + +if (!error) { +/* Assert that we have read all data */ +assert (state->buf_off >= acb->end); + +qemu_iovec_from_buf (acb->qiov,0 , + state->orig_buf + acb->start, + acb->end - acb->start); - acb->ret = -EIO ; - state->acb[i] =NULL ; -qemu_mutex_unlock (&s->mutex); -aio_co_wake (acb->co); -qemu_mutex_lock (&s->mutex); +if (acb->end - acb->start < acb->bytes) { +size_t offset = acb->end - acb->start; +qemu_iovec_memset (acb->qiov, offset,0 , + acb->bytes - offset); + } } + + acb->ret = error ? -EIO :0 ; + state->acb[i] =NULL ; +qemu_mutex_unlock (&s->mutex); +aio_co_wake (acb->co); +qemu_mutex_lock (&s->mutex); }curl_clean_state (state);
Signature · 1 line
2.17.1
PATCH 8/97 Makefile: add nit-picky mode to sphinx-build
In reply to Michael RothFrom: John Snow <[email protected]>
If we add references that don't resolve (or accidentally remove them), it will be helpful to have warning messages alerting us to that.
Further, turn those warnings into errors so we can be alerted to these problems sooner rather than later.
Message-id: [email protected]
--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
@@ -899,7 +899,7 @@ docs/version.texi: $(SRC_PATH)/VERSION sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html $(MANUAL_BUILDDIR)/interop/index.html # Canned command to build a single manual -build-manual = $(call quiet-command,sphinx-build $(if $(V),,-q) -b html -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1") +build-manual = $(call quiet-command,sphinx-build $(if $(V),,-q) -W -n -b html -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1") # We assume all RST files in the manual's directory are used in it manual-deps = $(wildcard $(SRC_PATH)/docs/$1/*.rst) $(SRC_PATH)/docs/$1/conf.py $(SRC_PATH)/docs/conf.py
Signature · 1 line
2.17.1
PATCH 89/97 curl: Pass CURLSocket to curl_multi_do()
In reply to Michael RothFrom: Max Reitz <[email protected]>
curl_multi_do_locked() currently marks all sockets as ready. That is not only inefficient, but in fact unsafe (the loop is). A follow-up patch will change that, but to do so, curl_multi_do_locked() needs to know exactly which socket is ready; and that is accomplished by this patch here.
Message-id: [email protected]
--- block/curl.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)
block/curl.c +11 -9
@@ -184,15 +184,15 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,switch (action) {case CURL_POLL_IN :aio_set_fd_handler (s->aio_context, fd, false, - curl_multi_do,NULL ,NULL , state); + curl_multi_do,NULL ,NULL , socket);break ;case CURL_POLL_OUT :aio_set_fd_handler (s->aio_context, fd, false, -NULL , curl_multi_do,NULL , state); +NULL , curl_multi_do,NULL , socket);break ;case CURL_POLL_INOUT :aio_set_fd_handler (s->aio_context, fd, false, - curl_multi_do, curl_multi_do,NULL , state); + curl_multi_do, curl_multi_do,NULL , socket);break ;case CURL_POLL_REMOVE : aio_set_fd_handler(s->aio_context, fd, false,
@@ -391,9 +391,10 @@ static void curl_multi_check_completion(BDRVCURLState *s) }/* Called with s->mutex held. */ -static void curl_multi_do_locked (CURLState *s) +static void curl_multi_do_locked (CURLSocket *ready_socket) {CURLSocket *socket, *next_socket; +CURLState *s = ready_socket->state;int running;int r;
@@ -412,12 +413,13 @@ static void curl_multi_do_locked(CURLState *s)static void curl_multi_do (void *arg) { -CURLState *s = (CURLState *)arg; +CURLSocket *socket = arg; +BDRVCURLState *s = socket->state->s; -qemu_mutex_lock (&s->s->mutex); -curl_multi_do_locked (s); -curl_multi_check_completion (s->s); -qemu_mutex_unlock (&s->s->mutex); +qemu_mutex_lock (&s->mutex); +curl_multi_do_locked (socket); +curl_multi_check_completion (s); +qemu_mutex_unlock (&s->mutex); }static void curl_multi_timeout_do (void *arg)
Signature · 1 line
2.17.1
PATCH 60/97 block/backup: refactor and tolerate unallocated cluster skipping
In reply to Michael RothFrom: Vladimir Sementsov-Ogievskiy <[email protected]>
Split allocation checking to separate function and reduce nesting. Consider bdrv_is_allocated() fail as allocated area, as copying more than needed is not wrong (and we do it anyway) and seems better than fail the whole job. And, most probably we will fail on the next read, if there are real problem with source.
Message-id: [email protected]
*prereq for 110571be4e
--- block/backup.c | 60 +++++++++++++++++++------------------------------- 1 file changed, 23 insertions(+), 37 deletions(-)
block/backup.c +23 -37
@@ -377,6 +377,22 @@ static bool coroutine_fn yield_and_check(BackupBlockJob *job)return false; } +static bool bdrv_is_unallocated_range (BlockDriverState *bs, +int64_t offset,int64_t bytes) +{ +int64_t end = offset + bytes; + +while (offset < end && !bdrv_is_allocated (bs, offset, bytes, &bytes)) { +if (bytes ==0 ) { +return true; + } + offset += bytes; + bytes = end - offset; + } + +return offset >= end; +} + static int coroutine_fn backup_run_incremental(BackupBlockJob *job) {int ret;
@@ -462,49 +478,19 @@ static int coroutine_fn backup_run(Job *job, Error **errp)for (offset =0 ; offset < s->len; offset += s->cluster_size) {bool error_is_read; -int alloced =0 ;if (yield_and_check (s)) {break ; } -if (s->sync_mode ==MIRROR_SYNC_MODE_TOP ) { -int i; -int64_t n; - -/* Check to see if these blocks are already in the -* backing file. */ - -for (i =0 ; i < s->cluster_size;) { -/* bdrv_is_allocated() only returns true/false based -* on the first set of sectors it comes across that -* are are all in the same state. -* For that reason we must verify each sector in the -* backup cluster length. We end up copying more than -* needed but at some point that is always the case. */ - alloced = -bdrv_is_allocated (bs, offset + i, - s->cluster_size - i, &n); - i += n; - -if (alloced || n ==0 ) { -break ; - } - } - -/* If the above loop never found any sectors that are in -* the topmost image, skip this backup. */ -if (alloced ==0 ) { -continue ; - } - } -/* FULL sync mode we copy the whole drive. */ -if (alloced <0 ) { - ret = alloced; - }else { - ret =backup_do_cow (s, offset, s->cluster_size, - &error_is_read, false); +if (s->sync_mode ==MIRROR_SYNC_MODE_TOP && +bdrv_is_unallocated_range (bs, offset, s->cluster_size)) + { +continue ; } + + ret =backup_do_cow (s, offset, s->cluster_size, + &error_is_read, false);if (ret <0 ) {/* Depending on error action, fail now or retry cluster */ BlockErrorAction action =
Signature · 1 line
2.17.1
PATCH 84/97 block/nfs: tear down aio before nfs_close
In reply to Michael RothFrom: Peter Lieven <[email protected]>
nfs_close is a sync call from libnfs and has its own event handler polling on the nfs FD. Avoid that both QEMU and libnfs are intefering here.
--- block/nfs.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
block/nfs.c +4 -2
@@ -389,12 +389,14 @@ static void nfs_attach_aio_context(BlockDriverState *bs,static void nfs_client_close (NFSClient *client) {if (client->context) { +qemu_mutex_lock (&client->mutex); +aio_set_fd_handler (client->aio_context,nfs_get_fd (client->context), + false,NULL ,NULL ,NULL ,NULL ); +qemu_mutex_unlock (&client->mutex);if (client->fh) {nfs_close (client->context, client->fh); client->fh =NULL ; } -aio_set_fd_handler (client->aio_context,nfs_get_fd (client->context), - false,NULL ,NULL ,NULL ,NULL );nfs_destroy_context (client->context); client->context =NULL ; }
Signature · 1 line
2.17.1
PATCH 71/97 qcow2: Fix the calculation of the maximum L2 cache size
In reply to Michael RothFrom: Alberto Garcia <[email protected]>
The size of the qcow2 L2 cache defaults to 32 MB, which can be easily larger than the maximum amount of L2 metadata that the image can have. For example: with 64 KB clusters the user would need a qcow2 image with a virtual size of 256 GB in order to have 32 MB of L2 metadata.
Because of that, since commit b749562d9822d14ef69c9eaa5f85903010b86c30 we forbid the L2 cache to become larger than the maximum amount of L2 metadata for the image, calculated using this formula:
uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size / 8);
The problem with this formula is that the result should be rounded up to the cluster size because an L2 table on disk always takes one full cluster.
For example, a 1280 MB qcow2 image with 64 KB clusters needs exactly 160 KB of L2 metadata, but we need 192 KB on disk (3 clusters) even if the last 32 KB of those are not going to be used.
However QEMU rounds the numbers down and only creates 2 cache tables (128 KB), which is not enough for the image.
A quick test doing 4KB random writes on a 1280 MB image gives me around 500 IOPS, while with the correct cache size I get 16K IOPS.
--- block/qcow2.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
block/qcow2.c +5 -1
@@ -831,7 +831,11 @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts,bool l2_cache_entry_size_set;int min_refcount_cache =MIN_REFCOUNT_CACHE_SIZE * s->cluster_size;uint64_t virtual_disk_size = bs->total_sectors *BDRV_SECTOR_SIZE ; -uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size /8 ); +uint64_t max_l2_entries =DIV_ROUND_UP (virtual_disk_size, s->cluster_size); +/* An L2 table is always one cluster in size so the max cache size +* should be a multiple of the cluster size. */ +uint64_t max_l2_cache =ROUND_UP (max_l2_entries *sizeof (uint64_t ), + s->cluster_size); combined_cache_size_set =qemu_opt_get (opts,QCOW2_OPT_CACHE_SIZE ); l2_cache_size_set =qemu_opt_get (opts,QCOW2_OPT_L2_CACHE_SIZE );
Signature · 1 line
2.17.1
PATCH 81/97 iotests: Test blockdev-create for vpc
In reply to Michael RothFrom: Max Reitz <[email protected]>
Conflicts:
tests/qemu-iotests/group
*drop context dep. on tests not in 4.0
--- tests/qemu-iotests/266 | 153 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/266.out | 137 +++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 291 insertions(+) create mode 100755 tests/qemu-iotests/266 create mode 100644 tests/qemu-iotests/266.out
tests/qemu-iotests/266 new file +153 -0
@@ -0,0 +1,153 @@ +#!/usr/bin/env python +# +# Test VPC and file image creation +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import iotests +from iotests import imgfmt + + +def blockdev_create(vm, options): + result = vm.qmp_log('blockdev-create', job_id='job0', options=options, + filters=[iotests.filter_qmp_testfiles]) + + if 'return' in result: + assert result['return'] == {} + vm.run_job('job0') + + +# Successful image creation (defaults) +def implicit_defaults(vm, file_path): + iotests.log("=== Successful image creation (defaults) ===") + iotests.log("") + + # 8 heads, 964 cyls/head, 17 secs/cyl + # (Close to 64 MB) + size = 8 * 964 * 17 * 512 + + blockdev_create(vm, { 'driver': imgfmt, + 'file': 'protocol-node', + 'size': size }) + + +# Successful image creation (explicit defaults) +def explicit_defaults(vm, file_path): + iotests.log("=== Successful image creation (explicit defaults) ===") + iotests.log("") + + # 16 heads, 964 cyls/head, 17 secs/cyl + # (Close to 128 MB) + size = 16 * 964 * 17 * 512 + + blockdev_create(vm, { 'driver': imgfmt, + 'file': 'protocol-node', + 'size': size, + 'subformat': 'dynamic', + 'force-size': False }) + + +# Successful image creation (non-default options) +def non_defaults(vm, file_path): + iotests.log("=== Successful image creation (non-default options) ===") + iotests.log("") + + # Not representable in CHS (fine with force-size=True) + size = 1048576 + + blockdev_create(vm, { 'driver': imgfmt, + 'file': 'protocol-node', + 'size': size, + 'subformat': 'fixed', + 'force-size': True }) + + +# Size not representable in CHS with force-size=False +def non_chs_size_without_force(vm, file_path): + iotests.log("=== Size not representable in CHS ===") + iotests.log("") + + # Not representable in CHS (will not work with force-size=False) + size = 1048576 + + blockdev_create(vm, { 'driver': imgfmt, + 'file': 'protocol-node', + 'size': size, + 'force-size': False }) + + +# Zero size +def zero_size(vm, file_path): + iotests.log("=== Zero size===") + iotests.log("") + + blockdev_create(vm, { 'driver': imgfmt, + 'file': 'protocol-node', + 'size': 0 }) + + +# Maximum CHS size +def maximum_chs_size(vm, file_path): + iotests.log("=== Maximum CHS size===") + iotests.log("") + + blockdev_create(vm, { 'driver': imgfmt, + 'file': 'protocol-node', + 'size': 16 * 65535 * 255 * 512 }) + + +# Actual maximum size +def maximum_size(vm, file_path): + iotests.log("=== Actual maximum size===") + iotests.log("") + + blockdev_create(vm, { 'driver': imgfmt, + 'file': 'protocol-node', + 'size': 0xff000000 * 512, + 'force-size': True }) + + +def main(): + for test_func in [implicit_defaults, explicit_defaults, non_defaults, + non_chs_size_without_force, zero_size, maximum_chs_size, + maximum_size]: + + with iotests.FilePath('t.vpc') as file_path, \ + iotests.VM() as vm: + + vm.launch() + + iotests.log('--- Creating empty file ---') + blockdev_create(vm, { 'driver': 'file', + 'filename': file_path, + 'size': 0 }) + + vm.qmp_log('blockdev-add', driver='file', filename=file_path, + node_name='protocol-node', + filters=[iotests.filter_qmp_testfiles]) + iotests.log('') + + print_info = test_func(vm, file_path) + iotests.log('') + + vm.shutdown() + iotests.img_info_log(file_path) + + +iotests.script_main(main, + supported_fmts=['vpc'], + supported_protocols=['file'])
tests/qemu-iotests/266.out new file +137 -0
137 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -251,3 +251,4 @@ 253 rw auto quick 255 rw auto quick 256 rw auto quick +266 rw quick
Signature · 1 line
2.17.1
PATCH 59/97 block/backup: move to copy_bitmap with granularity
In reply to Michael RothFrom: Vladimir Sementsov-Ogievskiy <[email protected]>
We are going to share this bitmap between backup and backup-top filter driver, so let's share something more meaningful. It also simplifies some calculations.
Message-id: [email protected]
*prereq for 4a5b91ca
--- block/backup.c | 48 +++++++++++++++++++++++------------------------- 1 file changed, 23 insertions(+), 25 deletions(-)
block/backup.c +23 -25
@@ -113,7 +113,8 @@ static int coroutine_fn backup_cow_with_bounce_buffer(BackupBlockJob *job,int read_flags = is_write_notifier ?BDRV_REQ_NO_SERIALISING :0 ;int write_flags = job->serialize_target_writes ?BDRV_REQ_SERIALISING :0 ; -hbitmap_reset (job->copy_bitmap, start / job->cluster_size,1 ); +assert (QEMU_IS_ALIGNED (start, job->cluster_size)); +hbitmap_reset (job->copy_bitmap, start, job->cluster_size); nbytes =MIN (job->cluster_size, job->len - start);if (!*bounce_buffer) { *bounce_buffer =blk_blockalign (blk, job->cluster_size);
@@ -147,7 +148,7 @@ static int coroutine_fn backup_cow_with_bounce_buffer(BackupBlockJob *job,return nbytes; fail: -hbitmap_set (job->copy_bitmap, start / job->cluster_size,1 ); +hbitmap_set (job->copy_bitmap, start, job->cluster_size);return ret; }
@@ -167,16 +168,15 @@ static int coroutine_fn backup_cow_with_offload(BackupBlockJob *job,int write_flags = job->serialize_target_writes ?BDRV_REQ_SERIALISING :0 ;assert (QEMU_IS_ALIGNED (job->copy_range_size, job->cluster_size)); +assert (QEMU_IS_ALIGNED (start, job->cluster_size)); nbytes =MIN (job->copy_range_size, end - start); nr_clusters =DIV_ROUND_UP (nbytes, job->cluster_size); -hbitmap_reset (job->copy_bitmap, start / job->cluster_size, - nr_clusters); +hbitmap_reset (job->copy_bitmap, start, job->cluster_size * nr_clusters); ret =blk_co_copy_range (blk, start, job->target, start, nbytes, read_flags, write_flags);if (ret <0 ) {trace_backup_do_cow_copy_range_fail (job, start, ret); -hbitmap_set (job->copy_bitmap, start / job->cluster_size, - nr_clusters); +hbitmap_set (job->copy_bitmap, start, job->cluster_size * nr_clusters);return ret; }
@@ -204,7 +204,7 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,cow_request_begin (&cow_request, job, start, end);while (start < end) { -if (!hbitmap_get (job->copy_bitmap, start / job->cluster_size)) { +if (!hbitmap_get (job->copy_bitmap, start)) {trace_backup_do_cow_skip (job, start); start += job->cluster_size;continue ;/* already copied */
@@ -300,6 +300,11 @@ static void backup_clean(Job *job)assert (s->target);blk_unref (s->target); s->target =NULL ; + +if (s->copy_bitmap) { +hbitmap_free (s->copy_bitmap); + s->copy_bitmap =NULL ; + } }static void backup_attached_aio_context (BlockJob *job,AioContext *aio_context)
@@ -312,7 +317,6 @@ static void backup_attached_aio_context(BlockJob *job, AioContext *aio_context)void backup_do_checkpoint (BlockJob *job,Error **errp) {BackupBlockJob *backup_job =container_of (job, BackupBlockJob, common); -int64_t len;assert (block_job_driver (job) == &backup_job_driver);
@@ -322,8 +326,7 @@ void backup_do_checkpoint(BlockJob *job, Error **errp)return ; } - len =DIV_ROUND_UP (backup_job->len, backup_job->cluster_size); -hbitmap_set (backup_job->copy_bitmap,0 , len); +hbitmap_set (backup_job->copy_bitmap,0 , backup_job->len); }static void backup_drain (BlockJob *job)
@@ -378,16 +381,16 @@ static int coroutine_fn backup_run_incremental(BackupBlockJob *job) {int ret;bool error_is_read; -int64_t cluster; +int64_t offset;HBitmapIter hbi;hbitmap_iter_init (&hbi, job->copy_bitmap,0 ); -while ((cluster =hbitmap_iter_next (&hbi)) !=-1 ) { +while ((offset =hbitmap_iter_next (&hbi)) !=-1 ) { do { if (yield_and_check(job)) { return 0; } - ret = backup_do_cow(job, cluster * job->cluster_size, + ret = backup_do_cow(job, offset, job->cluster_size, &error_is_read, false); if (ret < 0 && backup_error_action(job, error_is_read, -ret) == BLOCK_ERROR_ACTION_REPORT)
@@ -409,12 +412,9 @@ static void backup_incremental_init_copy_bitmap(BackupBlockJob *job)while (bdrv_dirty_bitmap_next_dirty_area (job->sync_bitmap, &offset, &bytes)) { -uint64_t cluster = offset / job->cluster_size; -uint64_t end_cluster =DIV_ROUND_UP (offset + bytes, job->cluster_size); +hbitmap_set (job->copy_bitmap, offset, bytes); -hbitmap_set (job->copy_bitmap, cluster, end_cluster - cluster); - - offset = end_cluster * job->cluster_size; + offset += bytes;if (offset >= job->len) {break ; }
@@ -423,30 +423,27 @@ static void backup_incremental_init_copy_bitmap(BackupBlockJob *job)/* TODO job_progress_set_remaining() would make more sense */ job_progress_update (&job->common.job, - job->len -hbitmap_count (job->copy_bitmap) * job->cluster_size); + job->len -hbitmap_count (job->copy_bitmap)); } static int coroutine_fn backup_run(Job *job, Error **errp) {BackupBlockJob *s =container_of (job, BackupBlockJob, common.job);BlockDriverState *bs =blk_bs (s->common.blk); -int64_t offset, nb_clusters; +int64_t offset;int ret =0 ;QLIST_INIT (&s->inflight_reqs);qemu_co_rwlock_init (&s->flush_rwlock); - nb_clusters =DIV_ROUND_UP (s->len, s->cluster_size);job_progress_set_remaining (job, s->len); - s->copy_bitmap =hbitmap_alloc (nb_clusters,0 );if (s->sync_mode ==MIRROR_SYNC_MODE_INCREMENTAL ) {backup_incremental_init_copy_bitmap (s); }else { -hbitmap_set (s->copy_bitmap,0 , nb_clusters); +hbitmap_set (s->copy_bitmap,0 , s->len); } - s->before_write.notify = backup_before_write_notify;bdrv_add_before_write_notifier (bs, &s->before_write);
@@ -527,7 +524,6 @@ static int coroutine_fn backup_run(Job *job, Error **errp)/* wait until pending backup_do_cow() calls have completed */ qemu_co_rwlock_wrlock (&s->flush_rwlock);qemu_co_rwlock_unlock (&s->flush_rwlock); -hbitmap_free (s->copy_bitmap);return ret; }
@@ -678,6 +674,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, }else { job->cluster_size =MAX (BACKUP_CLUSTER_SIZE_DEFAULT , bdi.cluster_size); } + + job->copy_bitmap =hbitmap_alloc (len,ctz32 (job->cluster_size)); job->use_copy_range = true; job->copy_range_size =MIN_NON_ZERO (blk_get_max_transfer (job->common.blk),blk_get_max_transfer (job->target));
Signature · 1 line
2.17.1
PATCH 97/97 scsi: lsi: exit infinite loop while executing script (CVE-2019-12068)
In reply to Michael RothFrom: Paolo Bonzini <[email protected]>
When executing script in lsi_execute_script(), the LSI scsi adapter emulator advances 's->dsp' index to read next opcode. This can lead to an infinite loop if the next opcode is empty. Move the existing loop exit after 10k iterations so that it covers no-op opcodes as well.
--- hw/scsi/lsi53c895a.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-)
hw/scsi/lsi53c895a.c +27 -14
@@ -184,6 +184,9 @@ static const char *names[] = {/* Flag set if this is a tagged command. */ #define LSI_TAG_VALID (1 << 16) +/* Maximum instructions to process. */ +#define LSI_MAX_INSN 10000 + typedef struct lsi_request { SCSIRequest *req; uint32_t tag;
@@ -1131,7 +1134,21 @@ static void lsi_execute_script(LSIState *s) s->istat1 |=LSI_ISTAT1_SRUN ; again: - insn_processed++; +if (++insn_processed >LSI_MAX_INSN ) { +/* Some windows drivers make the device spin waiting for a memory +location to change. If we have been executed a lot of code then +assume this is the case and force an unexpected device disconnect. +This is apparently sufficient to beat the drivers into submission. +*/ +if (!(s->sien0 &LSI_SIST0_UDC )) { +qemu_log_mask (LOG_GUEST_ERROR , +"lsi_scsi: inf. loop with UDC masked" ); + } +lsi_script_scsi_interrupt (s,LSI_SIST0_UDC ,0 ); +lsi_disconnect (s); +trace_lsi_execute_script_stop (); +return ; + } insn =read_dword (s, s->dsp);if (!insn) {/* If we receive an empty opcode increment the DSP by 4 bytes
@@ -1568,19 +1585,7 @@ again: } } } -if (insn_processed >10000 && s->waiting ==LSI_NOWAIT ) { -/* Some windows drivers make the device spin waiting for a memory -location to change. If we have been executed a lot of code then -assume this is the case and force an unexpected device disconnect. -This is apparently sufficient to beat the drivers into submission. -*/ -if (!(s->sien0 &LSI_SIST0_UDC )) { -qemu_log_mask (LOG_GUEST_ERROR , -"lsi_scsi: inf. loop with UDC masked" ); - } -lsi_script_scsi_interrupt (s,LSI_SIST0_UDC ,0 ); -lsi_disconnect (s); - }else if (s->istat1 &LSI_ISTAT1_SRUN && s->waiting ==LSI_NOWAIT ) { +if (s->istat1 &LSI_ISTAT1_SRUN && s->waiting ==LSI_NOWAIT ) {if (s->dcntl &LSI_DCNTL_SSM ) {lsi_script_dma_interrupt (s,LSI_DSTAT_SSI ); }else {
@@ -1968,6 +1973,10 @@ static void lsi_reg_writeb(LSIState *s, int offset, uint8_t val)case 0x2f :/* DSP[24:31] */ s->dsp &=0x00ffffff ; s->dsp |= val <<24 ; +/* +* FIXME: if s->waiting != LSI_NOWAIT, this will only execute one +* instruction. Is this correct? +*/ if ((s->dmode &LSI_DMODE_MAN ) ==0 && (s->istat1 &LSI_ISTAT1_SRUN ) ==0 )lsi_execute_script (s);
@@ -1986,6 +1995,10 @@ static void lsi_reg_writeb(LSIState *s, int offset, uint8_t val)break ;case 0x3b :/* DCNTL */ s->dcntl = val & ~(LSI_DCNTL_PFF |LSI_DCNTL_STD ); +/* +* FIXME: if s->waiting != LSI_NOWAIT, this will only execute one +* instruction. Is this correct? +*/ if ((val &LSI_DCNTL_STD ) && (s->istat1 &LSI_ISTAT1_SRUN ) ==0 )lsi_execute_script (s);break ;
Signature · 1 line
2.17.1
PATCH 90/97 curl: Report only ready sockets
In reply to Michael RothFrom: Max Reitz <[email protected]>
Instead of reporting all sockets to cURL, only report the one that has caused curl_multi_do_locked() to be called. This lets us get rid of the QLIST_FOREACH_SAFE() list, which was actually wrong: SAFE foreaches are only safe when the current element is removed in each iteration. If it possible for the list to be concurrently modified, we cannot guarantee that only the current element will be removed. Therefore, we must not use QLIST_FOREACH_SAFE() here.
Message-id: [email protected]
--- block/curl.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
block/curl.c +6 -11
@@ -391,24 +391,19 @@ static void curl_multi_check_completion(BDRVCURLState *s) }/* Called with s->mutex held. */ -static void curl_multi_do_locked (CURLSocket *ready_socket) +static void curl_multi_do_locked (CURLSocket *socket) { -CURLSocket *socket, *next_socket; -CURLState *s = ready_socket->state; +BDRVCURLState *s = socket->state->s;int running;int r; -if (!s->s->multi) { +if (!s->multi) {return ; } -/* Need to use _SAFE because curl_multi_socket_action() may trigger -* curl_sock_cb() which might modify this list */ - QLIST_FOREACH_SAFE(socket, &s->sockets, next, next_socket) { -do { - r =curl_multi_socket_action (s->s->multi, socket->fd,0 , &running); - }while (r ==CURLM_CALL_MULTI_PERFORM ); - } +do { + r =curl_multi_socket_action (s->multi, socket->fd,0 , &running); + }while (r ==CURLM_CALL_MULTI_PERFORM ); }static void curl_multi_do (void *arg)
Signature · 1 line
2.17.1
PATCH 7/97 cutils: Fix size_to_str() on 32-bit platforms
In reply to Michael RothFrom: Eric Blake <[email protected]>
When extracting a human-readable size formatter, we changed 'uint64_t div' pre-patch to 'unsigned long div' post-patch. Which breaks on 32-bit platforms, resulting in 'inf' instead of intended values larger than 999GB.
--- util/cutils.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
util/cutils.c +1 -1
@@ -825,7 +825,7 @@ const char *qemu_ether_ntoa(const MACAddr *mac)char *size_to_str (uint64_t val) {static const char *suffixes[] = {"" ,"Ki" ,"Mi" ,"Gi" ,"Ti" ,"Pi" ,"Ei" }; -unsigned long div; +uint64_t div;int i;/*
Signature · 1 line
2.17.1
PATCH 92/97 curl: Check curl_multi_add_handle()'s return code
In reply to Michael RothFrom: Max Reitz <[email protected]>
If we had done that all along, debugging would have been much simpler. (Also, I/O errors are better than hangs.)
Message-id: [email protected]
--- block/curl.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
block/curl.c +7 -1
@@ -881,7 +881,13 @@ static void curl_setup_preadv(BlockDriverState *bs, CURLAIOCB *acb)trace_curl_setup_preadv (acb->bytes, start, state->range);curl_easy_setopt (state->curl,CURLOPT_RANGE , state->range); -curl_multi_add_handle (s->multi, state->curl); +if (curl_multi_add_handle (s->multi, state->curl) !=CURLM_OK ) { + state->acb[0 ] =NULL ; + acb->ret = -EIO ; + +curl_clean_state (state); + goto out; + }/* Tell curl it needs to kick things off */ curl_multi_socket_action (s->multi,CURL_SOCKET_TIMEOUT ,0 , &running);
Signature · 1 line
2.17.1
PATCH 73/97 target/arm: Don't abort on M-profile exception return in linux-user mode
In reply to Michael RothFrom: Peter Maydell <[email protected]>
An attempt to do an exception-return (branch to one of the magic addresses) in linux-user mode for M-profile should behave like a normal branch, because linux-user mode is always going to be in 'handler' mode. This used to work, but we broke it when we added support for the M-profile security extension in commit d02a8698d7ae2bfed.
In that commit we allowed even handler-mode calls to magic return values to be checked for and dealt with by causing an EXCP_EXCEPTION_EXIT exception to be taken, because this is needed for the FNC_RETURN return-from-non-secure-function-call handling. For system mode we added a check in do_v7m_exception_exit() to make any spurious calls from Handler mode behave correctly, but forgot that linux-user mode would also be affected.
How an attempted return-from-non-secure-function-call in linux-user mode should be handled is not clear -- on real hardware it would result in return to secure code (not to the Linux kernel) which could then handle the error in any way it chose. For QEMU we take the simple approach of treating this erroneous return the same way it would be handled on a CPU without the security extensions -- treat it as a normal branch.
The upshot of all this is that for linux-user mode we should never do any of the bx_excret magic, so the code change is simple.
This ought to be a weird corner case that only affects broken guest code (because Linux user processes should never be attempting to do exception returns or NS function returns), except that the code that assigns addresses in RAM for the process and stack in our linux-user code does not attempt to avoid this magic address range, so legitimate code attempting to return to a trampoline routine on the stack can fall into this case. This change fixes those programs, but we should also look at restricting the range of memory we use for M-profile linux-user guests to the area that would be real RAM in hardware.
Message-id: [email protected]
--- target/arm/translate.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
target/arm/translate.c +20 -1
@@ -964,10 +964,27 @@ static inline void gen_bx(DisasContext *s, TCGv_i32 var)store_cpu_field (var, thumb); } -/* Set PC and Thumb state from var. var is marked as dead. +/* +* Set PC and Thumb state from var. var is marked as dead. * For M-profile CPUs, include logic to detect exception-return * branches and handle them. This is needed for Thumb POP/LDM to PC, LDR to PC, * and BX reg, and no others, and happens only for code in Handler mode. +* The Security Extension also requires us to check for the FNC_RETURN +* which signals a function return from non-secure state; this can happen +* in both Handler and Thread mode. +* To avoid having to do multiple comparisons in inline generated code, +* we make the check we do here loose, so it will match for EXC_RETURN +* in Thread mode. For system emulation do_v7m_exception_exit() checks +* for these spurious cases and returns without doing anything (giving +* the same behaviour as for a branch to a non-magic address). +* +* In linux-user mode it is unclear what the right behaviour for an +* attempted FNC_RETURN should be, because in real hardware this will go +* directly to Secure code (ie not the Linux kernel) which will then treat +* the error in any way it chooses. For QEMU we opt to make the FNC_RETURN +* attempt behave the way it would on a CPU without the security extension, +* which is to say "like a normal branch". That means we can simply treat +* all branches as normal with no magic address behaviour. */ static inline void gen_bx_excret (DisasContext *s,TCGv_i32 var) {
@@ -975,10 +992,12 @@ static inline void gen_bx_excret(DisasContext *s, TCGv_i32 var)* s->base.is_jmp that we need to do the rest of the work later. */ gen_bx (s, var); +#ifndef CONFIG_USER_ONLY if (arm_dc_feature (s,ARM_FEATURE_M_SECURITY ) || (s->v7m_handler_mode &&arm_dc_feature (s,ARM_FEATURE_M ))) { s->base.is_jmp =DISAS_BX_EXCRET ; } +#endif }static inline void gen_bx_excret_final_code (DisasContext *s)
Signature · 1 line
2.17.1
PATCH 61/97 block/backup: unify different modes code path
In reply to Michael RothFrom: Vladimir Sementsov-Ogievskiy <[email protected]>
Do full, top and incremental mode copying all in one place. This unifies the code path and helps further improvements.
Message-id: [email protected]
*prereq for 110571be4e
--- block/backup.c | 43 ++++++++++--------------------------------- 1 file changed, 10 insertions(+), 33 deletions(-)
block/backup.c +10 -33
@@ -393,15 +393,23 @@ static bool bdrv_is_unallocated_range(BlockDriverState *bs,return offset >= end; } -static int coroutine_fn backup_run_incremental(BackupBlockJob *job) +static int coroutine_fn backup_loop(BackupBlockJob *job) {int ret;bool error_is_read;int64_t offset;HBitmapIter hbi; +BlockDriverState *bs =blk_bs (job->common.blk);hbitmap_iter_init (&hbi, job->copy_bitmap,0 );while ((offset =hbitmap_iter_next (&hbi)) !=-1 ) { +if (job->sync_mode ==MIRROR_SYNC_MODE_TOP && +bdrv_is_unallocated_range (bs, offset, job->cluster_size)) + { +hbitmap_reset (job->copy_bitmap, offset, job->cluster_size); +continue ; + } + do { if (yield_and_check(job)) { return 0;
@@ -446,7 +454,6 @@ static int coroutine_fn backup_run(Job *job, Error **errp) {BackupBlockJob *s =container_of (job, BackupBlockJob, common.job);BlockDriverState *bs =blk_bs (s->common.blk); -int64_t offset;int ret =0 ;QLIST_INIT (&s->inflight_reqs);
@@ -471,38 +478,8 @@ static int coroutine_fn backup_run(Job *job, Error **errp)* notify callback service CoW requests. */ job_yield (job); } - }else if (s->sync_mode ==MIRROR_SYNC_MODE_INCREMENTAL ) { - ret =backup_run_incremental (s); }else { -/* Both FULL and TOP SYNC_MODE's require copying.. */ -for (offset =0 ; offset < s->len; - offset += s->cluster_size) { -bool error_is_read; - -if (yield_and_check (s)) { -break ; - } - -if (s->sync_mode ==MIRROR_SYNC_MODE_TOP && -bdrv_is_unallocated_range (bs, offset, s->cluster_size)) - { -continue ; - } - - ret =backup_do_cow (s, offset, s->cluster_size, - &error_is_read, false); -if (ret <0 ) { -/* Depending on error action, fail now or retry cluster */ -BlockErrorAction action = -backup_error_action (s, error_is_read, -ret); -if (action ==BLOCK_ERROR_ACTION_REPORT ) { -break ; - }else { - offset -= s->cluster_size; -continue ; - } - } - } + ret =backup_loop (s); }notifier_with_return_remove (&s->before_write);
Signature · 1 line
2.17.1
PATCH 17/97 block: Drain source node in bdrv_replace_node()
In reply to Michael RothFrom: Kevin Wolf <[email protected]>
Instead of just asserting that no requests are in flight in bdrv_replace_node(), which is a requirement that most callers ignore, we can just drain the source node right there. This fixes at least starting a commit job while I/O is active on the backing chain, but probably other callers, too.
Having requests in flight on the target node isn't a problem because the target just gets new parents, but the call path of running requests isn't modified. So we can just drop this assertion without a replacement.
*prereq for d81e1efb tests
--- block.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
block.c +4 -3
@@ -3987,13 +3987,13 @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,uint64_t perm =0 , shared =BLK_PERM_ALL ;int ret; -assert (!atomic_read (&from->in_flight)); -assert (!atomic_read (&to->in_flight)); -/* Make sure that @from doesn't go away until we have successfully attached * all of its parents to @to. */ bdrv_ref (from); +assert (qemu_get_current_aio_context () ==qemu_get_aio_context ()); +bdrv_drained_begin (from); +/* Put all parents into @list and calculate their cumulative permissions */ QLIST_FOREACH_SAFE(c, &from->parents, next_parent, next) {assert (c->bs == from);
@@ -4034,6 +4034,7 @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to, out:g_slist_free (list); +bdrv_drained_end (from);bdrv_unref (from); }
Signature · 1 line
2.17.1
PATCH 95/97 s390: PCI: fix IOMMU region init
In reply to Michael RothFrom: Matthew Rosato <[email protected]>
The fix in dbe9cf606c shrinks the IOMMU memory region to a size that seems reasonable on the surface, however is actually too small as it is based against a 0-mapped address space. This causes breakage with small guests as they can overrun the IOMMU window.
Let's go back to the prior method of initializing iommu for now.
Message-Id: <[email protected]>
--- hw/s390x/s390-pci-bus.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
hw/s390x/s390-pci-bus.c +6 -1
@@ -694,10 +694,15 @@ static const MemoryRegionOps s390_msi_ctrl_ops = {void s390_pci_iommu_enable (S390PCIIOMMU *iommu) { +/* +* The iommu region is initialized against a 0-mapped address space, +* so the smallest IOMMU region we can define runs from 0 to the end +* of the PCI address space. +*/ char *name =g_strdup_printf ("iommu-s390-%04x" , iommu->pbdev->uid);memory_region_init_iommu (&iommu->iommu_mr,sizeof (iommu->iommu_mr),TYPE_S390_IOMMU_MEMORY_REGION ,OBJECT (&iommu->mr), - name, iommu->pal - iommu->pba +1 ); + name, iommu->pal +1 ); iommu->enabled = true;memory_region_add_subregion (&iommu->mr,0 ,MEMORY_REGION (&iommu->iommu_mr));g_free (name);
Signature · 1 line
2.17.1
PATCH 94/97 slirp: ip_reass: Fix use after free
In reply to Michael RothUsing ip_deq after m_free might read pointers from an allocation reuse.
This would be difficult to exploit, but that is still related with CVE-2019-14378 which generates fragmented IP packets that would trigger this issue and at least produce a DoS.
(from libslirp.git commit c59279437eda91841b9d26079c70b8a540d41204)
--- slirp/src/ip_input.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
slirp/src/ip_input.c +4 -2
@@ -297,6 +297,7 @@ ip_reass(Slirp *slirp, struct ip *ip, struct ipq *fp)*/ while (q != (struct ipasfrag *)&fp->frag_link && ip->ip_off + ip->ip_len > q->ipf_off) { +struct ipasfrag *prev; i = (ip->ip_off + ip->ip_len) - q->ipf_off;if (i < q->ipf_len) { q->ipf_len -= i;
@@ -304,9 +305,10 @@ ip_reass(Slirp *slirp, struct ip *ip, struct ipq *fp)m_adj (dtom (slirp, q), i);break ; } + prev = q; q = q->ipf_next; -m_free (dtom (slirp, q->ipf_prev)); -ip_deq (q->ipf_prev); +ip_deq (prev); +m_free (dtom (slirp, prev)); } insert:
Signature · 1 line
2.17.1
PATCH 82/97 libvhost-user: fix SLAVE_SEND_FD handling
In reply to Michael RothFrom: Johannes Berg <[email protected]>
It doesn't look like this could possibly work properly since VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD is defined to 10, but the dev->protocol_features has a bitmap. I suppose the peer this was tested with also supported VHOST_USER_PROTOCOL_F_LOG_SHMFD, in which case the test would always be false, but nevertheless the code seems wrong.
Use has_feature() to fix this.
Message-Id: <[email protected]>
--- contrib/libvhost-user/libvhost-user.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
contrib/libvhost-user/libvhost-user.c +2 -1
@@ -1085,7 +1085,8 @@ bool vu_set_queue_host_notifier(VuDev *dev, VuVirtq *vq, int fd, vmsg.fd_num = fd_num; -if ((dev->protocol_features &VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD ) ==0 ) { +if (!has_feature (dev->protocol_features, +VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD )) {return false; }
Signature · 1 line
2.17.1
PATCH 96/97 hw/core/loader: Fix possible crash in rom_copy()
In reply to Michael RothFrom: Thomas Huth <[email protected]>
Both, "rom->addr" and "addr" are derived from the binary image that can be loaded with the "-kernel" paramer. The code in rom_copy() then calculates:
d = dest + (rom->addr - addr);
and uses "d" as destination in a memcpy() some lines later. Now with bad kernel images, it is possible that rom->addr is smaller than addr, thus "rom->addr - addr" gets negative and the memcpy() then tries to copy contents from the image to a bad memory location. This could maybe be used to inject code from a kernel image into the QEMU binary, so we better fix it with an additional sanity check here.
Message-Id: <[email protected]>
--- hw/core/loader.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
hw/core/loader.c +1 -1
@@ -1240,7 +1240,7 @@ int rom_copy(uint8_t *dest, hwaddr addr, size_t size)if (rom->addr + rom->romsize < addr) {continue ; } -if (rom->addr > end) { +if (rom->addr > end || rom->addr < addr) {break ; }
Signature · 1 line
2.17.1
PATCH 85/97 blockjob: update nodes head while removing all bdrv
In reply to Michael RothFrom: Sergio Lopez <[email protected]>
block_job_remove_all_bdrv() iterates through job->nodes, calling bdrv_root_unref_child() for each entry. The call to the latter may reach child_job_[can_]set_aio_ctx(), which will also attempt to traverse job->nodes, potentially finding entries that where freed on previous iterations.
To avoid this situation, update job->nodes head on each iteration to ensure that already freed entries are no longer linked to the list.
RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1746631
Message-id: [email protected]
--- blockjob.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
blockjob.c +13 -4
@@ -193,14 +193,23 @@ static const BdrvChildRole child_job = {void block_job_remove_all_bdrv (BlockJob *job) { -GSList *l; -for (l = job->nodes; l; l = l->next) { +/* +* bdrv_root_unref_child() may reach child_job_[can_]set_aio_ctx(), +* which will also traverse job->nodes, so consume the list one by +* one to make sure that such a concurrent access does not attempt +* to process an already freed BdrvChild. +*/ +while (job->nodes) { +GSList *l = job->nodes;BdrvChild *c = l->data; + + job->nodes = l->next; +bdrv_op_unblock_all (c->bs, job->blocker);bdrv_root_unref_child (c); + +g_slist_free_1 (l); } -g_slist_free (job->nodes); - job->nodes =NULL ; } int block_job_add_bdrv(BlockJob *job, const char *name, BlockDriverState *bs,
Signature · 1 line
2.17.1
PATCH 10/97 spapr/xive: fix EQ page addresses above 64GB
In reply to Michael RothFrom: Cédric Le Goater <[email protected]>
The high order bits of the address of the OS event queue is stored in bits [4-31] of word2 of the XIVE END internal structures and the low order bits in word3. This structure is using Big Endian ordering and computing the value requires some simple arithmetic which happens to be wrong. The mask removing bits [0-3] of word2 is applied to the wrong value and the resulting address is bogus when above 64GB.
Guests with more than 64GB of RAM will allocate pages for the OS event queues which will reside above the 64GB limit. In this case, the XIVE device model will wake up the CPUs in case of a notification, such as IPIs, but the update of the event queue will be written at the wrong place in memory. The result is uncertain as the guest memory is trashed and IPI are not delivered.
Introduce a helper xive_end_qaddr() to compute this value correctly in all places where it is used.
Message-Id: <[email protected]>
--- hw/intc/spapr_xive.c | 3 +-- hw/intc/xive.c | 9 +++------ include/hw/ppc/xive_regs.h | 6 ++++++ 3 files changed, 10 insertions(+), 8 deletions(-)
hw/intc/spapr_xive.c +1 -2
@@ -1144,8 +1144,7 @@ static target_ulong h_int_get_queue_config(PowerPCCPU *cpu, }if (xive_end_is_enqueue (end)) { - args[1 ] = (uint64_t )be32_to_cpu (end->w2 &0x0fffffff ) <<32 - |be32_to_cpu (end->w3); + args[1 ] =xive_end_qaddr (end); args[2 ] =xive_get_field32 (END_W0_QSIZE , end->w0) +12 ; }else { args[1 ] =0 ;
hw/intc/xive.c +3 -6
@@ -1042,8 +1042,7 @@ static const TypeInfo xive_source_info = {void xive_end_queue_pic_print_info (XiveEND *end,uint32_t width,Monitor *mon) { -uint64_t qaddr_base = (uint64_t )be32_to_cpu (end->w2 &0x0fffffff ) <<32 - |be32_to_cpu (end->w3); +uint64_t qaddr_base =xive_end_qaddr (end);uint32_t qsize =xive_get_field32 (END_W0_QSIZE , end->w0);uint32_t qindex =xive_get_field32 (END_W1_PAGE_OFF , end->w1);uint32_t qentries =1 << (qsize +10 );
@@ -1072,8 +1071,7 @@ void xive_end_queue_pic_print_info(XiveEND *end, uint32_t width, Monitor *mon)void xive_end_pic_print_info (XiveEND *end,uint32_t end_idx,Monitor *mon) { -uint64_t qaddr_base = (uint64_t )be32_to_cpu (end->w2 &0x0fffffff ) <<32 - |be32_to_cpu (end->w3); +uint64_t qaddr_base =xive_end_qaddr (end);uint32_t qindex =xive_get_field32 (END_W1_PAGE_OFF , end->w1);uint32_t qgen =xive_get_field32 (END_W1_GENERATION , end->w1);uint32_t qsize =xive_get_field32 (END_W0_QSIZE , end->w0);
@@ -1101,8 +1099,7 @@ void xive_end_pic_print_info(XiveEND *end, uint32_t end_idx, Monitor *mon)static void xive_end_enqueue (XiveEND *end,uint32_t data) { -uint64_t qaddr_base = (uint64_t )be32_to_cpu (end->w2 &0x0fffffff ) <<32 - |be32_to_cpu (end->w3); +uint64_t qaddr_base =xive_end_qaddr (end);uint32_t qsize =xive_get_field32 (END_W0_QSIZE , end->w0);uint32_t qindex =xive_get_field32 (END_W1_PAGE_OFF , end->w1);uint32_t qgen =xive_get_field32 (END_W1_GENERATION , end->w1);
include/hw/ppc/xive_regs.h +6 -0
@@ -208,6 +208,12 @@ typedef struct XiveEND {#define xive_end_is_backlog (end) (be32_to_cpu((end)->w0) & END_W0_BACKLOG)#define xive_end_is_escalate (end) (be32_to_cpu((end)->w0) & END_W0_ESCALATE_CTL) +static inline uint64_t xive_end_qaddr (XiveEND *end) +{ +return ((uint64_t )be32_to_cpu (end->w2) &0x0fffffff ) <<32 | +be32_to_cpu (end->w3); +} +/* Notification Virtual Target (NVT) */ typedef struct XiveNVT { uint32_t w0;
Signature · 1 line
2.17.1
PATCH 75/97 pr-manager: Fix invalid g_free() crash bug
In reply to Michael RothFrom: Markus Armbruster <[email protected]>
pr_manager_worker() passes its @opaque argument to g_free(). Wrong; it points to pr_manager_worker()'s automatic @data. Broken when commit 2f3a7ab39be converted @data from heap- to stack-allocated. Fix by deleting the g_free().
--- scsi/pr-manager.c | 1 - 1 file changed, 1 deletion(-)
scsi/pr-manager.c +0 -1
@@ -38,7 +38,6 @@ static int pr_manager_worker(void *opaque)int fd = data->fd;int r; -g_free (data);trace_pr_manager_run (fd, hdr->cmdp[0 ], hdr->cmdp[1 ]);/* The reference was taken in pr_manager_execute. */
Signature · 1 line
2.17.1
PATCH 87/97 curl: Keep *socket until the end of curl_sock_cb()
In reply to Michael RothFrom: Max Reitz <[email protected]>
This does not really change anything, but it makes the code a bit easier to follow once we use @socket as the opaque pointer for aio_set_fd_handler().
Message-id: [email protected]
--- block/curl.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
block/curl.c +5 -5
@@ -171,10 +171,6 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action, QLIST_FOREACH(socket, &state->sockets, next) {if (socket->fd == fd) { -if (action ==CURL_POLL_REMOVE ) { -QLIST_REMOVE (socket, next); -g_free (socket); - }break ; } }
@@ -184,7 +180,6 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action, socket->state = state;QLIST_INSERT_HEAD (&state->sockets, socket, next); } - socket =NULL ;trace_curl_sock_cb (action, (int )fd);switch (action) {
@@ -206,6 +201,11 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,break ; } +if (action ==CURL_POLL_REMOVE ) { +QLIST_REMOVE (socket, next); +g_free (socket); + } +return 0 ; }
Signature · 1 line
2.17.1
PATCH 9/97 docs/interop/bitmaps: rewrite and modernize doc
In reply to Michael RothFrom: John Snow <[email protected]>
This just about rewrites the entirety of the bitmaps.rst document to make it consistent with the 4.0 release. I have added new features seen in the 4.0 release, as well as tried to clarify some points that keep coming up when discussing this feature both in-house and upstream.
It does not yet cover pull backups or migration details, but I intend to keep extending this document to cover those cases.
Message-id: [email protected]
--- docs/interop/bitmaps.rst | 1573 ++++++++++++++++++++++++++++++-------- 1 file changed, 1251 insertions(+), 322 deletions(-)
docs/interop/bitmaps.rst +1251 -322
@@ -1,5 +1,5 @@ .. - Copyright 2015 John Snow <[email protected]> and Red Hat, Inc. + Copyright 2019 John Snow <[email protected]> and Red Hat, Inc. All rights reserved. This file is licensed via The FreeBSD Documentation License, the full
@@ -9,547 +9,1476 @@ Dirty Bitmaps and Incremental Backup ==================================== -- Dirty Bitmaps are objects that track which data needs to be backed up - for the next incremental backup. +Dirty Bitmaps are in-memory objects that track writes to block devices. They +can be used in conjunction with various block job operations to perform +incremental or differential backup regimens. -- Dirty bitmaps can be created at any time and attached to any node - (not just complete drives). +This document explains the conceptual mechanisms, as well as up-to-date, +complete and comprehensive documentation on the API to manipulate them. +(Hopefully, the "why", "what", and "how".) + +The intended audience for this document is developers who are adding QEMU +backup features to management applications, or power users who run and +administer QEMU directly via QMP. .. contents:: +Overview +-------- + +Bitmaps are bit vectors where each '1' bit in the vector indicates a modified +("dirty") segment of the corresponding block device. The size of the segment +that is tracked is the granularity of the bitmap. If the granularity of a +bitmap is 64K, each '1' bit means that a 64K region as a whole may have +changed in some way, possibly by as little as one byte. + +Smaller granularities mean more accurate tracking of modified disk data, but +requires more computational overhead and larger bitmap sizes. Larger +granularities mean smaller bitmap sizes, but less targeted backups. + +The size of a bitmap (in bytes) can be computed as such: + ``size`` = ceil(ceil(``image_size`` / ``granularity``) / 8) + +e.g. the size of a 64KiB granularity bitmap on a 2TiB image is: + ``size`` = ((2147483648K / 64K) / 8) + = 4194304B = 4MiB. + +QEMU uses these bitmaps when making incremental backups to know which sections +of the file to copy out. They are not enabled by default and must be +explicitly added in order to begin tracking writes. + +Bitmaps can be created at any time and can be attached to any arbitrary block +node in the storage graph, but are most useful conceptually when attached to +the root node attached to the guest's storage device model. + +That is to say: It's likely most useful to track the guest's writes to disk, +but you could theoretically track things like qcow2 metadata changes by +attaching the bitmap elsewhere in the storage graph. This is beyond the scope +of this document. + +QEMU supports persisting these bitmaps to disk via the qcow2 image format. +Bitmaps which are stored or loaded in this way are called "persistent", +whereas bitmaps that are not are called "transient". + +QEMU also supports the migration of both transient bitmaps (tracking any +arbitrary image format) or persistent bitmaps (qcow2) via live migration. + +Supported Image Formats +----------------------- + +QEMU supports all documented features below on the qcow2 image format. + +However, qcow2 is only strictly necessary for the persistence feature, which +writes bitmap data to disk upon close. If persistence is not required for a +specific use case, all bitmap features excepting persistence are available for +any arbitrary image format. + +For example, Dirty Bitmaps can be combined with the 'raw' image format, but +any changes to the bitmap will be discarded upon exit. + +.. warning:: Transient bitmaps will not be saved on QEMU exit! Persistent + bitmaps are available only on qcow2 images. + Dirty Bitmap Names ------------------ -- A dirty bitmap's name is unique to the node, but bitmaps attached to - different nodes can share the same name. +Bitmap objects need a method to reference them in the API. All API-created and +managed bitmaps have a human-readable name chosen by the user at creation +time. -- Dirty bitmaps created for internal use by QEMU may be anonymous and - have no name, but any user-created bitmaps must have a name. There - can be any number of anonymous bitmaps per node. +- A bitmap's name is unique to the node, but bitmaps attached to different + nodes can share the same name. Therefore, all bitmaps are addressed via + their (node, name) pair. -- The name of a user-created bitmap must not be empty (""). +- The name of a user-created bitmap cannot be empty (""). -Bitmap Modes ------------- +- Transient bitmaps can have JSON unicode names that are effectively not + length limited. (QMP protocol may restrict messages to less than 64MiB.) + +- Persistent storage formats may impose their own requirements on bitmap names + and namespaces. Presently, only qcow2 supports persistent bitmaps. See + docs/interop/qcow2.txt for more details on restrictions. Notably: + + - qcow2 bitmap names are limited to between 1 and 1023 bytes long. + + - No two bitmaps saved to the same qcow2 file may share the same name. + +- QEMU occasionally uses bitmaps for internal use which have no name. They are + hidden from API query calls, cannot be manipulated by the external API, are + never persistent, nor ever migrated. + +Bitmap Status +------------- + +Dirty Bitmap objects can be queried with the QMP command `query-block +<qemu-qmp-ref.html#index-query_002dblock>`_, and are visible via the +`BlockDirtyInfo <qemu-qmp-ref.html#index-BlockDirtyInfo>`_ QAPI structure. + +This struct shows the name, granularity, and dirty byte count for each bitmap. +Additionally, it shows several boolean status indicators: + +- ``recording``: This bitmap is recording writes. +- ``busy``: This bitmap is in-use by an operation. +- ``persistent``: This bitmap is a persistent type. +- ``inconsistent``: This bitmap is corrupted and cannot be used. + +The ``+busy`` status prohibits you from deleting, clearing, or otherwise +modifying a bitmap, and happens when the bitmap is being used for a backup +operation or is in the process of being loaded from a migration. Many of the +commands documented below will refuse to work on such bitmaps. + +The ``+inconsistent`` status similarly prohibits almost all operations, +notably allowing only the ``block-dirty-bitmap-remove`` operation. + +There is also a deprecated ``status`` field of type `DirtyBitmapStatus +<qemu-qmp-ref.html#index-DirtyBitmapStatus>`_. A bitmap historically had +five visible states: + + #. ``Frozen``: This bitmap is currently in-use by an operation and is + immutable. It can't be deleted, renamed, reset, etc. + + (This is now ``+busy``.) + + #. ``Disabled``: This bitmap is not recording new writes. + + (This is now ``-recording -busy``.) + + #. ``Active``: This bitmap is recording new writes. + + (This is now ``+recording -busy``.) -- A bitmap can be "frozen," which means that it is currently in-use by - a backup operation and cannot be deleted, renamed, written to, reset, - etc. + #. ``Locked``: This bitmap is in-use by an operation, and is immutable. + The difference from "Frozen" was primarily implementation details. -- The normal operating mode for a bitmap is "active." + (This is now ``+busy``.) + + #. ``Inconsistent``: This persistent bitmap was not saved to disk + correctly, and can no longer be used. It remains in memory to serve as + an indicator of failure. + + (This is now ``+inconsistent``.) + +These states are directly replaced by the status indicators and should not be +used. The difference between ``Frozen`` and ``Locked`` is an implementation +detail and should not be relevant to external users. Basic QMP Usage --------------- +The primary interface to manipulating bitmap objects is via the QMP +interface. If you are not familiar, see docs/interop/qmp-intro.txt for a broad +overview, and `qemu-qmp-ref <qemu-qmp-ref.html>`_ for a full reference of all +QMP commands. + Supported Commands ~~~~~~~~~~~~~~~~~~ +There are six primary bitmap-management API commands: + - ``block-dirty-bitmap-add`` - ``block-dirty-bitmap-remove`` - ``block-dirty-bitmap-clear`` +- ``block-dirty-bitmap-disable`` +- ``block-dirty-bitmap-enable`` +- ``block-dirty-bitmap-merge`` -Creation -~~~~~~~~ +And one related query command: -- To create a new bitmap, enabled, on the drive with id=drive0: +- ``query-block`` -.. code:: json +Creation: block-dirty-bitmap-add +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - { "execute": "block-dirty-bitmap-add", - "arguments": { - "node": "drive0", - "name": "bitmap0" - } - } +`block-dirty-bitmap-add +<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dadd>`_: -- This bitmap will have a default granularity that matches the cluster - size of its associated drive, if available, clamped to between [4KiB, - 64KiB]. The current default for qcow2 is 64KiB. +Creates a new bitmap that tracks writes to the specified node. granularity, +persistence, and recording state can be adjusted at creation time. -- To create a new bitmap that tracks changes in 32KiB segments: +.. admonition:: Example -.. code:: json + to create a new, actively recording persistent bitmap: - { "execute": "block-dirty-bitmap-add", - "arguments": { - "node": "drive0", - "name": "bitmap0", - "granularity": 32768 - } - } + .. code:: json -Deletion -~~~~~~~~ + -> { "execute": "block-dirty-bitmap-add", + "arguments": { + "node": "drive0", + "name": "bitmap0", + "persistent": true, + } + } -- Bitmaps that are frozen cannot be deleted. + <- { "return": {} } -- Deleting the bitmap does not impact any other bitmaps attached to the - same node, nor does it affect any backups already created from this - node. +- This bitmap will have a default granularity that matches the cluster size of + its associated drive, if available, clamped to between [4KiB, 64KiB]. The + current default for qcow2 is 64KiB. -- Because bitmaps are only unique to the node to which they are - attached, you must specify the node/drive name here, too. +.. admonition:: Example -.. code:: json + To create a new, disabled (``-recording``), transient bitmap that tracks + changes in 32KiB segments: - { "execute": "block-dirty-bitmap-remove", - "arguments": { - "node": "drive0", - "name": "bitmap0" - } - } + .. code:: json -Resetting -~~~~~~~~~ + -> { "execute": "block-dirty-bitmap-add", + "arguments": { + "node": "drive0", + "name": "bitmap1", + "granularity": 32768, + "disabled": true + } + } -- Resetting a bitmap will clear all information it holds. + <- { "return": {} } -- An incremental backup created from an empty bitmap will copy no data, - as if nothing has changed. +Deletion: block-dirty-bitmap-remove +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. code:: json +`block-dirty-bitmap-remove +<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dremove>`_: + +Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed. + +- Deleting a bitmap does not impact any other bitmaps attached to the same + node, nor does it affect any backups already created from this bitmap or + node. + +- Because bitmaps are only unique to the node to which they are attached, you + must specify the node/drive name here, too. + +- Deleting a persistent bitmap will remove it from the qcow2 file. + +.. admonition:: Example + + Remove a bitmap named ``bitmap0`` from node ``drive0``: + + .. code:: json + + -> { "execute": "block-dirty-bitmap-remove", + "arguments": { + "node": "drive0", + "name": "bitmap0" + } + } + + <- { "return": {} } + +Resetting: block-dirty-bitmap-clear +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`block-dirty-bitmap-clear +<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dclear>`_: + +Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared. + +- An incremental backup created from an empty bitmap will copy no data, as if + nothing has changed. + +.. admonition:: Example + + Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``: + + .. code:: json + + -> { "execute": "block-dirty-bitmap-clear", + "arguments": { + "node": "drive0", + "name": "bitmap0" + } + } + + <- { "return": {} } + +Enabling: block-dirty-bitmap-enable +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`block-dirty-bitmap-enable +<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002denable>`_: - { "execute": "block-dirty-bitmap-clear", - "arguments": { - "node": "drive0", - "name": "bitmap0" - } - } +"Enables" a bitmap, setting the ``recording`` bit to true, causing writes to +begin being recorded. ``+busy`` bitmaps cannot be enabled. + +- Bitmaps default to being enabled when created, unless configured otherwise. + +- Persistent enabled bitmaps will remember their ``+recording`` status on + load. + +.. admonition:: Example + + To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``: + + .. code:: json + + -> { "execute": "block-dirty-bitmap-enable", + "arguments": { + "node": "drive0", + "name": "bitmap0" + } + } + + <- { "return": {} } + +Enabling: block-dirty-bitmap-disable +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`block-dirty-bitmap-disable +<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002ddisable>`_: + +"Disables" a bitmap, setting the ``recording`` bit to false, causing further +writes to begin being ignored. ``+busy`` bitmaps cannot be disabled. + +.. warning:: + + This is potentially dangerous: QEMU makes no effort to stop any writes if + there are disabled bitmaps on a node, and will not mark any disabled bitmaps + as ``+inconsistent`` if any such writes do happen. Backups made from such + bitmaps will not be able to be used to reconstruct a coherent image. + +- Disabling a bitmap may be useful for examining which sectors of a disk + changed during a specific time period, or for explicit management of + differential backup windows. + +- Persistent disabled bitmaps will remember their ``-recording`` status on + load. + +.. admonition:: Example + + To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``: + + .. code:: json + + -> { "execute": "block-dirty-bitmap-disable", + "arguments": { + "node": "drive0", + "name": "bitmap0" + } + } + + <- { "return": {} } + +Merging, Copying: block-dirty-bitmap-merge +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`block-dirty-bitmap-merge +<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dmerge>`_: + +Merges one or more bitmaps into a target bitmap. For any segment that is dirty +in any one source bitmap, the target bitmap will mark that segment dirty. + +- Merge takes one or more bitmaps as a source and merges them together into a + single destination, such that any segment marked as dirty in any source + bitmap(s) will be marked dirty in the destination bitmap. + +- Merge does not create the destination bitmap if it does not exist. A blank + bitmap can be created beforehand to achieve the same effect. + +- The destination is not cleared prior to merge, so subsequent merge + operations will continue to cumulatively mark more segments as dirty. + +- If the merge operation should fail, the destination bitmap is guaranteed to + be unmodified. The operation may fail if the source or destination bitmaps + are busy, or have different granularities. + +- Bitmaps can only be merged on the same node. There is only one "node" + argument, so all bitmaps must be attached to that same node. + +- Copy can be achieved by merging from a single source to an empty + destination. + +.. admonition:: Example + + Merge the data from ``bitmap0`` into the bitmap ``new_bitmap`` on node + ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves + a copy. + + .. code:: json + + -> { "execute": "block-dirty-bitmap-merge", + "arguments": { + "node": "drive0", + "target": "new_bitmap", + "bitmaps: [ "bitmap0" ] + } + } + + <- { "return": {} } + +Querying: query-block +~~~~~~~~~~~~~~~~~~~~~ + +`query-block +<qemu-qmp-ref.html#index-query_002dblock>`_: + +Not strictly a bitmaps command, but will return information about any bitmaps +attached to nodes serving as the root for guest devices. + +- The "inconsistent" bit will not appear when it is false, appearing only when + the value is true to indicate there is a problem. + +.. admonition:: Example + + Query the block sub-system of QEMU. The following json has trimmed irrelevant + keys from the response to highlight only the bitmap-relevant portions of the + API. This result highlights a bitmap ``bitmap0`` attached to the root node of + device ``drive0``. + + .. code:: json + + -> { + "execute": "query-block", + "arguments": {} + } + + <- { + "return": [ { + "dirty-bitmaps": [ { + "status": "active", + "count": 0, + "busy": false, + "name": "bitmap0", + "persistent": false, + "recording": true, + "granularity": 65536 + } ], + "device": "drive0", + } ] + } + +Bitmap Persistence +------------------ + +As outlined in `Supported Image Formats`_, QEMU can persist bitmaps to qcow2 +files. Demonstrated in `Creation: block-dirty-bitmap-add`_, passing +``persistent: true`` to ``block-dirty-bitmap-add`` will persist that bitmap to +disk. + +Persistent bitmaps will be automatically loaded into memory upon load, and +will be written back to disk upon close. Their usage should be mostly +transparent. + +However, if QEMU does not get a chance to close the file cleanly, the bitmap +will be marked as ``+inconsistent`` at next load and considered unsafe to use +for any operation. At this point, the only valid operation on such bitmaps is +``block-dirty-bitmap-remove``. + +Losing a bitmap in this way does not invalidate any existing backups that have +been made from this bitmap, but no further backups will be able to be issued +for this chain. Transactions ------------ +Transactions are a QMP feature that allows you to submit multiple QMP commands +at once, being guaranteed that they will all succeed or fail atomically, +together. The interaction of bitmaps and transactions are demonstrated below. + +See `transaction <qemu-qmp.ref.html#index-transaction>`_ in the QMP reference +for more details. + Justification ~~~~~~~~~~~~~ -Bitmaps can be safely modified when the VM is paused or halted by using -the basic QMP commands. For instance, you might perform the following -actions: +Bitmaps can generally be modified at any time, but certain operations often +only make sense when paired directly with other commands. When a VM is paused, +it's easy to ensure that no guest writes occur between individual QMP +commands. When a VM is running, this is difficult to accomplish with +individual QMP commands that may allow guest writes to occur inbetween each +command. -1. Boot the VM in a paused state. -2. Create a full drive backup of drive0. -3. Create a new bitmap attached to drive0. -4. Resume execution of the VM. -5. Incremental backups are ready to be created. +For example, using only individual QMP commands, we could: -At this point, the bitmap and drive backup would be correctly in sync, -and incremental backups made from this point forward would be correctly -aligned to the full drive backup. +#. Boot the VM in a paused state. +#. Create a full drive backup of drive0. +#. Create a new bitmap attached to drive0, confident that nothing has been + written to drive0 in the meantime. +#. Resume execution of the VM. +#. At a later point, issue incremental backups from ``bitmap0``. -This is not particularly useful if we decide we want to start -incremental backups after the VM has been running for a while, for which -we will need to perform actions such as the following: +At this point, the bitmap and drive backup would be correctly in sync, and +incremental backups made from this point forward would be correctly aligned to +the full drive backup. -1. Boot the VM and begin execution. -2. Using a single transaction, perform the following operations: +This is not particularly useful if we decide we want to start incremental +backups after the VM has been running for a while, for which we would want to +perform actions such as the following: + +#. Boot the VM and begin execution. +#. Using a single transaction, perform the following operations: - Create ``bitmap0``. - Create a full drive backup of ``drive0``. -3. Incremental backups are now ready to be created. +#. At a later point, issue incremental backups from ``bitmap0``. + +.. note:: As a consideration, if ``bitmap0`` is created prior to the full + drive backup, incremental backups can still be authored from this + bitmap, but they will copy extra segments reflecting writes that + occurred prior to the backup operation. Transactions allow us to + narrow critical points in time to reduce waste, or, in the other + direction, to ensure that no segments are omitted. Supported Bitmap Transactions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ``block-dirty-bitmap-add`` - ``block-dirty-bitmap-clear`` +- ``block-dirty-bitmap-enable`` +- ``block-dirty-bitmap-disable`` +- ``block-dirty-bitmap-merge`` -The usages are identical to their respective QMP commands, but see below -for examples. +The usages for these commands are identical to their respective QMP commands, +but see the sections below for concrete examples. -Example: New Incremental Backup -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Incremental Backups - Push Model +-------------------------------- -As outlined in the justification, perhaps we want to create a new -incremental backup chain attached to a drive. +Incremental backups are simply partial disk images that can be combined with +other partial disk images on top of a base image to reconstruct a full backup +from the point in time at which the incremental backup was issued. -.. code:: json +The "Push Model" here references the fact that QEMU is "pushing" the modified +blocks out to a destination. We will be using the `drive-backup +<qemu-qmp-ref.html#index-drive_002dbackup>`_ and `blockdev-backup +<qemu-qmp-ref.html#index-blockdev_002dbackup>`_ QMP commands to create both +full and incremental backups. - { "execute": "transaction", - "arguments": { - "actions": [ - {"type": "block-dirty-bitmap-add", - "data": {"node": "drive0", "name": "bitmap0"} }, - {"type": "drive-backup", - "data": {"device": "drive0", "target": "/path/to/full_backup.img", - "sync": "full", "format": "qcow2"} } - ] - } - } +Both of these commands are jobs, which have their own QMP API for querying and +management documented in `Background jobs +<qemu-qmp-ref.html#Background-jobs>`_. Example: New Incremental Backup Anchor Point ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Maybe we just want to create a new full backup with an existing bitmap -and want to reset the bitmap to track the new chain. +As outlined in the Transactions - `Justification`_ section, perhaps we want to +create a new incremental backup chain attached to a drive. + +This example creates a new, full backup of "drive0" and accompanies it with a +new, empty bitmap that records writes from this point in time forward. + +.. note:: Any new writes that happen after this command is issued, even while + the backup job runs, will be written locally and not to the backup + destination. These writes will be recorded in the bitmap + accordingly. + +.. code:: json + + -> { + "execute": "transaction", + "arguments": { + "actions": [ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "drive0", + "name": "bitmap0" + } + }, + { + "type": "drive-backup", + "data": { + "device": "drive0", + "target": "/path/to/drive0.full.qcow2", + "sync": "full", + "format": "qcow2" + } + } + ] + } + } + + <- { "return": {} } + + <- { + "timestamp": { + "seconds": 1555436945, + "microseconds": 179620 + }, + "data": { + "status": "created", + "id": "drive0" + }, + "event": "JOB_STATUS_CHANGE" + } + + ... + + <- { + "timestamp": {...}, + "data": { + "device": "drive0", + "type": "backup", + "speed": 0, + "len": 68719476736, + "offset": 68719476736 + }, + "event": "BLOCK_JOB_COMPLETED" + } + + <- { + "timestamp": {...}, + "data": { + "status": "concluded", + "id": "drive0" + }, + "event": "JOB_STATUS_CHANGE" + } + + <- { + "timestamp": {...}, + "data": { + "status": "null", + "id": "drive0" + }, + "event": "JOB_STATUS_CHANGE" + } + +A full explanation of the job transition semantics and the JOB_STATUS_CHANGE +event are beyond the scope of this document and will be omitted in all +subsequent examples; above, several more events have been omitted for brevity. + +.. note:: Subsequent examples will omit all events except BLOCK_JOB_COMPLETED + except where necessary to illustrate workflow differences. + + Omitted events and json objects will be represented by ellipses: + ``...`` + +Example: Resetting an Incremental Backup Anchor Point +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If we want to start a new backup chain with an existing bitmap, we can also +use a transaction to reset the bitmap while making a new full backup: .. code:: json - { "execute": "transaction", - "arguments": { - "actions": [ - {"type": "block-dirty-bitmap-clear", - "data": {"node": "drive0", "name": "bitmap0"} }, - {"type": "drive-backup", - "data": {"device": "drive0", "target": "/path/to/new_full_backup.img", - "sync": "full", "format": "qcow2"} } - ] - } - } - -Incremental Backups -------------------- - -The star of the show. - -**Nota Bene!** Only incremental backups of entire drives are supported -for now. So despite the fact that you can attach a bitmap to any -arbitrary node, they are only currently useful when attached to the root -node. This is because drive-backup only supports drives/devices instead -of arbitrary nodes. + -> { + "execute": "transaction", + "arguments": { + "actions": [ + { + "type": "block-dirty-bitmap-clear", + "data": { + "node": "drive0", + "name": "bitmap0" + } + }, + { + "type": "drive-backup", + "data": { + "device": "drive0", + "target": "/path/to/drive0.new_full.qcow2", + "sync": "full", + "format": "qcow2" + } + } + ] + } + } + + <- { "return": {} } + + ... + + <- { + "timestamp": {...}, + "data": { + "device": "drive0", + "type": "backup", + "speed": 0, + "len": 68719476736, + "offset": 68719476736 + }, + "event": "BLOCK_JOB_COMPLETED" + } + + ... + +The result of this example is identical to the first, but we clear an existing +bitmap instead of adding a new one. + +.. tip:: In both of these examples, "bitmap0" is tied conceptually to the + creation of new, full backups. This relationship is not saved or + remembered by QEMU; it is up to the operator or management layer to + remember which bitmaps are associated with which backups. Example: First Incremental Backup ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. Create a full backup and sync it to the dirty bitmap, as in the - transactional examples above; or with the VM offline, manually create - a full copy and then create a new bitmap before the VM begins - execution. +#. Create a full backup and sync it to a dirty bitmap using any method: - - Let's assume the full backup is named ``full_backup.img``. - - Let's assume the bitmap you created is ``bitmap0`` attached to - ``drive0``. + - Either of the two live backup method demonstrated above, + - Using QMP commands with the VM paused as in the `Justification`_ section, + or + - With the VM offline, manually copy the image and start the VM in a paused + state, careful to add a new bitmap before the VM begins execution. -2. Create a destination image for the incremental backup that utilizes - the full backup as a backing image. + Whichever method is chosen, let's assume that at the end of this step: - - Let's assume the new incremental image is named - ``incremental.0.img``. + - The full backup is named ``drive0.full.qcow2``. + - The bitmap we created is named ``bitmap0``, attached to ``drive0``. + +#. Create a destination image for the incremental backup that utilizes the + full backup as a backing image. + + - Let's assume the new incremental image is named ``drive0.inc0.qcow2``: .. code:: bash - $ qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2 + $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ + -b drive0.full.qcow2 -F qcow2 -3. Issue the incremental backup command: +#. Issue an incremental backup command: .. code:: json - { "execute": "drive-backup", + -> { + "execute": "drive-backup", "arguments": { "device": "drive0", "bitmap": "bitmap0", - "target": "incremental.0.img", + "target": "drive0.inc0.qcow2", "format": "qcow2", "sync": "incremental", "mode": "existing" } } + <- { "return": {} } + + ... + + <- { + "timestamp": {...}, + "data": { + "device": "drive0", + "type": "backup", + "speed": 0, + "len": 68719476736, + "offset": 68719476736 + }, + "event": "BLOCK_JOB_COMPLETED" + } + + ... + +This copies any blocks modified since the full backup was created into the +``drive0.inc0.qcow2`` file. During the operation, ``bitmap0`` is marked +``+busy``. If the operation is successful, ``bitmap0`` will be cleared to +reflect the "incremental" backup regimen, which only copies out new changes +from each incremental backup. + +.. note:: Any new writes that occur after the backup operation starts do not + get copied to the destination. The backup's "point in time" is when + the backup starts, not when it ends. These writes are recorded in a + special bitmap that gets re-added to bitmap0 when the backup ends so + that the next incremental backup can copy them out. + Example: Second Incremental Backup ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. Create a new destination image for the incremental backup that points - to the previous one, e.g.: ``incremental.1.img`` +#. Create a new destination image for the incremental backup that points to + the previous one, e.g.: ``drive0.inc1.qcow2`` .. code:: bash - $ qemu-img create -f qcow2 incremental.1.img -b incremental.0.img -F qcow2 + $ qemu-img create -f qcow2 drive0.inc1.qcow2 \ + -b drive0.inc0.qcow2 -F qcow2 -2. Issue a new incremental backup command. The only difference here is - that we have changed the target image below. +#. Issue a new incremental backup command. The only difference here is that we + have changed the target image below. .. code:: json - { "execute": "drive-backup", + -> { + "execute": "drive-backup", "arguments": { "device": "drive0", "bitmap": "bitmap0", - "target": "incremental.1.img", + "target": "drive0.inc1.qcow2", "format": "qcow2", "sync": "incremental", "mode": "existing" } } -Errors ------- + <- { "return": {} } + + ... + + <- { + "timestamp": {...}, + "data": { + "device": "drive0", + "type": "backup", + "speed": 0, + "len": 68719476736, + "offset": 68719476736 + }, + "event": "BLOCK_JOB_COMPLETED" + } + + ... + +Because the first incremental backup from the previous example completed +successfully, ``bitmap0`` was synchronized with ``drive0.inc0.qcow2``. Here, +we use ``bitmap0`` again to create a new incremental backup that targets the +previous one, creating a chain of three images: + +.. admonition:: Diagram + + .. code:: text + + +-------------------+ +-------------------+ +-------------------+ + | drive0.full.qcow2 |<--| drive0.inc0.qcow2 |<--| drive0.inc1.qcow2 | + +-------------------+ +-------------------+ +-------------------+ + +Each new incremental backup re-synchronizes the bitmap to the latest backup +authored, allowing a user to continue to "consume" it to create new backups on +top of an existing chain. + +In the above diagram, neither drive0.inc1.qcow2 nor drive0.inc0.qcow2 are +complete images by themselves, but rely on their backing chain to reconstruct +a full image. The dependency terminates with each full backup. + +Each backup in this chain remains independent, and is unchanged by new entries +made later in the chain. For instance, drive0.inc0.qcow2 remains a perfectly +valid backup of the disk as it was when that backup was issued. + +Example: Incremental Push Backups without Backing Files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Backup images are best kept off-site, so we often will not have the preceding +backups in a chain available to link against. This is not a problem at backup +time; we simply do not set the backing image when creating the destination +image: + +#. Create a new destination image with no backing file set. We will need to + specify the size of the base image, because the backing file isn't + available for QEMU to use to determine it. + + .. code:: bash + + $ qemu-img create -f qcow2 drive0.inc2.qcow2 64G + + .. note:: Alternatively, you can omit ``mode: "existing"`` from the push + backup commands to have QEMU create an image without a backing + file for you, but you lose control over format options like + compatibility and preallocation presets. + +#. Issue a new incremental backup command. Apart from the new destination + image, there is no difference from the last two examples. + + .. code:: json + + -> { + "execute": "drive-backup", + "arguments": { + "device": "drive0", + "bitmap": "bitmap0", + "target": "drive0.inc2.qcow2", + "format": "qcow2", + "sync": "incremental", + "mode": "existing" + } + } + + <- { "return": {} } + + ... + + <- { + "timestamp": {...}, + "data": { + "device": "drive0", + "type": "backup", + "speed": 0, + "len": 68719476736, + "offset": 68719476736 + }, + "event": "BLOCK_JOB_COMPLETED" + } + + ... + +The only difference from the perspective of the user is that you will need to +set the backing image when attempting to restore the backup: + +.. code:: bash + + $ qemu-img rebase drive0.inc2.qcow2 \ + -u -b drive0.inc1.qcow2 + +This uses the "unsafe" rebase mode to simply set the backing file to a file +that isn't present. + +It is also possible to use ``--image-opts`` to specify the entire backing +chain by hand as an ephemeral property at runtime, but that is beyond the +scope of this document. + +Example: Multi-drive Incremental Backup +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Assume we have a VM with two drives, "drive0" and "drive1" and we wish to back +both of them up such that the two backups represent the same crash-consistent +point in time. + +#. For each drive, create an empty image: + + .. code:: bash + + $ qemu-img create -f qcow2 drive0.full.qcow2 64G + $ qemu-img create -f qcow2 drive1.full.qcow2 64G + +#. Create a full (anchor) backup for each drive, with accompanying bitmaps: + + .. code:: json + + -> { + "execute": "transaction", + "arguments": { + "actions": [ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "drive0", + "name": "bitmap0" + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "drive1", + "name": "bitmap0" + } + }, + { + "type": "drive-backup", + "data": { + "device": "drive0", + "target": "/path/to/drive0.full.qcow2", + "sync": "full", + "format": "qcow2" + } + }, + { + "type": "drive-backup", + "data": { + "device": "drive1", + "target": "/path/to/drive1.full.qcow2", + "sync": "full", + "format": "qcow2" + } + } + ] + } + } + + <- { "return": {} } + + ... + + <- { + "timestamp": {...}, + "data": { + "device": "drive0", + "type": "backup", + "speed": 0, + "len": 68719476736, + "offset": 68719476736 + }, + "event": "BLOCK_JOB_COMPLETED" + } + + ... + + <- { + "timestamp": {...}, + "data": { + "device": "drive1", + "type": "backup", + "speed": 0, + "len": 68719476736, + "offset": 68719476736 + }, + "event": "BLOCK_JOB_COMPLETED" + } + + ... + +#. Later, create new destination images for each of the incremental backups + that point to their respective full backups: + + .. code:: bash + + $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ + -b drive0.full.qcow2 -F qcow2 + $ qemu-img create -f qcow2 drive1.inc0.qcow2 \ + -b drive1.full.qcow2 -F qcow2 + +#. Issue a multi-drive incremental push backup transaction: + + .. code:: json + + -> { + "execute": "transaction", + "arguments": { + "actions": [ + { + "type": "drive-backup", + "data": { + "device": "drive0", + "bitmap": "bitmap0", + "format": "qcow2", + "mode": "existing", + "sync": "incremental", + "target": "drive0.inc0.qcow2" + } + }, + { + "type": "drive-backup", + "data": { + "device": "drive1", + "bitmap": "bitmap0", + "format": "qcow2", + "mode": "existing", + "sync": "incremental", + "target": "drive1.inc0.qcow2" + } + }, + ] + } + } + + <- { "return": {} } + + ... + + <- { + "timestamp": {...}, + "data": { + "device": "drive0", + "type": "backup", + "speed": 0, + "len": 68719476736, + "offset": 68719476736 + }, + "event": "BLOCK_JOB_COMPLETED" + } + + ... + + <- { + "timestamp": {...}, + "data": { + "device": "drive1", + "type": "backup", + "speed": 0, + "len": 68719476736, + "offset": 68719476736 + }, + "event": "BLOCK_JOB_COMPLETED" + } + + ... -- In the event of an error that occurs after a backup job is - successfully launched, either by a direct QMP command or a QMP - transaction, the user will receive a ``BLOCK_JOB_COMPLETE`` event with - a failure message, accompanied by a ``BLOCK_JOB_ERROR`` event. +Push Backup Errors & Recovery +----------------------------- -- In the case of an event being cancelled, the user will receive a - ``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR - events. +In the event of an error that occurs after a push backup job is successfully +launched, either by an individual QMP command or a QMP transaction, the user +will receive a ``BLOCK_JOB_COMPLETE`` event with a failure message, +accompanied by a ``BLOCK_JOB_ERROR`` event. -- In either case, the incremental backup data contained within the - bitmap is safely rolled back, and the data within the bitmap is not - lost. The image file created for the failed attempt can be safely - deleted. +In the case of a job being cancelled, the user will receive a +``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR +events. -- Once the underlying problem is fixed (e.g. more storage space is - freed up), you can simply retry the incremental backup command with - the same bitmap. +In either failure case, the bitmap used for the failed operation is not +cleared. It will contain all of the dirty bits it did at the start of the +operation, plus any new bits that got marked during the operation. -Example -~~~~~~~ +Effectively, the "point in time" that a bitmap is recording differences +against is kept at the issuance of the last successful incremental backup, +instead of being moved forward to the start of this now-failed backup. -1. Create a target image: +Once the underlying problem is addressed (e.g. more storage space is allocated +on the destination), the incremental backup command can be retried with the +same bitmap. + +Example: Individual Failures +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Incremental Push Backup jobs that fail individually behave simply as +described above. This example demonstrates the single-job failure case: + +#. Create a target image: .. code:: bash - $ qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2 + $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ + -b drive0.full.qcow2 -F qcow2 -2. Attempt to create an incremental backup via QMP: +#. Attempt to create an incremental backup via QMP: .. code:: json - { "execute": "drive-backup", + -> { + "execute": "drive-backup", "arguments": { "device": "drive0", "bitmap": "bitmap0", - "target": "incremental.0.img", + "target": "drive0.inc0.qcow2", "format": "qcow2", "sync": "incremental", "mode": "existing" } } -3. Receive an event notifying us of failure: + <- { "return": {} } + +#. Receive a pair of events indicating failure: .. code:: json - { "timestamp": { "seconds": 1424709442, "microseconds": 844524 }, - "data": { "speed": 0, "offset": 0, "len": 67108864, - "error": "No space left on device", - "device": "drive1", "type": "backup" }, - "event": "BLOCK_JOB_COMPLETED" } + <- { + "timestamp": {...}, + "data": { + "device": "drive0", + "action": "report", + "operation": "write" + }, + "event": "BLOCK_JOB_ERROR" + } + + <- { + "timestamp": {...}, + "data": { + "speed": 0, + "offset": 0, + "len": 67108864, + "error": "No space left on device", + "device": "drive0", + "type": "backup" + }, + "event": "BLOCK_JOB_COMPLETED" + } -4. Delete the failed incremental, and re-create the image. +#. Delete the failed image, and re-create it. .. code:: bash - $ rm incremental.0.img - $ qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2 + $ rm drive0.inc0.qcow2 + $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ + -b drive0.full.qcow2 -F qcow2 -5. Retry the command after fixing the underlying problem, such as +#. Retry the command after fixing the underlying problem, such as freeing up space on the backup volume: .. code:: json - { "execute": "drive-backup", + -> { + "execute": "drive-backup", "arguments": { "device": "drive0", "bitmap": "bitmap0", - "target": "incremental.0.img", + "target": "drive0.inc0.qcow2", "format": "qcow2", "sync": "incremental", "mode": "existing" } } -6. Receive confirmation that the job completed successfully: + <- { "return": {} } + +#. Receive confirmation that the job completed successfully: .. code:: json - { "timestamp": { "seconds": 1424709668, "microseconds": 526525 }, - "data": { "device": "drive1", "type": "backup", - "speed": 0, "len": 67108864, "offset": 67108864}, - "event": "BLOCK_JOB_COMPLETED" } + <- { + "timestamp": {...}, + "data": { + "device": "drive0", + "type": "backup", + "speed": 0, + "len": 67108864, + "offset": 67108864 + }, + "event": "BLOCK_JOB_COMPLETED" + } -Partial Transactional Failures -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Example: Partial Transactional Failures +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Sometimes, a transaction will succeed in launching and return - success, but then later the backup jobs themselves may fail. It is - possible that a management application may have to deal with a - partial backup failure after a successful transaction. +QMP commands like `drive-backup <qemu-qmp-ref.html#index-drive_002dbackup>`_ +conceptually only start a job, and so transactions containing these commands +may succeed even if the job it created later fails. This might have surprising +interactions with notions of how a "transaction" ought to behave. -- If multiple backup jobs are specified in a single transaction, when - one of them fails, it will not interact with the other backup jobs in - any way. +This distinction means that on occasion, a transaction containing such job +launching commands may appear to succeed and return success, but later +individual jobs associated with the transaction may fail. It is possible that +a management application may have to deal with a partial backup failure after +a "successful" transaction. -- The job(s) that succeeded will clear the dirty bitmap associated with - the operation, but the job(s) that failed will not. It is not "safe" - to delete any incremental backups that were created successfully in - this scenario, even though others failed. +If multiple backup jobs are specified in a single transaction, if one of those +jobs fails, it will not interact with the other backup jobs in any way by +default. The job(s) that succeeded will clear the dirty bitmap associated with +the operation, but the job(s) that failed will not. It is therefore not safe +to delete any incremental backups that were created successfully in this +scenario, even though others failed. -Example -^^^^^^^ +This example illustrates a transaction with two backup jobs, where one fails +and one succeeds: -- QMP example highlighting two backup jobs: +#. Issue the transaction to start a backup of both drives. .. code:: json - { "execute": "transaction", + -> { + "execute": "transaction", "arguments": { "actions": [ - { "type": "drive-backup", - "data": { "device": "drive0", "bitmap": "bitmap0", - "format": "qcow2", "mode": "existing", - "sync": "incremental", "target": "d0-incr-1.qcow2" } }, - { "type": "drive-backup", - "data": { "device": "drive1", "bitmap": "bitmap1", - "format": "qcow2", "mode": "existing", - "sync": "incremental", "target": "d1-incr-1.qcow2" } }, - ] + { + "type": "drive-backup", + "data": { + "device": "drive0", + "bitmap": "bitmap0", + "format": "qcow2", + "mode": "existing", + "sync": "incremental", + "target": "drive0.inc0.qcow2" + } + }, + { + "type": "drive-backup", + "data": { + "device": "drive1", + "bitmap": "bitmap0", + "format": "qcow2", + "mode": "existing", + "sync": "incremental", + "target": "drive1.inc0.qcow2" + } + }] } } -- QMP example response, highlighting one success and one failure: +#. Receive notice that the Transaction was accepted, and jobs were + launched: - - Acknowledgement that the Transaction was accepted and jobs were - launched: + .. code:: json - .. code:: json + <- { "return": {} } - { "return": {} } +#. Receive notice that the first job has completed: - - Later, QEMU sends notice that the first job was completed: + .. code:: json - .. code:: json + <- { + "timestamp": {...}, + "data": { + "device": "drive0", + "type": "backup", + "speed": 0, + "len": 67108864, + "offset": 67108864 + }, + "event": "BLOCK_JOB_COMPLETED" + } - { "timestamp": { "seconds": 1447192343, "microseconds": 615698 }, - "data": { "device": "drive0", "type": "backup", - "speed": 0, "len": 67108864, "offset": 67108864 }, - "event": "BLOCK_JOB_COMPLETED" - } +#. Receive notice that the second job has failed: - - Later yet, QEMU sends notice that the second job has failed: + .. code:: json + + <- { + "timestamp": {...}, + "data": { + "device": "drive1", + "action": "report", + "operation": "read" + }, + "event": "BLOCK_JOB_ERROR" + } + + ... + + <- { + "timestamp": {...}, + "data": { + "speed": 0, + "offset": 0, + "len": 67108864, + "error": "Input/output error", + "device": "drive1", + "type": "backup" + }, + "event": "BLOCK_JOB_COMPLETED" + } - .. code:: json +At the conclusion of the above example, ``drive0.inc0.qcow2`` is valid and +must be kept, but ``drive1.inc0.qcow2`` is incomplete and should be +deleted. If a VM-wide incremental backup of all drives at a point-in-time is +to be made, new backups for both drives will need to be made, taking into +account that a new incremental backup for drive0 needs to be based on top of +``drive0.inc0.qcow2``. - { "timestamp": { "seconds": 1447192399, "microseconds": 683015 }, - "data": { "device": "drive1", "action": "report", - "operation": "read" }, - "event": "BLOCK_JOB_ERROR" } +For this example, an incremental backup for ``drive0`` was created, but not +for ``drive1``. The last VM-wide crash-consistent backup that is available in +this case is the full backup: - .. code:: json +.. code:: text - { "timestamp": { "seconds": 1447192399, "microseconds": - 685853 }, "data": { "speed": 0, "offset": 0, "len": 67108864, - "error": "Input/output error", "device": "drive1", "type": - "backup" }, "event": "BLOCK_JOB_COMPLETED" } + [drive0.full.qcow2] <-- [drive0.inc0.qcow2] + [drive1.full.qcow2] -- In the above example, ``d0-incr-1.qcow2`` is valid and must be kept, - but ``d1-incr-1.qcow2`` is invalid and should be deleted. If a VM-wide - incremental backup of all drives at a point-in-time is to be made, - new backups for both drives will need to be made, taking into account - that a new incremental backup for drive0 needs to be based on top of - ``d0-incr-1.qcow2``. +To repair this, issue a new incremental backup across both drives. The result +will be backup chains that resemble the following: -Grouped Completion Mode -~~~~~~~~~~~~~~~~~~~~~~~ +.. code:: text -- While jobs launched by transactions normally complete or fail on - their own, it is possible to instruct them to complete or fail - together as a group. + [drive0.full.qcow2] <-- [drive0.inc0.qcow2] <-- [drive0.inc1.qcow2] + [drive1.full.qcow2] <-------------------------- [drive1.inc1.qcow2] -- QMP transactions take an optional properties structure that can - affect the semantics of the transaction. +Example: Grouped Completion Mode +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- The "completion-mode" transaction property can be either "individual" - which is the default, legacy behavior described above, or "grouped," - a new behavior detailed below. +While jobs launched by transactions normally complete or fail individually, +it's possible to instruct them to complete or fail together as a group. QMP +transactions take an optional properties structure that can affect the +behavior of the transaction. -- Delayed Completion: In grouped completion mode, no jobs will report - success until all jobs are ready to report success. +The ``completion-mode`` transaction property can be either ``individual`` +which is the default legacy behavior described above, or ``grouped``, detailed +below. -- Grouped failure: If any job fails in grouped completion mode, all - remaining jobs will be cancelled. Any incremental backups will - restore their dirty bitmap objects as if no backup command was ever - issued. +In ``grouped`` completion mode, no jobs will report success until all jobs are +ready to report success. If any job fails, all other jobs will be cancelled. - - Regardless of if QEMU reports a particular incremental backup job - as CANCELLED or as an ERROR, the in-memory bitmap will be - restored. +Regardless of if a participating incremental backup job failed or was +cancelled, their associated bitmaps will all be held at their existing +points-in-time, as in individual failure cases. -Example -^^^^^^^ +Here's the same multi-drive backup scenario from `Example: Partial +Transactional Failures`_, but with the ``grouped`` completion-mode property +applied: -- Here's the same example scenario from above with the new property: +#. Issue the multi-drive incremental backup transaction: .. code:: json - { "execute": "transaction", + -> { + "execute": "transaction", "arguments": { - "actions": [ - { "type": "drive-backup", - "data": { "device": "drive0", "bitmap": "bitmap0", - "format": "qcow2", "mode": "existing", - "sync": "incremental", "target": "d0-incr-1.qcow2" } }, - { "type": "drive-backup", - "data": { "device": "drive1", "bitmap": "bitmap1", - "format": "qcow2", "mode": "existing", - "sync": "incremental", "target": "d1-incr-1.qcow2" } }, - ], "properties": { "completion-mode": "grouped" - } + }, + "actions": [ + { + "type": "drive-backup", + "data": { + "device": "drive0", + "bitmap": "bitmap0", + "format": "qcow2", + "mode": "existing", + "sync": "incremental", + "target": "drive0.inc0.qcow2" + } + }, + { + "type": "drive-backup", + "data": { + "device": "drive1", + "bitmap": "bitmap0", + "format": "qcow2", + "mode": "existing", + "sync": "incremental", + "target": "drive1.inc0.qcow2" + } + }] } } -- QMP example response, highlighting a failure for ``drive2``: +#. Receive notice that the Transaction was accepted, and jobs were launched: - - Acknowledgement that the Transaction was accepted and jobs were - launched: + .. code:: json + + <- { "return": {} } - .. code:: json +#. Receive notification that the backup job for ``drive1`` has failed: - { "return": {} } + .. code:: json - - Later, QEMU sends notice that the second job has errored out, but - that the first job was also cancelled: + <- { + "timestamp": {...}, + "data": { + "device": "drive1", + "action": "report", + "operation": "read" + }, + "event": "BLOCK_JOB_ERROR" + } - .. code:: json + <- { + "timestamp": {...}, + "data": { + "speed": 0, + "offset": 0, + "len": 67108864, + "error": "Input/output error", + "device": "drive1", + "type": "backup" + }, + "event": "BLOCK_JOB_COMPLETED" + } - { "timestamp": { "seconds": 1447193702, "microseconds": 632377 }, - "data": { "device": "drive1", "action": "report", - "operation": "read" }, - "event": "BLOCK_JOB_ERROR" } +#. Receive notification that the job for ``drive0`` has been cancelled: - .. code:: json + .. code:: json - { "timestamp": { "seconds": 1447193702, "microseconds": 640074 }, - "data": { "speed": 0, "offset": 0, "len": 67108864, - "error": "Input/output error", - "device": "drive1", "type": "backup" }, - "event": "BLOCK_JOB_COMPLETED" } + <- { + "timestamp": {...} + "data": { + "device": "drive0", + "type": "backup", + "speed": 0, + "len": 67108864, + "offset": 16777216 + }, + "event": "BLOCK_JOB_CANCELLED" + } - .. code:: json +At the conclusion of *this* example, both jobs have been aborted due to a +failure. Both destination images should be deleted and are no longer of use. - { "timestamp": { "seconds": 1447193702, "microseconds": 640163 }, - "data": { "device": "drive0", "type": "backup", "speed": 0, - "len": 67108864, "offset": 16777216 }, - "event": "BLOCK_JOB_CANCELLED" } +The transaction as a whole can simply be re-issued at a later time. .. raw:: html <!-- The FreeBSD Documentation License - Redistribution and use in source (Markdown) and 'compiled' forms (SGML, HTML, - PDF, PostScript, RTF and so forth) with or without modification, are permitted - provided that the following conditions are met: - - Redistributions of source code (Markdown) must retain the above copyright - notice, this list of conditions and the following disclaimer of this file - unmodified. - - Redistributions in compiled form (transformed to other DTDs, converted to PDF, - PostScript, RTF and other formats) must reproduce the above copyright notice, - this list of conditions and the following disclaimer in the documentation and/or - other materials provided with the distribution. - - THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE - DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE - FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR - SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER - CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, - OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF - THIS DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + Redistribution and use in source (ReST) and 'compiled' forms (SGML, HTML, + PDF, PostScript, RTF and so forth) with or without modification, are + permitted provided that the following conditions are met: + + Redistributions of source code (ReST) must retain the above copyright notice, + this list of conditions and the following disclaimer of this file unmodified. + + Redistributions in compiled form (transformed to other DTDs, converted to + PDF, PostScript, RTF and other formats) must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS + IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE + LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENTATION, EVEN IF ADVISED OF + THE POSSIBILITY OF SUCH DAMAGE. -->
Signature · 1 line
2.17.1
PATCH 93/97 slirp: Fix heap overflow in ip_reass on big packet input
In reply to Michael RothWhen the first fragment does not fit in the preallocated buffer, q will already be pointing to the ext buffer, so we mustn't try to update it.
(from libslirp.git commit 126c04acbabd7ad32c2b018fe10dfac2a3bc1210) (from libslirp.git commit e0be80430c390bce181ea04dfcdd6ea3dfa97de1) *squash in e0be80 (clarifying comments)
--- slirp/src/ip_input.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
slirp/src/ip_input.c +7 -6
@@ -331,6 +331,8 @@ insert: q = fp->frag_link.next; m =dtom (slirp, q); +int was_ext = m->m_flags &M_EXT ; + q = (struct ipasfrag *) q->ipf_next;while (q != (struct ipasfrag *)&fp->frag_link) {struct mbuf *t =dtom (slirp, q);
@@ -347,13 +349,12 @@ insert: q = fp->frag_link.next;/* -* If the fragments concatenated to an mbuf that's -* bigger than the total size of the fragment, then and -* m_ext buffer was alloced. But fp->ipq_next points to -* the old buffer (in the mbuf), so we must point ip -* into the new buffer. +* If the fragments concatenated to an mbuf that's bigger than the total +* size of the fragment and the mbuf was not already using an m_ext buffer, +* then an m_ext buffer was alloced. But fp->ipq_next points to the old +* buffer (in the mbuf), so we must point ip into the new buffer. */ -if (m->m_flags &M_EXT ) { +if (!was_ext && m->m_flags &M_EXT ) {int delta = (char *)q - m->m_dat; q = (struct ipasfrag *)(m->m_ext + delta); }
Signature · 1 line
2.17.1
PATCH 57/97 tpm_emulator: Translate TPM error codes to strings
In reply to Michael RothFrom: Stefan Berger <[email protected]>
Implement a function to translate TPM error codes to strings so that at least the most common error codes can be translated to human readable strings.
--- hw/tpm/tpm_emulator.c | 60 +++++++++++++++++++++++++++++++++++-------- hw/tpm/tpm_int.h | 13 ++++++++++ 2 files changed, 63 insertions(+), 10 deletions(-)
hw/tpm/tpm_emulator.c +50 -10
@@ -81,6 +81,40 @@ typedef struct TPMEmulator {TPMBlobBuffers state_blobs; } TPMEmulator; +struct tpm_error { +uint32_t tpm_result; +const char *string; +}; + +static const struct tpm_error tpm_errors[] = { +/* TPM 1.2 error codes */ + {TPM_BAD_PARAMETER ,"a parameter is bad" }, + {TPM_FAIL ,"operation failed" }, + {TPM_KEYNOTFOUND ,"key could not be found" }, + {TPM_BAD_PARAM_SIZE ,"bad parameter size" }, + {TPM_ENCRYPT_ERROR ,"encryption error" }, + {TPM_DECRYPT_ERROR ,"decryption error" }, + {TPM_BAD_KEY_PROPERTY ,"bad key property" }, + {TPM_BAD_MODE ,"bad (encryption) mode" }, + {TPM_BAD_VERSION ,"bad version identifier" }, + {TPM_BAD_LOCALITY ,"bad locality" }, +/* TPM 2 error codes */ + {TPM_RC_FAILURE ,"operation failed" }, + {TPM_RC_LOCALITY ,"bad locality" }, + {TPM_RC_INSUFFICIENT ,"insufficient amount of data" }, +}; + +static const char *tpm_emulator_strerror (uint32_t tpm_result) +{ +size_t i; + +for (i =0 ; i <ARRAY_SIZE (tpm_errors); i++) { +if (tpm_errors[i].tpm_result == tpm_result) { +return tpm_errors[i].string; + } + } +return "" ; +}static int tpm_emulator_ctrlcmd (TPMEmulator *tpm,unsigned long cmd,void *msg,size_t msg_len_in,size_t msg_len_out)
@@ -263,7 +297,8 @@ static int tpm_emulator_stop_tpm(TPMBackend *tb) res =be32_to_cpu (res);if (res) { -error_report ("tpm-emulator: TPM result for CMD_STOP: 0x%x" , res); +error_report ("tpm-emulator: TPM result for CMD_STOP: 0x%x %s" , res, +tpm_emulator_strerror (res));return -1 ; }
@@ -292,8 +327,9 @@ static int tpm_emulator_set_buffer_size(TPMBackend *tb, psbs.u.resp.tpm_result =be32_to_cpu (psbs.u.resp.tpm_result);if (psbs.u.resp.tpm_result !=0 ) { -error_report ("tpm-emulator: TPM result for set buffer size : 0x%x" , - psbs.u.resp.tpm_result); +error_report ("tpm-emulator: TPM result for set buffer size : 0x%x %s" , + psbs.u.resp.tpm_result, +tpm_emulator_strerror (psbs.u.resp.tpm_result));return -1 ; }
@@ -338,7 +374,8 @@ static int tpm_emulator_startup_tpm_resume(TPMBackend *tb, size_t buffersize, res =be32_to_cpu (init.u.resp.tpm_result);if (res) { -error_report ("tpm-emulator: TPM result for CMD_INIT: 0x%x" , res); +error_report ("tpm-emulator: TPM result for CMD_INIT: 0x%x %s" , res, +tpm_emulator_strerror (res)); goto err_exit; }return 0 ;
@@ -398,8 +435,9 @@ static int tpm_emulator_reset_tpm_established_flag(TPMBackend *tb, res =be32_to_cpu (reset_est.u.resp.tpm_result);if (res) { -error_report ("tpm-emulator: TPM result for rest establixhed flag: 0x%x" , - res); +error_report ( +"tpm-emulator: TPM result for rest established flag: 0x%x %s" , + res,tpm_emulator_strerror (res));return -1 ; }
@@ -637,7 +675,8 @@ static int tpm_emulator_get_state_blob(TPMEmulator *tpm_emu, res =be32_to_cpu (pgs.u.resp.tpm_result);if (res !=0 && (res &0x800 ) ==0 ) {error_report ("tpm-emulator: Getting the stateblob (type %d) failed " -"with a TPM error 0x%x" , type, res); +"with a TPM error 0x%x %s" , type, res, +tpm_emulator_strerror (res));return -1 ; }
@@ -757,7 +796,8 @@ static int tpm_emulator_set_state_blob(TPMEmulator *tpm_emu, tpm_result =be32_to_cpu (pss.u.resp.tpm_result);if (tpm_result !=0 ) {error_report ("tpm-emulator: Setting the stateblob (type %d) failed " -"with a TPM error 0x%x" , type, tpm_result); +"with a TPM error 0x%x %s" , type, tpm_result, +tpm_emulator_strerror (tpm_result));return -1 ; }
@@ -887,8 +927,8 @@ static void tpm_emulator_shutdown(TPMEmulator *tpm_emu)error_report ("tpm-emulator: Could not cleanly shutdown the TPM: %s" ,strerror (errno)); }else if (res !=0 ) { -error_report ("tpm-emulator: TPM result for sutdown: 0x%x" , -be32_to_cpu (res)); +error_report ("tpm-emulator: TPM result for shutdown: 0x%x %s" , +be32_to_cpu (res),tpm_emulator_strerror (be32_to_cpu (res))); } }
hw/tpm/tpm_int.h +13 -0
@@ -39,7 +39,16 @@ struct tpm_resp_hdr {#define TPM_TAG_RSP_AUTH1_COMMAND 0xc5#define TPM_TAG_RSP_AUTH2_COMMAND 0xc6 +#define TPM_BAD_PARAMETER 3#define TPM_FAIL 9 +#define TPM_KEYNOTFOUND 13 +#define TPM_BAD_PARAM_SIZE 25 +#define TPM_ENCRYPT_ERROR 32 +#define TPM_DECRYPT_ERROR 33 +#define TPM_BAD_KEY_PROPERTY 40 +#define TPM_BAD_MODE 44 +#define TPM_BAD_VERSION 46 +#define TPM_BAD_LOCALITY 61#define TPM_ORD_ContinueSelfTest 0x53#define TPM_ORD_GetTicks 0xf1
@@ -59,4 +68,8 @@ struct tpm_resp_hdr {#define TPM2_PT_MAX_COMMAND_SIZE 0x11e +#define TPM_RC_INSUFFICIENT 0x9a +#define TPM_RC_FAILURE 0x101 +#define TPM_RC_LOCALITY 0x907 +#endif /* TPM_TPM_INT_H */
Signature · 1 line
2.17.1
PATCH 19/97 iotests.py: do not use infinite waits
In reply to Michael RothFrom: John Snow <[email protected]>
Cap waits to 60 seconds so that iotests can fail gracefully if something goes wrong.
Message-id: [email protected]
--- tests/qemu-iotests/iotests.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
tests/qemu-iotests/iotests.py +7 -7
@@ -521,7 +521,7 @@ class VM(qtest.QEMUQtestMachine): output_list += [key + '=' + obj[key]] return ','.join(output_list) - def get_qmp_events_filtered(self, wait=True): + def get_qmp_events_filtered(self, wait=60.0): result = [] for ev in self.get_qmp_events(wait=wait): result.append(filter_qmp_event(ev))
@@ -539,10 +539,10 @@ class VM(qtest.QEMUQtestMachine): # Returns None on success, and an error string on failure def run_job(self, job, auto_finalize=True, auto_dismiss=False, - pre_finalize=None): + pre_finalize=None, wait=60.0): error = None while True: - for ev in self.get_qmp_events_filtered(wait=True): + for ev in self.get_qmp_events_filtered(wait=wait): if ev['event'] == 'JOB_STATUS_CHANGE': status = ev['data']['status'] if status == 'aborting':
@@ -633,7 +633,7 @@ class QMPTestCase(unittest.TestCase): self.assertEqual(self.vm.flatten_qmp_object(json.loads(json_filename[5:])), self.vm.flatten_qmp_object(reference)) - def cancel_and_wait(self, drive='drive0', force=False, resume=False): + def cancel_and_wait(self, drive='drive0', force=False, resume=False, wait=60.0): '''Cancel a block job and wait for it to finish, returning the event''' result = self.vm.qmp('block-job-cancel', device=drive, force=force) self.assert_qmp(result, 'return', {})
@@ -644,7 +644,7 @@ class QMPTestCase(unittest.TestCase): cancelled = False result = None while not cancelled: - for event in self.vm.get_qmp_events(wait=True): + for event in self.vm.get_qmp_events(wait=wait): if event['event'] == 'BLOCK_JOB_COMPLETED' or \ event['event'] == 'BLOCK_JOB_CANCELLED': self.assert_qmp(event, 'data/device', drive)
@@ -657,10 +657,10 @@ class QMPTestCase(unittest.TestCase): self.assert_no_active_block_jobs() return result - def wait_until_completed(self, drive='drive0', check_offset=True): + def wait_until_completed(self, drive='drive0', check_offset=True, wait=60.0): '''Wait for a block job to finish, returning the event''' while True: - for event in self.vm.get_qmp_events(wait=True): + for event in self.vm.get_qmp_events(wait=wait): if event['event'] == 'BLOCK_JOB_COMPLETED': self.assert_qmp(event, 'data/device', drive) self.assert_qmp_absent(event, 'data/error')
Signature · 1 line
2.17.1
PATCH 86/97 curl: Keep pointer to the CURLState in CURLSocket
In reply to Michael RothFrom: Max Reitz <[email protected]>
A follow-up patch will make curl_multi_do() and curl_multi_read() take a CURLSocket instead of the CURLState. They still need the latter, though, so add a pointer to it to the former.
Message-id: [email protected]
--- block/curl.c | 3 +++ 1 file changed, 3 insertions(+)
block/curl.c +3 -0
@@ -79,6 +79,7 @@ static CURLMcode __curl_multi_socket_action(CURLM *multi_handle,#define CURL_BLOCK_OPT_TIMEOUT_DEFAULT 5struct BDRVCURLState ; +struct CURLState ;static bool libcurl_initialized;
@@ -96,6 +97,7 @@ typedef struct CURLAIOCB {typedef struct CURLSocket {int fd; +struct CURLState *state;QLIST_ENTRY (CURLSocket) next; }CURLSocket ;
@@ -179,6 +181,7 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,if (!socket) { socket =g_new0 (CURLSocket,1 ); socket->fd = fd; + socket->state = state;QLIST_INSERT_HEAD (&state->sockets, socket, next); } socket =NULL ;
Signature · 1 line
2.17.1
PATCH 23/97 iotests: add iotest 256 for testing blockdev-backup across iothread contexts
In reply to Michael RothFrom: John Snow <[email protected]>
Message-id: [email protected]
--- tests/qemu-iotests/256 | 122 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/256.out | 119 ++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 242 insertions(+) create mode 100755 tests/qemu-iotests/256 create mode 100644 tests/qemu-iotests/256.out
tests/qemu-iotests/256 new file +122 -0
@@ -0,0 +1,122 @@ +#!/usr/bin/env python +# +# Test incremental/backup across iothread contexts +# +# Copyright (c) 2019 John Snow for Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# [email protected] + +import os +import iotests +from iotests import log + +iotests.verify_image_format(supported_fmts=['qcow2']) +size = 64 * 1024 * 1024 + +with iotests.FilePath('img0') as img0_path, \ + iotests.FilePath('img1') as img1_path, \ + iotests.FilePath('img0-full') as img0_full_path, \ + iotests.FilePath('img1-full') as img1_full_path, \ + iotests.FilePath('img0-incr') as img0_incr_path, \ + iotests.FilePath('img1-incr') as img1_incr_path, \ + iotests.VM() as vm: + + def create_target(filepath, name, size): + basename = os.path.basename(filepath) + nodename = "file_{}".format(basename) + log(vm.command('blockdev-create', job_id='job1', + options={ + 'driver': 'file', + 'filename': filepath, + 'size': 0, + })) + vm.run_job('job1') + log(vm.command('blockdev-add', driver='file', + node_name=nodename, filename=filepath)) + log(vm.command('blockdev-create', job_id='job2', + options={ + 'driver': iotests.imgfmt, + 'file': nodename, + 'size': size, + })) + vm.run_job('job2') + log(vm.command('blockdev-add', driver=iotests.imgfmt, + node_name=name, + file=nodename)) + + log('--- Preparing images & VM ---\n') + vm.add_object('iothread,id=iothread0') + vm.add_object('iothread,id=iothread1') + vm.add_device('virtio-scsi-pci,id=scsi0,iothread=iothread0') + vm.add_device('virtio-scsi-pci,id=scsi1,iothread=iothread1') + iotests.qemu_img_create('-f', iotests.imgfmt, img0_path, str(size)) + iotests.qemu_img_create('-f', iotests.imgfmt, img1_path, str(size)) + vm.add_drive(img0_path, interface='none') + vm.add_device('scsi-hd,id=device0,drive=drive0,bus=scsi0.0') + vm.add_drive(img1_path, interface='none') + vm.add_device('scsi-hd,id=device1,drive=drive1,bus=scsi1.0') + + log('--- Starting VM ---\n') + vm.launch() + + log('--- Create Targets & Full Backups ---\n') + create_target(img0_full_path, 'img0-full', size) + create_target(img1_full_path, 'img1-full', size) + ret = vm.qmp_log('transaction', indent=2, actions=[ + { 'type': 'block-dirty-bitmap-add', + 'data': { 'node': 'drive0', 'name': 'bitmap0' }}, + { 'type': 'block-dirty-bitmap-add', + 'data': { 'node': 'drive1', 'name': 'bitmap1' }}, + { 'type': 'blockdev-backup', + 'data': { 'device': 'drive0', + 'target': 'img0-full', + 'sync': 'full', + 'job-id': 'j0' }}, + { 'type': 'blockdev-backup', + 'data': { 'device': 'drive1', + 'target': 'img1-full', + 'sync': 'full', + 'job-id': 'j1' }} + ]) + if "error" in ret: + raise Exception(ret['error']['desc']) + vm.run_job('j0', auto_dismiss=True) + vm.run_job('j1', auto_dismiss=True) + + log('\n--- Create Targets & Incremental Backups ---\n') + create_target(img0_incr_path, 'img0-incr', size) + create_target(img1_incr_path, 'img1-incr', size) + ret = vm.qmp_log('transaction', indent=2, actions=[ + { 'type': 'blockdev-backup', + 'data': { 'device': 'drive0', + 'target': 'img0-incr', + 'sync': 'incremental', + 'bitmap': 'bitmap0', + 'job-id': 'j2' }}, + { 'type': 'blockdev-backup', + 'data': { 'device': 'drive1', + 'target': 'img1-incr', + 'sync': 'incremental', + 'bitmap': 'bitmap1', + 'job-id': 'j3' }} + ]) + if "error" in ret: + raise Exception(ret['error']['desc']) + vm.run_job('j2', auto_dismiss=True) + vm.run_job('j3', auto_dismiss=True) + + log('\n--- Done ---') + vm.shutdown()
tests/qemu-iotests/256.out new file +119 -0
119 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -250,3 +250,4 @@ 248 rw auto quick 253 rw auto quick 255 rw auto quick +256 rw auto quick
Signature · 1 line
2.17.1
PATCH 79/97 iotests: Restrict file Python tests to file
In reply to Michael RothFrom: Max Reitz <[email protected]>
Most of our Python unittest-style tests only support the file protocol. You can run them with any other protocol, but the test will simply ignore your choice and use file anyway.
We should let them signal that they require the file protocol so they are skipped when you want to test some other protocol.
Conflicts:
tests/qemu-iotests/257
*drop changes for test 257, not present in 4.0
--- tests/qemu-iotests/030 | 3 ++- tests/qemu-iotests/040 | 3 ++- tests/qemu-iotests/041 | 3 ++- tests/qemu-iotests/044 | 3 ++- tests/qemu-iotests/045 | 3 ++- tests/qemu-iotests/055 | 3 ++- tests/qemu-iotests/056 | 3 ++- tests/qemu-iotests/057 | 3 ++- tests/qemu-iotests/065 | 3 ++- tests/qemu-iotests/096 | 3 ++- tests/qemu-iotests/118 | 3 ++- tests/qemu-iotests/124 | 3 ++- tests/qemu-iotests/129 | 3 ++- tests/qemu-iotests/132 | 3 ++- tests/qemu-iotests/139 | 3 ++- tests/qemu-iotests/148 | 3 ++- tests/qemu-iotests/151 | 3 ++- tests/qemu-iotests/152 | 3 ++- tests/qemu-iotests/155 | 3 ++- tests/qemu-iotests/163 | 3 ++- tests/qemu-iotests/165 | 3 ++- tests/qemu-iotests/169 | 3 ++- tests/qemu-iotests/196 | 3 ++- tests/qemu-iotests/199 | 3 ++- tests/qemu-iotests/245 | 3 ++- 25 files changed, 50 insertions(+), 25 deletions(-)
tests/qemu-iotests/030 +2 -1
@@ -865,4 +865,5 @@ class TestSetSpeed(iotests.QMPTestCase): self.cancel_and_wait(resume=True) if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2', 'qed']) + iotests.main(supported_fmts=['qcow2', 'qed'], + supported_protocols=['file'])
tests/qemu-iotests/040 +2 -1
@@ -395,4 +395,5 @@ class TestReopenOverlay(ImageCommitTestCase): self.run_commit_test(self.img1, self.img0) if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2', 'qed']) + iotests.main(supported_fmts=['qcow2', 'qed'], + supported_protocols=['file'])
tests/qemu-iotests/041 +2 -1
@@ -1068,4 +1068,5 @@ class TestOrphanedSource(iotests.QMPTestCase): self.assert_qmp(result, 'error/class', 'GenericError') if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2', 'qed']) + iotests.main(supported_fmts=['qcow2', 'qed'], + supported_protocols=['file'])
tests/qemu-iotests/044 +2 -1
@@ -118,4 +118,5 @@ class TestRefcountTableGrowth(iotests.QMPTestCase): pass if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2']) + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/045 +2 -1
@@ -175,4 +175,5 @@ class TestSCMFd(iotests.QMPTestCase): "File descriptor named '%s' not found" % fdname) if __name__ == '__main__': - iotests.main(supported_fmts=['raw']) + iotests.main(supported_fmts=['raw'], + supported_protocols=['file'])
tests/qemu-iotests/055 +2 -1
@@ -563,4 +563,5 @@ class TestDriveCompression(iotests.QMPTestCase): target='drive1') if __name__ == '__main__': - iotests.main(supported_fmts=['raw', 'qcow2']) + iotests.main(supported_fmts=['raw', 'qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/056 +2 -1
@@ -335,4 +335,5 @@ class BackupTest(iotests.QMPTestCase): self.dismissal_failure(True) if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2', 'qed']) + iotests.main(supported_fmts=['qcow2', 'qed'], + supported_protocols=['file'])
tests/qemu-iotests/057 +2 -1
@@ -256,4 +256,5 @@ class TestSnapshotDelete(ImageSnapshotTestCase): self.assert_qmp(result, 'error/class', 'GenericError') if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2']) + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/065 +2 -1
@@ -129,4 +129,5 @@ TestQemuImgInfo = None TestQMP = None if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2']) + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/096 +2 -1
@@ -67,4 +67,5 @@ class TestLiveSnapshot(iotests.QMPTestCase): self.checkConfig('target') if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2']) + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/118 +2 -1
@@ -703,4 +703,5 @@ if __name__ == '__main__': iotests.qemu_default_machine) # Need to support image creation iotests.main(supported_fmts=['vpc', 'parallels', 'qcow', 'vdi', 'qcow2', - 'vmdk', 'raw', 'vhdx', 'qed']) + 'vmdk', 'raw', 'vhdx', 'qed'], + supported_protocols=['file'])
tests/qemu-iotests/124 +2 -1
@@ -779,4 +779,5 @@ class TestIncrementalBackupBlkdebug(TestIncrementalBackupBase): if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2']) + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/129 +2 -1
@@ -83,4 +83,5 @@ class TestStopWithBlockJob(iotests.QMPTestCase): self.do_test_stop("block-commit", device="drive0") if __name__ == '__main__': - iotests.main(supported_fmts=["qcow2"]) + iotests.main(supported_fmts=["qcow2"], + supported_protocols=["file"])
tests/qemu-iotests/132 +2 -1
@@ -56,4 +56,5 @@ class TestSingleDrive(iotests.QMPTestCase): 'target image does not match source after mirroring') if __name__ == '__main__': - iotests.main(supported_fmts=['raw', 'qcow2']) + iotests.main(supported_fmts=['raw', 'qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/139 +2 -1
@@ -361,4 +361,5 @@ class TestBlockdevDel(iotests.QMPTestCase): if __name__ == '__main__': - iotests.main(supported_fmts=["qcow2"]) + iotests.main(supported_fmts=["qcow2"], + supported_protocols=["file"])
tests/qemu-iotests/148 +2 -1
@@ -137,4 +137,5 @@ class TestFifoQuorumEvents(TestQuorumEvents): if __name__ == '__main__': iotests.verify_quorum() - iotests.main(supported_fmts=["raw"]) + iotests.main(supported_fmts=["raw"], + supported_protocols=["file"])
tests/qemu-iotests/151 +2 -1
@@ -142,4 +142,5 @@ class TestActiveMirror(iotests.QMPTestCase): if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2', 'raw']) + iotests.main(supported_fmts=['qcow2', 'raw'], + supported_protocols=['file'])
tests/qemu-iotests/152 +2 -1
@@ -59,4 +59,5 @@ class TestUnaligned(iotests.QMPTestCase): if __name__ == '__main__': - iotests.main(supported_fmts=['raw', 'qcow2']) + iotests.main(supported_fmts=['raw', 'qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/155 +2 -1
@@ -258,4 +258,5 @@ BaseClass = None MirrorBaseClass = None if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2']) + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/163 +2 -1
@@ -170,4 +170,5 @@ class TestShrink1M(ShrinkBaseClass): ShrinkBaseClass = None if __name__ == '__main__': - iotests.main(supported_fmts=['raw', 'qcow2']) + iotests.main(supported_fmts=['raw', 'qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/165 +2 -1
@@ -103,4 +103,5 @@ class TestPersistentDirtyBitmap(iotests.QMPTestCase): self.vm.shutdown() if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2']) + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/169 +2 -1
@@ -222,4 +222,5 @@ for cmb in list(itertools.product((True, False), repeat=2)): 'do_test_migration_resume_source', *list(cmb)) if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2']) + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/196 +2 -1
@@ -63,4 +63,5 @@ class TestInvalidateAutoclear(iotests.QMPTestCase): self.assertEqual(f.read(1), b'\x00') if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2']) + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/199 +2 -1
@@ -115,4 +115,5 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase): self.assert_qmp(result, 'return/sha256', sha256); if __name__ == '__main__': - iotests.main(supported_fmts=['qcow2'], supported_cache_modes=['none']) + iotests.main(supported_fmts=['qcow2'], supported_cache_modes=['none'], + supported_protocols=['file'])
tests/qemu-iotests/245 +2 -1
@@ -994,4 +994,5 @@ class TestBlockdevReopen(iotests.QMPTestCase): self.reopen(opts, {'backing': 'hd2'}) if __name__ == '__main__': - iotests.main(supported_fmts=["qcow2"]) + iotests.main(supported_fmts=["qcow2"], + supported_protocols=["file"])
Signature · 1 line
2.17.1
PATCH 74/97 xen-bus: Fix backend state transition on device reset
In reply to Michael RothFrom: Anthony PERARD <[email protected]>
When a frontend wants to reset its state and the backend one, it starts with setting "Closing", then waits for the backend (QEMU) to do the same.
But when QEMU is setting "Closing" to its state, it triggers an event (xenstore watch) that re-execute xen_device_backend_changed() and set the backend state to "Closed". QEMU should wait for the frontend to set "Closed" before doing the same.
Before setting "Closed" to the backend_state, we are also going to check if there is a frontend. If that the case, when the backend state is set to "Closing" the frontend should react and sets its state to "Closing" then "Closed". The backend should wait for that to happen.
Message-Id: <[email protected]>
--- hw/xen/xen-bus.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
hw/xen/xen-bus.c +20 -3
@@ -515,6 +515,23 @@ static void xen_device_backend_set_online(XenDevice *xendev, bool online)xen_device_backend_printf (xendev,"online" ,"%u" , online); } +/* +* Tell from the state whether the frontend is likely alive, +* i.e. it will react to a change of state of the backend. +*/ +static bool xen_device_state_is_active (enum xenbus_state state) +{ +switch (state) { +case XenbusStateInitWait: +case XenbusStateInitialised: +case XenbusStateConnected: +case XenbusStateClosing: +return true; +default : +return false; + } +} +static void xen_device_backend_changed (void *opaque) {XenDevice *xendev = opaque;
@@ -538,11 +555,11 @@ static void xen_device_backend_changed(void *opaque)/* * If the toolstack (or unplug request callback) has set the backend -* state to Closing, but there is no active frontend (i.e. the -* state is not Connected) then set the backend state to Closed. +* state to Closing, but there is no active frontend then set the +* backend state to Closed. */ if (xendev->backend_state == XenbusStateClosing && - xendev->frontend_state != XenbusStateConnected) { + !xen_device_state_is_active (state)) {xen_device_backend_set_state (xendev, XenbusStateClosed); }
Signature · 1 line
2.17.1
PATCH 63/97 backup: Copy only dirty areas
In reply to Michael RothFrom: Max Reitz <[email protected]>
The backup job must only copy areas that the copy_bitmap reports as dirty. This is always the case when using traditional non-offloading backup, because it copies each cluster separately. When offloading the copy operation, we sometimes copy more than one cluster at a time, but we only check whether the first one is dirty.
Therefore, whenever copy offloading is possible, the backup job currently produces wrong output when the guest writes to an area of which an inner part has already been backed up, because that inner part will be re-copied.
Message-id: [email protected]
--- block/backup.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
block/backup.c +11 -2
@@ -204,22 +204,31 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,cow_request_begin (&cow_request, job, start, end);while (start < end) { +int64_t dirty_end; +if (!hbitmap_get (job->copy_bitmap, start)) {trace_backup_do_cow_skip (job, start); start += job->cluster_size;continue ;/* already copied */ } + dirty_end =hbitmap_next_zero (job->copy_bitmap, start, (end - start)); +if (dirty_end <0 ) { + dirty_end = end; + } +trace_backup_do_cow_process (job, start);if (job->use_copy_range) { - ret =backup_cow_with_offload (job, start, end, is_write_notifier); + ret =backup_cow_with_offload (job, start, dirty_end, + is_write_notifier);if (ret <0 ) { job->use_copy_range = false; } }if (!job->use_copy_range) { - ret =backup_cow_with_bounce_buffer (job, start, end, is_write_notifier, + ret =backup_cow_with_bounce_buffer (job, start, dirty_end, + is_write_notifier, error_is_read, &bounce_buffer); }if (ret <0 ) {
Signature · 1 line
2.17.1
PATCH 6/97 block: Fix AioContext switch for bs->drv == NULL
In reply to Michael RothFrom: Kevin Wolf <[email protected]>
Even for block nodes with bs->drv == NULL, we can't just ignore a bdrv_set_aio_context() call. Leaving the node in its old context can mean that it's still in an iothread context in bdrv_close_all() during shutdown, resulting in an attempted unlock of the AioContext lock which we don't hold.
This is an example stack trace of a related crash:
#0 0x00007ffff59da57f in raise () at /lib64/libc.so.6
#1 0x00007ffff59c4895 in abort () at /lib64/libc.so.6
#2 0x0000555555b97b1e in error_exit (err=<optimized out>, msg=msg@entry=0x555555d386d0 <__func__.19059> "qemu_mutex_unlock_impl") at util/qemu-thread-posix.c:36
#3 0x0000555555b97f7f in qemu_mutex_unlock_impl (mutex=mutex@entry=0x5555568002f0, file=file@entry=0x555555d378df "util/async.c", line=line@entry=507) at util/qemu-thread-posix.c:97
#4 0x0000555555b92f55 in aio_context_release (ctx=ctx@entry=0x555556800290) at util/async.c:507
#5 0x0000555555b05cf8 in bdrv_prwv_co (child=child@entry=0x7fffc80012f0, offset=offset@entry=131072, qiov=qiov@entry=0x7fffffffd4f0, is_write=is_write@entry=true, flags=flags@entry=0)
at block/io.c:833
#6 0x0000555555b060a9 in bdrv_pwritev (qiov=0x7fffffffd4f0, offset=131072, child=0x7fffc80012f0) at block/io.c:990
#7 0x0000555555b060a9 in bdrv_pwrite (child=0x7fffc80012f0, offset=131072, buf=<optimized out>, bytes=<optimized out>) at block/io.c:990
#8 0x0000555555ae172b in qcow2_cache_entry_flush (bs=bs@entry=0x555556810680, c=c@entry=0x5555568cc740, i=i@entry=0) at block/qcow2-cache.c:51
#9 0x0000555555ae18dd in qcow2_cache_write (bs=bs@entry=0x555556810680, c=0x5555568cc740) at block/qcow2-cache.c:248
#10 0x0000555555ae15de in qcow2_cache_flush (bs=0x555556810680, c=<optimized out>) at block/qcow2-cache.c:259
#11 0x0000555555ae16b1 in qcow2_cache_flush_dependency (c=0x5555568a1700, c=0x5555568a1700, bs=0x555556810680) at block/qcow2-cache.c:194
#12 0x0000555555ae16b1 in qcow2_cache_entry_flush (bs=bs@entry=0x555556810680, c=c@entry=0x5555568a1700, i=i@entry=0) at block/qcow2-cache.c:194
#13 0x0000555555ae18dd in qcow2_cache_write (bs=bs@entry=0x555556810680, c=0x5555568a1700) at block/qcow2-cache.c:248
#14 0x0000555555ae15de in qcow2_cache_flush (bs=bs@entry=0x555556810680, c=<optimized out>) at block/qcow2-cache.c:259
#15 0x0000555555ad242c in qcow2_inactivate (bs=bs@entry=0x555556810680) at block/qcow2.c:2124
#16 0x0000555555ad2590 in qcow2_close (bs=0x555556810680) at block/qcow2.c:2153
#17 0x0000555555ab0c62 in bdrv_close (bs=0x555556810680) at block.c:3358
#18 0x0000555555ab0c62 in bdrv_delete (bs=0x555556810680) at block.c:3542
#19 0x0000555555ab0c62 in bdrv_unref (bs=0x555556810680) at block.c:4598
#20 0x0000555555af4d72 in blk_remove_bs (blk=blk@entry=0x5555568103d0) at block/block-backend.c:785
#21 0x0000555555af4dbb in blk_remove_all_bs () at block/block-backend.c:483
#22 0x0000555555aae02f in bdrv_close_all () at block.c:3412
#23 0x00005555557f9796 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4776The reproducer I used is a qcow2 image on gluster volume, where the virtual disk size (4 GB) is larger than the gluster volume size (64M), so we can easily trigger an ENOSPC. This backend is assigned to a virtio-blk device using an iothread, and then from the guest a 'dd if=/dev/zero of=/dev/vda bs=1G count=1' causes the VM to stop because of an I/O error. qemu_gluster_co_flush_to_disk() sets bs->drv = NULL on error, so when virtio-blk stops the dataplane, the block nodes stay in the iothread AioContext. A 'quit' monitor command issued from this paused state crashes the process.
--- block.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
block.c +2 -10
@@ -5672,10 +5672,6 @@ void bdrv_detach_aio_context(BlockDriverState *bs)BdrvAioNotifier *baf, *baf_tmp;BdrvChild *child; -if (!bs->drv) { -return ; - } -assert (!bs->walking_aio_notifiers); bs->walking_aio_notifiers = true; QLIST_FOREACH_SAFE(baf, &bs->aio_notifiers, list, baf_tmp) {
@@ -5690,7 +5686,7 @@ void bdrv_detach_aio_context(BlockDriverState *bs)*/ bs->walking_aio_notifiers = false; -if (bs->drv->bdrv_detach_aio_context) { +if (bs->drv && bs->drv->bdrv_detach_aio_context) { bs->drv->bdrv_detach_aio_context (bs); } QLIST_FOREACH(child, &bs->children, next) {
@@ -5709,10 +5705,6 @@ void bdrv_attach_aio_context(BlockDriverState *bs,BdrvAioNotifier *ban, *ban_tmp;BdrvChild *child; -if (!bs->drv) { -return ; - } -if (bs->quiesce_counter) {aio_disable_external (new_context); }
@@ -5722,7 +5714,7 @@ void bdrv_attach_aio_context(BlockDriverState *bs, QLIST_FOREACH(child, &bs->children, next) {bdrv_attach_aio_context (child->bs, new_context); } -if (bs->drv->bdrv_attach_aio_context) { +if (bs->drv && bs->drv->bdrv_attach_aio_context) { bs->drv->bdrv_attach_aio_context (bs, new_context); }
Signature · 1 line
2.17.1
Quoted 4 lines · Michael Roth
Hi everyone,
The following new patches are queued for QEMU stable v4.0.1:
Quoted 9 lines
https://github.com/mdroth/qemu/commits/stable-4.0-staging
The release is planned for 2019-10-17:
https://wiki.qemu.org/Planning/4.0
Please respond here or CC [email protected] on any patches you think should be included in the release.
Would it make sense to include the slirp update:
e1a4a24d262ba5ac74ea1795adb3ab1cd574c7fb
"slirp: update with CVE-2019-14378 fix"
?
And maybe these commits:
22235bb609c18547cf6b215bad1f9d2ec56ad371 "pc-dimm: fix crash when invalid slot number is used"
95667c3be0c9f5fc62f58fe845879250f63f7d32 "nvme: Set number of queues later in nvme_init()"
c0bccee9b40ec58c9d165b406ae3d4f63652ce53 "hw/ssi/mss-spi: Avoid crash when reading empty RX FIFO"
a09ef5040477643a7026703199d8781fe048d3a8 "hw/display/xlnx_dp: Avoid crash when reading empty RX FIFO"
Thomas
Michael Roth
AppliedQuoting Thomas Huth (2019-10-01 23:40:49)
Quoted 11 lines · Michael Roth
Quoted 10 lines
Hi everyone,
The following new patches are queued for QEMU stable v4.0.1:
https://github.com/mdroth/qemu/commits/stable-4.0-staging
The release is planned for 2019-10-17:
Quoted 7 lines
Quoted 3 lines
Please respond here or CC [email protected] on any patches you think should be included in the release.
Would it make sense to include the slirp update:
e1a4a24d262ba5ac74ea1795adb3ab1cd574c7fb
"slirp: update with CVE-2019-14378 fix"
4.0 hadn't yet moved to using slirp submodule, but the following patches are included here and should provide the equivalent fixes:
slirp: ip_reass: Fix use after free
slirp: Fix heap overflow in ip_reass on big packet input
Quoted 8 lines
?
And maybe these commits:
22235bb609c18547cf6b215bad1f9d2ec56ad371"pc-dimm: fix crash when invalid slot number is used"
Quoted 8 lines
95667c3be0c9f5fc62f58fe845879250f63f7d32"nvme: Set number of queues later innvme_init()"
c0bccee9b40ec58c9d165b406ae3d4f63652ce53"hw/ssi/mss-spi: Avoid crash when reading empty RX FIFO"
a09ef5040477643a7026703199d8781fe048d3a8"hw/display/xlnx_dp: Avoid crash when reading empty RX FIFO"
Applied, thanks!
Quoted 3 lines
Thomas
Quoted 8 lines · Michael Roth
Hi everyone,
The following new patches are queued for QEMU stable v4.0.1:
https://github.com/mdroth/qemu/commits/stable-4.0-staging
The release is planned for 2019-10-17:
Quoted 8 lines
https://wiki.qemu.org/Planning/4.0Please respond here or CC [email protected] on any patches you think should be included in the release.
Note that this update falls outside the normal stable release support window (~1 development cycle), but is being released now since it was delayed from its intended release date.
This commit fix data corruption introduced by ee5d0f89de3:
commit 04da973501b591525ce68c2925c61c8886badd4d
Author: Li Qiang <[email protected]> Date: Wed Apr 24 07:06:41 2019 -0700
hw/nvram/fw_cfg: Store 'reboot-timeout' as little endian
Quoted 9 lines · Michael Roth
Hi everyone,
The following new patches are queued for QEMU stable v4.0.1:
https://github.com/mdroth/qemu/commits/stable-4.0-staging
The release is planned for 2019-10-17:
Hi,
How about
commit fd392cfa8e6fb0dc34bd0327fc356dfbf6edf1fd
Author: Yury Kotov <[email protected]> Date: Mon Apr 8 14:33:43 2019 +0300
migration: Fix use-after-free during process exit
commit ebb6ff25cd888a52a64a9adc3692541c6d1d9a42
Author: Philippe Mathieu-Daudé <[email protected]> Date: Fri Aug 16 19:15:03 2019 +0200
virtio-blk: Cancel the pending BH when the dataplane is reset
and https://lists.nongnu.org/archive/html/qemu-devel/2019-09/msg04618.html
?
Hi Michael,
Quoted 4 lines · Michael Roth
Hi everyone,
The following new patches are queued for QEMU stable v4.0.1:
Quoted 8 lines
https://github.com/mdroth/qemu/commits/stable-4.0-staging
The release is planned for 2019-10-17:
https://wiki.qemu.org/Planning/4.0Please respond here or CC [email protected] on any patches you think should be included in the release.
Since it is a "release", these probably fit:
commit 45c61c6c23918e3b05ed9ecac5b2328ebae5f774
Author: Michael Roth <[email protected]> Date: Thu Sep 12 18:12:01 2019 -0500
make-release: pull in edk2 submodules so we can build it from tarballs
commit f3e330e3c319160ac04954399b5a10afc965098c
Author: Michael Roth <[email protected]> Date: Thu Sep 12 18:12:02 2019 -0500
roms/Makefile.edk2: don't pull in submodules when building from tarball
When is the next qemu-stable release scheduled? Do we care about Python2 use for this one?
Quoting Philippe Mathieu-Daudé (2019-10-08 08:04:52)
Quoted 24 lines
Hi Michael,
Quoted 4 lines · Michael Roth
Hi everyone,
The following new patches are queued for QEMU stable v4.0.1:
Quoted 8 lines
https://github.com/mdroth/qemu/commits/stable-4.0-staging
The release is planned for 2019-10-17:
https://wiki.qemu.org/Planning/4.0Please respond here or CC [email protected] on any patches you think should be included in the release.
Since it is a "release", these probably fit:
commit
45c61c6c23918e3b05ed9ecac5b2328ebae5f774Author: Michael Roth <[email protected]> Date: Thu Sep 12 18:12:01 2019 -0500make-release: pull in edk2 submodules so we can build it from tarballs
Quoted 8 lines
commit
f3e330e3c319160ac04954399b5a10afc965098cAuthor: Michael Roth <[email protected]> Date: Thu Sep 12 18:12:02 2019 -0500roms/Makefile.edk2: don't pull in submodules when building from tarball
When is the next qemu-stable release scheduled? Do we care about Python2 use for this one?
4.1.1, likely early November during 4.2 hard-freeze. I think Python2 is okay since presumably downstreams would've crossed that bridge with 4.1.0, assuming you're thinking of the changes in your recent edk2 pull.
Quoted 29 lines · Michael Roth
Quoting Philippe Mathieu-Daudé (2019-10-08 08:04:52)
Quoted 28 lines
Hi Michael,
Quoted 4 lines · Michael Roth
Hi everyone,
The following new patches are queued for QEMU stable v4.0.1:
Quoted 8 lines
https://github.com/mdroth/qemu/commits/stable-4.0-staging
The release is planned for 2019-10-17:
https://wiki.qemu.org/Planning/4.0Please respond here or CC [email protected] on any patches you think should be included in the release.
Since it is a "release", these probably fit:
commit
45c61c6c23918e3b05ed9ecac5b2328ebae5f774Author: Michael Roth <[email protected]> Date: Thu Sep 12 18:12:01 2019 -0500make-release: pull in edk2 submodules so we can build it from tarballs
commit
f3e330e3c319160ac04954399b5a10afc965098cAuthor: Michael Roth <[email protected]> Date: Thu Sep 12 18:12:02 2019 -0500
Quoted 8 lines
Quoted 4 lines
roms/Makefile.edk2: don't pull in submodules when building from tarball
When is the next qemu-stable release scheduled? Do we care about Python2 use for this one?
4.1.1, likely early November during 4.2 hard-freeze. I think Python2 is okay since presumably downstreams would've crossed that bridge with 4.1.0, assuming you're thinking of the changes in your recent edk2 pull.
Yes.
Do you mind adding these dates to https://wiki.qemu.org/Planning/4.1?
Thanks,
Phil.
Quoted 1 line · Michael Roth
Hi everyone,
Hi Michael,
Quoted 2 lines
Please respond here or CC [email protected] on any patches you think should be included in the release.
[...]
Quoted 2 lines
Anthony PERARD (1):
xen-bus: Fix backend state transition on device reset
Could you cherry-pick df6180bb56cd03949c2c64083da58755fed81a61 "xen-bus: check whether the frontend is active during device reset..." as well, to fix the fix?
Thanks,
Signature · 1 line
Anthony PERARD
Quoting Philippe Mathieu-Daudé (2019-10-09 09:23:22)
Quoted 35 lines · Michael Roth
Quoted 34 lines
Quoting Philippe Mathieu-Daudé (2019-10-08 08:04:52)
Quoted 32 lines
Hi Michael,
Quoted 4 lines · Michael Roth
Hi everyone,
The following new patches are queued for QEMU stable v4.0.1:
Quoted 8 lines
https://github.com/mdroth/qemu/commits/stable-4.0-staging
The release is planned for 2019-10-17:
https://wiki.qemu.org/Planning/4.0Please respond here or CC [email protected] on any patches you think should be included in the release.
Since it is a "release", these probably fit:
commit
45c61c6c23918e3b05ed9ecac5b2328ebae5f774Author: Michael Roth <[email protected]> Date: Thu Sep 12 18:12:01 2019 -0500make-release: pull in edk2 submodules so we can build it from tarballs
commit
f3e330e3c319160ac04954399b5a10afc965098cAuthor: Michael Roth <[email protected]> Date: Thu Sep 12 18:12:02 2019 -0500roms/Makefile.edk2: don't pull in submodules when building from tarball
When is the next qemu-stable release scheduled? Do we care about Python2 use for this one?
Quoted 7 lines
Quoted 3 lines
4.1.1, likely early November during 4.2 hard-freeze. I think Python2 is okay since presumably downstreams would've crossed that bridge with 4.1.0, assuming you're thinking of the changes in your recent edk2 pull.
Yes.
Do you mind adding these dates to https://wiki.qemu.org/Planning/4.1?
Sure, I've set the freeze date for November 18th, and the release for the 21st.
Quoted 5 lines
Thanks,
Phil.