qcow2: Support refcount orders != 4
cover qcow2: Support refcount orders != 4
This should not need much of a cover letter, but here goes anyway:
As of version 3, the qcow2 file format supports different widths for refcount entries, ranging from 1 to 64 bit (only powers of two). Currently, qemu only supports 16 bit, which is the only width supported by version 2 (compat=0.10) images.
This series adds support to qemu for all other valid refcount orders. This is mainly done by adding two function pointers into the BDRVQcowState structure for reading and writing refcount values independently of the current refcount entry width; all in-memory refcount arrays (mostly cached refcount blocks) now are void pointers and are accessed through these functions alone.
Thanks to previous work of making the qemu code agnostic of e.g. the number of refcount entries per refcount block, the rest is fairly trivial. The most complex patch in this series is patch 18 which implements changing the refcount order through qemu-img amend.
To test different refcount widths, simply invoke the qemu-iotests check program with -o refcount_width=${your_desired_width}. The final test in this series adds some tests for operations which do not work with certain refcount orders and for refcount order amendment.
Max Reitz (21):
qcow2: Add two new fields to BDRVQcowState
qcow2: Add refcount_width to format-specific info
qcow2: Use 64 bits for refcount values
qcow2: Respect error in qcow2_alloc_bytes()
qcow2: Refcount overflow and qcow2_alloc_bytes()
qcow2: Helper function for refcount modification
qcow2: Helper for refcount array size calculation
qcow2: More helpers for refcount modification
qcow2: Open images with refcount order != 4
qcow2: refcount_order parameter for qcow2_create2
iotests: Prepare for refcount_width option
qcow2: Allow creation with refcount order != 4
block: Add opaque value to the amend CB
qcow2: Use error_report() in qcow2_amend_options()
qcow2: Use abort() instead of assert(false)
qcow2: Split upgrade/downgrade paths for amend
qcow2: Use intermediate helper CB for amend
qcow2: Add function for refcount order amendment
qcow2: Invoke refcount order amendment function
qcow2: Point to amend function in check
iotests: Add test for different refcount widths
block.c | 4 +- block/qcow2-cluster.c | 23 +- block/qcow2-refcount.c | 846 ++++++++++++++++++++++++++++++++++----- block/qcow2.c | 252 +++++++++--- block/qcow2.h | 24 +- include/block/block.h | 4 +- include/block/block_int.h | 4 +- qapi/block-core.json | 5 +- qemu-img.c | 5 +- tests/qemu-iotests/007 | 4 + tests/qemu-iotests/015 | 1 + tests/qemu-iotests/026 | 11 + tests/qemu-iotests/029 | 1 + tests/qemu-iotests/049.out | 112 +++--- tests/qemu-iotests/051 | 1 + tests/qemu-iotests/058 | 1 + tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 14 +- tests/qemu-iotests/065 | 23 +- tests/qemu-iotests/067 | 7 + tests/qemu-iotests/067.out | 10 +- tests/qemu-iotests/079 | 1 + tests/qemu-iotests/079.out | 18 +- tests/qemu-iotests/080 | 1 + tests/qemu-iotests/082.out | 48 ++- tests/qemu-iotests/085.out | 38 +- tests/qemu-iotests/089 | 7 + tests/qemu-iotests/089.out | 2 + tests/qemu-iotests/090 | 1 + tests/qemu-iotests/108 | 6 + tests/qemu-iotests/112 | 225 +++++++++++ tests/qemu-iotests/112.out | 123 ++++++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 34 files changed, 1546 insertions(+), 281 deletions(-) create mode 100755 tests/qemu-iotests/112 create mode 100644 tests/qemu-iotests/112.out
Signature · 1 line
1.9.3
PATCH 1/21 qcow2: Add two new fields to BDRVQcowState
Add two new fields regarding refcount information (the bit width of every entry and the maximum refcount value) to the BDRVQcowState.
--- block/qcow2-refcount.c | 2 +- block/qcow2.c | 9 +++++++++ block/qcow2.h | 2 ++ 3 files changed, 12 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +1 -1
@@ -584,7 +584,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs, refcount =be16_to_cpu (refcount_block[block_index]); refcount += addend; -if (refcount <0 || refcount >0xffff ) { +if (refcount <0 || refcount > s->refcount_max) { ret = -EINVAL ; goto fail; }
block/qcow2.c +9 -0
@@ -684,6 +684,15 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } s->refcount_order = header.refcount_order; + s->refcount_bits =1 << s->refcount_order; +if (s->refcount_order <6 ) { + s->refcount_max = (UINT64_C (1 ) << s->refcount_bits) -1 ; + }else { +/* The above shift would overflow with s->refcount_bits == 64; +* furthermore, we do not want to use UINT64_MAX because refcounts will +* be passed around in int64_ts (negative values for -errno) */ + s->refcount_max =INT64_MAX ; + }if (header.crypt_method >QCOW_CRYPT_AES ) { error_setg(errp, "Unsupported encryption method: %" PRIu32,
block/qcow2.h +2 -0
@@ -258,6 +258,8 @@ typedef struct BDRVQcowState {int qcow_version;bool use_lazy_refcounts;int refcount_order; +int refcount_bits; +uint64_t refcount_max;bool discard_passthrough[QCOW2_DISCARD_MAX ];
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 01/21] qcow2: Add two new fields to BDRVQcowState
Quoted 9 lines · Max Reitz
Add two new fields regarding refcount information (the bit width of every entry and the maximum refcount value) to the BDRVQcowState.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 2 +- block/qcow2.c | 9 +++++++++ block/qcow2.h | 2 ++ 3 files changed, 12 insertions(+), 1 deletion(-)
block/qcow2.c static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, · line 687
Quoted 6 lines
+++
b/block/qcow2.cblock/qcow2.c +1
-0@@ -684,6 +684,15 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } s->refcount_order = header.refcount_order; + s->refcount_bits =1 << s->refcount_order;
Not shown is the context where a few lines before still enforces refcount_order==4, so this doesn't overflow. When later patches relax that, I'll make sure we don't overflow here as well.
block/qcow2.c static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, · line 689
Quoted 2 lines
block/qcow2.c +2
-0+if (s->refcount_order <6 ) { + s->refcount_max = (UINT64_C (1 ) << s->refcount_bits) -1 ;
I don't see the UINT64_C macro get much use, but like it better than casting :)
block/qcow2.c static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, · line 690
Quoted 1 line
block/qcow2.c +1
-0+ }else {
I don't know if Coverity might complain about dead code during bisection (since we can't get here until we relax refcount_order to not be forced to 4), but that's a layer beyond making sure 'make check' works so I don't care.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 2/21 qcow2: Add refcount_width to format-specific info
In reply to Max ReitzAdd the bit width of every refcount entry to the format-specific information.
This breaks some test outputs, fix them.
--- block/qcow2.c | 4 +++- qapi/block-core.json | 5 ++++- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/065 | 23 +++++++++++++++-------- tests/qemu-iotests/067.out | 10 +++++----- tests/qemu-iotests/082.out | 7 +++++++ tests/qemu-iotests/089.out | 2 ++ 7 files changed, 37 insertions(+), 15 deletions(-)
block/qcow2.c +3 -1
@@ -2475,7 +2475,8 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs) };if (s->qcow_version ==2 ) { *spec_info->qcow2 = (ImageInfoSpecificQCow2 ){ - .compat =g_strdup ("0.10" ), + .compat =g_strdup ("0.10" ), + .refcount_width = s->refcount_bits, }; }else if (s->qcow_version ==3 ) { *spec_info->qcow2 = (ImageInfoSpecificQCow2 ){
@@ -2486,6 +2487,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs) .corrupt = s->incompatible_features & QCOW2_INCOMPAT_CORRUPT, .has_corrupt = true, + .refcount_width = s->refcount_bits, }; }
qapi/block-core.json +4 -1
@@ -41,13 +41,16 @@ # @corrupt: #optional true if the image has been marked corrupt; only valid for # compat >= 1.1 (since 2.2) # +# @refcount-width: width of a refcount entry in bits (since 2.3) +# # Since: 1.7 ## { 'type': 'ImageInfoSpecificQCow2', 'data': { 'compat': 'str', '*lazy-refcounts': 'bool', - '*corrupt': 'bool' + '*corrupt': 'bool', + 'refcount-width': 'int' } } ##
tests/qemu-iotests/060.out +1 -0
@@ -19,6 +19,7 @@ cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false + refcount width: 16 corrupt: true qemu-io: can't open device TEST_DIR/t.IMGFMT: IMGFMT: Image is corrupt; cannot be opened read/write read 512/512 bytes at offset 0
tests/qemu-iotests/065 +15 -8
@@ -88,34 +88,41 @@ class TestQMP(TestImageInfoSpecific): class TestQCow2(TestQemuImgInfo): '''Testing a qcow2 version 2 image''' img_options = 'compat=0.10' - json_compare = { 'compat': '0.10' } - human_compare = [ 'compat: 0.10' ] + json_compare = { 'compat': '0.10', 'refcount-width': 16 } + human_compare = [ 'compat: 0.10', 'refcount width: 16' ] class TestQCow3NotLazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts disabled''' img_options = 'compat=1.1,lazy_refcounts=off' - json_compare = { 'compat': '1.1', 'lazy-refcounts': False, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: false', 'corrupt: false' ] + json_compare = { 'compat': '1.1', 'lazy-refcounts': False, + 'refcount-width': 16, 'corrupt': False } + human_compare = [ 'compat: 1.1', 'lazy refcounts: false', + 'refcount width: 16', 'corrupt: false' ] class TestQCow3Lazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts enabled''' img_options = 'compat=1.1,lazy_refcounts=on' - json_compare = { 'compat': '1.1', 'lazy-refcounts': True, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: true', 'corrupt: false' ] + json_compare = { 'compat': '1.1', 'lazy-refcounts': True, + 'refcount-width': 16, 'corrupt': False } + human_compare = [ 'compat: 1.1', 'lazy refcounts: true', + 'refcount width: 16', 'corrupt: false' ] class TestQCow3NotLazyQMP(TestQMP): '''Testing a qcow2 version 3 image with lazy refcounts disabled, opening with lazy refcounts enabled''' img_options = 'compat=1.1,lazy_refcounts=off' qemu_options = 'lazy-refcounts=on' - compare = { 'compat': '1.1', 'lazy-refcounts': False, 'corrupt': False } + compare = { 'compat': '1.1', 'lazy-refcounts': False, + 'refcount-width': 16, 'corrupt': False } + class TestQCow3LazyQMP(TestQMP): '''Testing a qcow2 version 3 image with lazy refcounts enabled, opening with lazy refcounts disabled''' img_options = 'compat=1.1,lazy_refcounts=on' qemu_options = 'lazy-refcounts=off' - compare = { 'compat': '1.1', 'lazy-refcounts': True, 'corrupt': False } + compare = { 'compat': '1.1', 'lazy-refcounts': True, + 'refcount-width': 16, 'corrupt': False } TestImageInfoSpecific = None TestQemuImgInfo = None
tests/qemu-iotests/067.out +5 -5
@@ -6,7 +6,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,if=none,id=disk -device virtio-blk-pci,drive=disk,id=virtio0 QMP_VERSION {"return": {}} -{"return": [{"io-status": "ok", "device": "disk", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]} +{"return": [{"io-status": "ok", "device": "disk", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-width": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]} {"return": {}} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/virtio0/virtio-backend"}}
@@ -24,7 +24,7 @@ QMP_VERSION Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,if=none,id=disk QMP_VERSION {"return": {}} -{"return": [{"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]} +{"return": [{"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-width": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]} {"return": {}} {"return": {}} {"return": {}}
@@ -44,7 +44,7 @@ Testing: QMP_VERSION {"return": {}} {"return": "OK\r\n"} -{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]} +{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-width": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]} {"return": {}} {"return": {}} {"return": {}}
@@ -64,14 +64,14 @@ Testing: QMP_VERSION {"return": {}} {"return": {}} -{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]} +{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-width": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]} {"return": {}} {"return": {}} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/virtio0/virtio-backend"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "DEVICE_DELETED", "data": {"device": "virtio0", "path": "/machine/peripheral/virtio0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "RESET"} -{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"io-status": "ok", "device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]} +{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"io-status": "ok", "device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-width": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN"} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "DEVICE_TRAY_MOVED", "data": {"device": "ide1-cd0", "tray-open": true}}
tests/qemu-iotests/082.out +7 -0
@@ -21,6 +21,7 @@ cluster_size: 4096 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: create -f qcow2 -o cluster_size=4k -o lazy_refcounts=on -o cluster_size=8k TEST_DIR/t.qcow2 128M
@@ -35,6 +36,7 @@ cluster_size: 8192 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: create -f qcow2 -o cluster_size=4k,cluster_size=8k TEST_DIR/t.qcow2 128M
@@ -199,6 +201,7 @@ cluster_size: 4096 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: convert -O qcow2 -o cluster_size=4k -o lazy_refcounts=on -o cluster_size=8k TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
@@ -212,6 +215,7 @@ cluster_size: 8192 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: convert -O qcow2 -o cluster_size=4k,cluster_size=8k TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
@@ -361,6 +365,7 @@ cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: amend -f qcow2 -o size=130M -o lazy_refcounts=off TEST_DIR/t.qcow2
@@ -374,6 +379,7 @@ cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false + refcount width: 16 corrupt: false Testing: amend -f qcow2 -o size=8M -o lazy_refcounts=on -o size=132M TEST_DIR/t.qcow2
@@ -387,6 +393,7 @@ cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: amend -f qcow2 -o size=4M,size=148M TEST_DIR/t.qcow2
tests/qemu-iotests/089.out +2 -0
@@ -41,6 +41,7 @@ vm state offset: 512 MiB Format specific information: compat: 1.1 lazy refcounts: false + refcount width: 16 corrupt: false format name: IMGFMT cluster size: 64 KiB
@@ -48,5 +49,6 @@ vm state offset: 512 MiB Format specific information: compat: 1.1 lazy refcounts: false + refcount width: 16 corrupt: false *** done
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 02/21] qcow2: Add refcount_width to format-specific info
Quoted 21 lines · Max Reitz
Add the bit width of every refcount entry to the format-specific information.
This breaks some test outputs, fix them.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 4 +++- qapi/block-core.json | 5 ++++- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/065 | 23 +++++++++++++++-------- tests/qemu-iotests/067.out | 10 +++++----- tests/qemu-iotests/082.out | 7 +++++++ tests/qemu-iotests/089.out | 2 ++ 7 files changed, 37 insertions(+), 15 deletions(-)block/qcow2.c +0
-0
block/qcow2.c static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs) · line 2479
Quoted 6 lines
block/qcow2.c +2
-1@@ -2475,7 +2475,8 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs) };if (s->qcow_version ==2 ) { *spec_info->qcow2 = (ImageInfoSpecificQCow2 ){ - .compat =g_strdup ("0.10" ), + .compat =g_strdup ("0.10" ), + .refcount_width = s->refcount_bits,
Hmm - is it really worth displaying a constant? Since the 0.10 format cannot change the width from 16, I'm not sure if it adds anything to the output to display it. After all, there's other things we omit for the old format when they cannot be altered (such as the state of a lazy flag). On the other hand, if it makes your changes to later iotests easier for tests that operate on both image formats, I'm not opposed to it.
block/qcow2.c class TestQMP(TestImageInfoSpecific): · line 92
Quoted 9 lines
block/qcow2.c +3
-2+++ b/tests/qemu-iotests/065 @@ -88,34 +88,41 @@ class TestQMP(TestImageInfoSpecific): class TestQCow2(TestQemuImgInfo): '''Testing a qcow2 version 2 image''' img_options = 'compat=0.10' - json_compare = { 'compat': '0.10' } - human_compare = [ 'compat: 0.10' ] + json_compare = { 'compat': '0.10', 'refcount-width': 16 } + human_compare = [ 'compat: 0.10', 'refcount width: 16' ]
This would be a test that does not change if you decide to not output the constant.
block/qcow2.c class TestQMP(TestImageInfoSpecific): · line 93
Quoted 2 lines
block/qcow2.c +1
-1-{"return": [{"io-status": "ok", "device": "disk", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]} +{"return": [{"io-status": "ok", "device": "disk", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-width": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]}
It would be nice to figure out how to avoid long lines in the testsuite.
But that's a project for another day.
If you can make a strong argument for always outputting the constant width of 16 for 0.10 formats, then I can live with it, so:
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 02/21] qcow2: Add refcount_width to format-specific info
Quoted 26 lines · Eric Blake
Quoted 25 lines · Max Reitz
Add the bit width of every refcount entry to the format-specific information.
This breaks some test outputs, fix them.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 4 +++- qapi/block-core.json | 5 ++++- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/065 | 23 +++++++++++++++-------- tests/qemu-iotests/067.out | 10 +++++----- tests/qemu-iotests/082.out | 7 +++++++ tests/qemu-iotests/089.out | 2 ++ 7 files changed, 37 insertions(+), 15 deletions(-)block/qcow2.c +0
-1@@ -2475,7 +2475,8 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs) };if (s->qcow_version ==2 ) { *spec_info->qcow2 = (ImageInfoSpecificQCow2 ){ - .compat =g_strdup ("0.10" ),
Quoted 8 lines
block/qcow2.c static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs) · line 2479
Quoted 2 lines
block/qcow2.c +2
-0+ .compat = g_strdup("0.10"), + .refcount_width = s->refcount_bits,Hmm - is it really worth displaying a constant? Since the 0.10 format cannot change the width from 16, I'm not sure if it adds anything to the output to display it. After all, there's other things we omit for the old format when they cannot be altered (such as the state of a lazy flag). On the other hand, if it makes your changes to later iotests easier for tests that operate on both image formats, I'm not opposed to it.
Yes, I thought about not displaying it. But whereas "corrupt" or "lazy refcounts" simply do not make sense with compat=0.10 images (it's simply impossible), the refcount width does make sense. It's always 16 bits (I'm noticing myself how I keep swapping between "bit" and "bits", but I just can't help it) but I personally find it interesting enough to display. I'd be fine with dropping it from compat=0.10, though.
But in retrospect, I'd rather make the other two flags always visible than now drop this entry. However, not displaying a bool if it's always false makes more sense to me than not displaying an integer because it's always constant.
Quoted 12 lines
block/qcow2.c class TestQMP(TestImageInfoSpecific): · line 92
Quoted 9 lines
block/qcow2.c +3
-2+++ b/tests/qemu-iotests/065 @@ -88,34 +88,41 @@ class TestQMP(TestImageInfoSpecific): class TestQCow2(TestQemuImgInfo): '''Testing a qcow2 version 2 image''' img_options = 'compat=0.10' - json_compare = { 'compat': '0.10' } - human_compare = [ 'compat: 0.10' ] + json_compare = { 'compat': '0.10', 'refcount-width': 16 } + human_compare = [ 'compat: 0.10', 'refcount width: 16' ]This would be a test that does not change if you decide to not output the constant.
Quoted 4 lines
block/qcow2.c class TestQMP(TestImageInfoSpecific): · line 93
Quoted 2 lines
block/qcow2.c +1
-1-{"return": [{"io-status": "ok", "device": "disk", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]} +{"return": [{"io-status": "ok", "device": "disk", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-width": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]}It would be nice to figure out how to avoid long lines in the testsuite.
But that's a project for another day.
Yes, Kevin already proposed adding a "pretty" flag to -qmp which uses the pretty JSON formatter (which breaks lines and indents).
Quoted 2 lines
If you can make a strong argument for always outputting the constant width of 16 for 0.10 formats, then I can live with it, so:
You decide whether it's strong enough. :-)
My main argument is "If a bool is not displayed one can assume it to be false; if an integer is not displayed which naturally cannot be 0, I will have no idea what it would be, even if it's constant for that image version".
Quoted 1 line
Reviewed-by Eric Blake <[email protected]>
Thanks!
Max
Re: [Qemu-devel] [PATCH 02/21] qcow2: Add refcount_width to format-specific info
Quoted 26 lines · Max Reitz
Quoted 9 lines
block/qcow2.c static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs) · line 2479
Quoted 2 lines
block/qcow2.c +2
-0+ .compat = g_strdup("0.10"), + .refcount_width = s->refcount_bits,Hmm - is it really worth displaying a constant? Since the 0.10 format cannot change the width from 16, I'm not sure if it adds anything to the output to display it. After all, there's other things we omit for the old format when they cannot be altered (such as the state of a lazy flag). On the other hand, if it makes your changes to later iotests easier for tests that operate on both image formats, I'm not opposed to it.
Yes, I thought about not displaying it. But whereas "corrupt" or "lazy refcounts" simply do not make sense with compat=0.10 images (it's simply impossible), the refcount width does make sense. It's always 16 bits (I'm noticing myself how I keep swapping between "bit" and "bits", but I just can't help it) but I personally find it interesting enough to display. I'd be fine with dropping it from compat=0.10, though.
But in retrospect, I'd rather make the other two flags always visible than now drop this entry. However, not displaying a bool if it's always false makes more sense to me than not displaying an integer because it's always constant.
Quoted 2 lines
If you can make a strong argument for always outputting the constant width of 16 for 0.10 formats, then I can live with it, so:
Quoted 6 lines
You decide whether it's strong enough. :-)
My main argument is "If a bool is not displayed one can assume it to be false; if an integer is not displayed which naturally cannot be 0, I will have no idea what it would be, even if it's constant for that image version".
Sounds fairly convincing :) Add a paragraph like that to the commit message, and I'm sold!
Quoted 2 lines
Quoted 2 lines
Reviewed-by Eric Blake <[email protected]>
So looks like you get to keep this.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 3/21 qcow2: Use 64 bits for refcount values
In reply to Max ReitzRefcounts may have a width of up to 64 bit, so qemu should use the same width to represent refcount values internally.
--- block/qcow2-cluster.c | 9 ++++++--- block/qcow2-refcount.c | 37 ++++++++++++++++++++----------------- block/qcow2.h | 7 ++++--- 3 files changed, 30 insertions(+), 23 deletions(-)
block/qcow2-cluster.c +6 -3
@@ -1640,7 +1640,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,for (i =0 ; i < l1_size; i++) {uint64_t l2_offset = l1_table[i] &L1E_OFFSET_MASK ;bool l2_dirty = false; -int l2_refcount; +int64_t l2_refcount;if (!l2_offset) {/* unallocated */
@@ -1696,14 +1696,17 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, }if (l2_refcount >1 ) { +int64_t ret64; +/* For shared L2 tables, set the refcount accordingly (it is * already 1 and needs to be l2_refcount) */ - ret =qcow2_update_cluster_refcount (bs, + ret64 =qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits, l2_refcount -1 ,QCOW2_DISCARD_OTHER ); -if (ret <0 ) { +if (ret64 <0 ) {qcow2_free_clusters (bs, offset, s->cluster_size,QCOW2_DISCARD_OTHER ); + ret = ret64; goto fail; } }
block/qcow2-refcount.c +20 -17
@@ -91,14 +91,14 @@ static int load_refcount_block(BlockDriverState *bs,* return value is the refcount of the cluster, negative values are -errno * and indicate an error. */ -int qcow2_get_refcount (BlockDriverState *bs,int64_t cluster_index) +int64_t qcow2_get_refcount (BlockDriverState *bs,int64_t cluster_index) {BDRVQcowState *s = bs->opaque;uint64_t refcount_table_index, block_index;int64_t refcount_block_offset;int ret;uint16_t *refcount_block; -uint16_t refcount; +int64_t refcount; refcount_table_index = cluster_index >> s->refcount_block_bits;if (refcount_table_index >= s->refcount_table_size)
@@ -556,9 +556,10 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,for (cluster_offset = start; cluster_offset <= last; cluster_offset += s->cluster_size) { -int block_index, refcount; +int block_index;int64_t cluster_index = cluster_offset >> s->cluster_bits;int64_t table_index = cluster_index >> s->refcount_block_bits; +int64_t refcount;/* Load the refcount block and allocate it if needed */ if (table_index != old_table_index) {
@@ -634,10 +635,10 @@ fail:* If the return value is non-negative, it is the new refcount of the cluster. * If it is negative, it is -errno and indicates an error. */ -int qcow2_update_cluster_refcount (BlockDriverState *bs, -int64_t cluster_index, -int addend, -enum qcow2_discard_type type) +int64_t qcow2_update_cluster_refcount (BlockDriverState *bs, +int64_t cluster_index, +int addend, +enum qcow2_discard_type type) {BDRVQcowState *s = bs->opaque;int ret;
@@ -663,7 +664,7 @@ static int64_t alloc_clusters_noref(BlockDriverState *bs, uint64_t size) {BDRVQcowState *s = bs->opaque;uint64_t i, nb_clusters; -int refcount; +int64_t refcount; nb_clusters =size_to_clusters (s, size); retry:
@@ -722,7 +723,8 @@ int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,BDRVQcowState *s = bs->opaque;uint64_t cluster_index;uint64_t i; -int refcount, ret; +int64_t refcount; +int ret;assert (nb_clusters >=0 );if (nb_clusters ==0 ) {
@@ -878,8 +880,8 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,BDRVQcowState *s = bs->opaque;uint64_t *l1_table, *l2_table, l2_offset, offset, l1_size2;bool l1_allocated = false; -int64_t old_offset, old_l2_offset; -int i, j, l1_modified =0 , nb_csectors, refcount; +int64_t old_offset, old_l2_offset, refcount; +int i, j, l1_modified =0 , nb_csectors;int ret; l2_table =NULL ;
@@ -1341,7 +1343,7 @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res,BDRVQcowState *s = bs->opaque;uint64_t *l2_table =qemu_blockalign (bs, s->cluster_size);int ret; -int refcount; +int64_t refcount;int i, j;for (i =0 ; i < s->l1_size; i++) {
@@ -1360,7 +1362,7 @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res, }if ((refcount ==1 ) != ((l1_entry &QCOW_OFLAG_COPIED ) !=0 )) {fprintf (stderr,"%s OFLAG_COPIED L2 cluster: l1_index=%d " -"l1_entry=%" PRIx64" refcount=%d\n" , +"l1_entry=%" PRIx64" refcount=%" PRId64"\n" , fix &BDRV_FIX_ERRORS ?"Repairing" :"ERROR" , i, l1_entry, refcount);
@@ -1403,7 +1405,7 @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res, }if ((refcount ==1 ) != ((l2_entry &QCOW_OFLAG_COPIED ) !=0 )) {fprintf (stderr,"%s OFLAG_COPIED data cluster: " -"l2_entry=%" PRIx64" refcount=%d\n" , +"l2_entry=%" PRIx64" refcount=%" PRId64"\n" , fix &BDRV_FIX_ERRORS ?"Repairing" :"ERROR" , l2_entry, refcount);
@@ -1628,8 +1630,8 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res, uint16_t *refcount_table, int64_t nb_clusters) {BDRVQcowState *s = bs->opaque; -int64_t i; -int refcount1, refcount2, ret; +int64_t i, refcount1, refcount2; +int ret;for (i =0 , *highest_cluster =0 ; i < nb_clusters; i++) { refcount1 =qcow2_get_refcount (bs, i);
@@ -1657,7 +1659,8 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res, num_fixed = &res->corruptions_fixed; } - fprintf(stderr, "%s cluster %" PRId64 " refcount=%d reference=%d\n", + fprintf(stderr, "%s cluster %" PRId64 " refcount=%" PRId64 + " reference=%" PRId64 "\n", num_fixed != NULL ? "Repairing" : refcount1 < refcount2 ? "ERROR" : "Leaked",
block/qcow2.h +4 -3
@@ -489,10 +489,11 @@ void qcow2_signal_corruption(BlockDriverState *bs, bool fatal, int64_t offset,int qcow2_refcount_init (BlockDriverState *bs);void qcow2_refcount_close (BlockDriverState *bs); -int qcow2_get_refcount (BlockDriverState *bs,int64_t cluster_index); +int64_t qcow2_get_refcount (BlockDriverState *bs,int64_t cluster_index); -int qcow2_update_cluster_refcount (BlockDriverState *bs,int64_t cluster_index, -int addend,enum qcow2_discard_type type); +int64_t qcow2_update_cluster_refcount (BlockDriverState *bs, +int64_t cluster_index,int addend, +enum qcow2_discard_type type);int64_t qcow2_alloc_clusters (BlockDriverState *bs,uint64_t size); int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 03/21] qcow2: Use 64 bits for refcount values
Quoted 1 line · Max Reitz
Refcounts may have a width of up to 64 bit, so qemu should use the same
s/bit/bits/
Quoted 14 lines
width to represent refcount values internally.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-cluster.c | 9 ++++++--- block/qcow2-refcount.c | 37 ++++++++++++++++++++----------------- block/qcow2.h | 7 ++++--- 3 files changed, 30 insertions(+), 23 deletions(-)block/qcow2-cluster.c +0
-0
block/qcow2-cluster.c static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, · line 1643
Quoted 5 lines
block/qcow2-cluster.c +1
-1@@ -1640,7 +1640,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,for (i =0 ; i < l1_size; i++) {uint64_t l2_offset = l1_table[i] &L1E_OFFSET_MASK ;bool l2_dirty = false; -int l2_refcount; +int64_t l2_refcount;
You may want to mention in the commit message that you choose a signed type to allow negative for errors, and therefore we really allow only up to 63 useful bits. Or even mention that this is okay because no one can feasibly generate an image with more than 2^63 refs to the same cluster (there isn't that much storage or time to do such a task in our lifetime...)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 03/21] qcow2: Use 64 bits for refcount values
Quoted 3 lines · Eric Blake
Quoted 1 line · Max Reitz
Refcounts may have a width of up to 64 bit, so qemu should use the same
s/bit/bits/
See my reply to your review on patch 2. I keep swapping knowing which to use - now I know, thanks!
Quoted 9 lines
Quoted 9 lines
width to represent refcount values internally.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-cluster.c | 9 ++++++--- block/qcow2-refcount.c | 37 ++++++++++++++++++++----------------- block/qcow2.h | 7 ++++--- 3 files changed, 30 insertions(+), 23 deletions(-)
Quoted 16 lines
block/qcow2-cluster.c static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, · line 1643
Quoted 10 lines
block/qcow2-cluster.c +1
-1@@ -1640,7 +1640,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,for (i =0 ; i < l1_size; i++) {uint64_t l2_offset = l1_table[i] &L1E_OFFSET_MASK ;bool l2_dirty = false; -int l2_refcount; +int64_t l2_refcount;You may want to mention in the commit message that you choose a signed type to allow negative for errors, and therefore we really allow only up to 63 useful bits. Or even mention that this is okay because no one can feasibly generate an image with more than 2^63 refs to the same cluster (there isn't that much storage or time to do such a task in our lifetime...)
Will do.
Max
Quoted 1 line
Reviewed-by Eric Blake <[email protected]>
Re: [Qemu-devel] [PATCH 03/21] qcow2: Use 64 bits for refcount values
In reply to Eric BlakeAm 10.11.2014 um 21:59 hat Eric Blake geschrieben:
Quoted 23 lines · Max Reitz
Quoted 1 line
Refcounts may have a width of up to 64 bit, so qemu should use the same
s/bit/bits/
Quoted 18 lines
width to represent refcount values internally.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-cluster.c | 9 ++++++--- block/qcow2-refcount.c | 37 ++++++++++++++++++++----------------- block/qcow2.h | 7 ++++--- 3 files changed, 30 insertions(+), 23 deletions(-)block/qcow2-cluster.c +0
-1@@ -1640,7 +1640,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,for (i =0 ; i < l1_size; i++) {uint64_t l2_offset = l1_table[i] &L1E_OFFSET_MASK ;bool l2_dirty = false; -int l2_refcount;
Quoted 8 lines
block/qcow2-cluster.c static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, · line 1643
Quoted 1 line
block/qcow2-cluster.c +1
-0+int64_t l2_refcount;You may want to mention in the commit message that you choose a signed type to allow negative for errors, and therefore we really allow only up to 63 useful bits. Or even mention that this is okay because no one can feasibly generate an image with more than 2^63 refs to the same cluster (there isn't that much storage or time to do such a task in our lifetime...)
Should patch 1 then set refcount_max = 2^63 for refcount order 6?
Also note that while it might not be feasible to create a cluster with 2^63 references, this doesn't mean that it's impossible to create a cluster with a stored refcount of (more than) 2^63. We'll have to have checks there.
Kevin
Re: [Qemu-devel] [PATCH 03/21] qcow2: Use 64 bits for refcount values
Quoted 30 lines · Kevin Wolf
Am 10.11.2014 um 21:59 hat Eric Blake geschrieben:
Quoted 13 lines · Max Reitz
Quoted 1 line
Refcounts may have a width of up to 64 bit, so qemu should use the same
s/bit/bits/
Quoted 9 lines
width to represent refcount values internally.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-cluster.c | 9 ++++++--- block/qcow2-refcount.c | 37 ++++++++++++++++++++----------------- block/qcow2.h | 7 ++++--- 3 files changed, 30 insertions(+), 23 deletions(-)Quoted 16 lines
block/qcow2-cluster.c static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, · line 1643
Quoted 10 lines
block/qcow2-cluster.c +1
-1@@ -1640,7 +1640,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,for (i =0 ; i < l1_size; i++) {uint64_t l2_offset = l1_table[i] &L1E_OFFSET_MASK ;bool l2_dirty = false; -int l2_refcount; +int64_t l2_refcount;You may want to mention in the commit message that you choose a signed type to allow negative for errors, and therefore we really allow only up to 63 useful bits. Or even mention that this is okay because no one can feasibly generate an image with more than 2^63 refs to the same cluster (there isn't that much storage or time to do such a task in our lifetime...)
Quoted 1 line
Should patch 1 then set
refcount_max= 2^63 for refcount order 6?
It does set refcount_max to INT64_MAX (instead of UINT64_MAX, and there is a comment above that line why it's the signed maximum).
Quoted 4 lines
Also note that while it might not be feasible to create a cluster with 2^63 references, this doesn't mean that it's impossible to create a cluster with a stored refcount of (more than) 2^63. We'll have to have checks there.
Yes, the check is done in qcow2_get_refcount() (and needs to be done in update_refcount_discard() as well, which I forgot in this version) and consists of returning -ERANGE on error.
Max
Re: [Qemu-devel] [PATCH 03/21] qcow2: Use 64 bits for refcount values
Quoted 34 lines · Max Reitz
Quoted 31 lines · Kevin Wolf
Am 10.11.2014 um 21:59 hat Eric Blake geschrieben:
Quoted 22 lines · Max Reitz
Quoted 1 line
Refcounts may have a width of up to 64 bit, so qemu should use the same
s/bit/bits/
Quoted 18 lines
width to represent refcount values internally.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-cluster.c | 9 ++++++--- block/qcow2-refcount.c | 37 ++++++++++++++++++++----------------- block/qcow2.h | 7 ++++--- 3 files changed, 30 insertions(+), 23 deletions(-)block/qcow2-cluster.c +0
-0@@ -1640,7 +1640,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, for (i = 0; i < l1_size; i++) { uint64_t l2_offset = l1_table[i] & L1E_OFFSET_MASK; bool l2_dirty = false;Quoted 8 lines
block/qcow2-cluster.c static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, · line 1643
Quoted 2 lines
block/qcow2-cluster.c +1
-1-int l2_refcount; +int64_t l2_refcount;You may want to mention in the commit message that you choose a signed type to allow negative for errors, and therefore we really allow only up to 63 useful bits. Or even mention that this is okay because no one can feasibly generate an image with more than 2^63 refs to the same cluster (there isn't that much storage or time to do such a task in our lifetime...)
Quoted 1 line
Should patch 1 then set
refcount_max= 2^63 for refcount order 6?
Quoted 8 lines
It does set
refcount_maxtoINT64_MAX(instead ofUINT64_MAX, and there is a comment above that line why it's the signed maximum).Quoted 4 lines
Also note that while it might not be feasible to create a cluster with 2^63 references, this doesn't mean that it's impossible to create a cluster with a stored refcount of (more than) 2^63. We'll have to have checks there.
Yes, the check is done in
qcow2_get_refcount()(and needs to be done inupdate_refcount_discard()as well, which I forgot in this version) and consists of returning-ERANGEon error.
s/error/overflow/
Re: [Qemu-devel] [PATCH 03/21] qcow2: Use 64 bits for refcount values
In reply to Max ReitzAm 11.11.2014 um 10:25 hat Max Reitz geschrieben:
Quoted 31 lines · Kevin Wolf
Quoted 30 lines
Am 10.11.2014 um 21:59 hat Eric Blake geschrieben:
Quoted 29 lines · Max Reitz
Quoted 1 line
Refcounts may have a width of up to 64 bit, so qemu should use the same
s/bit/bits/
Quoted 14 lines
width to represent refcount values internally.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-cluster.c | 9 ++++++--- block/qcow2-refcount.c | 37 ++++++++++++++++++++----------------- block/qcow2.h | 7 ++++--- 3 files changed, 30 insertions(+), 23 deletions(-)block/qcow2-cluster.c +0
-0block/qcow2-cluster.c static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, · line 1643
Quoted 5 lines
block/qcow2-cluster.c +1
-1@@ -1640,7 +1640,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,for (i =0 ; i < l1_size; i++) {uint64_t l2_offset = l1_table[i] &L1E_OFFSET_MASK ;bool l2_dirty = false; -int l2_refcount; +int64_t l2_refcount;You may want to mention in the commit message that you choose a signed type to allow negative for errors, and therefore we really allow only up to 63 useful bits. Or even mention that this is okay because no one can feasibly generate an image with more than 2^63 refs to the same cluster (there isn't that much storage or time to do such a task in our lifetime...)
Quoted 4 lines
Quoted 1 line
Should patch 1 then set
refcount_max= 2^63 for refcount order 6?It does set
refcount_maxtoINT64_MAX(instead ofUINT64_MAX, and there is a comment above that line why it's the signed maximum).
Right, I should read patches instead of just Eric's reply before I reply something myself...
Kevin
PATCH 4/21 qcow2: Respect error in qcow2_alloc_bytes()
In reply to Max Reitzqcow2_update_cluster_refcount() may fail, and qcow2_alloc_bytes() should mind that case.
--- block/qcow2-refcount.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-)
block/qcow2-refcount.c +21 -11
@@ -761,7 +761,8 @@ int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,int64_t qcow2_alloc_bytes (BlockDriverState *bs,int size) {BDRVQcowState *s = bs->opaque; -int64_t offset, cluster_offset; +int64_t offset, cluster_offset, new_cluster; +int64_t ret;int free_in_cluster;BLKDBG_EVENT (bs->file,BLKDBG_CLUSTER_ALLOC_BYTES );
@@ -783,23 +784,32 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) free_in_cluster -= size;if (free_in_cluster ==0 ) s->free_byte_offset =0 ; -if (offset_into_cluster (s, offset) !=0 ) -qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits,1 , -QCOW2_DISCARD_NEVER ); +if (offset_into_cluster (s, offset) !=0 ) { + ret =qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits, +1 ,QCOW2_DISCARD_NEVER ); +if (ret <0 ) { +return ret; + } + } }else { - offset =qcow2_alloc_clusters (bs, s->cluster_size); -if (offset <0 ) { -return offset; + new_cluster =qcow2_alloc_clusters (bs, s->cluster_size); +if (new_cluster <0 ) { +return new_cluster; } cluster_offset =start_of_cluster (s, s->free_byte_offset); -if ((cluster_offset + s->cluster_size) == offset) { +if ((cluster_offset + s->cluster_size) == new_cluster) {/* we are lucky: contiguous data */ offset = s->free_byte_offset; -qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits,1 , -QCOW2_DISCARD_NEVER ); + ret =qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits, +1 ,QCOW2_DISCARD_NEVER ); +if (ret <0 ) { +qcow2_free_clusters (bs, new_cluster, s->cluster_size, +QCOW2_DISCARD_NEVER ); +return ret; + } s->free_byte_offset += size; }else { - s->free_byte_offset = offset; + s->free_byte_offset = new_cluster; goto redo; } }
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 04/21] qcow2: Respect error in qcow2_alloc_bytes()
Quoted 7 lines · Max Reitz
qcow2_update_cluster_refcount()may fail, andqcow2_alloc_bytes()should mind that case.Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 5/21 qcow2: Refcount overflow and qcow2_alloc_bytes()
In reply to Max Reitzqcow2_alloc_bytes() may reuse a cluster multiple times, in which case the refcount is increased accordingly. However, if this would lead to an overflow the function should instead just not reuse this cluster and allocate a new one.
--- block/qcow2-refcount.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
block/qcow2-refcount.c +30 -2
@@ -761,12 +761,13 @@ int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,int64_t qcow2_alloc_bytes (BlockDriverState *bs,int size) {BDRVQcowState *s = bs->opaque; -int64_t offset, cluster_offset, new_cluster; +int64_t offset, cluster_offset, new_cluster, refcount;int64_t ret;int free_in_cluster;BLKDBG_EVENT (bs->file,BLKDBG_CLUSTER_ALLOC_BYTES );assert (size >0 && size <= s->cluster_size); + redo:if (s->free_byte_offset ==0 ) { offset =qcow2_alloc_clusters (bs, s->cluster_size);if (offset <0 ) {
@@ -774,12 +775,25 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) } s->free_byte_offset = offset; } - redo: + free_in_cluster = s->cluster_size -offset_into_cluster (s, s->free_byte_offset);if (size <= free_in_cluster) {/* enough space in current cluster */ offset = s->free_byte_offset; + +if (offset_into_cluster (s, offset) !=0 ) { +/* We will have to increase the refcount of this cluster; if the +* maximum has been reached already, this cluster cannot be used */ + refcount =qcow2_get_refcount (bs, offset >> s->cluster_bits); +if (refcount <0 ) { +return refcount; + }else if (refcount == s->refcount_max) { + s->free_byte_offset =0 ; + goto redo; + } + } + s->free_byte_offset += size; free_in_cluster -= size;if (free_in_cluster ==0 )
@@ -800,6 +814,20 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)if ((cluster_offset + s->cluster_size) == new_cluster) {/* we are lucky: contiguous data */ offset = s->free_byte_offset; + +/* Same as above: In order to reuse the cluster, the refcount has to +* be increased; if that will not work, we are not so lucky after +* all */ + refcount =qcow2_get_refcount (bs, offset >> s->cluster_bits); +if (refcount <0 ) { +qcow2_free_clusters (bs, new_cluster, s->cluster_size, +QCOW2_DISCARD_NEVER ); +return refcount; + }else if (refcount == s->refcount_max) { + s->free_byte_offset = offset; + goto redo; + } + ret =qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits,1 ,QCOW2_DISCARD_NEVER );if (ret <0 ) {
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 05/21] qcow2: Refcount overflow and qcow2_alloc_bytes()
Quoted 4 lines · Max Reitz
qcow2_alloc_bytes()may reuse a cluster multiple times, in which case the refcount is increased accordingly. However, if this would lead to an overflow the function should instead just not reuse this cluster and allocate a new one.
So if recount_order is 1 (2 bits per refcount, max refcount of 4), and we encounter the same cluster 6 times (say by 5 back-to-back internal snapshots), does this code optimize to only 2 clusters (both with refcount 3) or does it result in each of the last 3 clusters spilling to its own 1-ref cluster for a total of 4 clusters? Short of Benoit's work on deduplication, is there even a way to avoid inefficient use of spilled clusters? But I guess answering that can be a separate patch; inefficiency is annoying, but not technically wrong and therefore not a reason to reject this one.
Quoted 6 lines
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 05/21] qcow2: Refcount overflow and qcow2_alloc_bytes()
Quoted 6 lines · Eric Blake
Quoted 4 lines · Max Reitz
qcow2_alloc_bytes()may reuse a cluster multiple times, in which case the refcount is increased accordingly. However, if this would lead to an overflow the function should instead just not reuse this cluster and allocate a new one.So if
recount_orderis 1 (2 bits per refcount, max refcount of 4
*max refcount of 3 (0b11)
Quoted 7 lines
), and
we encounter the same cluster 6 times (say by 5 back-to-back internal snapshots), does this code optimize to only 2 clusters (both with refcount 3) or does it result in each of the last 3 clusters spilling to its own 1-ref cluster for a total of 4 clusters? Short of Benoit's work on deduplication, is there even a way to avoid inefficient use of spilled clusters?
I'm not sure what you're referring to; maybe I should add that qcow2_alloc_bytes() is used for allocating compressed clusters (which ideally don't take up a full host cluster), so "reuse" in this context just means that several compressed clusters share one host cluster.
Maybe you're referring to the following situation: We have the default cluster size of 64k. Now we're trying to allocate 16k for each of the compressed clusters A, B, C and D. D won't fit into that cluster because the maximum refcount is three, so it will be put into a newly allocated host cluster. Finally, we're trying to allocate 32k for a compressed cluster E, which will then be put into the same cluster as D. We therefore have the following allocation (each sub-box representing 16k):
+---+---+---+---+ +---+---+---+---+ |A |B | C | | | D | E | | +---+---+---+---+ +---+---+---+---+
whereas the ideal allocation would be:
+---+---+---+---+ +---+---+---+---+ |A |B | E | | C | D | | | +---+---+---+---+ +---+---+---+---+
This is a problem, but I think first it's a minor one (just use a sufficiently large refcount width if you're going to use compressed clusters) and second it's about compressed clusters, whose performance I could hardly care less about, frankly.
Max
Quoted 11 lines
But I guess answering that can be a separate patch; inefficiency is annoying, but not technically wrong and therefore not a reason to reject this one.
Quoted 5 lines
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)Reviewed-by Eric Blake <[email protected]>
Re: [Qemu-devel] [PATCH 05/21] qcow2: Refcount overflow and qcow2_alloc_bytes()
Quoted 9 lines · Max Reitz
Quoted 6 lines · Eric Blake
Quoted 4 lines · Max Reitz
qcow2_alloc_bytes()may reuse a cluster multiple times, in which case the refcount is increased accordingly. However, if this would lead to an overflow the function should instead just not reuse this cluster and allocate a new one.So if
recount_orderis 1 (2 bits per refcount, max refcount of 4*max refcount of 3 (0b11)
Oh right, because 0 is special. Although I think I figured that out...
Quoted 5 lines
Quoted 5 lines
), and
we encounter the same cluster 6 times (say by 5 back-to-back internal snapshots), does this code optimize to only 2 clusters (both with refcount 3) or does it result in each of the last 3 clusters spilling to
...when talking about 3 shares of a cluster.
Quoted 8 lines
Quoted 3 lines
its own 1-ref cluster for a total of 4 clusters? Short of Benoit's work on deduplication, is there even a way to avoid inefficient use of spilled clusters?
I'm not sure what you're referring to; maybe I should add that
qcow2_alloc_bytes()is used for allocating compressed clusters (which ideally don't take up a full host cluster), so "reuse" in this context just means that several compressed clusters share one host cluster.
No, I was thinking about internal snapshots rather than compressed clusters (although there's probably some overlap on what happens).
Quoted 15 lines
Maybe you're referring to the following situation: We have the default cluster size of 64k. Now we're trying to allocate 16k for each of the compressed clusters A, B, C and D. D won't fit into that cluster because the maximum refcount is three, so it will be put into a newly allocated host cluster. Finally, we're trying to allocate 32k for a compressed cluster E, which will then be put into the same cluster as D. We therefore have the following allocation (each sub-box representing 16k):
+---+---+---+---+ +---+---+---+---+ |A |B | C | | | D | E | | +---+---+---+---+ +---+---+---+---+whereas the ideal allocation would be:
Quoted 8 lines
+---+---+---+---+ +---+---+---+---+ |A |B | E | | C | D | | | +---+---+---+---+ +---+---+---+---+This is a problem, but I think first it's a minor one (just use a sufficiently large refcount width if you're going to use compressed clusters) and second it's about compressed clusters, whose performance I could hardly care less about, frankly.
No, I was envisioning that we have a brand new image with one cluster allocated (cluster 1 has refcount 1), then 5 times in a row we do 'savevm' to take an internal snapshot. If I understand your code correctly, the first two snapshots increase the refcount, so cluster 1 has a refcount of 3. Then the next snapshot can't increase the refcount, so it instead copies the contents to cluster 2. The fourth and fifth snapshots also see that cluster 1 is full, and allocate cluster 3 and 4; whereas a more efficient usage would increase the refcount of cluster 2 instead of allocating.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 05/21] qcow2: Refcount overflow and qcow2_alloc_bytes()
Quoted 49 lines · Eric Blake
Quoted 8 lines · Max Reitz
Quoted 6 lines · Eric Blake
Quoted 4 lines · Max Reitz
qcow2_alloc_bytes()may reuse a cluster multiple times, in which case the refcount is increased accordingly. However, if this would lead to an overflow the function should instead just not reuse this cluster and allocate a new one.So if
recount_orderis 1 (2 bits per refcount, max refcount of 4*max refcount of 3 (0b11)
Oh right, because 0 is special. Although I think I figured that out...
Quoted 4 lines
Quoted 4 lines
), and
we encounter the same cluster 6 times (say by 5 back-to-back internal snapshots), does this code optimize to only 2 clusters (both with refcount 3) or does it result in each of the last 3 clusters spilling to...when talking about 3 shares of a cluster.
Quoted 7 lines
Quoted 3 lines
its own 1-ref cluster for a total of 4 clusters? Short of Benoit's work on deduplication, is there even a way to avoid inefficient use of spilled clusters?
I'm not sure what you're referring to; maybe I should add that
qcow2_alloc_bytes()is used for allocating compressed clusters (which ideally don't take up a full host cluster), so "reuse" in this context just means that several compressed clusters share one host cluster.No, I was thinking about internal snapshots rather than compressed clusters (although there's probably some overlap on what happens).
Quoted 14 lines
Maybe you're referring to the following situation: We have the default cluster size of 64k. Now we're trying to allocate 16k for each of the compressed clusters A, B, C and D. D won't fit into that cluster because the maximum refcount is three, so it will be put into a newly allocated host cluster. Finally, we're trying to allocate 32k for a compressed cluster E, which will then be put into the same cluster as D. We therefore have the following allocation (each sub-box representing 16k):
+---+---+---+---+ +---+---+---+---+ |A |B | C | | | D | E | | +---+---+---+---+ +---+---+---+---+whereas the ideal allocation would be:
Quoted 8 lines
+---+---+---+---+ +---+---+---+---+ |A |B | E | | C | D | | | +---+---+---+---+ +---+---+---+---+This is a problem, but I think first it's a minor one (just use a sufficiently large refcount width if you're going to use compressed clusters) and second it's about compressed clusters, whose performance I could hardly care less about, frankly.
Quoted 6 lines
No, I was envisioning that we have a brand new image with one cluster allocated (cluster 1 has refcount 1), then 5 times in a row we do 'savevm' to take an internal snapshot. If I understand your code correctly, the first two snapshots increase the refcount, so cluster 1 has a refcount of 3. Then the next snapshot can't increase the refcount, so it instead copies the contents to cluster 2.
No, it just errors out.
qcow2_alloc_bytes() is only used for allocating space for a compressed cluster. When taking a snapshot, update_refcount() will be called to increase the clusters' refcounts, and that function will simply throw an error.
Max
Quoted 5 lines
The fourth and fifth
snapshots also see that cluster 1 is full, and allocate cluster 3 and 4; whereas a more efficient usage would increase the refcount of cluster 2 instead of allocating.
Re: [Qemu-devel] [PATCH 05/21] qcow2: Refcount overflow and qcow2_alloc_bytes()
Quoted 7 lines · Max Reitz
Quoted 6 lines
No, I was envisioning that we have a brand new image with one cluster allocated (cluster 1 has refcount 1), then 5 times in a row we do 'savevm' to take an internal snapshot. If I understand your code correctly, the first two snapshots increase the refcount, so cluster 1 has a refcount of 3. Then the next snapshot can't increase the refcount, so it instead copies the contents to cluster 2.
Quoted 6 lines
No, it just errors out.
qcow2_alloc_bytes()is only used for allocating space for a compressed cluster. When taking a snapshot,update_refcount()will be called to increase the clusters' refcounts, and that function will simply throw an error.
That's okay for now (always better for an initial feature to be conservative, then expand it later if there is demand). But I wonder if we could be made smarter in the future and auto-COW any cluster that would otherwise exceed max refcount. Thus, for a refcount_order=0 (width=1) image, a snapshot now doubles the size of the image (as every single cluster would COW into a new cluster) rather than erroring out. Food for thought; maybe worth injecting comments into this series (whether in code or in commit messages, as appropriate) pointing out that we thought about the future possibility even though we chose not to allow it for now.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 05/21] qcow2: Refcount overflow and qcow2_alloc_bytes()
Quoted 14 lines · Eric Blake
Quoted 6 lines · Max Reitz
Quoted 6 lines
No, I was envisioning that we have a brand new image with one cluster allocated (cluster 1 has refcount 1), then 5 times in a row we do 'savevm' to take an internal snapshot. If I understand your code correctly, the first two snapshots increase the refcount, so cluster 1 has a refcount of 3. Then the next snapshot can't increase the refcount, so it instead copies the contents to cluster 2.
Quoted 6 lines
No, it just errors out.
qcow2_alloc_bytes()is only used for allocating space for a compressed cluster. When taking a snapshot,update_refcount()will be called to increase the clusters' refcounts, and that function will simply throw an error.
Quoted 10 lines
That's okay for now (always better for an initial feature to be conservative, then expand it later if there is demand). But I wonder if we could be made smarter in the future and auto-COW any cluster that would otherwise exceed max refcount. Thus, for a refcount_order=0 (width=1) image, a snapshot now doubles the size of the image (as every single cluster would COW into a new cluster) rather than erroring out. Food for thought; maybe worth injecting comments into this series (whether in code or in commit messages, as appropriate) pointing out that we thought about the future possibility even though we chose not to allow it for now.
Ah, right, thank you. Yes, that sounds like a good idea, I'll see to it at some later point in time.
I think adding comments will be hard because the snapshot functions aren't really modified. They just try to increase the refcount and that may now fail earlier than it did for a refcount width of 16 bits, so there's no real change in behavior there, it's just that it's now reasonably possible to hit that case. I will add appropriate comments to the test case (which tests this snapshotting issue), though.
Max
PATCH 6/21 qcow2: Helper function for refcount modification
In reply to Max ReitzSince refcounts do not always have to be a uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.
At the same time, replace all sizeof(**refcount_table) etc. in the qcow2 check code by s->refcount_bits / 8. Note that this might lead to wrong values due to truncating division, but currently s->refcount_bits is always 16, and before the upcoming patch which removes this limitation another patch will make the division round up correctly.
--- block/qcow2-refcount.c | 152 +++++++++++++++++++++++++++++-------------------- block/qcow2.h | 8 +++ 2 files changed, 98 insertions(+), 62 deletions(-)
block/qcow2-refcount.c +90 -62
@@ -32,6 +32,11 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs, int64_t offset, int64_t length, int addend, enum qcow2_discard_type type); +static uint64_t get_refcount_ro4 (const void *refcount_array,uint64_t index); + +static void set_refcount_ro4 (void *refcount_array,uint64_t index, +uint64_t value); +/*********************************************************/ /* refcount handling */
@@ -42,6 +47,9 @@ int qcow2_refcount_init(BlockDriverState *bs)unsigned int refcount_table_size2, i;int ret; + s->get_refcount = &get_refcount_ro4; + s->set_refcount = &set_refcount_ro4; +assert (s->refcount_table_size <=INT_MAX /sizeof (uint64_t )); refcount_table_size2 = s->refcount_table_size *sizeof (uint64_t ); s->refcount_table =g_try_malloc (refcount_table_size2);
@@ -72,6 +80,19 @@ void qcow2_refcount_close(BlockDriverState *bs) } +static uint64_t get_refcount_ro4 (const void *refcount_array,uint64_t index) +{ +return be16_to_cpu (((const uint16_t *)refcount_array)[index]); +} + +static void set_refcount_ro4 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>16 )); + ((uint16_t *)refcount_array)[index] =cpu_to_be16 (value); +} + +static int load_refcount_block (BlockDriverState *bs,int64_t refcount_block_offset,void **refcount_block)
@@ -97,7 +118,7 @@ int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index)uint64_t refcount_table_index, block_index;int64_t refcount_block_offset;int ret; -uint16_t *refcount_block; +void *refcount_block;int64_t refcount; refcount_table_index = cluster_index >> s->refcount_block_bits;
@@ -116,20 +137,24 @@ int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index) } ret =qcow2_cache_get (bs, s->refcount_block_cache, refcount_block_offset, - (void **) &refcount_block); + &refcount_block);if (ret <0 ) {return ret; } block_index = cluster_index & (s->refcount_block_size -1 ); - refcount =be16_to_cpu (refcount_block[block_index]); + refcount = s->get_refcount (refcount_block, block_index); - ret =qcow2_cache_put (bs, s->refcount_block_cache, - (void **) &refcount_block); + ret =qcow2_cache_put (bs, s->refcount_block_cache, &refcount_block);if (ret <0 ) {return ret; } +if (refcount <0 ) { +/* overflow */ +return -ERANGE ; + } +return refcount; }
@@ -169,7 +194,7 @@ static int in_same_refcount_block(BDRVQcowState *s, uint64_t offset_a,* Returns 0 on success or -errno in error case */ static int alloc_refcount_block (BlockDriverState *bs, -int64_t cluster_index,uint16_t **refcount_block) +int64_t cluster_index,void **refcount_block) {BDRVQcowState *s = bs->opaque;unsigned int refcount_table_index;
@@ -196,7 +221,7 @@ static int alloc_refcount_block(BlockDriverState *bs, }return load_refcount_block (bs, refcount_block_offset, - (void **) refcount_block); + refcount_block); } }
@@ -256,7 +281,7 @@ static int alloc_refcount_block(BlockDriverState *bs,/* The block describes itself, need to update the cache */ int block_index = (new_block >> s->cluster_bits) & (s->refcount_block_size -1 ); - (*refcount_block)[block_index] =cpu_to_be16 (1 ); + s->set_refcount (*refcount_block, block_index,1 ); }else {/* Described somewhere else. This can recurse at most twice before we * arrive at a block that describes itself. */
@@ -274,7 +299,7 @@ static int alloc_refcount_block(BlockDriverState *bs,/* Initialize the new refcount block only after updating its refcount, * update_refcount uses the refcount cache itself */ ret =qcow2_cache_get_empty (bs, s->refcount_block_cache, new_block, - (void **) refcount_block); + refcount_block);if (ret <0 ) { goto fail_block; }
@@ -308,7 +333,7 @@ static int alloc_refcount_block(BlockDriverState *bs,return -EAGAIN ; } - ret =qcow2_cache_put (bs, s->refcount_block_cache, (void **) refcount_block); + ret =qcow2_cache_put (bs, s->refcount_block_cache, refcount_block);if (ret <0 ) { goto fail_block; }
@@ -362,7 +387,7 @@ static int alloc_refcount_block(BlockDriverState *bs, s->cluster_size;uint64_t table_offset = meta_offset + blocks_clusters * s->cluster_size;uint64_t *new_table =g_try_new0 (uint64_t, table_size); -uint16_t *new_blocks =g_try_malloc0 (blocks_clusters * s->cluster_size); +void *new_blocks =g_try_malloc0 (blocks_clusters * s->cluster_size);assert (table_size >0 && blocks_clusters >0 );if (new_table ==NULL || new_blocks ==NULL ) {
@@ -384,7 +409,7 @@ static int alloc_refcount_block(BlockDriverState *bs,uint64_t table_clusters =size_to_clusters (s, table_size *sizeof (uint64_t ));int block =0 ;for (i =0 ; i < table_clusters + blocks_clusters; i++) { - new_blocks[block++] =cpu_to_be16 (1 ); + s->set_refcount (new_blocks, block++,1 ); }/* Write refcount blocks to disk */
@@ -437,7 +462,7 @@ static int alloc_refcount_block(BlockDriverState *bs,qcow2_free_clusters (bs, old_table_offset, old_table_size *sizeof (uint64_t ),QCOW2_DISCARD_OTHER ); - ret =load_refcount_block (bs, new_block, (void **) refcount_block); + ret =load_refcount_block (bs, new_block, refcount_block);if (ret <0 ) {return ret; }
@@ -452,7 +477,7 @@ fail_table:g_free (new_table); fail_block:if (*refcount_block !=NULL ) { -qcow2_cache_put (bs, s->refcount_block_cache, (void **) refcount_block); +qcow2_cache_put (bs, s->refcount_block_cache, refcount_block); }return ret; }
@@ -532,7 +557,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs, {BDRVQcowState *s = bs->opaque;int64_t start, last, cluster_offset; -uint16_t *refcount_block =NULL ; +void *refcount_block =NULL ;int64_t old_table_index =-1 ;int ret;
@@ -583,7 +608,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,/* we can update the count and save it */ block_index = cluster_index & (s->refcount_block_size -1 ); - refcount =be16_to_cpu (refcount_block[block_index]); + refcount = s->get_refcount (refcount_block, block_index); refcount += addend;if (refcount <0 || refcount > s->refcount_max) { ret = -EINVAL ;
@@ -592,7 +617,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,if (refcount ==0 && cluster_index < s->free_cluster_index) { s->free_cluster_index = cluster_index; } - refcount_block[block_index] =cpu_to_be16 (refcount); + s->set_refcount (refcount_block, block_index, refcount);if (refcount ==0 && s->discard_passthrough[type]) {update_refcount_discard (bs, cluster_offset, s->cluster_size);
@@ -608,8 +633,7 @@ fail:/* Write last changed block to disk */ if (refcount_block) {int wret; - wret =qcow2_cache_put (bs, s->refcount_block_cache, - (void **) &refcount_block); + wret =qcow2_cache_put (bs, s->refcount_block_cache, &refcount_block);if (wret <0 ) {return ret <0 ? ret : wret; }
@@ -1118,12 +1142,13 @@ fail:*/ static int inc_refcounts (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table, +void **refcount_table,int64_t *refcount_table_size,int64_t offset,int64_t size) {BDRVQcowState *s = bs->opaque; -uint64_t start, last, cluster_offset, k; +uint64_t start, last, cluster_offset, k, refcount; +int64_t i;if (size <=0 ) {return 0 ;
@@ -1136,12 +1161,12 @@ static int inc_refcounts(BlockDriverState *bs, k = cluster_offset >> s->cluster_bits;if (k >= *refcount_table_size) {int64_t old_refcount_table_size = *refcount_table_size; -uint16_t *new_refcount_table; +void *new_refcount_table; *refcount_table_size = k +1 ; new_refcount_table =g_try_realloc (*refcount_table, *refcount_table_size * -sizeof (**refcount_table)); + s->refcount_bits /8 );if (!new_refcount_table) { *refcount_table_size = old_refcount_table_size; res->check_errors++;
@@ -1149,16 +1174,19 @@ static int inc_refcounts(BlockDriverState *bs, } *refcount_table = new_refcount_table; -memset (*refcount_table + old_refcount_table_size,0 , - (*refcount_table_size - old_refcount_table_size) * -sizeof (**refcount_table)); +for (i = old_refcount_table_size; i < *refcount_table_size; i++) { + s->set_refcount (*refcount_table, i,0 ); + } } -if (++(*refcount_table)[k] ==0 ) { + refcount = s->get_refcount (*refcount_table, k); +if (refcount == s->refcount_max) {fprintf (stderr,"ERROR: overflow cluster offset=0x%" PRIx64"\n" , cluster_offset); res->corruptions++; +continue ; } + s->set_refcount (*refcount_table, k, refcount +1 ); }return 0 ;
@@ -1178,7 +1206,7 @@ enum {* error occurred. */ static int check_refcounts_l2 (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table,int64_t *refcount_table_size,int64_t l2_offset, +void **refcount_table,int64_t *refcount_table_size,int64_t l2_offset,int flags) {BDRVQcowState *s = bs->opaque;
@@ -1296,7 +1324,7 @@ fail:*/ static int check_refcounts_l1 (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table, +void **refcount_table,int64_t *refcount_table_size,int64_t l1_table_offset,int l1_size,int flags)
@@ -1493,10 +1521,10 @@ fail:*/ static int check_refblocks (BlockDriverState *bs,BdrvCheckResult *res,BdrvCheckMode fix,bool *rebuild, -uint16_t **refcount_table,int64_t *nb_clusters) +void **refcount_table,int64_t *nb_clusters) {BDRVQcowState *s = bs->opaque; -int64_t i, size; +int64_t i, j, size;int ret;for (i =0 ; i < s->refcount_table_size; i++) {
@@ -1519,7 +1547,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,if (fix &BDRV_FIX_ERRORS ) {int64_t old_nb_clusters = *nb_clusters; -uint16_t *new_refcount_table; +void *new_refcount_table;if (offset >INT64_MAX - s->cluster_size) { ret = -EINVAL ;
@@ -1541,7 +1569,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, new_refcount_table =g_try_realloc (*refcount_table, *nb_clusters * -sizeof (**refcount_table)); + s->refcount_bits /8 );if (!new_refcount_table) { *nb_clusters = old_nb_clusters; res->check_errors++;
@@ -1549,9 +1577,9 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, } *refcount_table = new_refcount_table; -memset (*refcount_table + old_nb_clusters,0 , - (*nb_clusters - old_nb_clusters) * -sizeof (**refcount_table)); +for (j = old_nb_clusters; j < *nb_clusters; j++) { + s->set_refcount (*refcount_table, j,0 ); + }if (cluster >= *nb_clusters) { ret = -EINVAL ;
@@ -1586,9 +1614,10 @@ resize_fail:if (ret <0 ) {return ret; } -if ((*refcount_table)[cluster] !=1 ) { +if (s->get_refcount (*refcount_table, cluster) !=1 ) {fprintf (stderr,"ERROR refcount block %" PRId64 -" refcount=%d\n" , i, (*refcount_table)[cluster]); +" refcount=%" PRIu64"\n" , i, + s->get_refcount (*refcount_table, cluster)); res->corruptions++; *rebuild = true; }
@@ -1603,7 +1632,7 @@ resize_fail:*/ static int calculate_refcounts (BlockDriverState *bs,BdrvCheckResult *res,BdrvCheckMode fix,bool *rebuild, -uint16_t **refcount_table,int64_t *nb_clusters) +void **refcount_table,int64_t *nb_clusters) {BDRVQcowState *s = bs->opaque;int64_t i;
@@ -1611,7 +1640,7 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,int ret;if (!*refcount_table) { - *refcount_table =g_try_new0 (uint16_t, *nb_clusters); + *refcount_table =g_try_malloc0 (*nb_clusters * s->refcount_bits /8 );if (*nb_clusters && *refcount_table ==NULL ) { res->check_errors++;return -ENOMEM ;
@@ -1665,7 +1694,7 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,static void compare_refcounts (BlockDriverState *bs,BdrvCheckResult *res,BdrvCheckMode fix,bool *rebuild,int64_t *highest_cluster, -uint16_t *refcount_table,int64_t nb_clusters) +void *refcount_table,int64_t nb_clusters) {BDRVQcowState *s = bs->opaque;int64_t i, refcount1, refcount2;
@@ -1680,7 +1709,7 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,continue ; } - refcount2 = refcount_table[i]; + refcount2 = s->get_refcount (refcount_table, i);if (refcount1 >0 || refcount2 >0 ) { *highest_cluster = i;
@@ -1738,7 +1767,7 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,*/ static int64_t alloc_clusters_imrt (BlockDriverState *bs,int cluster_count, -uint16_t **refcount_table, +void **refcount_table,int64_t *imrt_nb_clusters,int64_t *first_free_cluster) {
@@ -1754,7 +1783,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs, contiguous_free_clusters < cluster_count; cluster++) { -if (!(*refcount_table)[cluster]) { +if (!s->get_refcount (*refcount_table, cluster)) { contiguous_free_clusters++;if (first_gap) {/* If this is the first free cluster found, update
@@ -1776,7 +1805,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,* accordingly to append free clusters at the end of the image */ if (contiguous_free_clusters < cluster_count) {int64_t old_imrt_nb_clusters = *imrt_nb_clusters; -uint16_t *new_refcount_table; +void *new_refcount_table;/* contiguous_free_clusters clusters are already empty at the image end; * we need cluster_count clusters; therefore, we have to allocate
@@ -1787,22 +1816,22 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs, *imrt_nb_clusters = cluster + cluster_count - contiguous_free_clusters; new_refcount_table =g_try_realloc (*refcount_table, *imrt_nb_clusters * -sizeof (**refcount_table)); + s->refcount_bits /8 );if (!new_refcount_table) { *imrt_nb_clusters = old_imrt_nb_clusters;return -ENOMEM ; } *refcount_table = new_refcount_table; -memset (*refcount_table + old_imrt_nb_clusters,0 , - (*imrt_nb_clusters - old_imrt_nb_clusters) * -sizeof (**refcount_table)); +for (i = old_imrt_nb_clusters; i < *imrt_nb_clusters; i++) { + s->set_refcount (*refcount_table, i,0 ); + } }/* Go back to the first free cluster */ cluster -= contiguous_free_clusters;for (i =0 ; i < cluster_count; i++) { - (*refcount_table)[cluster + i] =1 ; + s->set_refcount (*refcount_table, cluster + i,1 ); }return cluster << s->cluster_bits;
@@ -1818,7 +1847,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,*/ static int rebuild_refcount_structure (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table, +void **refcount_table,int64_t *nb_clusters) {BDRVQcowState *s = bs->opaque;
@@ -1826,8 +1855,8 @@ static int rebuild_refcount_structure(BlockDriverState *bs,int64_t refblock_offset, refblock_start, refblock_index;uint32_t reftable_size =0 ;uint64_t *on_disk_reftable =NULL ; -uint16_t *on_disk_refblock; -int i, ret =0 ; +void *on_disk_refblock; +int ret =0 ;struct {uint64_t reftable_offset;uint32_t reftable_clusters;
@@ -1837,7 +1866,7 @@ static int rebuild_refcount_structure(BlockDriverState *bs, write_refblocks:for (; cluster < *nb_clusters; cluster++) { -if (!(*refcount_table)[cluster]) { +if (!s->get_refcount (*refcount_table, cluster)) {continue ; }
@@ -1911,12 +1940,11 @@ write_refblocks: } on_disk_refblock =qemu_blockalign0 (bs->file, s->cluster_size); -for (i =0 ; i < s->refcount_block_size && - refblock_start + i < *nb_clusters; i++) - { - on_disk_refblock[i] = -cpu_to_be16 ((*refcount_table)[refblock_start + i]); - } + +memcpy (on_disk_refblock, (void *)((uintptr_t )*refcount_table + + (refblock_index << s->refcount_block_bits)), +MIN (s->refcount_block_size, *nb_clusters - refblock_start) + * s->refcount_bits /8 ); ret =bdrv_write (bs->file, refblock_offset /BDRV_SECTOR_SIZE , (void *)on_disk_refblock, s->cluster_sectors);
@@ -2015,7 +2043,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,BDRVQcowState *s = bs->opaque;BdrvCheckResult pre_compare_res;int64_t size, highest_cluster, nb_clusters; -uint16_t *refcount_table =NULL ; +void *refcount_table =NULL ;bool rebuild = false;int ret;
@@ -2064,7 +2092,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,/* Because the old reftable has been exchanged for a new one the * references have to be recalculated */ rebuild = false; -memset (refcount_table,0 , nb_clusters *sizeof (uint16_t )); +memset (refcount_table,0 , nb_clusters * s->refcount_bits /8 ); ret =calculate_refcounts (bs, res,0 , &rebuild, &refcount_table, &nb_clusters);if (ret <0 ) {
block/qcow2.h +8 -0
@@ -213,6 +213,11 @@ typedef struct Qcow2DiscardRegion {QTAILQ_ENTRY (Qcow2DiscardRegion) next; } Qcow2DiscardRegion; +typedef uint64_t Qcow2GetRefcountFunc (const void *refcount_array, +uint64_t index); +typedef void Qcow2SetRefcountFunc (void *refcount_array, +uint64_t index,uint64_t value); + typedef struct BDRVQcowState { int cluster_bits; int cluster_size;
@@ -261,6 +266,9 @@ typedef struct BDRVQcowState {int refcount_bits;uint64_t refcount_max; +Qcow2GetRefcountFunc *get_refcount; +Qcow2SetRefcountFunc *set_refcount; +bool discard_passthrough[QCOW2_DISCARD_MAX ];int overlap_check;/* bitmask of Qcow2MetadataOverlap values */
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 06/21] qcow2: Helper function for refcount modification
Quoted 7 lines · Max Reitz
Since refcounts do not always have to be a
uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.
Quoted 5 lines
At the same time, replace all
sizeof(**refcount_table)etc. in the qcow2 check code bys->refcount_bits/ 8. Note that this might lead to wrong values due to truncating division, but currentlys->refcount_bitsis always 16, and before the upcoming patch which removes this limitation another patch will make the division round up correctly.
Thanks for pointing out that this transition is still in progress, and needs more patches. I agree that for this patch, the division is safe.
Quoted 23 lines
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 152 +++++++++++++++++++++++++++++-------------------- block/qcow2.h | 8 +++ 2 files changed, 98 insertions(+), 62 deletions(-)block/qcow2-refcount.c +2
-3@@ -116,20 +137,24 @@ int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index) } ret =qcow2_cache_get (bs, s->refcount_block_cache, refcount_block_offset, - (void **) &refcount_block); + &refcount_block);if (ret <0 ) {return ret; } block_index = cluster_index & (s->refcount_block_size -1 ); - refcount =be16_to_cpu (refcount_block[block_index]); + refcount = s->get_refcount (refcount_block, block_index); - ret = qcow2_cache_put(bs, s->refcount_block_cache,
block/qcow2-refcount.c int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index) · line 156
Quoted 10 lines
block/qcow2-refcount.c +5
-1- (void**) &refcount_block); + ret =qcow2_cache_put (bs, s->refcount_block_cache, &refcount_block);if (ret <0 ) {return ret; } +if (refcount <0 ) { +/* overflow */ +return -ERANGE ; + }
Should you be checking for overflow prior to calling qcow2_cache_put?
block/qcow2-refcount.c static int alloc_refcount_block(BlockDriverState *bs, · line 390
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -362,7 +387,7 @@ static int alloc_refcount_block(BlockDriverState *bs, s->cluster_size;uint64_t table_offset = meta_offset + blocks_clusters * s->cluster_size;uint64_t *new_table =g_try_new0 (uint64_t, table_size); -uint16_t *new_blocks =g_try_malloc0 (blocks_clusters * s->cluster_size); +void *new_blocks =g_try_malloc0 (blocks_clusters * s->cluster_size);
Can this multiplication ever overflow? Would we be better off with a g_try_new0 approach?
block/qcow2-refcount.c fail: · line 1150
Quoted 12 lines
block/qcow2-refcount.c +2
-2@@ -1118,12 +1142,13 @@ fail:*/ static int inc_refcounts (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table, +void **refcount_table,int64_t *refcount_table_size,int64_t offset,int64_t size) {BDRVQcowState *s = bs->opaque; -uint64_t start, last, cluster_offset, k; +uint64_t start, last, cluster_offset, k, refcount;
Why uint64_t, when you limit to int64_t in other patches?
Quoted 6 lines
block/qcow2-refcount.c +1
-0+int64_t i;if (size <=0 ) {return 0 ;@@ -1136,12 +1161,12 @@ static int inc_refcounts(BlockDriverState *bs, k = cluster_offset >> s->cluster_bits;
block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1169
Quoted 10 lines
block/qcow2-refcount.c +2
-2if (k >= *refcount_table_size) {int64_t old_refcount_table_size = *refcount_table_size; -uint16_t *new_refcount_table; +void *new_refcount_table; *refcount_table_size = k +1 ; new_refcount_table =g_try_realloc (*refcount_table, *refcount_table_size * -sizeof (**refcount_table)); + s->refcount_bits /8 );
This multiplies before dividing. Can it ever overflow, where writing *refcount_table_size * (s->refcount_bits / 8) would be safer? Also, is it better to use a malloc variant that checks for overflow (I think it is g_try_renew?) instead of open-coding the multiply?
block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1179
Quoted 13 lines
block/qcow2-refcount.c +3
-3if (!new_refcount_table) { *refcount_table_size = old_refcount_table_size; res->check_errors++;@@ -1149,16 +1174,19 @@ static int inc_refcounts(BlockDriverState *bs, } *refcount_table = new_refcount_table; -memset (*refcount_table + old_refcount_table_size,0 , - (*refcount_table_size - old_refcount_table_size) * -sizeof (**refcount_table)); +for (i = old_refcount_table_size; i < *refcount_table_size; i++) { + s->set_refcount (*refcount_table, i,0 ); + }
This feels slower than memset. Any chance we can add an optimization that brings back the speed of memset (may require an additional callback in addition to the getter and setter)?
block/qcow2-refcount.c enum { · line 1210
Quoted 7 lines
block/qcow2-refcount.c +1
-1@@ -1178,7 +1206,7 @@ enum {* error occurred. */ static int check_refcounts_l2 (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table,int64_t *refcount_table_size,int64_t l2_offset, +void **refcount_table,int64_t *refcount_table_size,int64_t l2_offset,int flags)
I noticed you cleaned up indentation in a lot of the patch, but not here. Any reason?
block/qcow2-refcount.c static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, · line 1572
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -1541,7 +1569,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, new_refcount_table =g_try_realloc (*refcount_table, *nb_clusters * -sizeof (**refcount_table)); + s->refcount_bits /8 );
Another possible overflow or g_try_renew site?
block/qcow2-refcount.c static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, · line 1582
Quoted 13 lines
block/qcow2-refcount.c +3
-3if (!new_refcount_table) { *nb_clusters = old_nb_clusters; res->check_errors++;@@ -1549,9 +1577,9 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, } *refcount_table = new_refcount_table; -memset (*refcount_table + old_nb_clusters,0 , - (*nb_clusters - old_nb_clusters) * -sizeof (**refcount_table)); +for (j = old_nb_clusters; j < *nb_clusters; j++) { + s->set_refcount (*refcount_table, j,0 ); + }
Another memset pessimation. Maybe even having a callback to expand the table, and factor out more of the common code of reallocating the table and clearing all new entries.
block/qcow2-refcount.c static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 1643
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -1611,7 +1640,7 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,int ret;if (!*refcount_table) { - *refcount_table =g_try_new0 (uint16_t, *nb_clusters); + *refcount_table =g_try_malloc0 (*nb_clusters * s->refcount_bits /8 );
Feels like a step backwards in overflow detection?
block/qcow2-refcount.c static int64_t alloc_clusters_imrt(BlockDriverState *bs, · line 1819
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -1787,22 +1816,22 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs, *imrt_nb_clusters = cluster + cluster_count - contiguous_free_clusters; new_refcount_table =g_try_realloc (*refcount_table, *imrt_nb_clusters * -sizeof (**refcount_table)); + s->refcount_bits /8 );
Another possible overflow
block/qcow2-refcount.c static int64_t alloc_clusters_imrt(BlockDriverState *bs, · line 1829
Quoted 13 lines
block/qcow2-refcount.c +3
-3if (!new_refcount_table) { *imrt_nb_clusters = old_imrt_nb_clusters;return -ENOMEM ; } *refcount_table = new_refcount_table; -memset (*refcount_table + old_imrt_nb_clusters,0 , - (*imrt_nb_clusters - old_imrt_nb_clusters) * -sizeof (**refcount_table)); +for (i = old_imrt_nb_clusters; i < *imrt_nb_clusters; i++) { + s->set_refcount (*refcount_table, i,0 ); + } }
and another resize where we pessimize memset
Quoted 6 lines
block/qcow2-refcount.c +0
-2@@ -1911,12 +1940,11 @@ write_refblocks: } on_disk_refblock =qemu_blockalign0 (bs->file, s->cluster_size); -for (i =0 ; i < s->refcount_block_size && - refblock_start + i < *nb_clusters; i++)
block/qcow2-refcount.c write_refblocks: · line 1948
Quoted 10 lines
block/qcow2-refcount.c +5
-4- { - on_disk_refblock[i] = -cpu_to_be16 ((*refcount_table)[refblock_start + i]); - } + +memcpy (on_disk_refblock, (void *)((uintptr_t )*refcount_table + + (refblock_index << s->refcount_block_bits)), +MIN (s->refcount_block_size, *nb_clusters - refblock_start) + * s->refcount_bits /8 );
This one's different in that you move TO a memcpy instead of open-coded loop. But I still worry if multiply before /8 could be a problem.
block/qcow2-refcount.c int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 2095
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -2064,7 +2092,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,/* Because the old reftable has been exchanged for a new one the * references have to be recalculated */ rebuild = false; -memset (refcount_table,0 , nb_clusters *sizeof (uint16_t )); +memset (refcount_table,0 , nb_clusters * s->refcount_bits /8 );
Another /8 possible overflow.
Quoted 8 lines
block/qcow2-refcount.c +0
-0ret =calculate_refcounts (bs, res,0 , &rebuild, &refcount_table, &nb_clusters);if (ret <0 ) {block/qcow2.h +0
-0
block/qcow2.h typedef struct Qcow2DiscardRegion { · line 219
Quoted 7 lines
block/qcow2.h +4
-0@@ -213,6 +213,11 @@ typedef struct Qcow2DiscardRegion {QTAILQ_ENTRY (Qcow2DiscardRegion) next; } Qcow2DiscardRegion; +typedef uint64_t Qcow2GetRefcountFunc (const void *refcount_array, +uint64_t index); +typedef void Qcow2SetRefcountFunc (void *refcount_array, +uint64_t index,uint64_t value);
Do you want int64_t for any of the types here, to make it obvious that you can't exceed 2^63?
Looks like you are on track to a sane conversion, but I'm worried enough about the math that it probably needs a respin (either comments stating why we know we don't overflow, or else safer constructs).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 06/21] qcow2: Helper function for refcount modification
Quoted 43 lines · Eric Blake
Quoted 7 lines · Max Reitz
Since refcounts do not always have to be a
uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.Quoted 5 lines
At the same time, replace all
sizeof(**refcount_table)etc. in the qcow2 check code bys->refcount_bits/ 8. Note that this might lead to wrong values due to truncating division, but currentlys->refcount_bitsis always 16, and before the upcoming patch which removes this limitation another patch will make the division round up correctly.Thanks for pointing out that this transition is still in progress, and needs more patches. I agree that for this patch, the division is safe.
Quoted 27 lines
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 152 +++++++++++++++++++++++++++++-------------------- block/qcow2.h | 8 +++ 2 files changed, 98 insertions(+), 62 deletions(-)block/qcow2-refcount.c +3
-4@@ -116,20 +137,24 @@ int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index) } ret =qcow2_cache_get (bs, s->refcount_block_cache, refcount_block_offset, - (void **) &refcount_block); + &refcount_block);if (ret <0 ) {return ret; } block_index = cluster_index & (s->refcount_block_size -1 ); - refcount =be16_to_cpu (refcount_block[block_index]); + refcount = s->get_refcount (refcount_block, block_index); - ret =qcow2_cache_put (bs, s->refcount_block_cache, - (void **) &refcount_block); + ret =qcow2_cache_put (bs, s->refcount_block_cache, &refcount_block);if (ret <0 ) {return ret; }
Quoted 5 lines
block/qcow2-refcount.c int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index) · line 156
Quoted 4 lines
block/qcow2-refcount.c +4
-0+if (refcount <0 ) { +/* overflow */ +return -ERANGE ; + }Should you be checking for overflow prior to calling
qcow2_cache_put?
I don't think so; we want to free the cache entry reference even if the refblock seems unusable.
Quoted 8 lines
block/qcow2-refcount.c static int alloc_refcount_block(BlockDriverState *bs, · line 390
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -362,7 +387,7 @@ static int alloc_refcount_block(BlockDriverState *bs, s->cluster_size;uint64_t table_offset = meta_offset + blocks_clusters * s->cluster_size;uint64_t *new_table =g_try_new0 (uint64_t, table_size); -uint16_t *new_blocks =g_try_malloc0 (blocks_clusters * s->cluster_size); +void *new_blocks =g_try_malloc0 (blocks_clusters * s->cluster_size);Can this multiplication ever overflow? Would we be better off with a
g_try_new0approach?
Yes, you're right. Patch 7 introduces a helper function which will allow checking such overflows in a central place, but I haven't done so in this version.
Quoted 5 lines
Quoted 5 lines
block/qcow2-refcount.c +0
-1@@ -1118,12 +1142,13 @@ fail:*/ static int inc_refcounts(BlockDriverState *bs, BdrvCheckResult *res, - uint16_t **refcount_table,
Quoted 8 lines
block/qcow2-refcount.c fail: · line 1150
Quoted 7 lines
block/qcow2-refcount.c +2
-1+void **refcount_table, int64_t *refcount_table_size, int64_t offset, int64_t size) { BDRVQcowState *s = bs->opaque; -uint64_t start, last, cluster_offset, k; +uint64_t start, last, cluster_offset, k, refcount;Why
uint64_t, when you limit toint64_tin other patches?
Because we don't need to check for errors here. But I guess we should really use int64_t everywhere to be consistent (so that if something breaks because a cluster has more than INT64_MAX references, it breaks everywhere).
Quoted 28 lines
Quoted 6 lines
block/qcow2-refcount.c +1
-0+int64_t i;if (size <=0 ) {return 0 ;@@ -1136,12 +1161,12 @@ static int inc_refcounts(BlockDriverState *bs, k = cluster_offset >> s->cluster_bits;block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1169
Quoted 10 lines
block/qcow2-refcount.c +2
-2if (k >= *refcount_table_size) {int64_t old_refcount_table_size = *refcount_table_size; -uint16_t *new_refcount_table; +void *new_refcount_table; *refcount_table_size = k +1 ; new_refcount_table =g_try_realloc (*refcount_table, *refcount_table_size * -sizeof (**refcount_table)); + s->refcount_bits /8 );This multiplies before dividing. Can it ever overflow, where writing
*refcount_table_size* (s->refcount_bits/ 8) would be safer? Also, is it better to use a malloc variant that checks for overflow (I think it isg_try_renew?) instead of open-coding the multiply?Quoted 7 lines
block/qcow2-refcount.c +0
-0if (!new_refcount_table) { *refcount_table_size = old_refcount_table_size; res->check_errors++;@@ -1149,16 +1174,19 @@ static int inc_refcounts(BlockDriverState *bs, } *refcount_table = new_refcount_table;
Quoted 7 lines
block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1179
Quoted 6 lines
block/qcow2-refcount.c +3
-3-memset (*refcount_table + old_refcount_table_size,0 , - (*refcount_table_size - old_refcount_table_size) * -sizeof (**refcount_table)); +for (i = old_refcount_table_size; i < *refcount_table_size; i++) { + s->set_refcount (*refcount_table, i,0 ); + }This feels slower than memset.
It is, yes. But this is the check function, I don't think performance is all that important here (especially not operations in RAM).
Quoted 3 lines
Any chance we can add an optimization
that brings back the speed of memset (may require an additional callback in addition to the getter and setter)?
For sub-byte refcount widths, we would have to manually set all non-byte aligned entries to 0 and then use memset() on the rest. Not impossible, but I think too complicated for a place where performance is not critical.
Quoted 9 lines
block/qcow2-refcount.c enum { · line 1210
Quoted 7 lines
block/qcow2-refcount.c +1
-1@@ -1178,7 +1206,7 @@ enum {* error occurred. */ static int check_refcounts_l2 (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table,int64_t *refcount_table_size,int64_t l2_offset, +void **refcount_table,int64_t *refcount_table_size,int64_t l2_offset,int flags)I noticed you cleaned up indentation in a lot of the patch, but not here. Any reason?
Maybe I remembered touching that header once already and not fixing up the indentation. Will do in v2.
Quoted 15 lines
block/qcow2-refcount.c static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, · line 1572
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -1541,7 +1569,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, new_refcount_table =g_try_realloc (*refcount_table, *nb_clusters * -sizeof (**refcount_table)); + s->refcount_bits /8 );Another possible overflow or
g_try_renewsite?Quoted 7 lines
block/qcow2-refcount.c +0
-0if (!new_refcount_table) { *nb_clusters = old_nb_clusters; res->check_errors++;@@ -1549,9 +1577,9 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, } *refcount_table = new_refcount_table;
Quoted 9 lines
block/qcow2-refcount.c static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, · line 1582
Quoted 6 lines
block/qcow2-refcount.c +3
-3-memset (*refcount_table + old_nb_clusters,0 , - (*nb_clusters - old_nb_clusters) * -sizeof (**refcount_table)); +for (j = old_nb_clusters; j < *nb_clusters; j++) { + s->set_refcount (*refcount_table, j,0 ); + }Another memset pessimation. Maybe even having a callback to expand the table, and factor out more of the common code of reallocating the table and clearing all new entries.
That sounds useful, yes. I'll look into it, but I'll probably still go without memset().
Quoted 35 lines
block/qcow2-refcount.c static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 1643
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -1611,7 +1640,7 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,int ret;if (!*refcount_table) { - *refcount_table =g_try_new0 (uint16_t, *nb_clusters); + *refcount_table =g_try_malloc0 (*nb_clusters * s->refcount_bits /8 );Feels like a step backwards in overflow detection?
block/qcow2-refcount.c static int64_t alloc_clusters_imrt(BlockDriverState *bs, · line 1819
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -1787,22 +1816,22 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs, *imrt_nb_clusters = cluster + cluster_count - contiguous_free_clusters; new_refcount_table =g_try_realloc (*refcount_table, *imrt_nb_clusters * -sizeof (**refcount_table)); + s->refcount_bits /8 );Another possible overflow
block/qcow2-refcount.c static int64_t alloc_clusters_imrt(BlockDriverState *bs, · line 1829
Quoted 13 lines
block/qcow2-refcount.c +3
-3if (!new_refcount_table) { *imrt_nb_clusters = old_imrt_nb_clusters;return -ENOMEM ; } *refcount_table = new_refcount_table; -memset (*refcount_table + old_imrt_nb_clusters,0 , - (*imrt_nb_clusters - old_imrt_nb_clusters) * -sizeof (**refcount_table)); +for (i = old_imrt_nb_clusters; i < *imrt_nb_clusters; i++) { + s->set_refcount (*refcount_table, i,0 ); + } }and another resize where we pessimize memset
Quoted 3 lines
(quoted diff) +0
-0@@ -1911,12 +1940,11 @@ write_refblocks: }
Quoted 15 lines
block/qcow2-refcount.c write_refblocks: · line 1948
Quoted 13 lines
block/qcow2-refcount.c +5
-6on_disk_refblock =qemu_blockalign0 (bs->file, s->cluster_size); -for (i =0 ; i < s->refcount_block_size && - refblock_start + i < *nb_clusters; i++) - { - on_disk_refblock[i] = -cpu_to_be16 ((*refcount_table)[refblock_start + i]); - } + +memcpy (on_disk_refblock, (void *)((uintptr_t )*refcount_table + + (refblock_index << s->refcount_block_bits)), +MIN (s->refcount_block_size, *nb_clusters - refblock_start) + * s->refcount_bits /8 );This one's different in that you move TO a memcpy instead of open-coded loop.
Yes, because the set_refcount() and get_refcount() helpers store the refcounts already in big endian, so now we can directly use the data without conversion.
Quoted 21 lines
But I still worry if multiply before /8 could be a problem.
block/qcow2-refcount.c int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 2095
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -2064,7 +2092,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,/* Because the old reftable has been exchanged for a new one the * references have to be recalculated */ rebuild = false; -memset (refcount_table,0 , nb_clusters *sizeof (uint16_t )); +memset (refcount_table,0 , nb_clusters * s->refcount_bits /8 );Another /8 possible overflow.
Quoted 11 lines
block/qcow2-refcount.c +0
-0ret =calculate_refcounts (bs, res,0 , &rebuild, &refcount_table, &nb_clusters);if (ret <0 ) {block/qcow2.h +0
-0@@ -213,6 +213,11 @@ typedef struct Qcow2DiscardRegion {QTAILQ_ENTRY (Qcow2DiscardRegion) next; } Qcow2DiscardRegion;
Quoted 6 lines
block/qcow2.h typedef struct Qcow2DiscardRegion { · line 219
Quoted 4 lines
block/qcow2.h +4
-0+typedef uint64_t Qcow2GetRefcountFunc (const void *refcount_array, +uint64_t index); +typedef void Qcow2SetRefcountFunc (void *refcount_array, +uint64_t index,uint64_t value);Do you want
int64_tfor any of the types here, to make it obvious that you can't exceed 2^63?
Yes, I do.
Quoted 3 lines
Looks like you are on track to a sane conversion, but I'm worried enough about the math that it probably needs a respin (either comments stating why we know we don't overflow, or else safer constructs).
I'll add a comment in the commit message why overflows do not really get more probable than they were before, but the real overflow prevention will happen in patch 7.
I think I'll factor out the refcount array resize which automatically sets the newly allocated entries to zero. Now that I think about it... I can actually get away without sub-byte operations. Since .*realloc() will only allocate full bytes anyway, I only need to set that newly allocated area to zero (with memset()). So it'll be back to memset(). (I'm still wondering why there's no g_try_realloc0() or something; probably because the glib does not require the libc heap manager to know the exact size of each heap object which would be necessary for that to work)
Max
Re: [Qemu-devel] [PATCH 06/21] qcow2: Helper function for refcount modification
Quoted 253 lines · Max Reitz
Quoted 40 lines · Eric Blake
Quoted 7 lines · Max Reitz
Since refcounts do not always have to be a
uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.Quoted 5 lines
At the same time, replace all
sizeof(**refcount_table)etc. in the qcow2 check code bys->refcount_bits/ 8. Note that this might lead to wrong values due to truncating division, but currentlys->refcount_bitsis always 16, and before the upcoming patch which removes this limitation another patch will make the division round up correctly.Thanks for pointing out that this transition is still in progress, and needs more patches. I agree that for this patch, the division is safe.
Quoted 24 lines
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 152 +++++++++++++++++++++++++++++-------------------- block/qcow2.h | 8 +++ 2 files changed, 98 insertions(+), 62 deletions(-)block/qcow2-refcount.c +3
-3@@ -116,20 +137,24 @@ int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index) } ret =qcow2_cache_get (bs, s->refcount_block_cache, refcount_block_offset, - (void **) &refcount_block); + &refcount_block);if (ret <0 ) {return ret; } block_index = cluster_index & (s->refcount_block_size -1 ); - refcount =be16_to_cpu (refcount_block[block_index]); + refcount = s->get_refcount (refcount_block, block_index); - ret =qcow2_cache_put (bs, s->refcount_block_cache, - (void **) &refcount_block); + ret =qcow2_cache_put (bs, s->refcount_block_cache, &refcount_block);Quoted 8 lines
block/qcow2-refcount.c int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index) · line 156
Quoted 7 lines
block/qcow2-refcount.c +3
-0if (ret <0 ) {return ret; } + if (refcount < 0) { +/* overflow */ +return -ERANGE ; + }Should you be checking for overflow prior to calling
qcow2_cache_put?I don't think so; we want to free the cache entry reference even if the refblock seems unusable.
Quoted 4 lines
Quoted 4 lines
(quoted diff) +0
-0@@ -362,7 +387,7 @@ static int alloc_refcount_block(BlockDriverState *bs, s->cluster_size; uint64_t table_offset = meta_offset + blocks_clusters *Quoted 8 lines
Quoted 6 lines
(quoted diff) +1
-1s->cluster_size; uint64_t *new_table = g_try_new0(uint64_t, table_size); - uint16_t *new_blocks = g_try_malloc0(blocks_clusters * s->cluster_size); + void *new_blocks = g_try_malloc0(blocks_clusters *
s->cluster_size);Can this multiplication ever overflow? Would we be better off with a
g_try_new0approach?Yes, you're right. Patch 7 introduces a helper function which will allow checking such overflows in a central place, but I haven't done so in this version.
Quoted 5 lines
Quoted 5 lines
block/qcow2-refcount.c +0
-1@@ -1118,12 +1142,13 @@ fail:*/ static int inc_refcounts(BlockDriverState *bs, BdrvCheckResult *res, - uint16_t **refcount_table,Quoted 8 lines
block/qcow2-refcount.c fail: · line 1150
Quoted 7 lines
block/qcow2-refcount.c +2
-1+void **refcount_table, int64_t *refcount_table_size, int64_t offset, int64_t size) { BDRVQcowState *s = bs->opaque; -uint64_t start, last, cluster_offset, k; +uint64_t start, last, cluster_offset, k, refcount;Why
uint64_t, when you limit toint64_tin other patches?Because we don't need to check for errors here. But I guess we should really use
int64_teverywhere to be consistent (so that if something breaks because a cluster has more thanINT64_MAXreferences, it breaks everywhere).Quoted 19 lines
Quoted 4 lines
block/qcow2-refcount.c +1
-0+int64_t i;if (size <=0 ) {return 0 ;block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1168
Quoted 10 lines
block/qcow2-refcount.c +2
-2@@ -1136,12 +1161,12 @@ static int inc_refcounts(BlockDriverState *bs, k = cluster_offset >> s->cluster_bits;if (k >= *refcount_table_size) {int64_t old_refcount_table_size = *refcount_table_size; -uint16_t *new_refcount_table; +void *new_refcount_table; *refcount_table_size = k +1 ; new_refcount_table =g_try_realloc (*refcount_table, *refcount_table_size * -sizeof (**refcount_table)); + s->refcount_bits /8 );This multiplies before dividing. Can it ever overflow, where writing
*refcount_table_size* (s->refcount_bits/ 8) would be safer? Also, is it better to use a malloc variant that checks for overflow (I think it isg_try_renew?) instead of open-coding the multiply?Quoted 14 lines
block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1180
Quoted 13 lines
block/qcow2-refcount.c +3
-2if (!new_refcount_table) { *refcount_table_size = old_refcount_table_size; res->check_errors++;@@ -1149,16 +1174,19 @@ static int inc_refcounts(BlockDriverState *bs, } *refcount_table = new_refcount_table; -memset (*refcount_table + old_refcount_table_size,0 , - (*refcount_table_size - old_refcount_table_size) * -sizeof (**refcount_table)); + for (i = old_refcount_table_size; i < *refcount_table_size; i++) { + s->set_refcount (*refcount_table, i,0 ); + }This feels slower than memset.
It is, yes. But this is the check function, I don't think performance is all that important here (especially not operations in RAM).
Quoted 3 lines
Any chance we can add an optimization
that brings back the speed of memset (may require an additional callback in addition to the getter and setter)?For sub-byte refcount widths, we would have to manually set all non-byte aligned entries to 0 and then use
memset()on the rest. Not impossible, but I think too complicated for a place where performance is not critical.Quoted 4 lines
Quoted 4 lines
block/qcow2-refcount.c +0
-0@@ -1178,7 +1206,7 @@ enum {* error occurred. */ static int check_refcounts_l2(BlockDriverState *bs,Quoted 8 lines
Quoted 6 lines
(quoted diff) +1
-1BdrvCheckResult *res, - uint16_t **refcount_table, int64_t *refcount_table_size, int64_t l2_offset, + void **refcount_table, int64_t *refcount_table_size, int64_t l2_offset,int flags)
I noticed you cleaned up indentation in a lot of the patch, but not here. Any reason?
Maybe I remembered touching that header once already and not fixing up the indentation. Will do in v2.
Quoted 8 lines
block/qcow2-refcount.c static int check_refblocks(BlockDriverState · line 1572
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -1541,7 +1569,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, new_refcount_table =g_try_realloc (*refcount_table, *nb_clusters * -sizeof (**refcount_table)); + s->refcount_bits /8 );Another possible overflow or
g_try_renewsite?Quoted 16 lines
block/qcow2-refcount.c static int check_refblocks(BlockDriverState · line 1583
Quoted 13 lines
block/qcow2-refcount.c +3
-2if (!new_refcount_table) { *nb_clusters = old_nb_clusters; res->check_errors++;@@ -1549,9 +1577,9 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, } *refcount_table = new_refcount_table; -memset (*refcount_table + old_nb_clusters,0 , - (*nb_clusters - old_nb_clusters) * -sizeof (**refcount_table)); + for (j = old_nb_clusters; j < *nb_clusters; j++) { + s->set_refcount (*refcount_table, j,0 ); + }Another memset pessimation. Maybe even having a callback to expand the table, and factor out more of the common code of reallocating the table and clearing all new entries.
That sounds useful, yes. I'll look into it, but I'll probably still go without
memset().Quoted 45 lines
block/qcow2-refcount.c static int · line 1644
Quoted 7 lines
block/qcow2-refcount.c +1
-1@@ -1611,7 +1640,7 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, int ret; if (!*refcount_table) { - *refcount_table = g_try_new0(uint16_t, *nb_clusters); + *refcount_table = g_try_malloc0(*nb_clusters * s->refcount_bits / 8);Feels like a step backwards in overflow detection?
block/qcow2-refcount.c static int64_t · line 1821
Quoted 8 lines
block/qcow2-refcount.c +1
-1@@ -1787,22 +1816,22 @@ static int64_talloc_clusters_imrt (BlockDriverState *bs, *imrt_nb_clusters = cluster + cluster_count - contiguous_free_clusters; new_refcount_table =g_try_realloc (*refcount_table, *imrt_nb_clusters * -sizeof (**refcount_table)); + s->refcount_bits /8 );Another possible overflow
block/qcow2-refcount.c static int64_t alloc_clusters_imrt(BlockDriverState *bs, · line 1829
Quoted 12 lines
block/qcow2-refcount.c +3
-2if (!new_refcount_table) { *imrt_nb_clusters = old_imrt_nb_clusters;return -ENOMEM ; } *refcount_table = new_refcount_table; - memset(*refcount_table + old_imrt_nb_clusters, 0, - (*imrt_nb_clusters - old_imrt_nb_clusters) * -sizeof (**refcount_table)); + for (i = old_imrt_nb_clusters; i < *imrt_nb_clusters; i++) { + s->set_refcount(*refcount_table, i, 0); + } }and another resize where we pessimize memset
Quoted 11 lines
block/qcow2-refcount.c +1
-6@@ -1911,12 +1940,11 @@ write_refblocks: } on_disk_refblock =qemu_blockalign0 (bs->file, s->cluster_size); -for (i =0 ; i < s->refcount_block_size && - refblock_start + i < *nb_clusters; i++) - { - on_disk_refblock[i] = -cpu_to_be16 ((*refcount_table)[refblock_start + i]); - } +Quoted 8 lines
Review write_refblocks: · line 1949
Quoted 6 lines
(quoted diff) +4
-0+ memcpy(on_disk_refblock, (void *)((uintptr_t)*refcount_table + + (refblock_index << s->refcount_block_bits)), + MIN(s->refcount_block_size, *nb_clusters - refblock_start) + * s->refcount_bits / 8);This one's different in that you move TO a memcpy instead of open-coded loop.
Yes, because the
set_refcount()andget_refcount()helpers store the refcounts already in big endian, so now we can directly use the data without conversion.Quoted 21 lines
But I still worry if multiply before /8 could be a problem.
block/qcow2-refcount.c int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 2095
Quoted 8 lines
(quoted diff) +0
-0@@ -2064,7 +2092,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res, /* Because the old reftable has been exchanged for a newone the
* references have to be recalculated */
rebuild = false;block/qcow2-refcount.c +1
-1-memset (refcount_table,0 , nb_clusters *sizeof (uint16_t )); +memset (refcount_table,0 , nb_clusters * s->refcount_bits /8 );Another /8 possible overflow.
Quoted 9 lines
ret = calculate_refcounts(bs, res, 0, &rebuild,
&refcount_table,&nb_clusters); if (ret < 0) {block/qcow2.h +0
-0@@ -213,6 +213,11 @@ typedef struct Qcow2DiscardRegion {Quoted 8 lines
block/qcow2.h typedef struct Qcow2DiscardRegion { · line 219
Quoted 6 lines
block/qcow2.h +3
-0QTAILQ_ENTRY (Qcow2DiscardRegion) next; } Qcow2DiscardRegion; +typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array, +uint64_t index); +typedef void Qcow2SetRefcountFunc (void *refcount_array, +uint64_t index,uint64_t value);Do you want
int64_tfor any of the types here, to make it obvious that you can't exceed 2^63?
Quoted 1 line
Yes, I do.
To be honest, I just read your reply and not the original code (which I'm doing now while working on v2). I think I don't want int64_t after all. I generally do want to use int64_t for all refcount values, but in this case, as these are just helpers for directly accessing refcount arrays (which I think should not fail, see my reply for patch 8), I'd rather keep them using uint64_t.
Max
Quoted 9 lines
Quoted 3 lines
Looks like you are on track to a sane conversion, but I'm worried enough about the math that it probably needs a respin (either comments stating why we know we don't overflow, or else safer constructs).
I'll add a comment in the commit message why overflows do not really get more probable than they were before, but the real overflow prevention will happen in patch 7.
I think I'll factor out the refcount array resize which automatically sets the newly allocated entries to zero. Now that I think about it... I can actually get away without sub-byte operations. Since .*realloc() will only allocate full bytes anyway, I only need to set that newly allocated area to zero (with
memset()). So it'll be back tomemset(). (I'm still wondering why there's nog_try_realloc0()or something; probably because the glib does not require the libc heap manager to know the exact size of each heap object which would be necessary for that to work)Max
Re: [Qemu-devel] [PATCH 06/21] qcow2: Helper function for refcount modification
In reply to Max ReitzQuoted 65 lines · Max Reitz
Quoted 40 lines · Eric Blake
Quoted 7 lines · Max Reitz
Since refcounts do not always have to be a
uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.Quoted 5 lines
At the same time, replace all
sizeof(**refcount_table)etc. in the qcow2 check code bys->refcount_bits/ 8. Note that this might lead to wrong values due to truncating division, but currentlys->refcount_bitsis always 16, and before the upcoming patch which removes this limitation another patch will make the division round up correctly.Thanks for pointing out that this transition is still in progress, and needs more patches. I agree that for this patch, the division is safe.
Quoted 24 lines
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 152 +++++++++++++++++++++++++++++-------------------- block/qcow2.h | 8 +++ 2 files changed, 98 insertions(+), 62 deletions(-)block/qcow2-refcount.c +3
-3@@ -116,20 +137,24 @@ int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index) } ret =qcow2_cache_get (bs, s->refcount_block_cache, refcount_block_offset, - (void **) &refcount_block); + &refcount_block);if (ret <0 ) {return ret; } block_index = cluster_index & (s->refcount_block_size -1 ); - refcount =be16_to_cpu (refcount_block[block_index]); + refcount = s->get_refcount (refcount_block, block_index); - ret =qcow2_cache_put (bs, s->refcount_block_cache, - (void **) &refcount_block); + ret =qcow2_cache_put (bs, s->refcount_block_cache, &refcount_block);Quoted 8 lines
block/qcow2-refcount.c int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index) · line 156
Quoted 7 lines
block/qcow2-refcount.c +3
-0if (ret <0 ) {return ret; } + if (refcount < 0) { +/* overflow */ +return -ERANGE ; + }Should you be checking for overflow prior to calling
qcow2_cache_put?I don't think so; we want to free the cache entry reference even if the refblock seems unusable.
Quoted 4 lines
Quoted 4 lines
(quoted diff) +0
-0@@ -362,7 +387,7 @@ static int alloc_refcount_block(BlockDriverState *bs, s->cluster_size; uint64_t table_offset = meta_offset + blocks_clusters *Quoted 8 lines
Quoted 6 lines
(quoted diff) +1
-1s->cluster_size; uint64_t *new_table = g_try_new0(uint64_t, table_size); - uint16_t *new_blocks = g_try_malloc0(blocks_clusters * s->cluster_size); + void *new_blocks = g_try_malloc0(blocks_clusters *
s->cluster_size);Can this multiplication ever overflow? Would we be better off with a
g_try_new0approach?
Quoted 1 line
Yes, you're right. Patch 7 introduces a helper function which will allow checking such overflows in a central place, but I haven't done so in this version.
I should really start reading my own code before responding; at least be assured that I do read my code when working on a respin. Of course this has nothing to do with patch 7, and I'll fix it. I don't think there'll be a problem here (because first, it would be pre-existing, and second, this would require offsets beyond 64 bits).
Max
Quoted 197 lines
Quoted 5 lines
Quoted 5 lines
block/qcow2-refcount.c +0
-1@@ -1118,12 +1142,13 @@ fail:*/ static int inc_refcounts(BlockDriverState *bs, BdrvCheckResult *res, - uint16_t **refcount_table,Quoted 8 lines
block/qcow2-refcount.c fail: · line 1150
Quoted 7 lines
block/qcow2-refcount.c +2
-1+void **refcount_table, int64_t *refcount_table_size, int64_t offset, int64_t size) { BDRVQcowState *s = bs->opaque; -uint64_t start, last, cluster_offset, k; +uint64_t start, last, cluster_offset, k, refcount;Why
uint64_t, when you limit toint64_tin other patches?Because we don't need to check for errors here. But I guess we should really use
int64_teverywhere to be consistent (so that if something breaks because a cluster has more thanINT64_MAXreferences, it breaks everywhere).Quoted 19 lines
Quoted 4 lines
block/qcow2-refcount.c +1
-0+int64_t i;if (size <=0 ) {return 0 ;block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1168
Quoted 10 lines
block/qcow2-refcount.c +2
-2@@ -1136,12 +1161,12 @@ static int inc_refcounts(BlockDriverState *bs, k = cluster_offset >> s->cluster_bits;if (k >= *refcount_table_size) {int64_t old_refcount_table_size = *refcount_table_size; -uint16_t *new_refcount_table; +void *new_refcount_table; *refcount_table_size = k +1 ; new_refcount_table =g_try_realloc (*refcount_table, *refcount_table_size * -sizeof (**refcount_table)); + s->refcount_bits /8 );This multiplies before dividing. Can it ever overflow, where writing
*refcount_table_size* (s->refcount_bits/ 8) would be safer? Also, is it better to use a malloc variant that checks for overflow (I think it isg_try_renew?) instead of open-coding the multiply?Quoted 14 lines
block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1180
Quoted 13 lines
block/qcow2-refcount.c +3
-2if (!new_refcount_table) { *refcount_table_size = old_refcount_table_size; res->check_errors++;@@ -1149,16 +1174,19 @@ static int inc_refcounts(BlockDriverState *bs, } *refcount_table = new_refcount_table; -memset (*refcount_table + old_refcount_table_size,0 , - (*refcount_table_size - old_refcount_table_size) * -sizeof (**refcount_table)); + for (i = old_refcount_table_size; i < *refcount_table_size; i++) { + s->set_refcount (*refcount_table, i,0 ); + }This feels slower than memset.
It is, yes. But this is the check function, I don't think performance is all that important here (especially not operations in RAM).
Quoted 3 lines
Any chance we can add an optimization
that brings back the speed of memset (may require an additional callback in addition to the getter and setter)?For sub-byte refcount widths, we would have to manually set all non-byte aligned entries to 0 and then use
memset()on the rest. Not impossible, but I think too complicated for a place where performance is not critical.Quoted 4 lines
Quoted 4 lines
block/qcow2-refcount.c +0
-0@@ -1178,7 +1206,7 @@ enum {* error occurred. */ static int check_refcounts_l2(BlockDriverState *bs,Quoted 8 lines
Quoted 6 lines
(quoted diff) +1
-1BdrvCheckResult *res, - uint16_t **refcount_table, int64_t *refcount_table_size, int64_t l2_offset, + void **refcount_table, int64_t *refcount_table_size, int64_t l2_offset,int flags)
I noticed you cleaned up indentation in a lot of the patch, but not here. Any reason?
Maybe I remembered touching that header once already and not fixing up the indentation. Will do in v2.
Quoted 8 lines
block/qcow2-refcount.c static int check_refblocks(BlockDriverState · line 1572
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -1541,7 +1569,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, new_refcount_table =g_try_realloc (*refcount_table, *nb_clusters * -sizeof (**refcount_table)); + s->refcount_bits /8 );Another possible overflow or
g_try_renewsite?Quoted 16 lines
block/qcow2-refcount.c static int check_refblocks(BlockDriverState · line 1583
Quoted 13 lines
block/qcow2-refcount.c +3
-2if (!new_refcount_table) { *nb_clusters = old_nb_clusters; res->check_errors++;@@ -1549,9 +1577,9 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, } *refcount_table = new_refcount_table; -memset (*refcount_table + old_nb_clusters,0 , - (*nb_clusters - old_nb_clusters) * -sizeof (**refcount_table)); + for (j = old_nb_clusters; j < *nb_clusters; j++) { + s->set_refcount (*refcount_table, j,0 ); + }Another memset pessimation. Maybe even having a callback to expand the table, and factor out more of the common code of reallocating the table and clearing all new entries.
That sounds useful, yes. I'll look into it, but I'll probably still go without
memset().Quoted 45 lines
block/qcow2-refcount.c static int · line 1644
Quoted 7 lines
block/qcow2-refcount.c +1
-1@@ -1611,7 +1640,7 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, int ret; if (!*refcount_table) { - *refcount_table = g_try_new0(uint16_t, *nb_clusters); + *refcount_table = g_try_malloc0(*nb_clusters * s->refcount_bits / 8);Feels like a step backwards in overflow detection?
block/qcow2-refcount.c static int64_t · line 1821
Quoted 8 lines
block/qcow2-refcount.c +1
-1@@ -1787,22 +1816,22 @@ static int64_talloc_clusters_imrt (BlockDriverState *bs, *imrt_nb_clusters = cluster + cluster_count - contiguous_free_clusters; new_refcount_table =g_try_realloc (*refcount_table, *imrt_nb_clusters * -sizeof (**refcount_table)); + s->refcount_bits /8 );Another possible overflow
block/qcow2-refcount.c static int64_t alloc_clusters_imrt(BlockDriverState *bs, · line 1829
Quoted 12 lines
block/qcow2-refcount.c +3
-2if (!new_refcount_table) { *imrt_nb_clusters = old_imrt_nb_clusters;return -ENOMEM ; } *refcount_table = new_refcount_table; - memset(*refcount_table + old_imrt_nb_clusters, 0, - (*imrt_nb_clusters - old_imrt_nb_clusters) * -sizeof (**refcount_table)); + for (i = old_imrt_nb_clusters; i < *imrt_nb_clusters; i++) { + s->set_refcount(*refcount_table, i, 0); + } }and another resize where we pessimize memset
Quoted 11 lines
block/qcow2-refcount.c +1
-6@@ -1911,12 +1940,11 @@ write_refblocks: } on_disk_refblock =qemu_blockalign0 (bs->file, s->cluster_size); -for (i =0 ; i < s->refcount_block_size && - refblock_start + i < *nb_clusters; i++) - { - on_disk_refblock[i] = -cpu_to_be16 ((*refcount_table)[refblock_start + i]); - } +Quoted 8 lines
Review write_refblocks: · line 1949
Quoted 6 lines
(quoted diff) +4
-0+ memcpy(on_disk_refblock, (void *)((uintptr_t)*refcount_table + + (refblock_index << s->refcount_block_bits)), + MIN(s->refcount_block_size, *nb_clusters - refblock_start) + * s->refcount_bits / 8);This one's different in that you move TO a memcpy instead of open-coded loop.
Yes, because the
set_refcount()andget_refcount()helpers store the refcounts already in big endian, so now we can directly use the data without conversion.Quoted 21 lines
But I still worry if multiply before /8 could be a problem.
block/qcow2-refcount.c int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 2095
Quoted 8 lines
(quoted diff) +0
-0@@ -2064,7 +2092,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res, /* Because the old reftable has been exchanged for a newone the
* references have to be recalculated */
rebuild = false;block/qcow2-refcount.c +1
-1-memset (refcount_table,0 , nb_clusters *sizeof (uint16_t )); +memset (refcount_table,0 , nb_clusters * s->refcount_bits /8 );Another /8 possible overflow.
Quoted 9 lines
ret = calculate_refcounts(bs, res, 0, &rebuild,
&refcount_table,&nb_clusters); if (ret < 0) {block/qcow2.h +0
-0@@ -213,6 +213,11 @@ typedef struct Qcow2DiscardRegion {Quoted 8 lines
block/qcow2.h typedef struct Qcow2DiscardRegion { · line 219
Quoted 6 lines
block/qcow2.h +3
-0QTAILQ_ENTRY (Qcow2DiscardRegion) next; } Qcow2DiscardRegion; +typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array, +uint64_t index); +typedef void Qcow2SetRefcountFunc (void *refcount_array, +uint64_t index,uint64_t value);Do you want
int64_tfor any of the types here, to make it obvious that you can't exceed 2^63?Yes, I do.
Quoted 3 lines
Looks like you are on track to a sane conversion, but I'm worried enough about the math that it probably needs a respin (either comments stating why we know we don't overflow, or else safer constructs).
I'll add a comment in the commit message why overflows do not really get more probable than they were before, but the real overflow prevention will happen in patch 7.
I think I'll factor out the refcount array resize which automatically sets the newly allocated entries to zero. Now that I think about it... I can actually get away without sub-byte operations. Since .*realloc() will only allocate full bytes anyway, I only need to set that newly allocated area to zero (with
memset()). So it'll be back tomemset(). (I'm still wondering why there's nog_try_realloc0()or something; probably because the glib does not require the libc heap manager to know the exact size of each heap object which would be necessary for that to work)Max
PATCH 7/21 qcow2: Helper for refcount array size calculation
In reply to Max ReitzAdd a helper function which correctly calculates the byte size of a refcount array for any refcount order, and use that function.
--- block/qcow2-refcount.c | 39 ++++++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 11 deletions(-)
block/qcow2-refcount.c +28 -11
@@ -1132,6 +1132,20 @@ fail:/* refcount checking functions */ +static size_t refcount_array_byte_size (BDRVQcowState *s,uint64_t entries) +{ +if (s->refcount_order <3 ) { +/* sub-byte width */ +int shift =3 - s->refcount_order; +return (entries + (1 << shift) -1 ) >> shift; + }else if (s->refcount_order ==3 ) { +/* byte width */ +return entries; + }else { +/* multiple bytes wide */ +return entries << (s->refcount_order -3 ); + } +}/* * Increases the refcount for a range of clusters in a given refcount table.
@@ -1161,12 +1175,13 @@ static int inc_refcounts(BlockDriverState *bs, k = cluster_offset >> s->cluster_bits;if (k >= *refcount_table_size) {int64_t old_refcount_table_size = *refcount_table_size; +size_t new_byte_size;void *new_refcount_table; *refcount_table_size = k +1 ; - new_refcount_table =g_try_realloc (*refcount_table, - *refcount_table_size * - s->refcount_bits /8 ); + new_byte_size =refcount_array_byte_size (s, *refcount_table_size); + + new_refcount_table =g_try_realloc (*refcount_table, new_byte_size);if (!new_refcount_table) { *refcount_table_size = old_refcount_table_size; res->check_errors++;
@@ -1547,6 +1562,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,if (fix &BDRV_FIX_ERRORS ) {int64_t old_nb_clusters = *nb_clusters; +size_t new_byte_size;void *new_refcount_table;if (offset >INT64_MAX - s->cluster_size) {
@@ -1567,9 +1583,9 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, *nb_clusters =size_to_clusters (s, size);assert (*nb_clusters >= old_nb_clusters); + new_byte_size =refcount_array_byte_size (s, *nb_clusters); new_refcount_table =g_try_realloc (*refcount_table, - *nb_clusters * - s->refcount_bits /8 ); + new_byte_size);if (!new_refcount_table) { *nb_clusters = old_nb_clusters; res->check_errors++;
@@ -1640,7 +1656,8 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,int ret;if (!*refcount_table) { - *refcount_table =g_try_malloc0 (*nb_clusters * s->refcount_bits /8 ); +size_t byte_size =refcount_array_byte_size (s, *nb_clusters); + *refcount_table =g_try_malloc0 (byte_size);if (*nb_clusters && *refcount_table ==NULL ) { res->check_errors++;return -ENOMEM ;
@@ -1805,6 +1822,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,* accordingly to append free clusters at the end of the image */ if (contiguous_free_clusters < cluster_count) {int64_t old_imrt_nb_clusters = *imrt_nb_clusters; +size_t new_byte_size;void *new_refcount_table;/* contiguous_free_clusters clusters are already empty at the image end;
@@ -1814,9 +1832,8 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,* may exceed old_imrt_nb_clusters if *first_free_cluster pointed beyond * the image end) */ *imrt_nb_clusters = cluster + cluster_count - contiguous_free_clusters; - new_refcount_table =g_try_realloc (*refcount_table, - *imrt_nb_clusters * - s->refcount_bits /8 ); + new_byte_size =refcount_array_byte_size (s, *imrt_nb_clusters); + new_refcount_table =g_try_realloc (*refcount_table, new_byte_size);if (!new_refcount_table) { *imrt_nb_clusters = old_imrt_nb_clusters;return -ENOMEM ;
@@ -1943,8 +1960,8 @@ write_refblocks:memcpy (on_disk_refblock, (void *)((uintptr_t )*refcount_table + (refblock_index << s->refcount_block_bits)), -MIN (s->refcount_block_size, *nb_clusters - refblock_start) - * s->refcount_bits /8 ); +refcount_array_byte_size (s,MIN (s->refcount_block_size, + *nb_clusters - refblock_start))); ret =bdrv_write (bs->file, refblock_offset /BDRV_SECTOR_SIZE , (void *)on_disk_refblock, s->cluster_sectors);
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 07/21] qcow2: Helper for refcount array size calculation
Quoted 19 lines · Max Reitz
Add a helper function which correctly calculates the byte size of a refcount array for any refcount order, and use that function.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 39 ++++++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 11 deletions(-)block/qcow2-refcount.c +3
-0@@ -1132,6 +1132,20 @@ fail:/* refcount checking functions */ +static size_t refcount_array_byte_size (BDRVQcowState *s,uint64_t entries) +{ +if (s->refcount_order <3 ) {
block/qcow2-refcount.c fail: · line 1147
Quoted 10 lines
block/qcow2-refcount.c +10
-0+/* sub-byte width */ +int shift =3 - s->refcount_order; +return (entries + (1 << shift) -1 ) >> shift; + }else if (s->refcount_order ==3 ) { +/* byte width */ +return entries; + }else { +/* multiple bytes wide */ +return entries << (s->refcount_order -3 ); + }
A comment proving why this can't overflow might be nice (if I analyzed correctly, entries will be computed by file size / clusters, and in the worst case, the smallest cluster and largest refcount_order results in '(size >> 9) << (6 - 3)' which is still safe).
Quoted 4 lines
block/qcow2-refcount.c +0
-0@@ -1161,12 +1175,13 @@ static int inc_refcounts(BlockDriverState *bs, k = cluster_offset >> s->cluster_bits;if (k >= *refcount_table_size) {int64_t old_refcount_table_size = *refcount_table_size;
block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1184
Quoted 10 lines
block/qcow2-refcount.c +4
-3+size_t new_byte_size;void *new_refcount_table; *refcount_table_size = k +1 ; - new_refcount_table =g_try_realloc (*refcount_table, - *refcount_table_size * - s->refcount_bits /8 ); + new_byte_size =refcount_array_byte_size (s, *refcount_table_size); + + new_refcount_table =g_try_realloc (*refcount_table, new_byte_size);
Yay - this addresses one of my possible overflow comments on 6/21.
I wonder if the series would have less churn if you rearranged this patch to come before 6/21.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 07/21] qcow2: Helper for refcount array size calculation
Quoted 26 lines · Eric Blake
Quoted 25 lines · Max Reitz
Add a helper function which correctly calculates the byte size of a refcount array for any refcount order, and use that function.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 39 ++++++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 11 deletions(-)block/qcow2-refcount.c +9
-0@@ -1132,6 +1132,20 @@ fail:/* refcount checking functions */ +static size_t refcount_array_byte_size (BDRVQcowState *s,uint64_t entries) +{ +if (s->refcount_order <3 ) { +/* sub-byte width */ +int shift =3 - s->refcount_order; +return (entries + (1 << shift) -1 ) >> shift; + }else if (s->refcount_order ==3 ) { +/* byte width */ +return entries;
Quoted 8 lines
block/qcow2-refcount.c fail: · line 1147
Quoted 4 lines
block/qcow2-refcount.c +4
-0+ }else { +/* multiple bytes wide */ +return entries << (s->refcount_order -3 ); + }A comment proving why this can't overflow might be nice (if I analyzed correctly, entries will be computed by file size / clusters, and in the worst case, the smallest cluster and largest
refcount_orderresults in '(size >> 9) << (6 - 3)' which is still safe).
Yes, will do.
Quoted 16 lines
Quoted 4 lines
block/qcow2-refcount.c +0
-0@@ -1161,12 +1175,13 @@ static int inc_refcounts(BlockDriverState *bs, k = cluster_offset >> s->cluster_bits;if (k >= *refcount_table_size) {int64_t old_refcount_table_size = *refcount_table_size;block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1184
Quoted 10 lines
block/qcow2-refcount.c +4
-3+size_t new_byte_size;void *new_refcount_table; *refcount_table_size = k +1 ; - new_refcount_table =g_try_realloc (*refcount_table, - *refcount_table_size * - s->refcount_bits /8 ); + new_byte_size =refcount_array_byte_size (s, *refcount_table_size); + + new_refcount_table =g_try_realloc (*refcount_table, new_byte_size);Yay - this addresses one of my possible overflow comments on 6/21.
Quoted 2 lines
I wonder if the series would have less churn if you rearranged this patch to come before 6/21.
Why not, I'll add an __attribute__((used)) to it (which should be fine for the duration of a single patch).
Max
Re: [Qemu-devel] [PATCH 07/21] qcow2: Helper for refcount array size calculation
Quoted 54 lines · Max Reitz
Quoted 25 lines · Eric Blake
Quoted 24 lines · Max Reitz
Add a helper function which correctly calculates the byte size of a refcount array for any refcount order, and use that function.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 39 ++++++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 11 deletions(-)block/qcow2-refcount.c +8
-0@@ -1132,6 +1132,20 @@ fail:/* refcount checking functions */ +static size_t refcount_array_byte_size(BDRVQcowState *s,uint64_t entries) +{ +if (s->refcount_order <3 ) { +/* sub-byte width */ +int shift =3 - s->refcount_order; +return (entries + (1 << shift) -1 ) >> shift; + }else if (s->refcount_order ==3 ) { +/* byte width */ +return entries;Quoted 8 lines
block/qcow2-refcount.c fail: · line 1147
Quoted 4 lines
block/qcow2-refcount.c +4
-0+ }else { +/* multiple bytes wide */ +return entries << (s->refcount_order -3 ); + }A comment proving why this can't overflow might be nice (if I analyzed correctly, entries will be computed by file size / clusters, and in the worst case, the smallest cluster and largest
refcount_orderresults in '(size >> 9) << (6 - 3)' which is still safe).Yes, will do.
Quoted 17 lines
Quoted 5 lines
block/qcow2-refcount.c +1
-0@@ -1161,12 +1175,13 @@ static int inc_refcounts(BlockDriverState *bs, k = cluster_offset >> s->cluster_bits;if (k >= *refcount_table_size) {int64_t old_refcount_table_size = *refcount_table_size; +size_t new_byte_size;block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1185
Quoted 10 lines
block/qcow2-refcount.c +3
-3void *new_refcount_table; *refcount_table_size = k +1 ; - new_refcount_table =g_try_realloc (*refcount_table, - *refcount_table_size * - s->refcount_bits /8 ); + new_byte_size =refcount_array_byte_size (s, *refcount_table_size); + + new_refcount_table =g_try_realloc (*refcount_table, new_byte_size);Yay - this addresses one of my possible overflow comments on 6/21.
Quoted 4 lines
Quoted 2 lines
I wonder if the series would have less churn if you rearranged this patch to come before 6/21.
Why not, I'll add an __attribute__((used)) to it (which should be fine for the duration of a single patch).
I'm not sure why I thought that might be necessary. Of course it isn't.
Max
PATCH 8/21 qcow2: More helpers for refcount modification
In reply to Max ReitzAdd helper functions for getting and setting refcounts in a refcount array for any possible refcount order, and choose the correct one during refcount initialization.
--- block/qcow2-refcount.c | 143 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 141 insertions(+), 2 deletions(-)
block/qcow2-refcount.c +141 -2
@@ -32,10 +32,73 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs, int64_t offset, int64_t length, int addend, enum qcow2_discard_type type); +static uint64_t get_refcount_ro0 (const void *refcount_array,uint64_t index); +static uint64_t get_refcount_ro1 (const void *refcount_array,uint64_t index); +static uint64_t get_refcount_ro2 (const void *refcount_array,uint64_t index); +static uint64_t get_refcount_ro3 (const void *refcount_array,uint64_t index);static uint64_t get_refcount_ro4 (const void *refcount_array,uint64_t index); +static uint64_t get_refcount_ro5 (const void *refcount_array,uint64_t index); +static uint64_t get_refcount_ro6 (const void *refcount_array,uint64_t index); +static void set_refcount_ro0 (void *refcount_array,uint64_t index, +uint64_t value); +static void set_refcount_ro1 (void *refcount_array,uint64_t index, +uint64_t value); +static void set_refcount_ro2 (void *refcount_array,uint64_t index, +uint64_t value); +static void set_refcount_ro3 (void *refcount_array,uint64_t index, +uint64_t value);static void set_refcount_ro4 (void *refcount_array,uint64_t index,uint64_t value); +static void set_refcount_ro5 (void *refcount_array,uint64_t index, +uint64_t value); +static void set_refcount_ro6 (void *refcount_array,uint64_t index, +uint64_t value); + +static void get_refcount_functions (int refcount_order, +Qcow2GetRefcountFunc **get, +Qcow2SetRefcountFunc **set) +{ +switch (refcount_order) { +case 0 : + *get = &get_refcount_ro0; + *set = &set_refcount_ro0; +break ; + +case 1 : + *get = &get_refcount_ro1; + *set = &set_refcount_ro1; +break ; + +case 2 : + *get = &get_refcount_ro2; + *set = &set_refcount_ro2; +break ; + +case 3 : + *get = &get_refcount_ro3; + *set = &set_refcount_ro3; +break ; + +case 4 : + *get = &get_refcount_ro4; + *set = &set_refcount_ro4; +break ; + +case 5 : + *get = &get_refcount_ro5; + *set = &set_refcount_ro5; +break ; + +case 6 : + *get = &get_refcount_ro6; + *set = &set_refcount_ro6; +break ; + +default : +abort (); + } +}/*********************************************************/
@@ -47,8 +110,8 @@ int qcow2_refcount_init(BlockDriverState *bs)unsigned int refcount_table_size2, i;int ret; - s->get_refcount = &get_refcount_ro4; - s->set_refcount = &set_refcount_ro4; +get_refcount_functions (s->refcount_order, + &s->get_refcount, &s->set_refcount);assert (s->refcount_table_size <=INT_MAX /sizeof (uint64_t )); refcount_table_size2 = s->refcount_table_size *sizeof (uint64_t );
@@ -80,6 +143,59 @@ void qcow2_refcount_close(BlockDriverState *bs) } +static uint64_t get_refcount_ro0 (const void *refcount_array,uint64_t index) +{ +return (((const uint8_t *)refcount_array)[index /8 ] >> (index %8 )) &0x1 ; +} + +static void set_refcount_ro0 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>1 )); + ((uint8_t *)refcount_array)[index /8 ] &= ~(0x1 << (index %8 )); + ((uint8_t *)refcount_array)[index /8 ] |= value << (index %8 ); +} + +static uint64_t get_refcount_ro1 (const void *refcount_array,uint64_t index) +{ +return (((const uint8_t *)refcount_array)[index /4 ] >> (2 * (index %4 ))) + &0x3 ; +} + +static void set_refcount_ro1 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>2 )); + ((uint8_t *)refcount_array)[index /4 ] &= ~(0x3 << (2 * (index %4 ))); + ((uint8_t *)refcount_array)[index /4 ] |= value << (2 * (index %4 )); +} + +static uint64_t get_refcount_ro2 (const void *refcount_array,uint64_t index) +{ +return (((const uint8_t *)refcount_array)[index /2 ] >> (4 * (index %2 ))) + &0xf ; +} + +static void set_refcount_ro2 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>4 )); + ((uint8_t *)refcount_array)[index /2 ] &= ~(0xf << (4 * (index %2 ))); + ((uint8_t *)refcount_array)[index /2 ] |= value << (4 * (index %2 )); +} + +static uint64_t get_refcount_ro3 (const void *refcount_array,uint64_t index) +{ +return ((const uint8_t *)refcount_array)[index]; +} + +static void set_refcount_ro3 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>8 )); + ((uint8_t *)refcount_array)[index] = value; +} +static uint64_t get_refcount_ro4 (const void *refcount_array,uint64_t index) {return be16_to_cpu (((const uint16_t *)refcount_array)[index]);
@@ -92,6 +208,29 @@ static void set_refcount_ro4(void *refcount_array, uint64_t index, ((uint16_t *)refcount_array)[index] =cpu_to_be16 (value); } +static uint64_t get_refcount_ro5 (const void *refcount_array,uint64_t index) +{ +return be32_to_cpu (((const uint32_t *)refcount_array)[index]); +} + +static void set_refcount_ro5 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>32 )); + ((uint32_t *)refcount_array)[index] =cpu_to_be32 (value); +} + +static uint64_t get_refcount_ro6 (const void *refcount_array,uint64_t index) +{ +return be64_to_cpu (((const uint64_t *)refcount_array)[index]); +} + +static void set_refcount_ro6 (void *refcount_array,uint64_t index, +uint64_t value) +{ + ((uint64_t *)refcount_array)[index] =cpu_to_be64 (value); +} + static int load_refcount_block(BlockDriverState *bs, int64_t refcount_block_offset,
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 08/21] qcow2: More helpers for refcount modification
Quoted 10 lines · Max Reitz
Add helper functions for getting and setting refcounts in a refcount array for any possible refcount order, and choose the correct one during refcount initialization.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 143 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 141 insertions(+), 2 deletions(-)
block/qcow2-refcount.c static void set_refcount_ro4(void *refcount_array, uint64_t index, · line 226
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ +static uint64_t get_refcount_ro6 (const void *refcount_array,uint64_t index) +{ +return be64_to_cpu (((const uint64_t *)refcount_array)[index]); +}
Should this return int64_t and error out if the user ever exceeded 2**63 via image fuzzing?
block/qcow2-refcount.c static void set_refcount_ro4(void *refcount_array, uint64_t index, · line 232
Quoted 6 lines
block/qcow2-refcount.c +6
-0+ +static void set_refcount_ro6 (void *refcount_array,uint64_t index, +uint64_t value) +{ + ((uint64_t *)refcount_array)[index] =cpu_to_be64 (value); +}
Should this assert that value <= INT64_MAX, since that's what the rest of the code caps it to?
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 08/21] qcow2: More helpers for refcount modification
Quoted 10 lines · Eric Blake
Quoted 9 lines · Max Reitz
Add helper functions for getting and setting refcounts in a refcount array for any possible refcount order, and choose the correct one during refcount initialization.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 143 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 141 insertions(+), 2 deletions(-)
Quoted 7 lines
block/qcow2-refcount.c static void set_refcount_ro4(void *refcount_array, uint64_t index, · line 226
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ +static uint64_t get_refcount_ro6 (const void *refcount_array,uint64_t index) +{ +return be64_to_cpu (((const uint64_t *)refcount_array)[index]); +}Should this return
int64_tand error out if the user ever exceeded 2**63 via image fuzzing?
I don't know. It's nice that these helper functions cannot return an error and thus we don't have to check for a error. I think checking that the value didn't overflow in qcow2_get_refcount() should be sufficient and relieves the other callers (mainly the image check for its in-memory refcount table/array) which know that the value cannot overflow from error checking.
Although I did forget an overflow check after the call to get_refcount() in update_refcount_discard().
Quoted 8 lines
block/qcow2-refcount.c static void set_refcount_ro4(void *refcount_array, uint64_t index, · line 232
Quoted 6 lines
block/qcow2-refcount.c +6
-0+ +static void set_refcount_ro6 (void *refcount_array,uint64_t index, +uint64_t value) +{ + ((uint64_t *)refcount_array)[index] =cpu_to_be64 (value); +}Should this assert that value <=
INT64_MAX, since that's what the rest of the code caps it to?
Yes, that it should most certainly do.
Max
PATCH 9/21 qcow2: Open images with refcount order != 4
In reply to Max ReitzNo longer refuse to open images with a different refcount entry width than 16 bits; only reject images with a refcount width larger than 64 bits (which is prohibited by the specification).
--- block/qcow2.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
block/qcow2.c +4 -4
@@ -677,10 +677,10 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, }/* Check support for various header values */ -if (header.refcount_order !=4 ) { -report_unsupported (bs, errp,"%d bit reference counts" , -1 << header.refcount_order); - ret = -ENOTSUP ; +if (header.refcount_order >6 ) { +error_setg (errp,"Reference count entry width too large (%i bit); may " +"not exceed 64 bit" ,1 << header.refcount_order); + ret = -EINVAL ; goto fail; } s->refcount_order = header.refcount_order;
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 09/21] qcow2: Open images with refcount order != 4
Quoted 14 lines · Max Reitz
No longer refuse to open images with a different refcount entry width than 16 bits; only reject images with a refcount width larger than 64 bits (which is prohibited by the specification).
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)block/qcow2.c +0
-0
block/qcow2.c static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, · line 682
Quoted 10 lines
block/qcow2.c +3
-4@@ -677,10 +677,10 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, }/* Check support for various header values */ -if (header.refcount_order !=4 ) { -report_unsupported (bs, errp,"%d bit reference counts" , -1 << header.refcount_order); - ret = -ENOTSUP ; +if (header.refcount_order >6 ) { +error_setg (errp,"Reference count entry width too large (%i bit); may " +"not exceed 64 bit" ,1 << header.refcount_order);
Overflows if I fuzz an image to put 32 or larger into header.refcount_order. It may be better to just tweak the error message to state that the order cannot exceed 6, rather than trying to display the actual bit width that the user is requesting, as then you avoid the '1 << problem'.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 09/21] qcow2: Open images with refcount order != 4
Quoted 17 lines · Eric Blake
Quoted 16 lines · Max Reitz
No longer refuse to open images with a different refcount entry width than 16 bits; only reject images with a refcount width larger than 64 bits (which is prohibited by the specification).
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)block/qcow2.c +0
-0@@ -677,10 +677,10 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, }
Quoted 13 lines
block/qcow2.c static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, · line 682
Quoted 8 lines
block/qcow2.c +3
-4/* Check support for various header values */ -if (header.refcount_order !=4 ) { -report_unsupported (bs, errp,"%d bit reference counts" , -1 << header.refcount_order); - ret = -ENOTSUP ; +if (header.refcount_order >6 ) { +error_setg (errp,"Reference count entry width too large (%i bit); may " +"not exceed 64 bit" ,1 << header.refcount_order);Overflows if I fuzz an image to put 32 or larger into header.refcount_order. It may be better to just tweak the error message to state that the order cannot exceed 6, rather than trying to display the actual bit width that the user is requesting, as then you avoid the '1 << problem'.
Yes, probably. Or we display it in binary: "0b1%0*i", header.refcount_order, 0 (which works since refcount_order is guaranteed to exceed 6).
Joking aside, yes, I'll omit the theoretical refcount width.
Max
PATCH 10/21 qcow2: refcount_order parameter for qcow2_create2
In reply to Max ReitzAdd a refcount_order parameter to qcow2_create2(), use that value for the image header and for calculating the size required for preallocation.
For now, always pass 4.
--- block/qcow2.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-)
block/qcow2.c +30 -11
@@ -1775,7 +1775,7 @@ static int preallocate(BlockDriverState *bs)static int qcow2_create2 (const char *filename,int64_t total_size,const char *backing_file,const char *backing_format,int flags,size_t cluster_size,PreallocMode prealloc, -QemuOpts *opts,int version, +QemuOpts *opts,int version,int refcount_order,Error **errp) {/* Calculate cluster_bits */
@@ -1811,6 +1811,13 @@ static int qcow2_create2(const char *filename, int64_t total_size,int64_t meta_size =0 ;uint64_t nreftablee, nrefblocke, nl1e, nl2e;int64_t aligned_total_size =align_offset (total_size, cluster_size); +int refblock_bits, refblock_size; +/* refcount entry size in bytes */ +double rces = (1 << refcount_order) /8. ; + +/* see qcow2_open() */ + refblock_bits = cluster_bits - (refcount_order -3 ); + refblock_size =1 << refblock_bits;/* header: 1 cluster */ meta_size += cluster_size;
@@ -1835,20 +1842,20 @@ static int qcow2_create2(const char *filename, int64_t total_size,* c = cluster size * y1 = number of refcount blocks entries * y2 = meta size including everything +* rces = refcount entry size in bytes * then, * y1 = (y2 + a)/c -* y2 = y1 * sizeof(u16) + y1 * sizeof(u16) * sizeof(u64) / c + m +* y2 = y1 * rces + y1 * rces * sizeof(u64) / c + m * we can get y1: -* y1 = (a + m) / (c - sizeof(u16) - sizeof(u16) * sizeof(u64) / c) +* y1 = (a + m) / (c - rces - rces * sizeof(u64) / c) */ - nrefblocke = (aligned_total_size + meta_size + cluster_size) / - (cluster_size -sizeof (uint16_t ) - -1.0 *sizeof (uint16_t ) *sizeof (uint64_t ) / cluster_size); - nrefblocke =align_offset (nrefblocke, cluster_size /sizeof (uint16_t )); - meta_size += nrefblocke *sizeof (uint16_t ); + nrefblocke = (aligned_total_size + meta_size + cluster_size) + / (cluster_size - rces - rces *sizeof (uint64_t ) + / cluster_size); + meta_size +=DIV_ROUND_UP (nrefblocke, refblock_size) * cluster_size;/* total size of refcount tables */ - nreftablee = nrefblocke *sizeof (uint16_t ) / cluster_size; + nreftablee = nrefblocke / refblock_size; nreftablee =align_offset (nreftablee, cluster_size /sizeof (uint64_t )); meta_size += nreftablee *sizeof (uint64_t );
@@ -1883,7 +1890,7 @@ static int qcow2_create2(const char *filename, int64_t total_size, .l1_size = cpu_to_be32(0), .refcount_table_offset = cpu_to_be64(cluster_size), .refcount_table_clusters = cpu_to_be32(1), - .refcount_order = cpu_to_be32(4), + .refcount_order = cpu_to_be32(refcount_order), .header_length = cpu_to_be32(sizeof(*header)), };
@@ -2003,6 +2010,7 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)size_t cluster_size =DEFAULT_CLUSTER_SIZE ;PreallocMode prealloc;int version =3 ; +int refcount_width =16 , refcount_order;Error *local_err =NULL ;int ret;
@@ -2057,8 +2065,19 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) goto finish; } +if (version <3 && refcount_width !=16 ) { +error_setg (errp,"Different refcount widths than 16 bits require " +"compatibility level 1.1 or above (use compat=1.1 or " +"greater)" ); + ret = -EINVAL ; + goto finish; + } + + refcount_order =ffs (refcount_width) -1 ; + ret =qcow2_create2 (filename, size, backing_file, backing_fmt, flags, - cluster_size, prealloc, opts, version, &local_err); + cluster_size, prealloc, opts, version, refcount_order, + &local_err);if (local_err) {error_propagate (errp, local_err); }
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 10/21] qcow2: refcount_order parameter for qcow2_create2
Quoted 13 lines · Max Reitz
Add a
refcount_orderparameter toqcow2_create2(), use that value for the image header and for calculating the size required for preallocation.For now, always pass 4.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-)
block/qcow2.c static int qcow2_create2(const char *filename, int64_t total_size, · line 1816
Quoted 6 lines
block/qcow2.c +3
-0@@ -1811,6 +1811,13 @@ static int qcow2_create2(const char *filename, int64_t total_size,int64_t meta_size =0 ;uint64_t nreftablee, nrefblocke, nl1e, nl2e;int64_t aligned_total_size =align_offset (total_size, cluster_size); +int refblock_bits, refblock_size; +/* refcount entry size in bytes */ +double rces = (1 << refcount_order) /8. ;
Would float be any simpler than double?
Quoted 8 lines
block/qcow2.c +4
-0+ +/* see qcow2_open() */ + refblock_bits = cluster_bits - (refcount_order -3 ); + refblock_size =1 << refblock_bits;/* header: 1 cluster */ meta_size += cluster_size;
block/qcow2.c static int qcow2_create2(const char *filename, int64_t total_size, · line 1848
Quoted 8 lines
block/qcow2.c +2
-1@@ -1835,20 +1842,20 @@ static int qcow2_create2(const char *filename, int64_t total_size,* c = cluster size * y1 = number of refcount blocks entries * y2 = meta size including everything +* rces = refcount entry size in bytes * then, * y1 = (y2 + a)/c -* y2 = y1 * sizeof(u16) + y1 * sizeof(u16) * sizeof(u64) / c + m +* y2 = y1 * rces + y1 * rces * sizeof(u64) / c + m
Hmm. This changes from integral to floating point. Are we going to suffer from any rounding problems? I guess you want double to ensure maximum precision; but whereas earlier patches were limited with refcount_order of 6 to 2^63, this now limits us around 2^53 if we are still trying to be accurate.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 10/21] qcow2: refcount_order parameter for qcow2_create2
Quoted 12 lines · Eric Blake
Quoted 11 lines · Max Reitz
Add a
refcount_orderparameter toqcow2_create2(), use that value for the image header and for calculating the size required for preallocation.For now, always pass 4.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-)
Quoted 8 lines
block/qcow2.c static int qcow2_create2(const char *filename, int64_t total_size, · line 1816
Quoted 7 lines
block/qcow2.c +3
-0@@ -1811,6 +1811,13 @@ static int qcow2_create2(const char *filename, int64_t total_size,int64_t meta_size =0 ;uint64_t nreftablee, nrefblocke, nl1e, nl2e;int64_t aligned_total_size =align_offset (total_size, cluster_size); +int refblock_bits, refblock_size; +/* refcount entry size in bytes */ +double rces = (1 << refcount_order) /8. ;Would float be any simpler than double?
Maybe. I'm generally in favor of using float whenever possible, but I don't think it's necessary here. This is really not a performance issue here and having more precision is probably better.
Quoted 5 lines
Quoted 5 lines
block/qcow2.c +4
-0+ +/* see qcow2_open() */ + refblock_bits = cluster_bits - (refcount_order -3 ); + refblock_size =1 << refblock_bits;
Quoted 13 lines
block/qcow2.c static int qcow2_create2(const char *filename, int64_t total_size, · line 1848
Quoted 11 lines
block/qcow2.c +2
-1/* header: 1 cluster */ meta_size += cluster_size;@@ -1835,20 +1842,20 @@ static int qcow2_create2(const char *filename, int64_t total_size,* c = cluster size * y1 = number of refcount blocks entries * y2 = meta size including everything +* rces = refcount entry size in bytes * then, * y1 = (y2 + a)/c -* y2 = y1 * sizeof(u16) + y1 * sizeof(u16) * sizeof(u64) / c + m +* y2 = y1 * rces + y1 * rces * sizeof(u64) / c + m Hmm. This changes from integral to floating point. Are we going to suffer from any rounding problems?
We are already suffering from rounding problems. See the actual code below this comment.
Quoted 4 lines
I guess you want double to ensure
maximum precision; but whereas earlier patches were limited withrefcount_orderof 6 to 2^63, this now limits us around 2^53 if we are still trying to be accurate.
We are not trying to be accurate, which is one of the reasons I rejected the first version of the patch which introduced this code (though that version had more issues which had been fixed in the final version). If you remember, I did write a very similar function which only used integral types in order not to suffer from precision problems. Some version of this code used that function, but then Kevin rejected my patch because it was too complicated, and this was taken in with the argument of not having to be really exact when it comes to preallocation: Being close enough should be fine.
So, in short, this code did not try to be exact from the beginning (it did use floating point arithmetic before this patch, see the "1.0 *"). If we want to be exact, there's still the code which I have the LaTeX PDF for. ;-)
Max
PATCH 11/21 iotests: Prepare for refcount_width option
In reply to Max ReitzSome tests do not work well with certain refcount widths (i.e. you cannot create internal snapshots with refcount_width=1), so make those widths unsupported.
Furthermore, add another filter to _filter_img_create in common.filter which filters out the refcount_width value.
--- tests/qemu-iotests/007 | 4 ++++ tests/qemu-iotests/015 | 1 + tests/qemu-iotests/026 | 11 +++++++++++ tests/qemu-iotests/029 | 1 + tests/qemu-iotests/051 | 1 + tests/qemu-iotests/058 | 1 + tests/qemu-iotests/067 | 7 +++++++ tests/qemu-iotests/079 | 1 + tests/qemu-iotests/080 | 1 + tests/qemu-iotests/089 | 7 +++++++ tests/qemu-iotests/090 | 1 + tests/qemu-iotests/108 | 6 ++++++ tests/qemu-iotests/common.filter | 3 ++- 13 files changed, 44 insertions(+), 1 deletion(-)
tests/qemu-iotests/007 +4 -0
@@ -43,6 +43,10 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _supported_os Linux +# refcount_width must be at least 4 bits so we can create ten internal snapshots +# (1 bit supports none, 2 bits support three, 4 bits support 15) +_unsupported_imgopts 'refcount_width=1[^0-9]' \ + 'refcount_width=2[^0-9]' echo echo "creating image"
tests/qemu-iotests/015 +1 -0
@@ -43,6 +43,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _supported_os Linux +_unsupported_imgopts 'refcount_width=1[^0-9]' echo echo "creating image"
tests/qemu-iotests/026 +11 -0
@@ -46,6 +46,17 @@ _supported_proto file _supported_os Linux _default_cache_mode "writethrough" _supported_cache_modes "writethrough" "none" +# The refcount table tests expect a certain minimum width for refcount entries +# (so that the refcount table actually needs to grow); that minimum is 16 bits, +# being the default refcount entry width. +# 32 and 64 bits do not work either, however, due to different leaked cluster +# count on error. +_unsupported_imgopts 'refcount_width=1[^0-9]' \ + 'refcount_width=2[^0-9]' \ + 'refcount_width=4[^0-9]' \ + 'refcount_width=8[^0-9]' \ + 'refcount_width=32[^0-9]' \ + 'refcount_width=64[^0-9]' \ echo "Errors while writing 128 kB" echo
tests/qemu-iotests/029 +1 -0
@@ -44,6 +44,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _supported_os Linux +_unsupported_imgopts 'refcount_width=1[^0-9]' offset_size=24 offset_l1_size=36
tests/qemu-iotests/051 +1 -0
@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +_unsupported_imgopts refcount_width function do_run_qemu() {
tests/qemu-iotests/058 +1 -0
@@ -88,6 +88,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _require_command QEMU_NBD +_unsupported_imgopts 'refcount_width=1[^0-9]' echo echo "== preparing image =="
tests/qemu-iotests/067 +7 -0
@@ -35,6 +35,13 @@ status=1 # failure is the default! _supported_fmt qcow2 _supported_proto file _supported_os Linux +# Because this would change the output of query-block +_unsupported_imgopts 'refcount_width=1[^0-9]' \ + 'refcount_width=2[^0-9]' \ + 'refcount_width=4[^0-9]' \ + 'refcount_width=8[^0-9]' \ + 'refcount_width=32[^0-9]' \ + 'refcount_width=64[^0-9]' function do_run_qemu() {
tests/qemu-iotests/079 +1 -0
@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file nfs _supported_os Linux +_unsupported_imgopts refcount_width function test_qemu_img() {
tests/qemu-iotests/080 +1 -0
@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +_unsupported_imgopts 'refcount_width=1[^0-9]' header_size=104
tests/qemu-iotests/089 +7 -0
@@ -41,6 +41,13 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +# Because this would change the output of qemu_io -c info +_unsupported_imgopts 'refcount_width=1[^0-9]' \ + 'refcount_width=2[^0-9]' \ + 'refcount_width=4[^0-9]' \ + 'refcount_width=8[^0-9]' \ + 'refcount_width=32[^0-9]' \ + 'refcount_width=64[^0-9]' # Using an image filename containing quotation marks will render the JSON data # below invalid. In that case, we have little choice but simply not to run this
tests/qemu-iotests/090 +1 -0
@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file nfs _supported_os Linux +_unsupported_imgopts 'refcount_width=1[^0-9]' IMG_SIZE=128K
tests/qemu-iotests/108 +6 -0
@@ -43,6 +43,12 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +_unsupported_imgopts 'refcount_width=1[^0-9]' \ + 'refcount_width=2[^0-9]' \ + 'refcount_width=4[^0-9]' \ + 'refcount_width=8[^0-9]' \ + 'refcount_width=32[^0-9]' \ + 'refcount_width=64[^0-9]' echo echo '=== Repairing an image without any refcount table ==='
tests/qemu-iotests/common.filter +2 -1
@@ -189,7 +189,8 @@ _filter_img_create() -e "s# block_size=[0-9]\\+##g" \ -e "s# block_state_zero=\\(on\\|off\\)##g" \ -e "s# log_size=[0-9]\\+##g" \ - -e "s/archipelago:a/TEST_DIR\//g" + -e "s/archipelago:a/TEST_DIR\//g" \ + -e "s# refcount_width=[0-9]\\+##g" } _filter_img_info()
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 11/21] iotests: Prepare for refcount_width option
Quoted 29 lines · Max Reitz
Some tests do not work well with certain refcount widths (i.e. you cannot create internal snapshots with refcount_width=1), so make those widths unsupported.
Furthermore, add another filter to
_filter_img_createin common.filter which filters out therefcount_widthvalue.Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/007 | 4 ++++ tests/qemu-iotests/015 | 1 + tests/qemu-iotests/026 | 11 +++++++++++ tests/qemu-iotests/029 | 1 + tests/qemu-iotests/051 | 1 + tests/qemu-iotests/058 | 1 + tests/qemu-iotests/067 | 7 +++++++ tests/qemu-iotests/079 | 1 + tests/qemu-iotests/080 | 1 + tests/qemu-iotests/089 | 7 +++++++ tests/qemu-iotests/090 | 1 + tests/qemu-iotests/108 | 6 ++++++ tests/qemu-iotests/common.filter | 3 ++- 13 files changed, 44 insertions(+), 1 deletion(-)tests/qemu-iotests/007 +0
-0
tests/qemu-iotests/007 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 47
Quoted 5 lines
tests/qemu-iotests/007 +2
-0@@ -43,6 +43,10 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _supported_os Linux +# refcount_width must be at least 4 bits so we can create ten internal snapshots +# (1 bit supports none, 2 bits support three, 4 bits support 15)
Feels like an off-by-one comment. A width of 1 bit support a max refcount of 1 (therefore no snapshots), a width of 2 bits supports a max refcount of 3 (therefore 2 snapshots in addition to the original), a width of 4 bits supports a max refcount of 15 (therefore only 14 snapshots).
tests/qemu-iotests/007 status=1 # failure is the default! · line 44
Quoted 12 lines
tests/qemu-iotests/007 +8
-0+++ b/tests/qemu-iotests/067@@ -35,6 +35,13 @@ status=1 # failure is the default! _supported_fmt qcow2 _supported_proto file _supported_os Linux +# Because this would change the output of query-block +_unsupported_imgopts 'refcount_width=1[^0-9]' \ + 'refcount_width=2[^0-9]' \ + 'refcount_width=4[^0-9]' \ + 'refcount_width=8[^0-9]' \ + 'refcount_width=32[^0-9]' \ + 'refcount_width=64[^0-9]'
It might be more compact to exploit globbing and just say:
_unsupported_imgopts 'refcount_width=?[^6]'
which leaves refcount_width=16 as the only pattern that doesn't match the glob. But that feels more fragile, so I can live with your longer list.
tests/qemu-iotests/007 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 45
Quoted 7 lines
tests/qemu-iotests/007 +3
-0+++ b/tests/qemu-iotests/089@@ -41,6 +41,13 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +# Because this would change the output of qemu_io -c info +_unsupported_imgopts 'refcount_width=1[^0-9]' \
I like how you give reasons for some tests...
tests/qemu-iotests/007 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 44
Quoted 6 lines
tests/qemu-iotests/007 +2
-0+++ b/tests/qemu-iotests/090@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file nfs _supported_os Linux +_unsupported_imgopts 'refcount_width=1[^0-9]'
...so why not do it for all tests?
At any rate, the patch makes sense, so whether or not you tweak comments,
I'm assuming that later in the series you add a test that explicitly covers the error message given when a refcount_order=0 (width=1) image is attempted to be used with snapshots, since that will fail (internal snapshots are simply not possible without a refcount that can't exceed 1).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 11/21] iotests: Prepare for refcount_width option
Quoted 25 lines · Eric Blake
Quoted 24 lines · Max Reitz
Some tests do not work well with certain refcount widths (i.e. you cannot create internal snapshots with refcount_width=1), so make those widths unsupported.
Furthermore, add another filter to
_filter_img_createin common.filter which filters out therefcount_widthvalue.Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/007 | 4 ++++ tests/qemu-iotests/015 | 1 + tests/qemu-iotests/026 | 11 +++++++++++ tests/qemu-iotests/029 | 1 + tests/qemu-iotests/051 | 1 + tests/qemu-iotests/058 | 1 + tests/qemu-iotests/067 | 7 +++++++ tests/qemu-iotests/079 | 1 + tests/qemu-iotests/080 | 1 + tests/qemu-iotests/089 | 7 +++++++ tests/qemu-iotests/090 | 1 + tests/qemu-iotests/108 | 6 ++++++ tests/qemu-iotests/common.filter | 3 ++- 13 files changed, 44 insertions(+), 1 deletion(-)
Quoted 14 lines
tests/qemu-iotests/007 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 47
Quoted 10 lines
tests/qemu-iotests/007 +2
-0@@ -43,6 +43,10 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _supported_os Linux +# refcount_width must be at least 4 bits so we can create ten internal snapshots +# (1 bit supports none, 2 bits support three, 4 bits support 15)Feels like an off-by-one comment. A width of 1 bit support a max refcount of 1 (therefore no snapshots), a width of 2 bits supports a max refcount of 3 (therefore 2 snapshots in addition to the original), a width of 4 bits supports a max refcount of 15 (therefore only 14 snapshots).
Telling you how to correctly get to the right maximum refcount and then getting it wrong myself is a bit embarrassing...
Quoted 14 lines
tests/qemu-iotests/007 status=1 # failure is the default! · line 44
Quoted 12 lines
tests/qemu-iotests/007 +8
-0+++ b/tests/qemu-iotests/067@@ -35,6 +35,13 @@ status=1 # failure is the default! _supported_fmt qcow2 _supported_proto file _supported_os Linux +# Because this would change the output of query-block +_unsupported_imgopts 'refcount_width=1[^0-9]' \ + 'refcount_width=2[^0-9]' \ + 'refcount_width=4[^0-9]' \ + 'refcount_width=8[^0-9]' \ + 'refcount_width=32[^0-9]' \ + 'refcount_width=64[^0-9]'It might be more compact to exploit globbing and just say:
Quoted 4 lines
_unsupported_imgopts'refcount_width=?[^6]'which leaves refcount_width=16 as the only pattern that doesn't match the glob. But that feels more fragile, so I can live with your longer list.
Well, maybe using ?[^6] is even better because this isn't about ruling out the options 1, 2, 4, 8, 32 and 64, but rather only allowing 16. Thus, using ?[^6] seems more explicit.
Quoted 9 lines
tests/qemu-iotests/007 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 45
Quoted 7 lines
tests/qemu-iotests/007 +3
-0+++ b/tests/qemu-iotests/089@@ -41,6 +41,13 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +# Because this would change the output of qemu_io -c info +_unsupported_imgopts 'refcount_width=1[^0-9]' \I like how you give reasons for some tests...
Quoted 7 lines
tests/qemu-iotests/007 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 44
Quoted 6 lines
tests/qemu-iotests/007 +2
-0+++ b/tests/qemu-iotests/090@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file nfs _supported_os Linux +_unsupported_imgopts 'refcount_width=1[^0-9]'...so why not do it for all tests?
Because the ones I didn't give reasons for are the ones I spotted first, so they seemed obvious to me. ;-) (I wondered the same thing when looking through the patches before submitting them, but decided to just leave it at that)
I'll add a comment for every test.
Quoted 8 lines
At any rate, the patch makes sense, so whether or not you tweak comments,
Reviewed-by Eric Blake <[email protected]>I'm assuming that later in the series you add a test that explicitly covers the error message given when a refcount_order=0 (width=1) image is attempted to be used with snapshots, since that will fail (internal snapshots are simply not possible without a refcount that can't exceed 1).
Well, as you yourself explained, they are indeed possible if done right (immediate COW). But that'll go into another series.
Max
PATCH 12/21 qcow2: Allow creation with refcount order != 4
In reply to Max ReitzAdd a creation option to qcow2 for setting the refcount order of images to be created, and respect that option's value.
This breaks some test outputs, fix them.
--- block/qcow2.c | 20 ++++++++ include/block/block_int.h | 1 + tests/qemu-iotests/049.out | 112 ++++++++++++++++++++++----------------------- tests/qemu-iotests/079.out | 18 ++++---- tests/qemu-iotests/082.out | 41 ++++++++++++++--- tests/qemu-iotests/085.out | 38 +++++++-------- 6 files changed, 139 insertions(+), 91 deletions(-)
block/qcow2.c +20 -0
@@ -2065,6 +2065,17 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) goto finish; } + refcount_width =qemu_opt_get_number_del (opts,BLOCK_OPT_REFCOUNT_WIDTH , + refcount_width); +if (refcount_width <=0 || refcount_width >64 || + !is_power_of_2 (refcount_width)) + { +error_setg (errp,"Refcount width must be a power of two and may not " +"exceed 64 bits" ); + ret = -EINVAL ; + goto finish; + } +if (version <3 && refcount_width !=16 ) {error_setg (errp,"Different refcount widths than 16 bits require " "compatibility level 1.1 or above (use compat=1.1 or "
@@ -2704,6 +2715,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, }else if (!strcmp (desc->name,"lazy_refcounts" )) { lazy_refcounts =qemu_opt_get_bool (opts,"lazy_refcounts" , lazy_refcounts); + }else if (!strcmp (desc->name,"refcount_width" )) { +error_report ("Cannot change refcount entry width" ); +return -ENOTSUP ; }else {/* if this assertion fails, this probably means a new option was * added without having it covered here */
@@ -2873,6 +2887,12 @@ static QemuOptsList qcow2_create_opts = { .help = "Postpone refcount updates", .def_value_str = "off" }, + { + .name = BLOCK_OPT_REFCOUNT_WIDTH, + .type = QEMU_OPT_NUMBER, + .help = "Width of a reference count entry in bits", + .def_value_str = "16" + }, {/* end of list */ } } };
include/block/block_int.h +1 -0
@@ -56,6 +56,7 @@#define BLOCK_OPT_ADAPTER_TYPE "adapter_type"#define BLOCK_OPT_REDUNDANCY "redundancy"#define BLOCK_OPT_NOCOW "nocow" +#define BLOCK_OPT_REFCOUNT_WIDTH "refcount_width" typedef struct BdrvTrackedRequest { BlockDriverState *bs;
tests/qemu-iotests/049.out +56 -56
254 lines not on this page · raw message
tests/qemu-iotests/079.out +9 -9
@@ -2,31 +2,31 @@ QA output created by 079 === Check option preallocation and cluster_size === qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=16384 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=16384 preallocation='metadata' lazy_refcounts=off +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=16384 preallocation='metadata' lazy_refcounts=off refcount_width=16 qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=32768 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=32768 preallocation='metadata' lazy_refcounts=off +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=32768 preallocation='metadata' lazy_refcounts=off refcount_width=16 qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=65536 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 preallocation='metadata' lazy_refcounts=off +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 preallocation='metadata' lazy_refcounts=off refcount_width=16 qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=131072 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=131072 preallocation='metadata' lazy_refcounts=off +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=131072 preallocation='metadata' lazy_refcounts=off refcount_width=16 qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=262144 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=262144 preallocation='metadata' lazy_refcounts=off +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=262144 preallocation='metadata' lazy_refcounts=off refcount_width=16 qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=524288 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=524288 preallocation='metadata' lazy_refcounts=off +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=524288 preallocation='metadata' lazy_refcounts=off refcount_width=16 qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=1048576 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=1048576 preallocation='metadata' lazy_refcounts=off +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=1048576 preallocation='metadata' lazy_refcounts=off refcount_width=16 qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=2097152 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=2097152 preallocation='metadata' lazy_refcounts=off +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=2097152 preallocation='metadata' lazy_refcounts=off refcount_width=16 qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=4194304 TEST_DIR/t.qcow2 4G qemu-img: TEST_DIR/t.qcow2: Cluster size must be a power of two between 512 and 2048k -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=4194304 preallocation='metadata' lazy_refcounts=off +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=4194304 preallocation='metadata' lazy_refcounts=off refcount_width=16 *** done
tests/qemu-iotests/082.out +34 -7
238 lines not on this page · raw message
tests/qemu-iotests/085.out +19 -19
@@ -11,7 +11,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 === Create a single snapshot on virtio0 === -Formatting 'TEST_DIR/1-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2.orig' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/1-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2.orig' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} === Invalid command - missing device and nodename ===
@@ -25,31 +25,31 @@ Formatting 'TEST_DIR/1-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file === Create several transactional group snapshots === -Formatting 'TEST_DIR/2-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/1-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/2-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/2-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/1-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/2-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/3-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/3-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/3-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/3-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/4-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/4-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/4-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/4-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/5-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/5-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/5-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/5-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/6-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/6-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/6-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/6-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/7-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/7-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/7-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/7-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/8-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/8-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/8-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/8-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/9-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/9-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/9-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/9-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/10-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/10-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/10-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/10-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} *** done
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 12/21] qcow2: Allow creation with refcount order != 4
Quoted 6 lines · Max Reitz
Add a creation option to qcow2 for setting the refcount order of images to be created, and respect that option's value.
This breaks some test outputs, fix them.
Signed-off-by Max Reitz <[email protected]>
Quoted 8 lines
--- block/qcow2.c | 20 ++++++++ include/block/block_int.h | 1 + tests/qemu-iotests/049.out | 112 ++++++++++++++++++++++----------------------- tests/qemu-iotests/079.out | 18 ++++---- tests/qemu-iotests/082.out | 41 ++++++++++++++--- tests/qemu-iotests/085.out | 38 +++++++-------- 6 files changed, 139 insertions(+), 91 deletions(-)
Is there any .json file that needs to be modified to allow this option to be set via QMP? I guess that can be a followup.
tests/qemu-iotests/049.out qemu-img: Image size must be less than 8 EiB! · line 112
Quoted 4 lines
qemu-img create -f qcow2 TEST_DIR/t.qcow2 -- 1kilobyte
tests/qemu-iotests/049.out +1
-1-qemu-img: Invalid image size specified! You may use k, M, G, T, P or E suffixes for +qemu-img: Invalid image size specified! You may use k, M, G, T, P or E suffixes for qemu-img: kilobytes, megabytes, gigabytes, terabytes, petabytes and exabytes.
Nice that you are also getting rid of trailing whitespace.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 12/21] qcow2: Allow creation with refcount order != 4
Quoted 15 lines · Eric Blake
Quoted 6 lines · Max Reitz
Add a creation option to qcow2 for setting the refcount order of images to be created, and respect that option's value.
This breaks some test outputs, fix them.
Signed-off-by Max Reitz <[email protected]>Quoted 8 lines
--- block/qcow2.c | 20 ++++++++ include/block/block_int.h | 1 + tests/qemu-iotests/049.out | 112 ++++++++++++++++++++++----------------------- tests/qemu-iotests/079.out | 18 ++++---- tests/qemu-iotests/082.out | 41 ++++++++++++++--- tests/qemu-iotests/085.out | 38 +++++++-------- 6 files changed, 139 insertions(+), 91 deletions(-)
Quoted 2 lines
Is there any .json file that needs to be modified to allow this option to be set via QMP? I guess that can be a followup.
Good point, but I can't find any JSON object which contains a "lazy-refcounts" (or "lazy_refcounts") field and which seems to be related to image creation (if "lazy-refcounts" is somewhere not relating to options for opening an existing image, "refcount-width" should be there, too). I guess there probably is no way of directly creating an image over QMP yet at all.
Quoted 5 lines
tests/qemu-iotests/049.out qemu-img: Image size must be less than 8 EiB! · line 112
Quoted 4 lines
qemu-img create -f qcow2 TEST_DIR/t.qcow2 -- 1kilobyte
tests/qemu-iotests/049.out +1
-1-qemu-img: Invalid image size specified! You may use k, M, G, T, P or E suffixes for +qemu-img: Invalid image size specified! You may use k, M, G, T, P or E suffixes for qemu-img: kilobytes, megabytes, gigabytes, terabytes, petabytes and exabytes.Nice that you are also getting rid of trailing whitespace.
That was actually unintentional. You seem fine with it, though, so I'll remove all trailing whitespace in all the test outputs I'm touching in v2.
Max
Quoted 1 line
Reviewed-by Eric Blake <[email protected]>
PATCH 13/21 block: Add opaque value to the amend CB
In reply to Max ReitzAdd an opaque value which is to be passed to the bdrv_amend_options() status callback.
--- block.c | 4 ++-- block/qcow2-cluster.c | 14 ++++++++------ block/qcow2.c | 9 +++++---- block/qcow2.h | 3 ++- include/block/block.h | 4 ++-- include/block/block_int.h | 3 ++- qemu-img.c | 5 +++-- 7 files changed, 24 insertions(+), 18 deletions(-)
block.c +2 -2
@@ -5790,12 +5790,12 @@ void bdrv_add_before_write_notifier(BlockDriverState *bs, }int bdrv_amend_options (BlockDriverState *bs,QemuOpts *opts, -BlockDriverAmendStatusCB *status_cb) +BlockDriverAmendStatusCB *status_cb,void *cb_opaque) {if (!bs->drv->bdrv_amend_options) {return -ENOTSUP ; } -return bs->drv->bdrv_amend_options (bs, opts, status_cb); +return bs->drv->bdrv_amend_options (bs, opts, status_cb, cb_opaque); }/* This function will be called by the bdrv_recurse_is_first_non_filter method
block/qcow2-cluster.c +8 -6
@@ -1620,7 +1620,8 @@ fail:static int expand_zero_clusters_in_l1 (BlockDriverState *bs,uint64_t *l1_table,int l1_size,int64_t *visited_l1_entries,int64_t l1_entries, -BlockDriverAmendStatusCB *status_cb) +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque) {BDRVQcowState *s = bs->opaque;bool is_active_l1 = (l1_table == s->l1_table);
@@ -1646,7 +1647,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,/* unallocated */ (*visited_l1_entries)++;if (status_cb) { -status_cb (bs, *visited_l1_entries, l1_entries); +status_cb (bs, *visited_l1_entries, l1_entries, cb_opaque); }continue ; }
@@ -1768,7 +1769,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, (*visited_l1_entries)++;if (status_cb) { -status_cb (bs, *visited_l1_entries, l1_entries); +status_cb (bs, *visited_l1_entries, l1_entries, cb_opaque); } }
@@ -1797,7 +1798,8 @@ fail:* qcow2 version which doesn't yet support metadata zero clusters. */ int qcow2_expand_zero_clusters (BlockDriverState *bs, -BlockDriverAmendStatusCB *status_cb) +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque) {BDRVQcowState *s = bs->opaque;uint64_t *l1_table =NULL ;
@@ -1814,7 +1816,7 @@ int qcow2_expand_zero_clusters(BlockDriverState *bs, ret =expand_zero_clusters_in_l1 (bs, s->l1_table, s->l1_size, &visited_l1_entries, l1_entries, - status_cb); + status_cb, cb_opaque);if (ret <0 ) { goto fail; }
@@ -1849,7 +1851,7 @@ int qcow2_expand_zero_clusters(BlockDriverState *bs, ret =expand_zero_clusters_in_l1 (bs, l1_table, s->snapshots[i].l1_size, &visited_l1_entries, l1_entries, - status_cb); + status_cb, cb_opaque);if (ret <0 ) { goto fail; }
block/qcow2.c +5 -4
@@ -2592,7 +2592,7 @@ static int qcow2_load_vmstate(BlockDriverState *bs, uint8_t *buf,* have to be removed. */ static int qcow2_downgrade (BlockDriverState *bs,int target_version, -BlockDriverAmendStatusCB *status_cb) +BlockDriverAmendStatusCB *status_cb,void *cb_opaque) {BDRVQcowState *s = bs->opaque;int current_version = s->qcow_version;
@@ -2641,7 +2641,7 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,/* clearing autoclear features is trivial */ s->autoclear_features =0 ; - ret =qcow2_expand_zero_clusters (bs, status_cb); + ret =qcow2_expand_zero_clusters (bs, status_cb, cb_opaque);if (ret <0 ) {return ret; }
@@ -2656,7 +2656,8 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version, }static int qcow2_amend_options (BlockDriverState *bs,QemuOpts *opts, -BlockDriverAmendStatusCB *status_cb) +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque) {BDRVQcowState *s = bs->opaque;int old_version = s->qcow_version, new_version = old_version;
@@ -2737,7 +2738,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,return ret; } }else { - ret =qcow2_downgrade (bs, new_version, status_cb); + ret =qcow2_downgrade (bs, new_version, status_cb, cb_opaque);if (ret <0 ) {return ret; }
block/qcow2.h +2 -1
@@ -551,7 +551,8 @@ int qcow2_discard_clusters(BlockDriverState *bs, uint64_t offset,int qcow2_zero_clusters (BlockDriverState *bs,uint64_t offset,int nb_sectors);int qcow2_expand_zero_clusters (BlockDriverState *bs, -BlockDriverAmendStatusCB *status_cb); +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque);/* qcow2-snapshot.c functions */ int qcow2_snapshot_create (BlockDriverState *bs,QEMUSnapshotInfo *sn_info);
include/block/block.h +2 -2
@@ -272,9 +272,9 @@ int bdrv_check(BlockDriverState *bs, BdrvCheckResult *res, BdrvCheckMode fix);* block driver; total_work_size may change during the course of the amendment * operation */ typedef void BlockDriverAmendStatusCB (BlockDriverState *bs,int64_t offset, -int64_t total_work_size); +int64_t total_work_size,void *opaque);int bdrv_amend_options (BlockDriverState *bs_new,QemuOpts *opts, -BlockDriverAmendStatusCB *status_cb); +BlockDriverAmendStatusCB *status_cb,void *cb_opaque);/* external snapshots */ bool bdrv_recurse_is_first_non_filter(BlockDriverState *bs,
include/block/block_int.h +2 -1
@@ -234,7 +234,8 @@ struct BlockDriver { BdrvCheckMode fix);int (*bdrv_amend_options)(BlockDriverState *bs,QemuOpts *opts, -BlockDriverAmendStatusCB *status_cb); +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque);void (*bdrv_debug_event)(BlockDriverState *bs,BlkDebugEvent event);
qemu-img.c +3 -2
@@ -2869,7 +2869,8 @@ out: }static void amend_status_cb (BlockDriverState *bs, -int64_t offset,int64_t total_work_size) +int64_t offset,int64_t total_work_size, +void *opaque) {qemu_progress_print (100.f * offset / total_work_size,0 ); }
@@ -2982,7 +2983,7 @@ static int img_amend(int argc, char **argv)/* In case the driver does not call amend_status_cb() */ qemu_progress_print (0.f ,0 ); - ret =bdrv_amend_options (bs, opts, &amend_status_cb); + ret =bdrv_amend_options (bs, opts, &amend_status_cb,NULL );qemu_progress_print (100.f ,0 );if (ret <0 ) {error_report ("Error while amending options: %s" ,strerror (-ret));
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 13/21] block: Add opaque value to the amend CB
Quoted 4 lines · Max Reitz
Add an opaque value which is to be passed to the
bdrv_amend_options()status callback.Signed-off-by Max Reitz <[email protected]>
Quoted 10 lines
--- block.c | 4 ++-- block/qcow2-cluster.c | 14 ++++++++------ block/qcow2.c | 9 +++++---- block/qcow2.h | 3 ++- include/block/block.h | 4 ++-- include/block/block_int.h | 3 ++- qemu-img.c | 5 +++-- 7 files changed, 24 insertions(+), 18 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 14/21 qcow2: Use error_report() in qcow2_amend_options()
In reply to Max Reitz--- block/qcow2.c | 14 ++++++-------- tests/qemu-iotests/061.out | 14 +++++++------- 2 files changed, 13 insertions(+), 15 deletions(-)
block/qcow2.c +6 -8
@@ -2686,11 +2686,11 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, }else if (!strcmp (compat,"1.1" )) { new_version =3 ; }else { -fprintf (stderr,"Unknown compatibility level %s.\n" , compat); +error_report ("Unknown compatibility level %s." , compat);return -EINVAL ; } }else if (!strcmp (desc->name,"preallocation" )) { -fprintf (stderr,"Cannot change preallocation mode.\n" ); +error_report ("Cannot change preallocation mode." );return -ENOTSUP ; }else if (!strcmp (desc->name,"size" )) { new_size =qemu_opt_get_size (opts,"size" ,0 );
@@ -2701,16 +2701,14 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, }else if (!strcmp (desc->name,"encryption" )) { encrypt =qemu_opt_get_bool (opts,"encryption" , s->crypt_method);if (encrypt != !!s->crypt_method) { -fprintf (stderr,"Changing the encryption flag is not " -"supported.\n" ); +error_report ("Changing the encryption flag is not supported." );return -ENOTSUP ; } }else if (!strcmp (desc->name,"cluster_size" )) { cluster_size =qemu_opt_get_size (opts,"cluster_size" , cluster_size);if (cluster_size != s->cluster_size) { -fprintf (stderr,"Changing the cluster size is not " -"supported.\n" ); +error_report ("Changing the cluster size is not supported." );return -ENOTSUP ; } }else if (!strcmp (desc->name,"lazy_refcounts" )) {
@@ -2756,8 +2754,8 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,if (s->use_lazy_refcounts != lazy_refcounts) {if (lazy_refcounts) {if (s->qcow_version <3 ) { -fprintf (stderr,"Lazy refcounts only supported with compatibility " -"level 1.1 and above (use compat=1.1 or greater)\n" ); +error_report ("Lazy refcounts only supported with compatibility " +"level 1.1 and above (use compat=1.1 or greater)" );return -EINVAL ; } s->compatible_features |=QCOW2_COMPAT_LAZY_REFCOUNTS ;
tests/qemu-iotests/061.out +7 -7
@@ -281,19 +281,19 @@ No errors were found on the image. === Testing invalid configurations === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) +qemu-img: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) qemu-img: Error while amending options: Invalid argument -Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) +qemu-img: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) qemu-img: Error while amending options: Invalid argument -Unknown compatibility level 0.42. +qemu-img: Unknown compatibility level 0.42. qemu-img: Error while amending options: Invalid argument qemu-img: Invalid parameter 'foo' qemu-img: Invalid options for file format 'qcow2' -Changing the cluster size is not supported. +qemu-img: Changing the cluster size is not supported. qemu-img: Error while amending options: Operation not supported -Changing the encryption flag is not supported. +qemu-img: Changing the encryption flag is not supported. qemu-img: Error while amending options: Operation not supported -Cannot change preallocation mode. +qemu-img: Cannot change preallocation mode. qemu-img: Error while amending options: Operation not supported === Testing correct handling of unset value ===
@@ -301,7 +301,7 @@ qemu-img: Error while amending options: Operation not supported Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 Should work: Should not work: -Changing the cluster size is not supported. +qemu-img: Changing the cluster size is not supported. qemu-img: Error while amending options: Operation not supported === Testing zero expansion on inactive clusters ===
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 14/21] qcow2: Use error_report() in qcow2_amend_options()
Quoted 11 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 14 ++++++-------- tests/qemu-iotests/061.out | 14 +++++++------- 2 files changed, 13 insertions(+), 15 deletions(-)block/qcow2.c +0
-0
block/qcow2.c static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 2689
Quoted 5 lines
block/qcow2.c +1
-1@@ -2686,11 +2686,11 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, }else if (!strcmp (compat,"1.1" )) { new_version =3 ; }else { -fprintf (stderr,"Unknown compatibility level %s.\n" , compat); +error_report ("Unknown compatibility level %s." , compat);
Not many error_report() locations include a trailing '.'
Quoted 15 lines
block/qcow2.c +4
-3+++ b/tests/qemu-iotests/061.out@@ -281,19 +281,19 @@ No errors were found on the image. === Testing invalid configurations === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) +qemu-img: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) qemu-img: Error while amending options: Invalid argument -Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) +qemu-img: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) qemu-img: Error while amending options: Invalid argument -Unknown compatibility level 0.42. +qemu-img: Unknown compatibility level 0.42. qemu-img: Error while amending options: Invalid argument qemu-img: Invalid parameter 'foo'
block/qcow2.c No errors were found on the image. · line 297
Quoted 10 lines
block/qcow2.c +3
-3qemu-img: Invalid options for file format 'qcow2' -Changing the cluster size is not supported. +qemu-img: Changing the cluster size is not supported. qemu-img: Error while amending options: Operation not supported -Changing the encryption flag is not supported. +qemu-img: Changing the encryption flag is not supported. qemu-img: Error while amending options: Operation not supported -Cannot change preallocation mode. +qemu-img: Cannot change preallocation mode. qemu-img: Error while amending options: Operation not supported
See - most of the messages do not end in '.'. Probably worth cleaning up if you respin. But it's not a show-stopper if you leave it, so:
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 14/21] qcow2: Use error_report() in qcow2_amend_options()
Quoted 38 lines · Eric Blake
Quoted 11 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 14 ++++++-------- tests/qemu-iotests/061.out | 14 +++++++------- 2 files changed, 13 insertions(+), 15 deletions(-)block/qcow2.c +0
-0block/qcow2.c static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 2689
Quoted 5 lines
block/qcow2.c +1
-1@@ -2686,11 +2686,11 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, }else if (!strcmp (compat,"1.1" )) { new_version =3 ; }else { -fprintf (stderr,"Unknown compatibility level %s.\n" , compat); +error_report ("Unknown compatibility level %s." , compat);Not many
error_report()locations include a trailing '.'Quoted 19 lines
block/qcow2.c +5
-4+++ b/tests/qemu-iotests/061.out@@ -281,19 +281,19 @@ No errors were found on the image. === Testing invalid configurations === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) +qemu-img: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) qemu-img: Error while amending options: Invalid argument -Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) +qemu-img: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) qemu-img: Error while amending options: Invalid argument -Unknown compatibility level 0.42. +qemu-img: Unknown compatibility level 0.42. qemu-img: Error while amending options: Invalid argument qemu-img: Invalid parameter 'foo' qemu-img: Invalid options for file format 'qcow2' -Changing the cluster size is not supported. +qemu-img: Changing the cluster size is not supported. qemu-img: Error while amending options: Operation not supported
Quoted 8 lines
block/qcow2.c No errors were found on the image. · line 297
Quoted 6 lines
block/qcow2.c +2
-2-Changing the encryption flag is not supported. +qemu-img: Changing the encryption flag is not supported. qemu-img: Error while amending options: Operation not supported -Cannot change preallocation mode. +qemu-img: Cannot change preallocation mode. qemu-img: Error while amending options: Operation not supportedSee - most of the messages do not end in '.'. Probably worth cleaning up if you respin. But it's not a show-stopper if you leave it, so:
I'll clean it up.
Max
Quoted 1 line
Reviewed-by Eric Blake <[email protected]>
PATCH 15/21 qcow2: Use abort() instead of assert(false)
In reply to Max Reitz--- block/qcow2.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block/qcow2.c +2 -2
@@ -2718,9 +2718,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,error_report ("Cannot change refcount entry width" );return -ENOTSUP ; }else { -/* if this assertion fails, this probably means a new option was +/* if this point is reached, this probably means a new option was * added without having it covered here */ -assert (false); +abort (); } desc++;
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 15/21] qcow2: Use abort() instead of assert(false)
Quoted 4 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Is it worth hoisting this one into 2.2 via the -trivial tree?
Quoted 18 lines
block/qcow2.c +2
-2@@ -2718,9 +2718,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,error_report ("Cannot change refcount entry width" );return -ENOTSUP ; }else { -/* if this assertion fails, this probably means a new option was +/* if this point is reached, this probably means a new option was * added without having it covered here */ -assert (false); +abort (); } desc++;
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 15/21] qcow2: Use abort() instead of assert(false)
Quoted 8 lines · Eric Blake
Quoted 4 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)Reviewed-by Eric Blake <[email protected]>Is it worth hoisting this one into 2.2 via the -trivial tree?
No, as explained this point can only be reached if there is some creation option for qcow2 images which is not handled by any of the branches in this function. Since there is no such thing currently in master and there most certainly won't be in 2.2 (thanks to hard freeze), it's fine to keep it out of 2.2.
Max
Quoted 17 lines
Quoted 17 lines
block/qcow2.c +2
-2@@ -2718,9 +2718,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,error_report ("Cannot change refcount entry width" );return -ENOTSUP ; }else { -/* if this assertion fails, this probably means a new option was +/* if this point is reached, this probably means a new option was * added without having it covered here */ -assert (false); +abort (); } desc++;
PATCH 16/21 qcow2: Split upgrade/downgrade paths for amend
In reply to Max ReitzIf the image version should be upgraded, that is the first we should do; if it should be downgraded, that is the last we should do. So split the version change block into an upgrade part at the start and a downgrade part at the end.
--- block/qcow2.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-)
block/qcow2.c +16 -15
@@ -2726,20 +2726,13 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, desc++; } -if (new_version != old_version) { -if (new_version > old_version) { -/* Upgrade */ - s->qcow_version = new_version; - ret =qcow2_update_header (bs); -if (ret <0 ) { - s->qcow_version = old_version; -return ret; - } - }else { - ret =qcow2_downgrade (bs, new_version, status_cb, cb_opaque); -if (ret <0 ) { -return ret; - } +/* Upgrade first (some features may require compat=1.1) */ +if (new_version > old_version) { + s->qcow_version = new_version; + ret =qcow2_update_header (bs); +if (ret <0 ) { + s->qcow_version = old_version; +return ret; } }
@@ -2753,7 +2746,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,if (s->use_lazy_refcounts != lazy_refcounts) {if (lazy_refcounts) { -if (s->qcow_version <3 ) { +if (new_version <3 ) {error_report ("Lazy refcounts only supported with compatibility " "level 1.1 and above (use compat=1.1 or greater)" );return -EINVAL ;
@@ -2789,6 +2782,14 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, } } +/* Downgrade last (so unsupported features can be removed before) */ +if (new_version < old_version) { + ret =qcow2_downgrade (bs, new_version, status_cb, cb_opaque); +if (ret <0 ) { +return ret; + } + } +return 0 ; }
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 16/21] qcow2: Split upgrade/downgrade paths for amend
Quoted 10 lines · Max Reitz
If the image version should be upgraded, that is the first we should do; if it should be downgraded, that is the last we should do. So split the version change block into an upgrade part at the start and a downgrade part at the end.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 17/21 qcow2: Use intermediate helper CB for amend
In reply to Max ReitzIf there is more than one time-consuming operation to be performed for qcow2_amend_options(), we need an intermediate CB which coordinates the progress of the individual operations and passes the result to the original status callback.
--- block/qcow2.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-)
block/qcow2.c +75 -1
@@ -2655,6 +2655,71 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,return 0 ; } +typedef enum Qcow2AmendOperation { +/* This is the value Qcow2AmendHelperCBInfo::last_operation will be +* statically initialized to so that the helper CB can discern the first +* invocation from an operation change */ +QCOW2_NO_OPERATION =0 , + +QCOW2_DOWNGRADING , +}Qcow2AmendOperation ; + +typedef struct Qcow2AmendHelperCBInfo { +/* The code coordinating the amend operations should only modify +* these four fields; the rest will be managed by the CB */ +BlockDriverAmendStatusCB *original_status_cb; +void *original_cb_opaque; + +Qcow2AmendOperation current_operation; + +/* Total number of operations to perform (only set once) */ +int total_operations; + +/* The following fields are managed by the CB */ + +/* Number of operations completed */ +int operations_completed; + +/* Cumulative offset of all completed operations */ +int64_t offset_completed; + +Qcow2AmendOperation last_operation; +int64_t last_work_size; +}Qcow2AmendHelperCBInfo ; + +static void qcow2_amend_helper_cb (BlockDriverState *bs,int64_t offset, +int64_t total_work_size,void *opaque) +{ +Qcow2AmendHelperCBInfo *info = opaque; +int64_t current_work_size; +int64_t projected_work_size; + +if (info->current_operation != info->last_operation) { +if (info->last_operation !=QCOW2_NO_OPERATION ) { + info->offset_completed += info->last_work_size; + info->operations_completed++; + } + + info->last_operation = info->current_operation; + } + + info->last_work_size = total_work_size; + + current_work_size = info->offset_completed + total_work_size; + +/* current_work_size is the total work size for (operations_completed + 1) +* operations (which includes this one), so multiply it by the number of +* operations not covered and divide it by the number of operations +* covered to get a projection for the operations not covered */ + projected_work_size = current_work_size * (info->total_operations - + info->operations_completed -1 ) + / (info->operations_completed +1 ); + + info->original_status_cb (bs, info->offset_completed + offset, + current_work_size + projected_work_size, + info->original_cb_opaque); +} +static int qcow2_amend_options (BlockDriverState *bs,QemuOpts *opts,BlockDriverAmendStatusCB *status_cb,void *cb_opaque)
@@ -2669,6 +2734,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,bool encrypt;int ret;QemuOptDesc *desc = opts->list->desc; +Qcow2AmendHelperCBInfo helper_cb_info;while (desc && desc->name) {if (!qemu_opt_find (opts, desc->name)) {
@@ -2726,6 +2792,12 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, desc++; } + helper_cb_info = (Qcow2AmendHelperCBInfo ){ + .original_status_cb = status_cb, + .original_cb_opaque = cb_opaque, + .total_operations = (new_version < old_version) + }; +/* Upgrade first (some features may require compat=1.1) */ if (new_version > old_version) { s->qcow_version = new_version;
@@ -2784,7 +2856,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,/* Downgrade last (so unsupported features can be removed before) */ if (new_version < old_version) { - ret =qcow2_downgrade (bs, new_version, status_cb, cb_opaque); + helper_cb_info.current_operation =QCOW2_DOWNGRADING ; + ret =qcow2_downgrade (bs, new_version, &qcow2_amend_helper_cb, + &helper_cb_info);if (ret <0 ) {return ret; }
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 17/21] qcow2: Use intermediate helper CB for amend
Quoted 9 lines · Max Reitz
If there is more than one time-consuming operation to be performed for
qcow2_amend_options(), we need an intermediate CB which coordinates the progress of the individual operations and passes the result to the original status callback.Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-)
Getting trickier to review.
Quoted 7 lines
block/qcow2.c +0
-0@@ -2655,6 +2655,71 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,return 0 ;
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2665
Quoted 10 lines
block/qcow2.c +8
-0} +typedef enum Qcow2AmendOperation { +/* This is the value Qcow2AmendHelperCBInfo::last_operation will be +* statically initialized to so that the helper CB can discern the first +* invocation from an operation change */ +QCOW2_NO_OPERATION =0 , + +QCOW2_DOWNGRADING , +}Qcow2AmendOperation ;
So for this patch, you still have just one operation, but later in the series, you add a second (and the goal of THIS patch is that it will work even if there are 3 or more operations, even though this series doesn't add that many).
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2691
Quoted 2 lines
block/qcow2.c +2
-0+static void qcow2_amend_helper_cb (BlockDriverState *bs,int64_t offset, +int64_t total_work_size,void *opaque)
indentation looks off
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2695
Quoted 4 lines
block/qcow2.c +4
-0+{ +Qcow2AmendHelperCBInfo *info = opaque; +int64_t current_work_size; +int64_t projected_work_size;
Worth asserting that info->total_operations is non-zero? Or is there ever a valid case for calling the callback even when there are no sub-operations, and therefore we are automatically complete (offset == total_work_size)?
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2701
Quoted 6 lines
block/qcow2.c +6
-0+ +if (info->current_operation != info->last_operation) { +if (info->last_operation !=QCOW2_NO_OPERATION ) { + info->offset_completed += info->last_work_size; + info->operations_completed++; + }
Would it be any easier to guarantee that we come to 100% completion by requiring the coordinator to pass a final completion callback? [1]
info->current_operation = QCOW2_NO_OPERATION;
cb(bs, 0, 0, info)
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2706
Quoted 5 lines
block/qcow2.c +5
-0+ + info->last_operation = info->current_operation; + } + + info->last_work_size = total_work_size;
Took me a while to realize that total_work_size is the incoming (estimated) total size for the current sub-operation, and not the total over the combination of all sub-operations...
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2710
Quoted 4 lines
block/qcow2.c +4
-0+ + current_work_size = info->offset_completed + total_work_size; + +/* current_work_size is the total work size for (operations_completed + 1)
but this comment helped.
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2716
Quoted 6 lines
block/qcow2.c +6
-0+* operations (which includes this one), so multiply it by the number of +* operations not covered and divide it by the number of operations +* covered to get a projection for the operations not covered */ + projected_work_size = current_work_size * (info->total_operations - + info->operations_completed -1 ) + / (info->operations_completed +1 );
So, when there is just one sub-operation (which is the case until later patches add a second), this results in the following calculation for ALL calls during the intermediate steps of the sub-operation:
projected_work_size = total_work_size * (1 - 0 - 1) / (0 + 1)
that is, we are projecting 0 additional work because we have zero additional stages to complete. Am I correct that we will never enter the callback in a state where info->operations_completed==info->total_operations? (because if we do, you'd have a computation of final_size * (1 - 1 - 1) / (1 + 1) which looks weird). Worth an assert()? Then again, my proposal above [1] to guarantee a 100% completion by use of a final cleanup callback would indeed reach the point where operations_completed==total_operations.
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2720
Quoted 4 lines
block/qcow2.c +4
-0+ + info->original_status_cb (bs, info->offset_completed + offset, + current_work_size + projected_work_size, + info->original_cb_opaque);
So, as long as we don't add a second phase, this is strictly equivalent to calling the original callback with the original offset (since info->offset_completed remains 0) and original work size (since projected_work_size remains 0). That part works fine.
Let's see what happens if we had three phases. To make it more interesting, let's pick some numbers - how about the first phase progresses from 0-10, the second from 0-100, and the third from 0-10, and where none of the sub-operations change predicted total_work_size. The caller would first set info->current_operation to 1, then call the callback a few times; how about twice with 5/10 and 10/10. For both calls, current_work_size is 0+10, then projected_work_size is 10*(3-0-1)/(0+1) == 20, and we call the original callback with (0+5)/(10+20) and (0+10)/(10+20). Pretty good (5/30 and 10/30 are right on if the first sub-command is exactly one-third of the time of the overall command; and even if it is not, it still shows reasonable progress).
Then we move on to the second sub-command where the coordinator updates info->current_operation to 2 before triggering several callbacks; let's suppose it reports at 0/100, 30/100, 60/100, and 100/100. The first call updates info to track that we've detected a change in sub-command (offset_completed is now 10, operations_completed is now 1). Then for all four calls, current_work_size is 10+100, and projected_work_size is 110*(3-1-1)/(1+1) == 55. So we call the original callback with (10+0)/(110+55), (10+30)/(110+55), (10+60)/(110+55), (10+100)/(110+55).
The first report of 10/165 looks like we jumped backwards (much smaller
progress than our previous report of 10/30), but that's merely a representation that this phase is estimating a larger total_work count, and we have no way of correlating whether 1 unit of work count in each phase is equivalent to an equal amount of time. But by the end, we report 110/165, which is spot on for being two-thirds complete.
Another assignment to info->current_operation, and a couple more callbacks; let's again use 5/10 and 10/10. The first callback updates info (offset_completed is now 110, operations_completed is now 2). For each call, current_work_size is 110+10, and projected_work_size is 120*(3-2-1/(2+1) == 0. We call the original callback with (120+5)/(120+10) and (120+10)/(120+10). We've done a very rapid jump from 2/3 to 125/130, but end the overall operation with the two values equal. So the function is not very smooth, but at least it is as good an estimate as possible along each stage of the operation, and we never violate the premise of reporting equal values until all sub-commands are complete.
Quoted 14 lines
block/qcow2.c +3
-0+} +static int qcow2_amend_options (BlockDriverState *bs,QemuOpts *opts,BlockDriverAmendStatusCB *status_cb,void *cb_opaque)@@ -2669,6 +2734,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,bool encrypt;int ret;QemuOptDesc *desc = opts->list->desc; +Qcow2AmendHelperCBInfo helper_cb_info;while (desc && desc->name) {if (!qemu_opt_find (opts, desc->name)) {
block/qcow2.c static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 2799
Quoted 8 lines
block/qcow2.c +5
-0@@ -2726,6 +2792,12 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, desc++; } + helper_cb_info = (Qcow2AmendHelperCBInfo ){ + .original_status_cb = status_cb, + .original_cb_opaque = cb_opaque, + .total_operations = (new_version < old_version) + };
Slick.
block/qcow2.c static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 2861
Quoted 12 lines
block/qcow2.c +4
-1+/* Upgrade first (some features may require compat=1.1) */ if (new_version > old_version) { s->qcow_version = new_version;@@ -2784,7 +2856,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,/* Downgrade last (so unsupported features can be removed before) */ if (new_version < old_version) { - ret =qcow2_downgrade (bs, new_version, status_cb, cb_opaque); + helper_cb_info.current_operation =QCOW2_DOWNGRADING ; + ret =qcow2_downgrade (bs, new_version, &qcow2_amend_helper_cb, + &helper_cb_info);
Looks correct to me. Other than the indentation issue and possible addition of some asserts, this is good to go.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 17/21] qcow2: Use intermediate helper CB for amend
Quoted 11 lines · Eric Blake
Quoted 9 lines · Max Reitz
If there is more than one time-consuming operation to be performed for
qcow2_amend_options(), we need an intermediate CB which coordinates the progress of the individual operations and passes the result to the original status callback.Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-)Getting trickier to review.
Yes, and 18 is the worst.
Quoted 8 lines
Quoted 8 lines
block/qcow2.c +0
-0@@ -2655,6 +2655,71 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,return 0 ; }
Quoted 12 lines
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2665
Quoted 8 lines
block/qcow2.c +8
-0+typedef enum Qcow2AmendOperation { +/* This is the value Qcow2AmendHelperCBInfo::last_operation will be +* statically initialized to so that the helper CB can discern the first +* invocation from an operation change */ +QCOW2_NO_OPERATION =0 , + +QCOW2_DOWNGRADING , +}Qcow2AmendOperation ;So for this patch, you still have just one operation, but later in the series, you add a second (and the goal of THIS patch is that it will work even if there are 3 or more operations, even though this series doesn't add that many).
Right.
Quoted 3 lines
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2691
Quoted 2 lines
block/qcow2.c +2
-0+static void qcow2_amend_helper_cb (BlockDriverState *bs,int64_t offset, +int64_t total_work_size,void *opaque)indentation looks off
Right, will fix.
Quoted 8 lines
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2695
Quoted 4 lines
block/qcow2.c +4
-0+{ +Qcow2AmendHelperCBInfo *info = opaque; +int64_t current_work_size; +int64_t projected_work_size;Worth asserting that
info->total_operationsis non-zero? Or is there ever a valid case for calling the callback even when there are no sub-operations, and therefore we are automatically complete (offset ==total_work_size)?
No, the driver does not have to call the status CB; qemu-img amend will mind that case by first printing 0 %, then invoking the amend operation, and then printing 100 %. Will add an assertion.
Quoted 10 lines
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2701
Quoted 6 lines
block/qcow2.c +6
-0+ +if (info->current_operation != info->last_operation) { +if (info->last_operation !=QCOW2_NO_OPERATION ) { + info->offset_completed += info->last_work_size; + info->operations_completed++; + }Would it be any easier to guarantee that we come to 100% completion by requiring the coordinator to pass a final completion callback? [1]
info->current_operation=QCOW2_NO_OPERATION;
cb(bs, 0, 0, info)
No, because the amend CB does not have to be called on completion; img_amend() in qemu-img.c takes care of that case.
Quoted 8 lines
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2706
Quoted 5 lines
block/qcow2.c +5
-0+ + info->last_operation = info->current_operation; + } + + info->last_work_size = total_work_size;Took me a while to realize that
total_work_sizeis the incoming (estimated) total size for the current sub-operation, and not the total over the combination of all sub-operations...
Ah, right, I'll change the variable name, maybe to operation_work_size or something like that.
Quoted 16 lines
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2710
Quoted 4 lines
block/qcow2.c +4
-0+ + current_work_size = info->offset_completed + total_work_size; + +/* current_work_size is the total work size for (operations_completed + 1) but this comment helped.
block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2716
Quoted 6 lines
block/qcow2.c +6
-0+* operations (which includes this one), so multiply it by the number of +* operations not covered and divide it by the number of operations +* covered to get a projection for the operations not covered */ + projected_work_size = current_work_size * (info->total_operations - + info->operations_completed -1 ) + / (info->operations_completed +1 );So, when there is just one sub-operation (which is the case until later patches add a second), this results in the following calculation for ALL calls during the intermediate steps of the sub-operation:
Quoted 6 lines
projected_work_size=total_work_size* (1 - 0 - 1) / (0 + 1)that is, we are projecting 0 additional work because we have zero additional stages to complete. Am I correct that we will never enter the callback in a state where info->operations_completed==info->total_operations?
Yes, we won't.
Quoted 3 lines
(because if we do,
you'd have a computation offinal_size* (1 - 1 - 1) / (1 + 1) which looks weird). Worth anassert()?
assert()s are always worth it.
Quoted 40 lines
Then again, my proposal above [1] to
guarantee a 100% completion by use of a final cleanup callback would indeed reach the point where operations_completed==total_operations.block/qcow2.c static int qcow2_downgrade(BlockDriverState *bs, int target_version, · line 2720
Quoted 4 lines
block/qcow2.c +4
-0+ + info->original_status_cb (bs, info->offset_completed + offset, + current_work_size + projected_work_size, + info->original_cb_opaque);So, as long as we don't add a second phase, this is strictly equivalent to calling the original callback with the original offset (since
info->offset_completedremains 0) and original work size (sinceprojected_work_sizeremains 0). That part works fine.Let's see what happens if we had three phases. To make it more interesting, let's pick some numbers - how about the first phase progresses from 0-10, the second from 0-100, and the third from 0-10, and where none of the sub-operations change predicted
total_work_size. The caller would first setinfo->current_operationto 1, then call the callback a few times; how about twice with 5/10 and 10/10. For both calls,current_work_sizeis 0+10, thenprojected_work_sizeis 10*(3-0-1)/(0+1) == 20, and we call the original callback with (0+5)/(10+20) and (0+10)/(10+20). Pretty good (5/30 and 10/30 are right on if the first sub-command is exactly one-third of the time of the overall command; and even if it is not, it still shows reasonable progress).Then we move on to the second sub-command where the coordinator updates
info->current_operationto 2 before triggering several callbacks; let's suppose it reports at 0/100, 30/100, 60/100, and 100/100. The first call updates info to track that we've detected a change in sub-command (offset_completedis now 10,operations_completedis now 1). Then for all four calls,current_work_sizeis 10+100, andprojected_work_sizeis 110*(3-1-1)/(1+1) == 55. So we call the original callback with (10+0)/(110+55), (10+30)/(110+55), (10+60)/(110+55), (10+100)/(110+55).
The first report of 10/165 looks like we jumped backwards (much smaller
progress than our previous report of 10/30), but that's merely a representation that this phase is estimating a largertotal_workcount, and we have no way of correlating whether 1 unit of work count in each phase is equivalent to an equal amount of time. But by the end, we report 110/165, which is spot on for being two-thirds complete.
Quoted 11 lines
Another assignment to
info->current_operation, and a couple more callbacks; let's again use 5/10 and 10/10. The first callback updates info (offset_completedis now 110,operations_completedis now 2). For each call,current_work_sizeis 110+10, andprojected_work_sizeis 120*(3-2-1/(2+1) == 0. We call the original callback with (120+5)/(120+10) and (120+10)/(120+10). We've done a very rapid jump from 2/3 to 125/130, but end the overall operation with the two values equal. So the function is not very smooth, but at least it is as good an estimate as possible along each stage of the operation, and we never violate the premise of reporting equal values until all sub-commands are complete.
Yes, it's not pretty but it's the best we can do without either hard-coding some estimations or trying some kind of dry-run to determine each operation's work size beforehand.
Quoted 39 lines
Quoted 14 lines
block/qcow2.c +3
-0+} +static int qcow2_amend_options (BlockDriverState *bs,QemuOpts *opts,BlockDriverAmendStatusCB *status_cb,void *cb_opaque)@@ -2669,6 +2734,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,bool encrypt;int ret;QemuOptDesc *desc = opts->list->desc; +Qcow2AmendHelperCBInfo helper_cb_info;while (desc && desc->name) {if (!qemu_opt_find (opts, desc->name)) {block/qcow2.c static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 2799
Quoted 8 lines
block/qcow2.c +5
-0@@ -2726,6 +2792,12 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, desc++; } + helper_cb_info = (Qcow2AmendHelperCBInfo ){ + .original_status_cb = status_cb, + .original_cb_opaque = cb_opaque, + .total_operations = (new_version < old_version) + };Slick.
block/qcow2.c static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 2861
Quoted 12 lines
block/qcow2.c +4
-1+/* Upgrade first (some features may require compat=1.1) */ if (new_version > old_version) { s->qcow_version = new_version;@@ -2784,7 +2856,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,/* Downgrade last (so unsupported features can be removed before) */ if (new_version < old_version) { - ret =qcow2_downgrade (bs, new_version, status_cb, cb_opaque); + helper_cb_info.current_operation =QCOW2_DOWNGRADING ; + ret =qcow2_downgrade (bs, new_version, &qcow2_amend_helper_cb, + &helper_cb_info);Looks correct to me. Other than the indentation issue and possible addition of some asserts, this is good to go.
Quoted 1 line
Reviewed-by Eric Blake <[email protected]>
Thanks!
Max
PATCH 18/21 qcow2: Add function for refcount order amendment
In reply to Max ReitzAdd a function qcow2_change_refcount_order() which allows changing the refcount order of a qcow2 image.
--- block/qcow2-refcount.c | 424 +++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 + 2 files changed, 428 insertions(+)
block/qcow2-refcount.c +424 -0
@@ -2473,3 +2473,427 @@ int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset,return 0 ; } + +/** +* This "operation" for walk_over_reftable() allocates the refblock on disk (if +* it is not empty) and inserts its offset into the new reftable. The size of +* this new reftable is increased as required. +*/ +static int alloc_refblock (BlockDriverState *bs,uint64_t **reftable, +uint64_t reftable_index,uint64_t *reftable_size, +void *refblock,bool refblock_empty,Error **errp) +{ +BDRVQcowState *s = bs->opaque; +int64_t offset; + +if (!refblock_empty && reftable_index >= *reftable_size) { +uint64_t *new_reftable; +uint64_t new_reftable_size; + + new_reftable_size =ROUND_UP (reftable_index +1 , + s->cluster_size /sizeof (uint64_t )); +if (new_reftable_size >QCOW_MAX_REFTABLE_SIZE /sizeof (uint64_t )) { +error_setg (errp, +"This operation would make the refcount table grow " +"beyond the maximum size supported by QEMU, aborting" ); +return -ENOTSUP ; + } + + new_reftable =g_try_realloc (*reftable, new_reftable_size * +sizeof (uint64_t )); +if (!new_reftable) { +error_setg (errp,"Failed to increase reftable buffer size" ); +return -ENOMEM ; + } + +memset (new_reftable + *reftable_size,0 , + (new_reftable_size - *reftable_size) *sizeof (uint64_t )); + + *reftable = new_reftable; + *reftable_size = new_reftable_size; + } + +if (refblock_empty) { +if (reftable_index < *reftable_size) { + (*reftable)[reftable_index] =0 ; + } + }else { + offset =qcow2_alloc_clusters (bs, s->cluster_size); +if (offset <0 ) { +error_setg_errno (errp, -offset,"Failed to allocate refblock" ); +return offset; + } + (*reftable)[reftable_index++] = offset; + } + +return 0 ; +} + +/** +* This "operation" for walk_over_reftable() writes the refblock to disk at the +* offset specified by the new reftable's entry. It does not modify the new +* reftable or change any refcounts. +*/ +static int flush_refblock (BlockDriverState *bs,uint64_t **reftable, +uint64_t reftable_index,uint64_t *reftable_size, +void *refblock,bool refblock_empty,Error **errp) +{ +BDRVQcowState *s = bs->opaque; +int64_t offset; +int ret; + +if (refblock_empty) { +if (reftable_index < *reftable_size) { +assert ((*reftable)[reftable_index] ==0 ); + } + }else { +/* The first pass with alloc_refblock() made the reftable large enough +*/ +assert (reftable_index < *reftable_size); + offset = (*reftable)[reftable_index]; +assert (offset !=0 ); + + ret =qcow2_pre_write_overlap_check (bs,0 , offset, s->cluster_size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Overlap check failed" ); +return ret; + } + + ret =bdrv_pwrite (bs->file, offset, refblock, s->cluster_size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to write refblock" ); +return ret; + } + } + +return 0 ; +} + +/** +* This function walks over the existing reftable and every referenced refblock; +* if @new_set_refcount is non-NULL, it is called for every refcount entry to +* create an equal new entry in the passed @new_refblock. Once that +* @new_refblock is completely filled, @operation will be called. +* +* @operation is expected to combine the @new_refblock and its entry in the new +* reftable (which is described by the parameters starting with "reftable"). +* @refblock_empty is set if all entries in the refblock are zero. +* +* @status_cb and @cb_opaque are used for the amend operation's status callback. +* @index is the index of the walk_over_reftable() calls and @total is the total +* number of walk_over_reftable() calls per amend operation. Both are used for +* calculating the parameters for the status callback. +*/ +static int walk_over_reftable (BlockDriverState *bs,uint64_t **new_reftable, +uint64_t *new_reftable_index, +uint64_t *new_reftable_size, +void *new_refblock,int new_refblock_size, +int new_refcount_bits, +int (*operation)(BlockDriverState *bs, +uint64_t **reftable, +uint64_t reftable_index, +uint64_t *reftable_size, +void *refblock, +bool refblock_empty, +Error **errp), +Qcow2SetRefcountFunc *new_set_refcount, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,int index,int total, +Error **errp) +{ +BDRVQcowState *s = bs->opaque; +uint64_t reftable_index; +bool new_refblock_empty = true; +int refblock_index; +int new_refblock_index =0 ; +int ret; + +for (reftable_index =0 ; reftable_index < s->refcount_table_size; + reftable_index++) + { +uint64_t refblock_offset = s->refcount_table[reftable_index] + &REFT_OFFSET_MASK ; + +status_cb (bs, (uint64_t )index * s->refcount_table_size + reftable_index, + (uint64_t )total * s->refcount_table_size, cb_opaque); + +if (refblock_offset) { +void *refblock; + +if (offset_into_cluster (s, refblock_offset)) { +qcow2_signal_corruption (bs, true,-1 ,-1 ,"Refblock offset %#" + PRIx64" unaligned (reftable index: %#" + PRIx64")" , refblock_offset, + reftable_index); +error_setg (errp, +"Image is corrupt (unaligned refblock offset)" ); +return -EIO ; + } + + ret =qcow2_cache_get (bs, s->refcount_block_cache, refblock_offset, + &refblock); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to retrieve refblock" ); +return ret; + } + +for (refblock_index =0 ; refblock_index < s->refcount_block_size; + refblock_index++) + { +uint64_t refcount; + +if (new_refblock_index >= new_refblock_size) { +/* new_refblock is now complete */ + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, + new_refblock_empty, errp); +if (ret <0 ) { +qcow2_cache_put (bs, s->refcount_block_cache, &refblock); +return ret; + } + + (*new_reftable_index)++; + new_refblock_index =0 ; + new_refblock_empty = true; + } + + refcount = s->get_refcount (refblock, refblock_index); +if (new_refcount_bits <64 && refcount >> new_refcount_bits) { +uint64_t offset; + +qcow2_cache_put (bs, s->refcount_block_cache, &refblock); + + offset = ((reftable_index << s->refcount_block_bits) + + refblock_index) << s->cluster_bits; + +error_setg (errp,"Cannot decrease refcount entry width to " +"%i bits: Cluster at offset %#" PRIx64" has a " +"refcount of %" PRIu64, new_refcount_bits, + offset, refcount); +return -EINVAL ; + } + +if (new_set_refcount) { +new_set_refcount (new_refblock, new_refblock_index++, refcount); + }else { + new_refblock_index++; + } + new_refblock_empty = new_refblock_empty && refcount ==0 ; + } + + ret =qcow2_cache_put (bs, s->refcount_block_cache, &refblock); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to put refblock back into " +"the cache" ); +return ret; + } + }else { +/* No refblock means every refcount is 0 */ +for (refblock_index =0 ; refblock_index < s->refcount_block_size; + refblock_index++) + { +if (new_refblock_index >= new_refblock_size) { +/* new_refblock is now complete */ + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, + new_refblock_empty, errp); +if (ret <0 ) { +return ret; + } + + (*new_reftable_index)++; + new_refblock_index =0 ; + new_refblock_empty = true; + } + +if (new_set_refcount) { +new_set_refcount (new_refblock, new_refblock_index++,0 ); + }else { + new_refblock_index++; + } + } + } + } + +if (new_refblock_index >0 ) { +/* Complete the potentially existing partially filled final refblock */ +if (new_set_refcount) { +for (; new_refblock_index < new_refblock_size; + new_refblock_index++) + { +new_set_refcount (new_refblock, new_refblock_index,0 ); + } + } + + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, new_refblock_empty, + errp); +if (ret <0 ) { +return ret; + } + + (*new_reftable_index)++; + } + +return 0 ; +} + +int qcow2_change_refcount_order (BlockDriverState *bs,int refcount_order, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,Error **errp) +{ +BDRVQcowState *s = bs->opaque; +Qcow2GetRefcountFunc *new_get_refcount; +Qcow2SetRefcountFunc *new_set_refcount; +void *new_refblock =qemu_blockalign (bs->file, s->cluster_size); +uint64_t *new_reftable =NULL , new_reftable_size =0 ; +uint64_t *old_reftable, old_reftable_size, old_reftable_offset; +uint64_t new_reftable_index =0 ; +uint64_t i; +int64_t new_reftable_offset; +int new_refblock_size, new_refcount_bits =1 << refcount_order; +int old_refcount_order; +int ret; + +assert (s->qcow_version >=3 ); +assert (refcount_order >=0 && refcount_order <=6 ); + +/* see qcow2_open() */ + new_refblock_size =1 << (s->cluster_bits - (refcount_order -3 )); + +get_refcount_functions (refcount_order, + &new_get_refcount, &new_set_refcount); + + +/* First, allocate the structures so they are present in the refcount +* structures */ + ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size,NULL , new_refblock_size, + new_refcount_bits, &alloc_refblock,NULL , + status_cb, cb_opaque,0 ,2 , errp); +if (ret <0 ) { + goto done; + } + +/* The new_reftable_size is now valid and will not be changed anymore, +* so we can now allocate the reftable */ + new_reftable_offset =qcow2_alloc_clusters (bs, new_reftable_size * +sizeof (uint64_t )); +if (new_reftable_offset <0 ) { +error_setg_errno (errp, -new_reftable_offset, +"Failed to allocate the new reftable" ); + ret = new_reftable_offset; + goto done; + } + + new_reftable_index =0 ; + +/* Second, write the new refblocks */ + ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size, new_refblock, + new_refblock_size, new_refcount_bits, + &flush_refblock, new_set_refcount, + status_cb, cb_opaque,1 ,2 , errp); +if (ret <0 ) { + goto done; + } + + +/* Write the new reftable */ + ret =qcow2_pre_write_overlap_check (bs,0 , new_reftable_offset, + new_reftable_size *sizeof (uint64_t )); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Overlap check failed" ); + goto done; + } + +for (i =0 ; i < new_reftable_size; i++) { +cpu_to_be64s (&new_reftable[i]); + } + + ret =bdrv_pwrite (bs->file, new_reftable_offset, new_reftable, + new_reftable_size *sizeof (uint64_t )); + +for (i =0 ; i < new_reftable_size; i++) { +be64_to_cpus (&new_reftable[i]); + } + +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to write the new reftable" ); + goto done; + } + + +/* Empty the refcount cache */ + ret =qcow2_cache_flush (bs, s->refcount_block_cache); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to flush the refblock cache" ); + goto done; + } + +/* Update the image header to point to the new reftable; this only updates +* the fields which are relevant to qcow2_update_header(); other fields +* such as s->refcount_table or s->refcount_bits stay stale for now +* (because we have to restore everything if qcow2_update_header() fails) */ + old_refcount_order = s->refcount_order; + old_reftable_size = s->refcount_table_size; + old_reftable_offset = s->refcount_table_offset; + + s->refcount_order = refcount_order; + s->refcount_table_size = new_reftable_size; + s->refcount_table_offset = new_reftable_offset; + + ret =qcow2_update_header (bs); +if (ret <0 ) { + s->refcount_order = old_refcount_order; + s->refcount_table_size = old_reftable_size; + s->refcount_table_offset = old_reftable_offset; +error_setg_errno (errp, -ret,"Failed to update the qcow2 header" ); + goto done; + } + +/* Now update the rest of the in-memory information */ + old_reftable = s->refcount_table; + s->refcount_table = new_reftable; + +/* For cleaning up all old refblocks */ + new_reftable = old_reftable; + new_reftable_size = old_reftable_size; + + s->refcount_bits =1 << refcount_order; +if (refcount_order <6 ) { + s->refcount_max = (UINT64_C (1 ) << s->refcount_bits) -1 ; + }else { + s->refcount_max =INT64_MAX ; + } + + s->refcount_block_bits = s->cluster_bits - (refcount_order -3 ); + s->refcount_block_size =1 << s->refcount_block_bits; + + s->get_refcount = new_get_refcount; + s->set_refcount = new_set_refcount; + +/* And free the old reftable (the old refblocks are freed below the "done" +* label) */ +qcow2_free_clusters (bs, old_reftable_offset, + old_reftable_size *sizeof (uint64_t ), +QCOW2_DISCARD_NEVER ); + +done: +if (new_reftable) { +/* On success, new_reftable actually points to the old reftable (and +* new_reftable_size is the old reftable's size); but that is just +* fine */ +for (i =0 ; i < new_reftable_size; i++) { +uint64_t offset = new_reftable[i] &REFT_OFFSET_MASK ; +if (offset) { +qcow2_free_clusters (bs, offset, s->cluster_size, +QCOW2_DISCARD_NEVER ); + } + } +g_free (new_reftable); + } + +qemu_vfree (new_refblock); +return ret; +}
block/qcow2.h +4 -0
@@ -526,6 +526,10 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,int qcow2_pre_write_overlap_check (BlockDriverState *bs,int ign,int64_t offset,int64_t size); +int qcow2_change_refcount_order (BlockDriverState *bs,int refcount_order, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,Error **errp); +/* qcow2-cluster.c functions */ int qcow2_grow_l1_table (BlockDriverState *bs,uint64_t min_size,bool exact_size);
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 18/21] qcow2: Add function for refcount order amendment
Quoted 8 lines · Max Reitz
Add a function
qcow2_change_refcount_order()which allows changing the refcount order of a qcow2 image.Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 424 +++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 + 2 files changed, 428 insertions(+)
This is fairly big; you may want to get a second review from a maintainer rather than blindly trusting me.
My review was not linear, but I left the email in linear order. Feel free to ask for clarification if my presentation is too hard to follow.
Quoted 18 lines
block/qcow2-refcount.c +18
-0+ +/** +* This "operation" for walk_over_reftable() allocates the refblock on disk (if +* it is not empty) and inserts its offset into the new reftable. The size of +* this new reftable is increased as required. +*/ +static int alloc_refblock (BlockDriverState *bs,uint64_t **reftable, +uint64_t reftable_index,uint64_t *reftable_size, +void *refblock,bool refblock_empty,Error **errp) +{ +BDRVQcowState *s = bs->opaque; +int64_t offset; + +if (!refblock_empty && reftable_index >= *reftable_size) { +uint64_t *new_reftable; +uint64_t new_reftable_size; + + new_reftable_size = ROUND_UP(reftable_index + 1,
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2503
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ s->cluster_size / sizeof(uint64_t)); +if (new_reftable_size >QCOW_MAX_REFTABLE_SIZE /sizeof (uint64_t )) { +error_setg (errp, +"This operation would make the refcount table grow " +"beyond the maximum size supported by QEMU, aborting" ); +return -ENOTSUP ; + } + + new_reftable =g_try_realloc (*reftable, new_reftable_size * +sizeof (uint64_t ));
Safe from overflow based on checks a few lines earlier. Good.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2513
Quoted 10 lines
block/qcow2-refcount.c +10
-0+if (!new_reftable) { +error_setg (errp,"Failed to increase reftable buffer size" ); +return -ENOMEM ; + } + +memset (new_reftable + *reftable_size,0 , + (new_reftable_size - *reftable_size) *sizeof (uint64_t )); + + *reftable = new_reftable; + *reftable_size = new_reftable_size;
Just to check my math here:
Suppose we have an image with 512-byte clusters, and are changing from 16-bit refcount (order 4) to 64-bit refcount. Also, suppose the existing image has exactly filled a one-cluster refcount table (that is, there are 64 refcount blocks, each describing at a refblock with all 256 refcount entries full, for a total image size of exactly 8M). The original image occupies a header (1 cluster), L1 and L2 tables, and data; but 65 of the 16k clusters tied up in the image are dedicated to the refcount structures.
Meanwhile, the new refcount table will have to point to 256 refcount blocks, each holding only 64 entries, which in turn implies that the refcount table now has to be at least 4 clusters long. But as this requires at least 260 clusters to represent, then even if we were able to reuse the 65 clusters of the original table, we'd still be allocating at least 195 clusters; in reality, your code doesn't free any old clusters until after allocating the new, because it is easier to keep the old table live until the new table is populated. The process of allocating the new clusters means we actually end up with a new refcount table of 5 clusters long, where not all 320 refblocks will be populated.
But as long as we are keeping the old table up-to-date for the refblock
allocations, it ALSO means that we caused a rollover of the old table from 1 cluster into 2, which itself consumes several clusters (the larger table must be contiguous, and we must also set up a refblock to describe the larger table, so we've added at least three clusters associated to the original table during the course of preparing the new table).
Hmm - that means I found a bug in your implementation. See [3] below.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2518
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ } + +if (refblock_empty) { +if (reftable_index < *reftable_size) { + (*reftable)[reftable_index] =0 ;
Necessary since you used g_try_realloc which leaves the new reftable uninitialized. Reasonable (rather than a memset) since the caller will be visiting every single refblock in the table anyways.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2521
Quoted 3 lines
block/qcow2-refcount.c +3
-0+ } + }else { + offset =qcow2_alloc_clusters (bs, s->cluster_size);
As mentioned above, this action will potentially change s->refcount_table_size of the original table, which in turn makes the caller execute its loops more often to cover the increased allocation. Does qcow2_alloc_clusters() guarantee that the just-allocated cluster is zero-initialized (and/or should we add a flag to the function to allow the caller to choose whether to force zero allocation instead of leaving uninitialized)? See [4] below for why I ask.
Quoted 21 lines
block/qcow2-refcount.c +21
-0+if (offset <0 ) { +error_setg_errno (errp, -offset,"Failed to allocate refblock" ); +return offset; + } + (*reftable)[reftable_index++] = offset; + } + +return 0 ; +} + +/** +* This "operation" for walk_over_reftable() writes the refblock to disk at the +* offset specified by the new reftable's entry. It does not modify the new +* reftable or change any refcounts. +*/ +static int flush_refblock (BlockDriverState *bs,uint64_t **reftable, +uint64_t reftable_index,uint64_t *reftable_size, +void *refblock,bool refblock_empty,Error **errp) +{ +BDRVQcowState *s = bs->opaque; +int64_t offset;
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2552
Quoted 10 lines
block/qcow2-refcount.c +10
-0+int ret; + +if (refblock_empty) { +if (reftable_index < *reftable_size) { +assert ((*reftable)[reftable_index] ==0 ); + } + }else { +/* The first pass with alloc_refblock() made the reftable large enough +*/ +assert (reftable_index < *reftable_size);
Okay, I see why you couldn't hoist this assert outside of the if - the caller may call this with refblock_empty for any refblocks at the tail of the final partial reftable cluster.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2565
Quoted 13 lines
block/qcow2-refcount.c +13
-0+ offset = (*reftable)[reftable_index]; +assert (offset !=0 ); + + ret =qcow2_pre_write_overlap_check (bs,0 , offset, s->cluster_size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Overlap check failed" ); +return ret; + } + + ret =bdrv_pwrite (bs->file, offset, refblock, s->cluster_size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to write refblock" ); +return ret;
If we fail here, do we leak all clusters written so far? At least the image is still consistent. After reading further, I think I answered myself at point [5].
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ } + } + +return 0 ; +} + +/** +* This function walks over the existing reftable and every referenced refblock; +* if @new_set_refcount is non-NULL, it is called for every refcount entry to +* create an equal new entry in the passed @new_refblock. Once that
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2585
Quoted 10 lines
block/qcow2-refcount.c +10
-0+* @new_refblock is completely filled, @operation will be called. +* +* @operation is expected to combine the @new_refblock and its entry in the new +* reftable (which is described by the parameters starting with "reftable"). +* @refblock_empty is set if all entries in the refblock are zero. +* +* @status_cb and @cb_opaque are used for the amend operation's status callback. +* @index is the index of the walk_over_reftable() calls and @total is the total +* number of walk_over_reftable() calls per amend operation. Both are used for +* calculating the parameters for the status callback.
Nice writeup; I was referring to it frequently during review.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2598
Quoted 13 lines
+ */
block/qcow2-refcount.c +12
-0+static int walk_over_reftable(BlockDriverState *bs, uint64_t **new_reftable, + uint64_t *new_reftable_index, + uint64_t *new_reftable_size, + void *new_refblock, int new_refblock_size, + int new_refcount_bits, + int (*operation)(BlockDriverState *bs, + uint64_t **reftable, + uint64_t reftable_index, + uint64_t *reftable_size, + void *refblock, + bool refblock_empty, + Error **errp),
Worth a typedef? Maybe not; I managed.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2602
Quoted 4 lines
block/qcow2-refcount.c +4
-0+ Qcow2SetRefcountFunc *new_set_refcount, + BlockDriverAmendStatusCB *status_cb, + void *cb_opaque, int index, int total, + Error **errp)
After several reads of the patch, I see that this walk function gets called twice - first with a NULL new_set_refcount (merely to figure out how big the new reftable should be, as well as allocating all necessary non-zero refcount blocks, but not committing the top-level reftable to any particular file location); the second walk then commits the new refcounts to disk (updating each non-zero entry in all the new refcount blocks to match their original counterparts, but no allocation required). Pretty slick to ensure that we are sure that the new table is feasible before actually swapping over to it, while still allowing a fairly clean rollback on early failure.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2612
Quoted 10 lines
block/qcow2-refcount.c +10
-0+{ +BDRVQcowState *s = bs->opaque; +uint64_t reftable_index; +bool new_refblock_empty = true; +int refblock_index; +int new_refblock_index =0 ; +int ret; + +for (reftable_index =0 ; reftable_index < s->refcount_table_size; + reftable_index++)
Outer loop - for each cluster of the top-level reference table, visit each child table and update the status callback. On the first walk, s->refcount_table_size might be increasing during calls to operation().
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2619
Quoted 7 lines
block/qcow2-refcount.c +7
-0+ { +uint64_t refblock_offset = s->refcount_table[reftable_index] + &REFT_OFFSET_MASK ; + +status_cb (bs, (uint64_t )index * s->refcount_table_size + reftable_index, + (uint64_t )total * s->refcount_table_size, cb_opaque); +
This never quite reaches 100%, and the caller also never reaches 100%. I think you want one more call to status_cb() at the end of the loop (at either site [1] or [2]) that passes an equal index and total to make it obvious that this (portion of the) long-running conversion is complete.
Since s->refcount_table_size may grow during the loop, the callback
does not necessarily have a constant total size; good thing we already documented that progress bars need not have a constant total.
Quoted 13 lines
block/qcow2-refcount.c +13
-0+if (refblock_offset) { +void *refblock; + +if (offset_into_cluster (s, refblock_offset)) { +qcow2_signal_corruption (bs, true,-1 ,-1 ,"Refblock offset %#" + PRIx64" unaligned (reftable index: %#" + PRIx64")" , refblock_offset, + reftable_index); +error_setg (errp, +"Image is corrupt (unaligned refblock offset)" ); +return -EIO ; + } +
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2642
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ ret =qcow2_cache_get (bs, s->refcount_block_cache, refblock_offset, + &refblock); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to retrieve refblock" ); +return ret; + } + +for (refblock_index =0 ; refblock_index < s->refcount_block_size; + refblock_index++) + {
If a child table (refcount block) exists, visit each refcount entry within the table (at least one refcount in that visit should be non-empty, otherwise we could garbage-collect the refblock and put a 0 entry in the outer loop).
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2649
Quoted 7 lines
block/qcow2-refcount.c +7
-0+uint64_t refcount; + +if (new_refblock_index >= new_refblock_size) { +/* new_refblock is now complete */ + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, + new_refblock_empty, errp);
The new refcount table will either be filled faster than the original (when going from small to large refcount - calling operation() multiple times per inner loop) or will be filled slower than the original (when going from large to small; operation() will only be called after several outer loops).
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2661
Quoted 12 lines
block/qcow2-refcount.c +12
-0+if (ret <0 ) { +qcow2_cache_put (bs, s->refcount_block_cache, &refblock); +return ret; + } + + (*new_reftable_index)++; + new_refblock_index =0 ; + new_refblock_empty = true; + } + + refcount = s->get_refcount (refblock, refblock_index); +if (new_refcount_bits <64 && refcount >> new_refcount_bits) {
Technically, this get_refcount() call is dead code on the second walk, since the first walk already validated things, so you could push all of this code...
Quoted 7 lines
block/qcow2-refcount.c +7
-0+uint64_t offset; + +qcow2_cache_put (bs, s->refcount_block_cache, &refblock); + + offset = ((reftable_index << s->refcount_block_bits) + + refblock_index) << s->cluster_bits; +
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2678
Quoted 10 lines
block/qcow2-refcount.c +10
-0+error_setg (errp,"Cannot decrease refcount entry width to " +"%i bits: Cluster at offset %#" PRIx64" has a " +"refcount of %" PRIu64, new_refcount_bits, + offset, refcount); +return -EINVAL ; + } + +if (new_set_refcount) { +new_set_refcount (new_refblock, new_refblock_index++, refcount); + }else {
...here, in the branch only run on the first walk.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2681
Quoted 3 lines
block/qcow2-refcount.c +3
-0+ new_refblock_index++; + } + new_refblock_empty = new_refblock_empty && refcount ==0 ;
Worth condensing to 'new_refblock_empty &= !refcount'? Maybe not.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2693
Quoted 12 lines
block/qcow2-refcount.c +12
-0+ } + + ret =qcow2_cache_put (bs, s->refcount_block_cache, &refblock); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to put refblock back into " +"the cache" ); +return ret; + } + }else { +/* No refblock means every refcount is 0 */ +for (refblock_index =0 ; refblock_index < s->refcount_block_size; + refblock_index++)
Again, visiting each (implied) entry for the given refcount block of the outer loop. When enlarging the width, each of the new blocks will also be all zero; but when shrinking the width, even though all entries on this pass are zero, we may be combining this pass with another outer loop with non-zero data for a non-zero block in the resulting new table.
Quoted 7 lines
block/qcow2-refcount.c +7
-0+ { +if (new_refblock_index >= new_refblock_size) { +/* new_refblock is now complete */ + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, + new_refblock_empty, errp); +if (ret <0 ) {
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2710
Quoted 10 lines
block/qcow2-refcount.c +10
-0+return ret; + } + + (*new_reftable_index)++; + new_refblock_index =0 ; + new_refblock_empty = true; + } + +if (new_set_refcount) { +new_set_refcount (new_refblock, new_refblock_index++,0 );
Would it be worth guaranteeing that every new refblock is 0-initialized when allocated, so that you can skip setting a refcount to 0? This question depends on the answer about block allocation asked at [4] above.
Quoted 4 lines
block/qcow2-refcount.c +4
-0+ }else { + new_refblock_index++; + } + }
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2724
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ } + } + +if (new_refblock_index >0 ) { +/* Complete the potentially existing partially filled final refblock */ +if (new_set_refcount) { +for (; new_refblock_index < new_refblock_size; + new_refblock_index++) + { +new_set_refcount (new_refblock, new_refblock_index,0 );
Again, if you 0-initialize refblocks when allocated, you could skip this (another instance of [4] above).
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2736
Quoted 12 lines
block/qcow2-refcount.c +12
-0+ } + } + + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, new_refblock_empty, + errp); +if (ret <0 ) { +return ret; + } + + (*new_reftable_index)++; + }
site [1] mentioned above, as a good place to make a final status callback at 100%. But if you do it here, it means that we call the status callback twice with the same values (the 100% value of the first loop is the 0% value of the second loop) - not the end of the world, but may impact any testsuite that tracks progress reports.
Quoted 16 lines
block/qcow2-refcount.c +16
-0+ +return 0 ; +} + +int qcow2_change_refcount_order (BlockDriverState *bs,int refcount_order, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,Error **errp) +{ +BDRVQcowState *s = bs->opaque; +Qcow2GetRefcountFunc *new_get_refcount; +Qcow2SetRefcountFunc *new_set_refcount; +void *new_refblock =qemu_blockalign (bs->file, s->cluster_size); +uint64_t *new_reftable =NULL , new_reftable_size =0 ; +uint64_t *old_reftable, old_reftable_size, old_reftable_offset; +uint64_t new_reftable_index =0 ; +uint64_t i;
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2762
Quoted 10 lines
block/qcow2-refcount.c +10
-0+int64_t new_reftable_offset; +int new_refblock_size, new_refcount_bits =1 << refcount_order; +int old_refcount_order; +int ret; + +assert (s->qcow_version >=3 ); +assert (refcount_order >=0 && refcount_order <=6 ); + +/* see qcow2_open() */ + new_refblock_size =1 << (s->cluster_bits - (refcount_order -3 ));
Safe (cluster_bits is always at least 9, and at most 21 in our current implementation, so we are shifting anywhere from 6 to 24 positions).
Quoted 9 lines
block/qcow2-refcount.c +9
-0+ +get_refcount_functions (refcount_order, + &new_get_refcount, &new_set_refcount); + + +/* First, allocate the structures so they are present in the refcount +* structures */ + ret = walk_over_reftable(bs, &new_reftable, &new_reftable_index, + &new_reftable_size, NULL, new_refblock_size,
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2781
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ new_refcount_bits, &alloc_refblock,NULL , + status_cb, cb_opaque, 0, 2, errp); +if (ret <0 ) { + goto done; + } + +/* The new_reftable_size is now valid and will not be changed anymore, +* so we can now allocate the reftable */ + new_reftable_offset =qcow2_alloc_clusters (bs, new_reftable_size * +sizeof (uint64_t ));
And here is your bug, that I hinted at with the mention of [3] above. This allocation can potentially cause an overflow of the existing reftable to occupy one more cluster. Remember my thought experiment above, how an 8 megabyte image rolls from 1 to 2 clusters during the course of allocating refblocks for the new table? What if the original image wasn't completely full, but things are perfectly sized with enough free clusters, then all of the refblock allocations done during the first walk will still fit, and it is only this final allocation of the new reftable that will cause the rollover, at which point we've failed to account for the new refblock size. That is, I think I could craft an image that would trigger either an assertion failure or an out-of-bounds array access during the second walk.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2786
Quoted 5 lines
block/qcow2-refcount.c +5
-0+if (new_reftable_offset <0 ) { +error_setg_errno (errp, -new_reftable_offset, +"Failed to allocate the new reftable" ); + ret = new_reftable_offset; + goto done;
If we fail here, do we leak allocations of the refblocks? I guess not; based on another forward reference to point [5].
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2799
Quoted 13 lines
block/qcow2-refcount.c +13
-0+ } + + new_reftable_index =0 ; + +/* Second, write the new refblocks */ + ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size, new_refblock, + new_refblock_size, new_refcount_bits, + &flush_refblock, new_set_refcount, + status_cb, cb_opaque,1 ,2 , errp); +if (ret <0 ) { + goto done; + }
If we fail here, it looks like we DO leak the clusters allocated for the new reftable (again, point [5]).
Quoted 15 lines
block/qcow2-refcount.c +15
-0+ + +/* Write the new reftable */ + ret =qcow2_pre_write_overlap_check (bs,0 , new_reftable_offset, + new_reftable_size *sizeof (uint64_t )); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Overlap check failed" ); + goto done; + } + +for (i =0 ; i < new_reftable_size; i++) { +cpu_to_be64s (&new_reftable[i]); + } + + ret = bdrv_pwrite(bs->file, new_reftable_offset, new_reftable,
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2824
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ new_reftable_size * sizeof(uint64_t)); + +for (i =0 ; i < new_reftable_size; i++) { +be64_to_cpus (&new_reftable[i]); + } + +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to write the new reftable" ); + goto done; + }
Looks like you correctly maintain the in-memory copy in preferred cpu byte order, while writing to disk in big-endian order.
Quoted 19 lines
block/qcow2-refcount.c +19
-0+ + +/* Empty the refcount cache */ + ret =qcow2_cache_flush (bs, s->refcount_block_cache); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to flush the refblock cache" ); + goto done; + } + +/* Update the image header to point to the new reftable; this only updates +* the fields which are relevant to qcow2_update_header(); other fields +* such as s->refcount_table or s->refcount_bits stay stale for now +* (because we have to restore everything if qcow2_update_header() fails) */ + old_refcount_order = s->refcount_order; + old_reftable_size = s->refcount_table_size; + old_reftable_offset = s->refcount_table_offset; + + s->refcount_order = refcount_order; + s->refcount_table_size = new_reftable_size;
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2853
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ s->refcount_table_offset = new_reftable_offset; + + ret =qcow2_update_header (bs); +if (ret <0 ) { + s->refcount_order = old_refcount_order; + s->refcount_table_size = old_reftable_size; + s->refcount_table_offset = old_reftable_offset; +error_setg_errno (errp, -ret,"Failed to update the qcow2 header" ); + goto done; + }
Failures up to here still have issues leaking the new reftable allocation (point [5]).
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ +/* Now update the rest of the in-memory information */ + old_reftable = s->refcount_table; + s->refcount_table = new_reftable; +
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2868
Quoted 10 lines
block/qcow2-refcount.c +10
-0+/* For cleaning up all old refblocks */ + new_reftable = old_reftable; + new_reftable_size = old_reftable_size; + + s->refcount_bits =1 << refcount_order; +if (refcount_order <6 ) { + s->refcount_max = (UINT64_C (1 ) << s->refcount_bits) -1 ; + }else { + s->refcount_max =INT64_MAX ; + }
Is it worth factoring this computation into a common helper, since it appeared in an earlier patch as well?
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2880
Quoted 12 lines
block/qcow2-refcount.c +12
-0+ + s->refcount_block_bits = s->cluster_bits - (refcount_order -3 ); + s->refcount_block_size =1 << s->refcount_block_bits; + + s->get_refcount = new_get_refcount; + s->set_refcount = new_set_refcount; + +/* And free the old reftable (the old refblocks are freed below the "done" +* label) */ +qcow2_free_clusters (bs, old_reftable_offset, + old_reftable_size *sizeof (uint64_t ), +QCOW2_DISCARD_NEVER );
site [2] mentioned above, as a possible point where you might want to ensure the callback is called with equal progress and total values to ensure the caller knows the job is done. Except this site doesn't have quite as much information as site [1] about what total size all the other status callbacks were using.
Quoted 4 lines
block/qcow2-refcount.c +4
-0+ +done: +if (new_reftable) { +/* On success, new_reftable actually points to the old reftable (and
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2894
Quoted 10 lines
block/qcow2-refcount.c +10
-0+* new_reftable_size is the old reftable's size); but that is just +* fine */ +for (i =0 ; i < new_reftable_size; i++) { +uint64_t offset = new_reftable[i] &REFT_OFFSET_MASK ; +if (offset) { +qcow2_free_clusters (bs, offset, s->cluster_size, +QCOW2_DISCARD_NEVER ); + } + } +g_free (new_reftable);
So here is point [5] - if we failed early, this tries to clean up all allocated refblocks associated with the new table. It does NOT clean up any refblocks allocated due to resizing the old table to be slightly larger, but that should be fine (not a leak, so much as an image that is now a couple clusters larger than the minimum required size). However, while you clean up the clusters associated with refblocks (layer 2), the cleanup of old clusters associated with the reftable (layer 1) happened before the done: label on success, but that means that on failure, you are NOT cleaning up the clusters associated with the new reftable.
Quoted 11 lines
block/qcow2-refcount.c +5
-0+ } + +qemu_vfree (new_refblock); +return ret; +}block/qcow2.h +0
-0
block/qcow2.h int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, · line 535
Quoted 10 lines
block/qcow2.h +4
-0@@ -526,6 +526,10 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,int qcow2_pre_write_overlap_check (BlockDriverState *bs,int ign,int64_t offset,int64_t size); +int qcow2_change_refcount_order (BlockDriverState *bs,int refcount_order, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,Error **errp); +/* qcow2-cluster.c functions */ int qcow2_grow_l1_table (BlockDriverState *bs,uint64_t min_size,bool exact_size);
Interesting patch. Hope my review helps you prepare a better v2.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 18/21] qcow2: Add function for refcount order amendment
Quoted 11 lines · Eric Blake
Quoted 8 lines · Max Reitz
Add a function
qcow2_change_refcount_order()which allows changing the refcount order of a qcow2 image.Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 424 +++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 + 2 files changed, 428 insertions(+)This is fairly big; you may want to get a second review from a maintainer rather than blindly trusting me.
I didn't really see the point in splitting it up. Introducing the static helper functions first would not be very helpful either, so I thought what I'd do as a reviewer, and that was "apply the patch and just read through all the code". Splitting it into multiple patches would not have helped there (and I don't see how I could split this patch into logical changes, where at the start we have some inefficient but simple implementation and it gets better over time).
I'll need a review from a maintainer anyway, but I won't get one without being able to show another review first...
Quoted 294 lines
My review was not linear, but I left the email in linear order. Feel free to ask for clarification if my presentation is too hard to follow.
Quoted 18 lines
block/qcow2-refcount.c +18
-0+ +/** +* This "operation" for walk_over_reftable() allocates the refblock on disk (if +* it is not empty) and inserts its offset into the new reftable. The size of +* this new reftable is increased as required. +*/ +static int alloc_refblock (BlockDriverState *bs,uint64_t **reftable, +uint64_t reftable_index,uint64_t *reftable_size, +void *refblock,bool refblock_empty,Error **errp) +{ +BDRVQcowState *s = bs->opaque; +int64_t offset; + +if (!refblock_empty && reftable_index >= *reftable_size) { +uint64_t *new_reftable; +uint64_t new_reftable_size; + + new_reftable_size = ROUND_UP(reftable_index + 1,block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2503
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ s->cluster_size / sizeof(uint64_t)); +if (new_reftable_size >QCOW_MAX_REFTABLE_SIZE /sizeof (uint64_t )) { +error_setg (errp, +"This operation would make the refcount table grow " +"beyond the maximum size supported by QEMU, aborting" ); +return -ENOTSUP ; + } + + new_reftable =g_try_realloc (*reftable, new_reftable_size * +sizeof (uint64_t ));Safe from overflow based on checks a few lines earlier. Good.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2513
Quoted 10 lines
block/qcow2-refcount.c +10
-0+if (!new_reftable) { +error_setg (errp,"Failed to increase reftable buffer size" ); +return -ENOMEM ; + } + +memset (new_reftable + *reftable_size,0 , + (new_reftable_size - *reftable_size) *sizeof (uint64_t )); + + *reftable = new_reftable; + *reftable_size = new_reftable_size;Just to check my math here:
Suppose we have an image with 512-byte clusters, and are changing from 16-bit refcount (order 4) to 64-bit refcount. Also, suppose the existing image has exactly filled a one-cluster refcount table (that is, there are 64 refcount blocks, each describing at a refblock with all 256 refcount entries full, for a total image size of exactly 8M). The original image occupies a header (1 cluster), L1 and L2 tables, and data; but 65 of the 16k clusters tied up in the image are dedicated to the refcount structures.
Meanwhile, the new refcount table will have to point to 256 refcount blocks, each holding only 64 entries, which in turn implies that the refcount table now has to be at least 4 clusters long. But as this requires at least 260 clusters to represent, then even if we were able to reuse the 65 clusters of the original table, we'd still be allocating at least 195 clusters; in reality, your code doesn't free any old clusters until after allocating the new, because it is easier to keep the old table live until the new table is populated. The process of allocating the new clusters means we actually end up with a new refcount table of 5 clusters long, where not all 320 refblocks will be populated.
But as long as we are keeping the old table up-to-date for the refblock
allocations, it ALSO means that we caused a rollover of the old table from 1 cluster into 2, which itself consumes several clusters (the larger table must be contiguous, and we must also set up a refblock to describe the larger table, so we've added at least three clusters associated to the original table during the course of preparing the new table).Hmm - that means I found a bug in your implementation. See [3] below.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2518
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ } + +if (refblock_empty) { +if (reftable_index < *reftable_size) { + (*reftable)[reftable_index] =0 ;Necessary since you used
g_try_reallocwhich leaves the new reftable uninitialized. Reasonable (rather than a memset) since the caller will be visiting every single refblock in the table anyways.block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2521
Quoted 3 lines
block/qcow2-refcount.c +3
-0+ } + }else { + offset =qcow2_alloc_clusters (bs, s->cluster_size);As mentioned above, this action will potentially change
s->refcount_table_sizeof the original table, which in turn makes the caller execute its loops more often to cover the increased allocation. Doesqcow2_alloc_clusters()guarantee that the just-allocated cluster is zero-initialized (and/or should we add a flag to the function to allow the caller to choose whether to force zero allocation instead of leaving uninitialized)? See [4] below for why I ask.Quoted 21 lines
block/qcow2-refcount.c +21
-0+if (offset <0 ) { +error_setg_errno (errp, -offset,"Failed to allocate refblock" ); +return offset; + } + (*reftable)[reftable_index++] = offset; + } + +return 0 ; +} + +/** +* This "operation" for walk_over_reftable() writes the refblock to disk at the +* offset specified by the new reftable's entry. It does not modify the new +* reftable or change any refcounts. +*/ +static int flush_refblock (BlockDriverState *bs,uint64_t **reftable, +uint64_t reftable_index,uint64_t *reftable_size, +void *refblock,bool refblock_empty,Error **errp) +{ +BDRVQcowState *s = bs->opaque; +int64_t offset;block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2552
Quoted 10 lines
block/qcow2-refcount.c +10
-0+int ret; + +if (refblock_empty) { +if (reftable_index < *reftable_size) { +assert ((*reftable)[reftable_index] ==0 ); + } + }else { +/* The first pass with alloc_refblock() made the reftable large enough +*/ +assert (reftable_index < *reftable_size);Okay, I see why you couldn't hoist this assert outside of the if - the caller may call this with
refblock_emptyfor any refblocks at the tail of the final partial reftable cluster.block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2565
Quoted 13 lines
block/qcow2-refcount.c +13
-0+ offset = (*reftable)[reftable_index]; +assert (offset !=0 ); + + ret =qcow2_pre_write_overlap_check (bs,0 , offset, s->cluster_size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Overlap check failed" ); +return ret; + } + + ret =bdrv_pwrite (bs->file, offset, refblock, s->cluster_size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to write refblock" ); +return ret;If we fail here, do we leak all clusters written so far? At least the image is still consistent. After reading further, I think I answered myself at point [5].
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ } + } + +return 0 ; +} + +/** +* This function walks over the existing reftable and every referenced refblock; +* if @new_set_refcount is non-NULL, it is called for every refcount entry to +* create an equal new entry in the passed @new_refblock. Once that block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2585
Quoted 10 lines
block/qcow2-refcount.c +10
-0+* @new_refblock is completely filled, @operation will be called. +* +* @operation is expected to combine the @new_refblock and its entry in the new +* reftable (which is described by the parameters starting with "reftable"). +* @refblock_empty is set if all entries in the refblock are zero. +* +* @status_cb and @cb_opaque are used for the amend operation's status callback. +* @index is the index of the walk_over_reftable() calls and @total is the total +* number of walk_over_reftable() calls per amend operation. Both are used for +* calculating the parameters for the status callback. Nice writeup; I was referring to it frequently during review.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2598
Quoted 13 lines
+ */
block/qcow2-refcount.c +12
-0+static int walk_over_reftable(BlockDriverState *bs, uint64_t **new_reftable, + uint64_t *new_reftable_index, + uint64_t *new_reftable_size, + void *new_refblock, int new_refblock_size, + int new_refcount_bits, + int (*operation)(BlockDriverState *bs, + uint64_t **reftable, + uint64_t reftable_index, + uint64_t *reftable_size, + void *refblock, + bool refblock_empty, + Error **errp),Worth a typedef? Maybe not; I managed.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2602
Quoted 4 lines
block/qcow2-refcount.c +4
-0+ Qcow2SetRefcountFunc *new_set_refcount, + BlockDriverAmendStatusCB *status_cb, + void *cb_opaque, int index, int total, + Error **errp)After several reads of the patch, I see that this walk function gets called twice - first with a NULL
new_set_refcount(merely to figure out how big the new reftable should be, as well as allocating all necessary non-zero refcount blocks, but not committing the top-level reftable to any particular file location); the second walk then commits the new refcounts to disk (updating each non-zero entry in all the new refcount blocks to match their original counterparts, but no allocation required). Pretty slick to ensure that we are sure that the new table is feasible before actually swapping over to it, while still allowing a fairly clean rollback on early failure.block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2612
Quoted 10 lines
block/qcow2-refcount.c +10
-0+{ +BDRVQcowState *s = bs->opaque; +uint64_t reftable_index; +bool new_refblock_empty = true; +int refblock_index; +int new_refblock_index =0 ; +int ret; + +for (reftable_index =0 ; reftable_index < s->refcount_table_size; + reftable_index++)Outer loop - for each cluster of the top-level reference table, visit each child table and update the status callback. On the first walk,
s->refcount_table_sizemight be increasing during calls tooperation().block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2619
Quoted 7 lines
block/qcow2-refcount.c +7
-0+ { +uint64_t refblock_offset = s->refcount_table[reftable_index] + &REFT_OFFSET_MASK ; + +status_cb (bs, (uint64_t )index * s->refcount_table_size + reftable_index, + (uint64_t )total * s->refcount_table_size, cb_opaque); +This never quite reaches 100%, and the caller also never reaches 100%. I think you want one more call to
status_cb()at the end of the loop (at either site [1] or [2]) that passes an equal index and total to make it obvious that this (portion of the) long-running conversion is complete.
Sinces->refcount_table_sizemay grow during the loop, the callback
does not necessarily have a constant total size; good thing we already documented that progress bars need not have a constant total.Quoted 13 lines
block/qcow2-refcount.c +13
-0+if (refblock_offset) { +void *refblock; + +if (offset_into_cluster (s, refblock_offset)) { +qcow2_signal_corruption (bs, true,-1 ,-1 ,"Refblock offset %#" + PRIx64" unaligned (reftable index: %#" + PRIx64")" , refblock_offset, + reftable_index); +error_setg (errp, +"Image is corrupt (unaligned refblock offset)" ); +return -EIO ; + } +block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2642
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ ret =qcow2_cache_get (bs, s->refcount_block_cache, refblock_offset, + &refblock); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to retrieve refblock" ); +return ret; + } + +for (refblock_index =0 ; refblock_index < s->refcount_block_size; + refblock_index++) + {If a child table (refcount block) exists, visit each refcount entry within the table (at least one refcount in that visit should be non-empty, otherwise we could garbage-collect the refblock and put a 0 entry in the outer loop).
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2649
Quoted 7 lines
block/qcow2-refcount.c +7
-0+uint64_t refcount; + +if (new_refblock_index >= new_refblock_size) { +/* new_refblock is now complete */ + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, + new_refblock_empty, errp);The new refcount table will either be filled faster than the original (when going from small to large refcount - calling
operation()multiple times per inner loop) or will be filled slower than the original (when going from large to small;operation()will only be called after several outer loops).block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2661
Quoted 12 lines
block/qcow2-refcount.c +12
-0+if (ret <0 ) { +qcow2_cache_put (bs, s->refcount_block_cache, &refblock); +return ret; + } + + (*new_reftable_index)++; + new_refblock_index =0 ; + new_refblock_empty = true; + } + + refcount = s->get_refcount (refblock, refblock_index); +if (new_refcount_bits <64 && refcount >> new_refcount_bits) {Technically, this
get_refcount()call is dead code on the second walk, since the first walk already validated things, so you could push all of this code...Quoted 10 lines
block/qcow2-refcount.c +10
-0+uint64_t offset; + +qcow2_cache_put (bs, s->refcount_block_cache, &refblock); + + offset = ((reftable_index << s->refcount_block_bits) + + refblock_index) << s->cluster_bits; + + error_setg(errp, "Cannot decrease refcount entry width to " + "%i bits: Cluster at offset %#" PRIx64 " has a " + "refcount of %" PRIu64, new_refcount_bits,
Quoted 8 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2678
Quoted 7 lines
block/qcow2-refcount.c +7
-0+ offset, refcount); +return -EINVAL ; + } + +if (new_set_refcount) { +new_set_refcount (new_refblock, new_refblock_index++, refcount); + }else {...here, in the branch only run on the first walk.
Well, yes, but I wanted to keep this function as agnostic to what the caller wants to do with it as possible. I'd rather decide depending on whether index == 0 because that's a better way of discerning the first walk.
Quoted 4 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2681
Quoted 3 lines
block/qcow2-refcount.c +3
-0+ new_refblock_index++; + } + new_refblock_empty = new_refblock_empty && refcount ==0 ;Worth condensing to '
new_refblock_empty&= !refcount'? Maybe not.
I personally would find that harder to read.
Quoted 30 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2693
Quoted 12 lines
block/qcow2-refcount.c +12
-0+ } + + ret =qcow2_cache_put (bs, s->refcount_block_cache, &refblock); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to put refblock back into " +"the cache" ); +return ret; + } + }else { +/* No refblock means every refcount is 0 */ +for (refblock_index =0 ; refblock_index < s->refcount_block_size; + refblock_index++)Again, visiting each (implied) entry for the given refcount block of the outer loop. When enlarging the width, each of the new blocks will also be all zero; but when shrinking the width, even though all entries on this pass are zero, we may be combining this pass with another outer loop with non-zero data for a non-zero block in the resulting new table.
Quoted 12 lines
block/qcow2-refcount.c +12
-0+ { +if (new_refblock_index >= new_refblock_size) { +/* new_refblock is now complete */ + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, + new_refblock_empty, errp); +if (ret <0 ) { +return ret; + } + + (*new_reftable_index)++; + new_refblock_index =0 ;
Quoted 8 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2710
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ new_refblock_empty = true; + } + +if (new_set_refcount) { +new_set_refcount (new_refblock, new_refblock_index++,0 );Would it be worth guaranteeing that every new refblock is 0-initialized when allocated, so that you can skip setting a refcount to 0? This question depends on the answer about block allocation asked at [4] above.
This function sets a value in the buffer new_refblock, not in the cluster on disk. Therefore, in order to be able to omit this call, we'd have to call a memset() with 0 on new_refblock after each call to operation(). I don't think it's worth it. This is more explicit and won't cost much performance.
Quoted 26 lines
Quoted 4 lines
block/qcow2-refcount.c +4
-0+ }else { + new_refblock_index++; + } + }block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2724
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ } + } + +if (new_refblock_index >0 ) { +/* Complete the potentially existing partially filled final refblock */ +if (new_set_refcount) { +for (; new_refblock_index < new_refblock_size; + new_refblock_index++) + { +new_set_refcount (new_refblock, new_refblock_index,0 );Again, if you 0-initialize refblocks when allocated, you could skip this (another instance of [4] above).
Quoted 9 lines
block/qcow2-refcount.c +9
-0+ } + } + + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, new_refblock_empty, + errp); +if (ret <0 ) { +return ret; + }
Quoted 8 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2658
Quoted 3 lines
block/qcow2-refcount.c +3
-0+ + (*new_reftable_index)++; + }site [1] mentioned above, as a good place to make a final status callback at 100%. But if you do it here, it means that we call the status callback twice with the same values (the 100% value of the first loop is the 0% value of the second loop) - not the end of the world, but may impact any testsuite that tracks progress reports.
Well, why not.
Quoted 43 lines
Quoted 16 lines
block/qcow2-refcount.c +16
-0+ +return 0 ; +} + +int qcow2_change_refcount_order (BlockDriverState *bs,int refcount_order, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,Error **errp) +{ +BDRVQcowState *s = bs->opaque; +Qcow2GetRefcountFunc *new_get_refcount; +Qcow2SetRefcountFunc *new_set_refcount; +void *new_refblock =qemu_blockalign (bs->file, s->cluster_size); +uint64_t *new_reftable =NULL , new_reftable_size =0 ; +uint64_t *old_reftable, old_reftable_size, old_reftable_offset; +uint64_t new_reftable_index =0 ; +uint64_t i;block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2762
Quoted 10 lines
block/qcow2-refcount.c +10
-0+int64_t new_reftable_offset; +int new_refblock_size, new_refcount_bits =1 << refcount_order; +int old_refcount_order; +int ret; + +assert (s->qcow_version >=3 ); +assert (refcount_order >=0 && refcount_order <=6 ); + +/* see qcow2_open() */ + new_refblock_size =1 << (s->cluster_bits - (refcount_order -3 ));Safe (
cluster_bitsis always at least 9, and at most 21 in our current implementation, so we are shifting anywhere from 6 to 24 positions).Quoted 14 lines
block/qcow2-refcount.c +14
-0+ +get_refcount_functions (refcount_order, + &new_get_refcount, &new_set_refcount); + + +/* First, allocate the structures so they are present in the refcount +* structures */ + ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size,NULL , new_refblock_size, + new_refcount_bits, &alloc_refblock,NULL , + status_cb, cb_opaque,0 ,2 , errp); +if (ret <0 ) { + goto done; + }
Quoted 8 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2781
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ +/* The new_reftable_size is now valid and will not be changed anymore, +* so we can now allocate the reftable */ + new_reftable_offset =qcow2_alloc_clusters (bs, new_reftable_size * +sizeof (uint64_t ));And here is your bug, that I hinted at with the mention of [3] above. This allocation can potentially cause an overflow of the existing reftable to occupy one more cluster.
An additional bug is that the new reftable may be referenced by an existing refblock which was completely empty, though (or at least referenced by part of an existing refblock which was to be turned into a new refblock which was then completely empty, and thus omitted from allocation).
Quoted 10 lines
Remember my thought experiment
above, how an 8 megabyte image rolls from 1 to 2 clusters during the course of allocating refblocks for the new table? What if the original image wasn't completely full, but things are perfectly sized with enough free clusters, then all of the refblock allocations done during the first walk will still fit, and it is only this final allocation of the new reftable that will cause the rollover, at which point we've failed to account for the new refblock size. That is, I think I could craft an image that would trigger either an assertion failure or an out-of-bounds array access during the second walk.
You're completely right, this will be a pain to fix, though... The simplest way would probably to check whether the new_reftable_size should be increased due to this operation and if it did, rerun walk_over_reftable() with the alloc_refblock() function only allocating refblocks if the new reftable does not already point to a refblock for a certain index. This would be repeated until the new_reftable_size is constant. And the really simplest incarnation of this would be to have a flag whether any allocations were done and repeat until everything is fine.
Another way would be to somehow integrate the allocation of the new reftable into walk_over_reftable() and then to only mind the additional reftable entries.
But probably the first way is the correct one because due to reallocation of the old reftable, some intermediate refblocks which were empty before are now filled.
I'll have to craft some test images myself, not least to be able to include them in the iotest.
Quoted 93 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2786
Quoted 5 lines
block/qcow2-refcount.c +5
-0+if (new_reftable_offset <0 ) { +error_setg_errno (errp, -new_reftable_offset, +"Failed to allocate the new reftable" ); + ret = new_reftable_offset; + goto done;If we fail here, do we leak allocations of the refblocks? I guess not; based on another forward reference to point [5].
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2799
Quoted 13 lines
block/qcow2-refcount.c +13
-0+ } + + new_reftable_index =0 ; + +/* Second, write the new refblocks */ + ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size, new_refblock, + new_refblock_size, new_refcount_bits, + &flush_refblock, new_set_refcount, + status_cb, cb_opaque,1 ,2 , errp); +if (ret <0 ) { + goto done; + }If we fail here, it looks like we DO leak the clusters allocated for the new reftable (again, point [5]).
Quoted 15 lines
block/qcow2-refcount.c +15
-0+ + +/* Write the new reftable */ + ret =qcow2_pre_write_overlap_check (bs,0 , new_reftable_offset, + new_reftable_size *sizeof (uint64_t )); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Overlap check failed" ); + goto done; + } + +for (i =0 ; i < new_reftable_size; i++) { +cpu_to_be64s (&new_reftable[i]); + } + + ret = bdrv_pwrite(bs->file, new_reftable_offset, new_reftable,block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2824
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ new_reftable_size * sizeof(uint64_t)); + +for (i =0 ; i < new_reftable_size; i++) { +be64_to_cpus (&new_reftable[i]); + } + +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to write the new reftable" ); + goto done; + }Looks like you correctly maintain the in-memory copy in preferred cpu byte order, while writing to disk in big-endian order.
Quoted 19 lines
block/qcow2-refcount.c +19
-0+ + +/* Empty the refcount cache */ + ret =qcow2_cache_flush (bs, s->refcount_block_cache); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to flush the refblock cache" ); + goto done; + } + +/* Update the image header to point to the new reftable; this only updates +* the fields which are relevant to qcow2_update_header(); other fields +* such as s->refcount_table or s->refcount_bits stay stale for now +* (because we have to restore everything if qcow2_update_header() fails) */ + old_refcount_order = s->refcount_order; + old_reftable_size = s->refcount_table_size; + old_reftable_offset = s->refcount_table_offset; + + s->refcount_order = refcount_order; + s->refcount_table_size = new_reftable_size;block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2853
Quoted 10 lines
block/qcow2-refcount.c +10
-0+ s->refcount_table_offset = new_reftable_offset; + + ret =qcow2_update_header (bs); +if (ret <0 ) { + s->refcount_order = old_refcount_order; + s->refcount_table_size = old_reftable_size; + s->refcount_table_offset = old_reftable_offset; +error_setg_errno (errp, -ret,"Failed to update the qcow2 header" ); + goto done; + }Failures up to here still have issues leaking the new reftable allocation (point [5]).
Quoted 9 lines
block/qcow2-refcount.c +9
-0+ +/* Now update the rest of the in-memory information */ + old_reftable = s->refcount_table; + s->refcount_table = new_reftable; + +/* For cleaning up all old refblocks */ + new_reftable = old_reftable; + new_reftable_size = old_reftable_size; +
Quoted 8 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2868
Quoted 6 lines
block/qcow2-refcount.c +6
-0+ s->refcount_bits =1 << refcount_order; +if (refcount_order <6 ) { + s->refcount_max = (UINT64_C (1 ) << s->refcount_bits) -1 ; + }else { + s->refcount_max =INT64_MAX ; + }Is it worth factoring this computation into a common helper, since it appeared in an earlier patch as well?
Well, it does appear in qcow2_open(), as do the next two lines. The only reason to factor them out would be so that neither place is forgotten if one of them is changed; but the only reason I could imagine for this would be to replace the INT64_MAX by UINT64_MAX at some point in the future, but I guess that point is still far away (because there's no reasonable way someone would be needing the 64th bit, as you agreed on) , so it should be fine that way.
Quoted 33 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2880
Quoted 12 lines
block/qcow2-refcount.c +12
-0+ + s->refcount_block_bits = s->cluster_bits - (refcount_order -3 ); + s->refcount_block_size =1 << s->refcount_block_bits; + + s->get_refcount = new_get_refcount; + s->set_refcount = new_set_refcount; + +/* And free the old reftable (the old refblocks are freed below the "done" +* label) */ +qcow2_free_clusters (bs, old_reftable_offset, + old_reftable_size *sizeof (uint64_t ), +QCOW2_DISCARD_NEVER );site [2] mentioned above, as a possible point where you might want to ensure the callback is called with equal progress and total values to ensure the caller knows the job is done. Except this site doesn't have quite as much information as site [1] about what total size all the other status callbacks were using.
Quoted 4 lines
block/qcow2-refcount.c +4
-0+ +done: +if (new_reftable) { +/* On success, new_reftable actually points to the old reftable (and block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2894
Quoted 10 lines
block/qcow2-refcount.c +10
-0+* new_reftable_size is the old reftable's size); but that is just +* fine */ +for (i =0 ; i < new_reftable_size; i++) { +uint64_t offset = new_reftable[i] &REFT_OFFSET_MASK ; +if (offset) { +qcow2_free_clusters (bs, offset, s->cluster_size, +QCOW2_DISCARD_NEVER ); + } + } +g_free (new_reftable);So here is point [5] - if we failed early, this tries to clean up all
Quoted 8 lines
allocated refblocks associated with the new table. It does NOT clean up any refblocks allocated due to resizing the old table to be slightly larger, but that should be fine (not a leak, so much as an image that is now a couple clusters larger than the minimum required size). However, while you clean up the clusters associated with refblocks (layer 2), the cleanup of old clusters associated with the reftable (layer 1) happened before the done: label on success, but that means that on failure, you are NOT cleaning up the clusters associated with the new reftable.
Oops, right, will fix.
Quoted 13 lines
Quoted 13 lines
block/qcow2-refcount.c +5
-0+ } + +qemu_vfree (new_refblock); +return ret; +}block/qcow2.h +0
-0@@ -526,6 +526,10 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,int qcow2_pre_write_overlap_check (BlockDriverState *bs,int ign,int64_t offset,int64_t size);
Quoted 9 lines
block/qcow2.h int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, · line 535
Quoted 8 lines
block/qcow2.h +4
-0+int qcow2_change_refcount_order (BlockDriverState *bs,int refcount_order, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,Error **errp); +/* qcow2-cluster.c functions */ int qcow2_grow_l1_table (BlockDriverState *bs,uint64_t min_size,bool exact_size);Interesting patch. Hope my review helps you prepare a better v2.
If everything else fails, I'll just split the amend stuff from this series. But I'll work it out somehow. And your review will definitely help, thanks a lot!
Max
Re: [Qemu-devel] [PATCH 18/21] qcow2: Add function for refcount order amendment
Quoted 14 lines · Max Reitz
Quoted 4 lines · Eric Blake
Quoted 3 lines · Max Reitz
Add a function
qcow2_change_refcount_order()which allows changing the refcount order of a qcow2 image.Quoted 8 lines
Quoted 6 lines
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 424 +++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 + 2 files changed, 428 insertions(+)This is fairly big; you may want to get a second review from a maintainer rather than blindly trusting me.
Quoted 7 lines
I didn't really see the point in splitting it up. Introducing the static helper functions first would not be very helpful either, so I thought what I'd do as a reviewer, and that was "apply the patch and just read through all the code". Splitting it into multiple patches would not have helped there (and I don't see how I could split this patch into logical changes, where at the start we have some inefficient but simple implementation and it gets better over time).
Yes, I agree that your patch is not divisible. Big doesn't always mean bad :)
Quoted 32 lines
Quoted 23 lines
Quoted 3 lines
(quoted diff) +2
-0+ refcount = s->get_refcount(refblock, refblock_index); + if (new_refcount_bits < 64 && refcount >>
new_refcount_bits) {Technically, this
get_refcount()call is dead code on the second walk, since the first walk already validated things, so you could push all of this code...Quoted 16 lines
block/qcow2-refcount.c +3
-0+uint64_t offset; + + qcow2_cache_put(bs, s->refcount_block_cache,&refblock);
(quoted diff) +2
-0+ + offset = ((reftable_index <<
s->refcount_block_bits)block/qcow2-refcount.c +3
-0+ + refblock_index) << s->cluster_bits; + + error_setg(errp, "Cannot decrease refcount entrywidth to "
+ "%i bits: Cluster at offset %#"
PRIx64 " has a "
+ "refcount of %" PRIu64,new_refcount_bits,
+ offset, refcount);Quoted 8 lines
Quoted 7 lines
block/qcow2-refcount.c +5
-0+return -EINVAL ; + } + +if (new_set_refcount) { + new_set_refcount(new_refblock,new_refblock_index++, refcount);
+ } else {...here, in the branch only run on the first walk.
Quoted 4 lines
Well, yes, but I wanted to keep this function as agnostic to what the caller wants to do with it as possible. I'd rather decide depending on whether index == 0 because that's a better way of discerning the first walk.
I was thinking a bit more about how to avoid the allocation corner case bug, and wonder if three walks instead of 2 is the right solution, in which case the first two walks are both allocations, and index==0 is no longer a reliable witness of whether these checks are needed. More on that below.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
block/qcow2-refcount.c +3
-0+ new_refblock_index++; + } + new_refblock_empty = new_refblock_empty && refcount== 0;
Worth condensing to '
new_refblock_empty&= !refcount'? Maybe not.I personally would find that harder to read.
No need to change it then.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
(quoted diff) +2
-0+ if (new_set_refcount) { + new_set_refcount(new_refblock,new_refblock_index++, 0);
Would it be worth guaranteeing that every new refblock is 0-initialized when allocated, so that you can skip setting a refcount to 0? This question depends on the answer about block allocation asked at [4] above.
Quoted 5 lines
This function sets a value in the buffer
new_refblock, not in the cluster on disk. Therefore, in order to be able to omit this call, we'd have to call amemset()with 0 onnew_refblockafter each call tooperation(). I don't think it's worth it. This is more explicit and won't cost much performance.
Yeah, that's about the same conclusion I came to after finishing the whole review, although I didn't state it very well (this was one of my earlier comments in my non-linear review, that I didn't touch up later).
Quoted 39 lines
Quoted 8 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2781
Quoted 5 lines
+ /* The
new_reftable_sizeis now valid and will not be changed
anymore,block/qcow2-refcount.c +3
-0+* so we can now allocate the reftable */ + new_reftable_offset =qcow2_alloc_clusters (bs, new_reftable_size * +sizeof (uint64_t ));And here is your bug, that I hinted at with the mention of [3] above. This allocation can potentially cause an overflow of the existing reftable to occupy one more cluster.
An additional bug is that the new reftable may be referenced by an existing refblock which was completely empty, though (or at least referenced by part of an existing refblock which was to be turned into a new refblock which was then completely empty, and thus omitted from allocation).
Quoted 10 lines
Remember my thought experiment
above, how an 8 megabyte image rolls from 1 to 2 clusters during the course of allocating refblocks for the new table? What if the original image wasn't completely full, but things are perfectly sized with enough free clusters, then all of the refblock allocations done during the first walk will still fit, and it is only this final allocation of the new reftable that will cause the rollover, at which point we've failed to account for the new refblock size. That is, I think I could craft an image that would trigger either an assertion failure or an out-of-bounds array access during the second walk.You're completely right, this will be a pain to fix, though... The simplest way would probably to check whether the
new_reftable_sizeshould be increased due to this operation and if it did, rerunwalk_over_reftable()with thealloc_refblock()function only allocating refblocks if the new reftable does not already point to a refblock for a certain index. This would be repeated until thenew_reftable_sizeis constant. And the really simplest incarnation of this would be to have a flag whether any allocations were done and repeat until everything is fine.Another way would be to somehow integrate the allocation of the new reftable into
walk_over_reftable()and then to only mind the additional reftable entries.
Quoted 6 lines
But probably the first way is the correct one because due to reallocation of the old reftable, some intermediate refblocks which were empty before are now filled.
I'll have to craft some test images myself, not least to be able to include them in the iotest.
Overnight, I had been thinking about it (I don't know if it's a good or a bad thing when a patch is so mentally engaging that it becomes the thing on my mind). My initial idea was to teach the walk function how to start and stop at given limits, maybe by passing an address to dereference as the end point. Then do something like:
original_limit = s->refcount_table_size;
walk allocations with limits of 0, &original_limit
allocate a contiguous block of limit+1 clusters for the new reftable walk allocations with limits of original_limit, &s->refcount_table_size if limit+1 is too big after all (most of the time), then free the tail walk refcount assignment with limits of 0, &s->refcount_table_size
Allocating limit+1 for the new refblock should always be safe. In the majority of cases, it overallocates; but as we are already having several small holes in the image after freeing the original reftable means it won't hurt to have one more cluster hole. It's better to have the algorithm be safe and waste a few clusters than to figure out how to pack it into the most efficient space possible at the expense of more code complexity. And maybe someday we'll implement a cluster defragmenter for packing an image down into no holes. In the minority of cases, the +1 should always be sufficient to cover any additional allocation spillovers.
But you raise a good point that the original image may have holes on the first walk, where allocating refblocks for the new table will turn those holes into refcounts that must be picked up, so I think you are right that you have to walk the ENTIRE image on the second pass, rather than just the tail of the image where you stopped early on the first pass.
Good luck on coming up with a plan to tackle it.
Quoted 5 lines
Quoted 1 line
Interesting patch. Hope my review helps you prepare a better v2.
If everything else fails, I'll just split the amend stuff from this series. But I'll work it out somehow. And your review will definitely help, thanks a lot!
Glad to hear it.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 18/21] qcow2: Add function for refcount order amendment
In reply to Max ReitzQuoted 2 lines · Max Reitz
Add a function
qcow2_change_refcount_order()which allows changing the refcount order of a qcow2 image.
A thought: didn't you just submit a patch that marked the image as dirty, nuked the on-disk refcount, then rebuilt one using the in-memory refcounts? Would reusing THAT code be any better than writing this patch?
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 18/21] qcow2: Add function for refcount order amendment
Quoted 6 lines · Eric Blake
Quoted 2 lines · Max Reitz
Add a function
qcow2_change_refcount_order()which allows changing the refcount order of a qcow2 image.A thought: didn't you just submit a patch that marked the image as dirty, nuked the on-disk refcount, then rebuilt one using the in-memory refcounts? Would reusing THAT code be any better than writing this patch?
Yes, I thought about that, too... The problem is that that patch requires all refcount blocks to fit in memory at the same time (or generally, the qcow2 check function requires that, for now). I'd really like to avoid that, if possible, but maybe it isn't possible after all.
But if you say it like that ("nuke"), I guess I'll give it a try. Maybe it looks funny enough.
Max
Re: [Qemu-devel] [PATCH 18/21] qcow2: Add function for refcount order amendment
Quoted 11 lines · Max Reitz
Quoted 6 lines · Eric Blake
Quoted 2 lines · Max Reitz
Add a function
qcow2_change_refcount_order()which allows changing the refcount order of a qcow2 image.A thought: didn't you just submit a patch that marked the image as dirty, nuked the on-disk refcount, then rebuilt one using the in-memory refcounts? Would reusing THAT code be any better than writing this patch?
Yes, I thought about that, too... The problem is that that patch requires all refcount blocks to fit in memory at the same time (or generally, the qcow2 check function requires that, for now). I'd really like to avoid that, if possible, but maybe it isn't possible after all.
But if you say it like that ("nuke"), I guess I'll give it a try. Maybe it looks funny enough.
Okay, I gave it a try. It does work, but one problem that would require larger changes (which would be relatively easy to review, though, I guess...) is a refcount overflow during the image check.
What I did was the following: First, check all refcounts against the maximum after the amendment. Second, do an image check without repairing anything, because we only want to do this on clean images. Third, wipe all references to the existing reftable. Fourth, use a function normally used by the image check to calculate the refcounts (which results in the in-memory refcount table). Fifth, create new refcount structures using the rebuild_refcount_structure() function for image repairing.
There are (at least) three problems with this approach. The first is a rather cosmetic one: You can't easily give progress reports. There are four time-consuming steps here (wiping references to the existing reftable is not time-consuming), so this approach can only say when 25 % are done, 50 %, 75 % and 100 %.
The second is that if an error occurs during rebuilding the refcount structures, it's close to impossible to restore the old ones, because the new structures may have been partially written thus overwriting the old ones. But having marked the image dirty should suffice to "solve" this.
And the third one is that the initial check (whether the image is consistent at all) may throw an error because of refcount overflows. This error will tell you to use amend to increase the refcount width. Well, too bad. To solve this, we'd have to be able to do the refcount consistency check with an arbitrary refcount order (in this case the target refcount order), which would require some work on the check functions.
I'll just go with the original idea for now.
Max
Re: [Qemu-devel] [PATCH 18/21] qcow2: Add function for refcount order amendment
Quoted 11 lines · Max Reitz
Quoted 9 lines · Max Reitz
Quoted 7 lines · Eric Blake
Quoted 2 lines · Max Reitz
Add a function
qcow2_change_refcount_order()which allows changing the refcount order of a qcow2 image.A thought: didn't you just submit a patch that marked the image as dirty, nuked the on-disk refcount, then rebuilt one using the in-memory refcounts? Would reusing THAT code be any better than writing this patch?
Quoted 5 lines
There are (at least) three problems with this approach. The first is a rather cosmetic one: You can't easily give progress reports. There are four time-consuming steps here (wiping references to the existing reftable is not time-consuming), so this approach can only say when 25 % are done, 50 %, 75 % and 100 %.
Yep, definitely annoying.
Quoted 5 lines
The second is that if an error occurs during rebuilding the refcount structures, it's close to impossible to restore the old ones, because the new structures may have been partially written thus overwriting the old ones. But having marked the image dirty should suffice to "solve" this.
Solved, but not as clean as your current patch that maintains image consistency throughout and doesn't require a rebuild. I guess a tradeoff of clean images vs. code reuse can go either way.
Quoted 10 lines
And the third one is that the initial check (whether the image is consistent at all) may throw an error because of refcount overflows. This error will tell you to use amend to increase the refcount width. Well, too bad. To solve this, we'd have to be able to do the refcount consistency check with an arbitrary refcount order (in this case the target refcount order), which would require some work on the check functions.
I'll just go with the original idea for now.
Okay. Just making sure we are considering alternatives when justifying why we go with a particular solution.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 19/21 qcow2: Invoke refcount order amendment function
In reply to Max ReitzMake use of qcow2_change_refcount_order() to support changing the refcount order with qemu-img amend.
--- block/qcow2.c | 44 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 9 deletions(-)
block/qcow2.c +35 -9
@@ -2607,13 +2607,7 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version, }if (s->refcount_order !=4 ) { -/* we would have to convert the image to a refcount_order == 4 image -* here; however, since qemu (at the time of writing this) does not -* support anything different than 4 anyway, there is no point in doing -* so right now; however, we should error out (if qemu supports this in -* the future and this code has not been adapted) */ -error_report ("qcow2_downgrade: Image refcount orders other than 4 are " -"currently not supported." ); +error_report ("compat=0.10 requires refcount_width=16" );return -ENOTSUP ; }
@@ -2661,6 +2655,7 @@ typedef enum Qcow2AmendOperation {* invocation from an operation change */ QCOW2_NO_OPERATION =0 , +QCOW2_CHANGING_REFCOUNT_ORDER ,QCOW2_DOWNGRADING , } Qcow2AmendOperation;
@@ -2732,6 +2727,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,const char *compat =NULL ;uint64_t cluster_size = s->cluster_size;bool encrypt; +int refcount_width = s->refcount_bits;int ret;QemuOptDesc *desc = opts->list->desc;Qcow2AmendHelperCBInfo helper_cb_info;
@@ -2781,8 +2777,16 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, lazy_refcounts =qemu_opt_get_bool (opts,"lazy_refcounts" , lazy_refcounts); }else if (!strcmp (desc->name,"refcount_width" )) { -error_report ("Cannot change refcount entry width" ); -return -ENOTSUP ; + refcount_width =qemu_opt_get_number (opts,"refcount_width" , + refcount_width); + +if (refcount_width <=0 || refcount_width >64 || + !is_power_of_2 (refcount_width)) + { +error_report ("Refcount width must be a power of two and may " +"not exceed 64 bits" ); +return -EINVAL ; + } }else {/* if this point is reached, this probably means a new option was * added without having it covered here */
@@ -2796,6 +2800,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, .original_status_cb = status_cb, .original_cb_opaque = cb_opaque, .total_operations = (new_version < old_version) + + (s->refcount_bits != refcount_width) };/* Upgrade first (some features may require compat=1.1) */
@@ -2808,6 +2813,27 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, } } +if (s->refcount_bits != refcount_width) { +int refcount_order =ffs (refcount_width) -1 ; +Error *local_error =NULL ; + +if (new_version <3 && refcount_width !=16 ) { +error_report ("Different refcount widths than 16 bits require " +"compatibility level 1.1 or above (use compat=1.1 or " +"greater)" ); +return -EINVAL ; + } + + helper_cb_info.current_operation =QCOW2_CHANGING_REFCOUNT_ORDER ; + ret =qcow2_change_refcount_order (bs, refcount_order, + &qcow2_amend_helper_cb, + &helper_cb_info, &local_error); +if (ret <0 ) { +qerror_report_err (local_error); +return ret; + } + } +if (backing_file || backing_format) { ret =qcow2_change_backing_file (bs, backing_file ?: bs->backing_file, backing_format ?: bs->backing_format);
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 19/21] qcow2: Invoke refcount order amendment function
Quoted 8 lines · Max Reitz
Make use of
qcow2_change_refcount_order()to support changing the refcount order with qemu-img amend.Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 44 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 9 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 20/21 qcow2: Point to amend function in check
In reply to Max ReitzIf a reference count is not representable with the current refcount order, the image check should point to qemu-img amend for increasing the refcount order. However, qemu-img amend needs write access to the image which cannot be provided if the image is marked corrupt; and the image check will not mark the image consistent unless everything actually is consistent.
Therefore, if an image is marked corrupt and the image check encounters a reference count overflow, it cannot be fixed by using qemu-img amend to increase the refcount order. Instead, one has to use qemu-img convert to create a completely new copy of the image in this case.
Alternatively, we may want to give the user a way of manually removing the corrupt flag, maybe through qemu-img amend, but this is not part of this patch.
--- block/qcow2-refcount.c | 3 +++ 1 file changed, 3 insertions(+)
block/qcow2-refcount.c +3 -0
@@ -1337,6 +1337,9 @@ static int inc_refcounts(BlockDriverState *bs,if (refcount == s->refcount_max) {fprintf (stderr,"ERROR: overflow cluster offset=0x%" PRIx64"\n" , cluster_offset); +fprintf (stderr,"Use qemu-img amend to increase the refcount entry " +"width or qemu-img convert to create a clean copy if the " +"image cannot be opened for writing\n" ); res->corruptions++;continue ; }
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 20/21] qcow2: Point to amend function in check
Quoted 12 lines · Max Reitz
If a reference count is not representable with the current refcount order, the image check should point to qemu-img amend for increasing the refcount order. However, qemu-img amend needs write access to the image which cannot be provided if the image is marked corrupt; and the image check will not mark the image consistent unless everything actually is consistent.
Therefore, if an image is marked corrupt and the image check encounters a reference count overflow, it cannot be fixed by using qemu-img amend to increase the refcount order. Instead, one has to use qemu-img convert to create a completely new copy of the image in this case.
Quoted 8 lines
Alternatively, we may want to give the user a way of manually removing the corrupt flag, maybe through qemu-img amend, but this is not part of this patch.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 3 +++ 1 file changed, 3 insertions(+)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 21/21 iotests: Add test for different refcount widths
In reply to Max ReitzAdd a test for conversion between different refcount widths and errors specific to certain widths (i.e. snapshots with refcount_width=1).
--- tests/qemu-iotests/112 | 225 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/112.out | 123 +++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 349 insertions(+) create mode 100755 tests/qemu-iotests/112 create mode 100644 tests/qemu-iotests/112.out
tests/qemu-iotests/112 new file +225 -0
@@ -0,0 +1,225 @@ +#!/bin/bash +# +# Test cases for different refcount_widths +# +# Copyright (C) 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq="$(basename $0)" +echo "QA output created by $seq" + +here="$PWD" +tmp=/tmp/$$ +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +# This tests qocw2-specific low-level functionality +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux +# This test will set refcount_width on its own which would conflict with the +# manual setting; compat will be overridden as well +_unsupported_imgopts refcount_width 'compat=0.10' + +function print_refcount_width() +{ + $QEMU_IMG info "$TEST_IMG" | grep 'refcount width:' | sed -e 's/^ *//' +} + +echo +echo '=== refcount_width limits ===' +echo + +# Must be positive (non-zero) +IMGOPTS="$IMGOPTS,refcount_width=0" _make_test_img 64M +# Must be positive (non-negative) +IMGOPTS="$IMGOPTS,refcount_width=-1" _make_test_img 64M +# May not exceed 64 +IMGOPTS="$IMGOPTS,refcount_width=128" _make_test_img 64M +# Must be a power of two +IMGOPTS="$IMGOPTS,refcount_width=42" _make_test_img 64M + +# 1 is the minimum +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# 64 is the maximum +IMGOPTS="$IMGOPTS,refcount_width=64" _make_test_img 64M +print_refcount_width + +# 16 is the default +_make_test_img 64M +print_refcount_width + +echo +echo '=== refcount_width and compat=0.10 ===' +echo + +# Should work +IMGOPTS="$IMGOPTS,compat=0.10,refcount_width=16" _make_test_img 64M +print_refcount_width + +# Should not work +IMGOPTS="$IMGOPTS,compat=0.10,refcount_width=1" _make_test_img 64M +IMGOPTS="$IMGOPTS,compat=0.10,refcount_width=64" _make_test_img 64M + + +echo +echo '=== Snapshot limit on refcount_width=1 ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io + +# Should fail +$QEMU_IMG snapshot -c foo "$TEST_IMG" + +# The new L1 table could/shoud be leaked +_check_test_img + +echo +echo '=== Snapshot limit on refcount_width=2 ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=2" _make_test_img 64M +print_refcount_width + +$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io + +# Should succeed +$QEMU_IMG snapshot -c foo "$TEST_IMG" +$QEMU_IMG snapshot -c bar "$TEST_IMG" +# Should fail (4th reference) +$QEMU_IMG snapshot -c baz "$TEST_IMG" + +# The new L1 table could/shoud be leaked +_check_test_img + +echo +echo '=== Compressed clusters with refcount_width=1 ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# Both should fit into a single host cluster; instead of failing to increase the +# refcount of that cluster, qemu should just allocate a new cluster and make +# this operation succeed +$QEMU_IO -c 'write -P 0 -c 0 64k' \ + -c 'write -P 1 -c 64k 64k' \ + "$TEST_IMG" | _filter_qemu_io + +_check_test_img + +echo +echo '=== Amend from refcount_width=16 to refcount_width=1 ===' +echo + +_make_test_img 64M +print_refcount_width + +$QEMU_IO -c 'write 16M 32M' "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG amend -o refcount_width=1 "$TEST_IMG" +_check_test_img +print_refcount_width + +echo +echo '=== Amend from refcount_width=1 to refcount_width=64 ===' +echo + +$QEMU_IMG amend -o refcount_width=64 "$TEST_IMG" +_check_test_img +print_refcount_width + +echo +echo '=== Amend to compat=0.10 ===' +echo + +# Should not work because refcount_width needs to be 16 for compat=0.10 +$QEMU_IMG amend -o compat=0.10 "$TEST_IMG" +print_refcount_width +# Should work +$QEMU_IMG amend -o compat=0.10,refcount_width=16 "$TEST_IMG" +_check_test_img +print_refcount_width + +# Get back to compat=1.1 and refcount_width=16 +$QEMU_IMG amend -o compat=1.1 "$TEST_IMG" +print_refcount_width +# Should not work +$QEMU_IMG amend -o refcount_width=32,compat=0.10 "$TEST_IMG" +print_refcount_width + +echo +echo '=== Amend with snapshot ===' +echo + +$QEMU_IMG snapshot -c foo "$TEST_IMG" +# Just to have different refcounts across the image +$QEMU_IO -c 'write 0 16M' "$TEST_IMG" | _filter_qemu_io + +# Should not work +$QEMU_IMG amend -o refcount_width=1 "$TEST_IMG" +_check_test_img +print_refcount_width + +# Should work +$QEMU_IMG amend -o refcount_width=2 "$TEST_IMG" +_check_test_img +print_refcount_width + +echo +echo '=== Testing too many references for check ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# This cluster should be created at 0x50000 +$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io +# Now make the second L2 entriy (the L2 table should be at 0x40000) point to +# that cluster, so we have two references +poke_file "$TEST_IMG" $((0x40008)) "\x80\x00\x00\x00\x00\x05\x00\x00" + +# This should say "please use amend" +_check_test_img -r all + +# So we do that +$QEMU_IMG amend -o refcount_width=2 "$TEST_IMG" +print_refcount_width + +# And try again +_check_test_img -r all + + +# success, all done +echo '*** done' +rm -f $seq.full +status=0
tests/qemu-iotests/112.out new file +123 -0
123 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -112,3 +112,4 @@ 107 rw auto quick 108 rw auto quick 111 rw auto quick +112 rw auto
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH 21/21] iotests: Add test for different refcount widths
Quoted 13 lines · Max Reitz
Add a test for conversion between different refcount widths and errors specific to certain widths (i.e. snapshots with refcount_width=1).
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/112 | 225 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/112.out | 123 +++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 349 insertions(+) create mode 100755 tests/qemu-iotests/112 create mode 100644 tests/qemu-iotests/112.out
tests/qemu-iotests/112 line 44
Quoted 5 lines
tests/qemu-iotests/112 new file +5
-0+ +# This tests qocw2-specific low-level functionality +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux
Might work on more than just Linux, but then again, it's probably worth scrubbing the whole testsuite for situations like that, so don't worry about it here.
tests/qemu-iotests/112 line 51
Quoted 7 lines
tests/qemu-iotests/112 new file +7
-0+# This test will set refcount_width on its own which would conflict with the +# manual setting; compat will be overridden as well +_unsupported_imgopts refcount_width 'compat=0.10' + +function print_refcount_width() +{ + $QEMU_IMG info "$TEST_IMG" | grep 'refcount width:' | sed -e 's/^ *//'
grep|sed is almost always a waste. This is equivalent:
$QEMU_IMG info "$TEST_IMG" | sed -n '/refcount width:/ s/^ *//p'
tests/qemu-iotests/112 line 104
Quoted 13 lines
tests/qemu-iotests/112 new file +13
-0+echo +echo '=== Snapshot limit on refcount_width=1 ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io + +# Should fail +$QEMU_IMG snapshot -c foo "$TEST_IMG" + +# The new L1 table could/shoud be leaked
s/shoud/should/
Quoted 8 lines
tests/qemu-iotests/112 new file +8
-0+_check_test_img + +echo +echo '=== Snapshot limit on refcount_width=2 ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=2" _make_test_img 64M +print_refcount_width
tests/qemu-iotests/112 line 122
Quoted 10 lines
tests/qemu-iotests/112 new file +10
-0+ +$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io + +# Should succeed +$QEMU_IMG snapshot -c foo "$TEST_IMG" +$QEMU_IMG snapshot -c bar "$TEST_IMG" +# Should fail (4th reference) +$QEMU_IMG snapshot -c baz "$TEST_IMG" + +# The new L1 table could/shoud be leaked
again
tests/qemu-iotests/112 line 191
Quoted 12 lines
tests/qemu-iotests/112 new file +12
-0+echo +echo '=== Amend with snapshot ===' +echo + +$QEMU_IMG snapshot -c foo "$TEST_IMG" +# Just to have different refcounts across the image +$QEMU_IO -c 'write 0 16M' "$TEST_IMG" | _filter_qemu_io + +# Should not work +$QEMU_IMG amend -o refcount_width=1 "$TEST_IMG" +_check_test_img +print_refcount_width
This matches your initial implementation. Someday, though, we may decide to auto-COW any overflowed cluster, and thus allow the conversion to succeed. Worth a comment?
tests/qemu-iotests/112 line 207
Quoted 9 lines
tests/qemu-iotests/112 new file +9
-0+echo '=== Testing too many references for check ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# This cluster should be created at 0x50000 +$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io +# Now make the second L2 entriy (the L2 table should be at 0x40000) point to
s/entriy/entry/
tests/qemu-iotests/112 line 225
Quoted 4 lines
tests/qemu-iotests/112 new file +4
-0+# success, all done +echo '*** done' +rm -f $seq.full +status=0
Overall a nice set of tests!
tests/qemu-iotests/112.out line 36
Quoted 8 lines
+=== Snapshot limit on refcount_width=1 ===
tests/qemu-iotests/112.out new file +7
-0+ +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +refcount width: 1 +wrote 512/512 bytes at offset 0 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +qemu-img: Could not create snapshot 'foo': -22 (Invalid argument) +Leaked cluster 6 refcount=1 reference=0
Bummer that the error message did not state WHY (because a cluster would overflow refcounts), but I'm not sure how hard it would be to make that better, and at least we correctly errored out.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 21/21] iotests: Add test for different refcount widths
Quoted 31 lines · Eric Blake
Quoted 12 lines · Max Reitz
Add a test for conversion between different refcount widths and errors specific to certain widths (i.e. snapshots with refcount_width=1).
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/112 | 225 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/112.out | 123 +++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 349 insertions(+)create mode 100755 tests/qemu-iotests/112
create mode 100644 tests/qemu-iotests/112.outtests/qemu-iotests/112 line 44
Quoted 5 lines
tests/qemu-iotests/112 new file +5
-0+ +# This tests qocw2-specific low-level functionality +_supported_fmt qcow2 +_supported_proto file +_supported_os LinuxMight work on more than just Linux, but then again, it's probably worth scrubbing the whole testsuite for situations like that, so don't worry about it here.
tests/qemu-iotests/112 line 51
Quoted 7 lines
tests/qemu-iotests/112 new file +7
-0+# This test will set refcount_width on its own which would conflict with the +# manual setting; compat will be overridden as well +_unsupported_imgopts refcount_width 'compat=0.10' + +function print_refcount_width() +{ + $QEMU_IMG info "$TEST_IMG" | grep 'refcount width:' | sed -e 's/^ *//'grep|sed is almost always a waste. This is equivalent:
Quoted 1 line
$QEMU_IMG info "$TEST_IMG" | sed -n '/refcount width:/ s/^ *//p'
As said before, I just don't know sed well enough. My knowledge of sed is "you can use it for regex replacement with -e", and that's about it. Oh, and "you can do it in-place with -i" and "someone wrote Sokoban in sed". And maybe even "sed better fits the term Standard EDitor than ed does".
But thanks a lot for telling me, I'm just always afraid to learn sed because it seems even more unreadable than perl to me...
Quoted 6 lines
Quoted 6 lines
tests/qemu-iotests/112 new file +6
-0+echo +echo '=== Snapshot limit on refcount_width=1 ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width
Quoted 8 lines
tests/qemu-iotests/112 line 104
Quoted 7 lines
tests/qemu-iotests/112 new file +7
-0+ +$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io + +# Should fail +$QEMU_IMG snapshot -c foo "$TEST_IMG" + +# The new L1 table could/shoud be leakeds/shoud/should/
Right.
Quoted 11 lines
Quoted 11 lines
tests/qemu-iotests/112 new file +11
-0+_check_test_img + +echo +echo '=== Snapshot limit on refcount_width=2 ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=2" _make_test_img 64M +print_refcount_width + +$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io +
Quoted 8 lines
tests/qemu-iotests/112 line 122
Quoted 7 lines
tests/qemu-iotests/112 new file +7
-0+# Should succeed +$QEMU_IMG snapshot -c foo "$TEST_IMG" +$QEMU_IMG snapshot -c bar "$TEST_IMG" +# Should fail (4th reference) +$QEMU_IMG snapshot -c baz "$TEST_IMG" + +# The new L1 table could/shoud be leakedagain
yyp is dangerous.
Quoted 7 lines
Quoted 7 lines
tests/qemu-iotests/112 new file +7
-0+echo +echo '=== Amend with snapshot ===' +echo + +$QEMU_IMG snapshot -c foo "$TEST_IMG" +# Just to have different refcounts across the image +$QEMU_IO -c 'write 0 16M' "$TEST_IMG" | _filter_qemu_io
Quoted 8 lines
tests/qemu-iotests/112 line 151
Quoted 5 lines
tests/qemu-iotests/112 new file +5
-0+ +# Should not work +$QEMU_IMG amend -o refcount_width=1 "$TEST_IMG" +_check_test_img +print_refcount_widthThis matches your initial implementation. Someday, though, we may decide to auto-COW any overflowed cluster, and thus allow the conversion to succeed. Worth a comment?
Yes, will do.
Quoted 10 lines
tests/qemu-iotests/112 line 207
Quoted 9 lines
tests/qemu-iotests/112 new file +9
-0+echo '=== Testing too many references for check ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# This cluster should be created at 0x50000 +$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io +# Now make the second L2 entriy (the L2 table should be at 0x40000) point tos/entriy/entry/
I think this happened because at one point in time it said something about "L2 entries".
Quoted 7 lines
tests/qemu-iotests/112 line 225
Quoted 4 lines
tests/qemu-iotests/112 new file +4
-0+# success, all done +echo '*** done' +rm -f $seq.full +status=0Overall a nice set of tests!
Quoted 11 lines
tests/qemu-iotests/112.out line 36
Quoted 8 lines
+=== Snapshot limit on refcount_width=1 ===
tests/qemu-iotests/112.out new file +7
-0+ +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +refcount width: 1 +wrote 512/512 bytes at offset 0 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +qemu-img: Could not create snapshot 'foo': -22 (Invalid argument) +Leaked cluster 6 refcount=1 reference=0Bummer that the error message did not state WHY (because a cluster would overflow refcounts), but I'm not sure how hard it would be to make that better, and at least we correctly errored out.
Yes, I know, it's a really bad error. The problem is that no Error object is used in that path at all so it will be rather cumbersome, but I'll look into it one more time.
Max