qcow2: Support refcount orders != 4
v2 cover qcow2: Support refcount orders != 4
As of version 3, the qcow2 file format supports different widths for refcount entries, ranging from 1 to 64 bit (only powers of two). Currently, qemu only supports 16 bit, which is the only width supported by version 2 (compat=0.10) images.
This series adds support to qemu for all other valid refcount orders. This is mainly done by adding two function pointers into the BDRVQcowState structure for reading and writing refcount values independently of the current refcount entry width; all in-memory refcount arrays (mostly cached refcount blocks) now are void pointers and are accessed through these functions alone.
Thanks to previous work of making the qemu code agnostic of e.g. the number of refcount entries per refcount block, the rest is fairly trivial. The most complex patch in this series is patch 18 which implements changing the refcount order through qemu-img amend.
To test different refcount widths, simply invoke the qemu-iotests check program with -o refcount_width=${your_desired_width}. The final test in this series adds some tests for operations which do not work with certain refcount orders and for refcount order amendment.
As of this version, this series depends on version 3 of my "chardev: Add -qmp-pretty" series (due to different test output of test 067, which makes changing it here much nicer).
v2:
- Patch 2:
- Added justification for always emitting refcount-width to the commit message [Eric]
- Due to making this series dependent on the -qmp-pretty series, the reference output of iotest 067 is changed differently
- Patch 3: Added justification for using int64_t instead of uint64_t to the commit message [Eric]
- Patch 6: This replaces patch 7 from v1.
- Added an assertion and an explanation why refcount_array_byte_size() cannot overflow [Eric]
- Added a helper function for reallocating a refcount array (this is what we really want); make use of that function
- Patch 7: Was patch 6 in v1, swapped the order to reduce the diffstat and not having to explain why a truncating division does not truncate here [Eric]
- Added overflow check in update_refcount()
- Consequent use of int64_t for refcounts [Eric]
- Use g_try_malloc0_n() instead of g_try_malloc0() once to prevent a multiplication overflow [Eric]
- Dropped the temporary on_disk_refblock buffer in rebuild_refcount_structure(); just directly write the in-memory refcount array to disk
- Patch 8: Added an assertion in set_refcount_ro6() that the MSb of the given value is not set (which makes the value fit into an int64_t) [Eric]
- Patch 9: Dropped the refcount width specified by the image header from the output to prevent overflows [Eric]
- Patch 10: Kept the same, because this calculation was never meant to be exact
- Patch 11:
- Added comments why certain refcount_widths to do not work with some tests [Eric]
- Changed the _unsupported_imgopts argument for allowing only refcount_width=16 to disallow any other refcount width instead of listing them all (your proposal did not work, though, Eric)
- Fixed comment in test 007 [Eric]
- 079 does support any refcount width, we only need to use _make_test_img instead of calling qemu-img create manually, so do that
- Disallowing 090 for refcount_width=1 was an artifact from before v1 (where qcow2_alloc_bytes() simply returned an error when a refcount overflow was about to occur, instead of skipping to the next cluster), so remove that limitation
- Patch 12: Dropped 079 reference output change because patch 11 took care of that (by using _make_test_img)
- Patch 14: Removed trailing full stops in error_report() calls [Eric]
- Patch 17:
- Fixed indentation for the function header of qcow2_amend_helper_cb()
[Eric]
- Renamed arguments of qcow2_amend_helper_cb() to make it more obvious what they mean in the context of this function [Eric, in a way]
- Added to assertions regarding total_operations [Eric]
- Patch 18:
- Add a typedef for the function pointer which is given to walk_over_reftable(); the benefit is that this makes documentation of the parameters easier [Eric]
- Add an "allocated" parameter to that RefblockFinishOp and consequently to walk_over_reftable() so that changes in the original refcount structures can be tracked; if anything has been changed, walk_over_reftable() is rerun with alloc_refblock() until no new allocations were necessary anymore
- Some changes to alloc_refblock() and flush_refblock() to accommodate to this change
- Call the status_cb additionally once at the end of walk_over_reftable() [Eric]
- Fix leak of the new reftable clusters on error [Eric]
- Patch 21:
- Added comment that snapshotting an image with a small refcount width fails for now, but may work in the future [Eric]
- %s/shoud/should/ [Eric]
- 64s/qocw2/qcow2/
- Added comment that failing to amend an image because of refcount overflows may work in the future [Eric]
- 232s/entriy/entry/
- Added a test for multiple walks during refcount width amendment (for the changes to patch 18)
git-backport-diff against v1:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/21:[----] [--] 'qcow2: Add two new fields to BDRVQcowState' 002/21:[0015] [FC] 'qcow2: Add refcount_width to format-specific info' 003/21:[----] [--] 'qcow2: Use 64 bits for refcount values' 004/21:[----] [--] 'qcow2: Respect error in qcow2_alloc_bytes()' 005/21:[----] [--] 'qcow2: Refcount overflow and qcow2_alloc_bytes()' 006/21:[down] 'qcow2: Helper for refcount array reallocation' 007/21:[0062] [FC] 'qcow2: Helper function for refcount modification' 008/21:[0003] [FC] 'qcow2: More helpers for refcount modification' 009/21:[0004] [FC] 'qcow2: Open images with refcount order != 4' 010/21:[----] [--] 'qcow2: refcount_order parameter for qcow2_create2' 011/21:[0096] [FC] 'iotests: Prepare for refcount_width option' 012/21:[0018] [FC] 'qcow2: Allow creation with refcount order != 4' 013/21:[----] [--] 'block: Add opaque value to the amend CB' 014/21:[0018] [FC] 'qcow2: Use error_report() in qcow2_amend_options()' 015/21:[----] [--] 'qcow2: Use abort() instead of assert(false)' 016/21:[----] [--] 'qcow2: Split upgrade/downgrade paths for amend' 017/21:[0014] [FC] 'qcow2: Use intermediate helper CB for amend' 018/21:[0159] [FC] 'qcow2: Add function for refcount order amendment' 019/21:[----] [--] 'qcow2: Invoke refcount order amendment function' 020/21:[----] [--] 'qcow2: Point to amend function in check' 021/21:[0051] [FC] 'iotests: Add test for different refcount widths'
Max Reitz (21):
qcow2: Add two new fields to BDRVQcowState
qcow2: Add refcount_width to format-specific info
qcow2: Use 64 bits for refcount values
qcow2: Respect error in qcow2_alloc_bytes()
qcow2: Refcount overflow and qcow2_alloc_bytes()
qcow2: Helper for refcount array reallocation
qcow2: Helper function for refcount modification
qcow2: More helpers for refcount modification
qcow2: Open images with refcount order != 4
qcow2: refcount_order parameter for qcow2_create2
iotests: Prepare for refcount_width option
qcow2: Allow creation with refcount order != 4
block: Add opaque value to the amend CB
qcow2: Use error_report() in qcow2_amend_options()
qcow2: Use abort() instead of assert(false)
qcow2: Split upgrade/downgrade paths for amend
qcow2: Use intermediate helper CB for amend
qcow2: Add function for refcount order amendment
qcow2: Invoke refcount order amendment function
qcow2: Point to amend function in check
iotests: Add test for different refcount widths
block.c | 4 +- block/qcow2-cluster.c | 23 +- block/qcow2-refcount.c | 946 +++++++++++++++++++++++++++++++++------ block/qcow2.c | 256 ++++++++--- block/qcow2.h | 24 +- include/block/block.h | 4 +- include/block/block_int.h | 4 +- qapi/block-core.json | 5 +- qemu-img.c | 5 +- tests/qemu-iotests/007 | 3 + tests/qemu-iotests/015 | 2 + tests/qemu-iotests/026 | 7 + tests/qemu-iotests/029 | 1 + tests/qemu-iotests/049.out | 112 ++--- tests/qemu-iotests/051 | 3 + tests/qemu-iotests/058 | 2 + tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 14 +- tests/qemu-iotests/065 | 23 +- tests/qemu-iotests/067 | 2 + tests/qemu-iotests/067.out | 5 + tests/qemu-iotests/079 | 10 +- tests/qemu-iotests/079.out | 38 +- tests/qemu-iotests/080 | 2 + tests/qemu-iotests/082.out | 48 +- tests/qemu-iotests/085.out | 38 +- tests/qemu-iotests/089 | 2 + tests/qemu-iotests/089.out | 2 + tests/qemu-iotests/108 | 2 + tests/qemu-iotests/112 | 252 +++++++++++ tests/qemu-iotests/112.out | 131 ++++++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 33 files changed, 1644 insertions(+), 331 deletions(-) create mode 100755 tests/qemu-iotests/112 create mode 100644 tests/qemu-iotests/112.out
Signature · 1 line
1.9.3
PATCH v2 1/21 qcow2: Add two new fields to BDRVQcowState
Add two new fields regarding refcount information (the bit width of every entry and the maximum refcount value) to the BDRVQcowState.
--- block/qcow2-refcount.c | 2 +- block/qcow2.c | 9 +++++++++ block/qcow2.h | 2 ++ 3 files changed, 12 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +1 -1
@@ -584,7 +584,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs, refcount =be16_to_cpu (refcount_block[block_index]); refcount += addend; -if (refcount <0 || refcount >0xffff ) { +if (refcount <0 || refcount > s->refcount_max) { ret = -EINVAL ; goto fail; }
block/qcow2.c +9 -0
@@ -684,6 +684,15 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } s->refcount_order = header.refcount_order; + s->refcount_bits =1 << s->refcount_order; +if (s->refcount_order <6 ) { + s->refcount_max = (UINT64_C (1 ) << s->refcount_bits) -1 ; + }else { +/* The above shift would overflow with s->refcount_bits == 64; +* furthermore, we do not want to use UINT64_MAX because refcounts will +* be passed around in int64_ts (negative values for -errno) */ + s->refcount_max =INT64_MAX ; + }if (header.crypt_method >QCOW_CRYPT_AES ) { error_setg(errp, "Unsupported encryption method: %" PRIu32,
block/qcow2.h +2 -0
@@ -258,6 +258,8 @@ typedef struct BDRVQcowState {int qcow_version;bool use_lazy_refcounts;int refcount_order; +int refcount_bits; +uint64_t refcount_max;bool discard_passthrough[QCOW2_DISCARD_MAX ];
Signature · 1 line
1.9.3
PATCH v2 2/21 qcow2: Add refcount_width to format-specific info
In reply to Max ReitzAdd the bit width of every refcount entry to the format-specific information.
In contrast to lazy_refcounts and the corrupt flag, this should be always emitted, even for compat=0.10 although it does not support any refcount width other than 16 bits. This is because if a boolean is optional, one normally assumes it to be false when omitted; but if an integer is not specified, it is rather difficult to guess its value.
This new field breaks some test outputs, fix them.
--- block/qcow2.c | 4 +++- qapi/block-core.json | 5 ++++- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/065 | 23 +++++++++++++++-------- tests/qemu-iotests/067.out | 5 +++++ tests/qemu-iotests/082.out | 7 +++++++ tests/qemu-iotests/089.out | 2 ++ 7 files changed, 37 insertions(+), 10 deletions(-)
block/qcow2.c +3 -1
@@ -2475,7 +2475,8 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs) };if (s->qcow_version ==2 ) { *spec_info->qcow2 = (ImageInfoSpecificQCow2 ){ - .compat =g_strdup ("0.10" ), + .compat =g_strdup ("0.10" ), + .refcount_width = s->refcount_bits, }; }else if (s->qcow_version ==3 ) { *spec_info->qcow2 = (ImageInfoSpecificQCow2 ){
@@ -2486,6 +2487,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs) .corrupt = s->incompatible_features & QCOW2_INCOMPAT_CORRUPT, .has_corrupt = true, + .refcount_width = s->refcount_bits, }; }
qapi/block-core.json +4 -1
@@ -41,13 +41,16 @@ # @corrupt: #optional true if the image has been marked corrupt; only valid for # compat >= 1.1 (since 2.2) # +# @refcount-width: width of a refcount entry in bits (since 2.3) +# # Since: 1.7 ## { 'type': 'ImageInfoSpecificQCow2', 'data': { 'compat': 'str', '*lazy-refcounts': 'bool', - '*corrupt': 'bool' + '*corrupt': 'bool', + 'refcount-width': 'int' } } ##
tests/qemu-iotests/060.out +1 -0
@@ -19,6 +19,7 @@ cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false + refcount width: 16 corrupt: true qemu-io: can't open device TEST_DIR/t.IMGFMT: IMGFMT: Image is corrupt; cannot be opened read/write read 512/512 bytes at offset 0
tests/qemu-iotests/065 +15 -8
@@ -88,34 +88,41 @@ class TestQMP(TestImageInfoSpecific): class TestQCow2(TestQemuImgInfo): '''Testing a qcow2 version 2 image''' img_options = 'compat=0.10' - json_compare = { 'compat': '0.10' } - human_compare = [ 'compat: 0.10' ] + json_compare = { 'compat': '0.10', 'refcount-width': 16 } + human_compare = [ 'compat: 0.10', 'refcount width: 16' ] class TestQCow3NotLazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts disabled''' img_options = 'compat=1.1,lazy_refcounts=off' - json_compare = { 'compat': '1.1', 'lazy-refcounts': False, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: false', 'corrupt: false' ] + json_compare = { 'compat': '1.1', 'lazy-refcounts': False, + 'refcount-width': 16, 'corrupt': False } + human_compare = [ 'compat: 1.1', 'lazy refcounts: false', + 'refcount width: 16', 'corrupt: false' ] class TestQCow3Lazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts enabled''' img_options = 'compat=1.1,lazy_refcounts=on' - json_compare = { 'compat': '1.1', 'lazy-refcounts': True, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: true', 'corrupt: false' ] + json_compare = { 'compat': '1.1', 'lazy-refcounts': True, + 'refcount-width': 16, 'corrupt': False } + human_compare = [ 'compat: 1.1', 'lazy refcounts: true', + 'refcount width: 16', 'corrupt: false' ] class TestQCow3NotLazyQMP(TestQMP): '''Testing a qcow2 version 3 image with lazy refcounts disabled, opening with lazy refcounts enabled''' img_options = 'compat=1.1,lazy_refcounts=off' qemu_options = 'lazy-refcounts=on' - compare = { 'compat': '1.1', 'lazy-refcounts': False, 'corrupt': False } + compare = { 'compat': '1.1', 'lazy-refcounts': False, + 'refcount-width': 16, 'corrupt': False } + class TestQCow3LazyQMP(TestQMP): '''Testing a qcow2 version 3 image with lazy refcounts enabled, opening with lazy refcounts disabled''' img_options = 'compat=1.1,lazy_refcounts=on' qemu_options = 'lazy-refcounts=off' - compare = { 'compat': '1.1', 'lazy-refcounts': True, 'corrupt': False } + compare = { 'compat': '1.1', 'lazy-refcounts': True, + 'refcount-width': 16, 'corrupt': False } TestImageInfoSpecific = None TestQemuImgInfo = None
tests/qemu-iotests/067.out +5 -0
@@ -32,6 +32,7 @@ Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,if=none,id=disk -device virti "data": { "compat": "1.1", "lazy-refcounts": false, + "refcount-width": 16, "corrupt": false } },
@@ -202,6 +203,7 @@ Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,if=none,id=disk "data": { "compat": "1.1", "lazy-refcounts": false, + "refcount-width": 16, "corrupt": false } },
@@ -402,6 +404,7 @@ Testing: "data": { "compat": "1.1", "lazy-refcounts": false, + "refcount-width": 16, "corrupt": false } },
@@ -581,6 +584,7 @@ Testing: "data": { "compat": "1.1", "lazy-refcounts": false, + "refcount-width": 16, "corrupt": false } },
@@ -686,6 +690,7 @@ Testing: "data": { "compat": "1.1", "lazy-refcounts": false, + "refcount-width": 16, "corrupt": false } },
tests/qemu-iotests/082.out +7 -0
@@ -21,6 +21,7 @@ cluster_size: 4096 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: create -f qcow2 -o cluster_size=4k -o lazy_refcounts=on -o cluster_size=8k TEST_DIR/t.qcow2 128M
@@ -35,6 +36,7 @@ cluster_size: 8192 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: create -f qcow2 -o cluster_size=4k,cluster_size=8k TEST_DIR/t.qcow2 128M
@@ -199,6 +201,7 @@ cluster_size: 4096 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: convert -O qcow2 -o cluster_size=4k -o lazy_refcounts=on -o cluster_size=8k TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
@@ -212,6 +215,7 @@ cluster_size: 8192 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: convert -O qcow2 -o cluster_size=4k,cluster_size=8k TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
@@ -361,6 +365,7 @@ cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: amend -f qcow2 -o size=130M -o lazy_refcounts=off TEST_DIR/t.qcow2
@@ -374,6 +379,7 @@ cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false + refcount width: 16 corrupt: false Testing: amend -f qcow2 -o size=8M -o lazy_refcounts=on -o size=132M TEST_DIR/t.qcow2
@@ -387,6 +393,7 @@ cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: true + refcount width: 16 corrupt: false Testing: amend -f qcow2 -o size=4M,size=148M TEST_DIR/t.qcow2
tests/qemu-iotests/089.out +2 -0
@@ -41,6 +41,7 @@ vm state offset: 512 MiB Format specific information: compat: 1.1 lazy refcounts: false + refcount width: 16 corrupt: false format name: IMGFMT cluster size: 64 KiB
@@ -48,5 +49,6 @@ vm state offset: 512 MiB Format specific information: compat: 1.1 lazy refcounts: false + refcount width: 16 corrupt: false *** done
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v2 02/21] qcow2: Add refcount_width to format-specific info
Quoted 12 lines · Max Reitz
Add the bit width of every refcount entry to the format-specific information.
In contrast to
lazy_refcountsand the corrupt flag, this should be always emitted, even for compat=0.10 although it does not support any refcount width other than 16 bits. This is because if a boolean is optional, one normally assumes it to be false when omitted; but if an integer is not specified, it is rather difficult to guess its value.This new field breaks some test outputs, fix them.
Signed-off-by Max Reitz <[email protected]>
Quoted 9 lines
--- block/qcow2.c | 4 +++- qapi/block-core.json | 5 ++++- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/065 | 23 +++++++++++++++-------- tests/qemu-iotests/067.out | 5 +++++ tests/qemu-iotests/082.out | 7 +++++++ tests/qemu-iotests/089.out | 2 ++ 7 files changed, 37 insertions(+), 10 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v2 3/21 qcow2: Use 64 bits for refcount values
In reply to Max ReitzRefcounts may have a width of up to 64 bits, so qemu should use the same width to represent refcount values internally.
Since for instance qcow2_get_refcount() signals an error by returning a negative value, refcount values are generally signed to be able to represent those error values correctly. This limits the maximum refcount value supported by qemu to INT64_MAX (= 63 bits), as established in "qcow2: Add two new fields to BDRVQcowState".
This limitation should have no implications in practice for normal valid images. If the MSb in a 64 bit refcount value is set, we can safely assume the value to be invalid (because reaching such high refcounts is impossible due to other limitations of the qcow2 format).
--- block/qcow2-cluster.c | 9 ++++++--- block/qcow2-refcount.c | 37 ++++++++++++++++++++----------------- block/qcow2.h | 7 ++++--- 3 files changed, 30 insertions(+), 23 deletions(-)
block/qcow2-cluster.c +6 -3
@@ -1640,7 +1640,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,for (i =0 ; i < l1_size; i++) {uint64_t l2_offset = l1_table[i] &L1E_OFFSET_MASK ;bool l2_dirty = false; -int l2_refcount; +int64_t l2_refcount;if (!l2_offset) {/* unallocated */
@@ -1696,14 +1696,17 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, }if (l2_refcount >1 ) { +int64_t ret64; +/* For shared L2 tables, set the refcount accordingly (it is * already 1 and needs to be l2_refcount) */ - ret =qcow2_update_cluster_refcount (bs, + ret64 =qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits, l2_refcount -1 ,QCOW2_DISCARD_OTHER ); -if (ret <0 ) { +if (ret64 <0 ) {qcow2_free_clusters (bs, offset, s->cluster_size,QCOW2_DISCARD_OTHER ); + ret = ret64; goto fail; } }
block/qcow2-refcount.c +20 -17
@@ -91,14 +91,14 @@ static int load_refcount_block(BlockDriverState *bs,* return value is the refcount of the cluster, negative values are -errno * and indicate an error. */ -int qcow2_get_refcount (BlockDriverState *bs,int64_t cluster_index) +int64_t qcow2_get_refcount (BlockDriverState *bs,int64_t cluster_index) {BDRVQcowState *s = bs->opaque;uint64_t refcount_table_index, block_index;int64_t refcount_block_offset;int ret;uint16_t *refcount_block; -uint16_t refcount; +int64_t refcount; refcount_table_index = cluster_index >> s->refcount_block_bits;if (refcount_table_index >= s->refcount_table_size)
@@ -556,9 +556,10 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,for (cluster_offset = start; cluster_offset <= last; cluster_offset += s->cluster_size) { -int block_index, refcount; +int block_index;int64_t cluster_index = cluster_offset >> s->cluster_bits;int64_t table_index = cluster_index >> s->refcount_block_bits; +int64_t refcount;/* Load the refcount block and allocate it if needed */ if (table_index != old_table_index) {
@@ -634,10 +635,10 @@ fail:* If the return value is non-negative, it is the new refcount of the cluster. * If it is negative, it is -errno and indicates an error. */ -int qcow2_update_cluster_refcount (BlockDriverState *bs, -int64_t cluster_index, -int addend, -enum qcow2_discard_type type) +int64_t qcow2_update_cluster_refcount (BlockDriverState *bs, +int64_t cluster_index, +int addend, +enum qcow2_discard_type type) {BDRVQcowState *s = bs->opaque;int ret;
@@ -663,7 +664,7 @@ static int64_t alloc_clusters_noref(BlockDriverState *bs, uint64_t size) {BDRVQcowState *s = bs->opaque;uint64_t i, nb_clusters; -int refcount; +int64_t refcount; nb_clusters =size_to_clusters (s, size); retry:
@@ -722,7 +723,8 @@ int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,BDRVQcowState *s = bs->opaque;uint64_t cluster_index;uint64_t i; -int refcount, ret; +int64_t refcount; +int ret;assert (nb_clusters >=0 );if (nb_clusters ==0 ) {
@@ -878,8 +880,8 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,BDRVQcowState *s = bs->opaque;uint64_t *l1_table, *l2_table, l2_offset, offset, l1_size2;bool l1_allocated = false; -int64_t old_offset, old_l2_offset; -int i, j, l1_modified =0 , nb_csectors, refcount; +int64_t old_offset, old_l2_offset, refcount; +int i, j, l1_modified =0 , nb_csectors;int ret; l2_table =NULL ;
@@ -1341,7 +1343,7 @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res,BDRVQcowState *s = bs->opaque;uint64_t *l2_table =qemu_blockalign (bs, s->cluster_size);int ret; -int refcount; +int64_t refcount;int i, j;for (i =0 ; i < s->l1_size; i++) {
@@ -1360,7 +1362,7 @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res, }if ((refcount ==1 ) != ((l1_entry &QCOW_OFLAG_COPIED ) !=0 )) {fprintf (stderr,"%s OFLAG_COPIED L2 cluster: l1_index=%d " -"l1_entry=%" PRIx64" refcount=%d\n" , +"l1_entry=%" PRIx64" refcount=%" PRId64"\n" , fix &BDRV_FIX_ERRORS ?"Repairing" :"ERROR" , i, l1_entry, refcount);
@@ -1403,7 +1405,7 @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res, }if ((refcount ==1 ) != ((l2_entry &QCOW_OFLAG_COPIED ) !=0 )) {fprintf (stderr,"%s OFLAG_COPIED data cluster: " -"l2_entry=%" PRIx64" refcount=%d\n" , +"l2_entry=%" PRIx64" refcount=%" PRId64"\n" , fix &BDRV_FIX_ERRORS ?"Repairing" :"ERROR" , l2_entry, refcount);
@@ -1628,8 +1630,8 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res, uint16_t *refcount_table, int64_t nb_clusters) {BDRVQcowState *s = bs->opaque; -int64_t i; -int refcount1, refcount2, ret; +int64_t i, refcount1, refcount2; +int ret;for (i =0 , *highest_cluster =0 ; i < nb_clusters; i++) { refcount1 =qcow2_get_refcount (bs, i);
@@ -1657,7 +1659,8 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res, num_fixed = &res->corruptions_fixed; } - fprintf(stderr, "%s cluster %" PRId64 " refcount=%d reference=%d\n", + fprintf(stderr, "%s cluster %" PRId64 " refcount=%" PRId64 + " reference=%" PRId64 "\n", num_fixed != NULL ? "Repairing" : refcount1 < refcount2 ? "ERROR" : "Leaked",
block/qcow2.h +4 -3
@@ -489,10 +489,11 @@ void qcow2_signal_corruption(BlockDriverState *bs, bool fatal, int64_t offset,int qcow2_refcount_init (BlockDriverState *bs);void qcow2_refcount_close (BlockDriverState *bs); -int qcow2_get_refcount (BlockDriverState *bs,int64_t cluster_index); +int64_t qcow2_get_refcount (BlockDriverState *bs,int64_t cluster_index); -int qcow2_update_cluster_refcount (BlockDriverState *bs,int64_t cluster_index, -int addend,enum qcow2_discard_type type); +int64_t qcow2_update_cluster_refcount (BlockDriverState *bs, +int64_t cluster_index,int addend, +enum qcow2_discard_type type);int64_t qcow2_alloc_clusters (BlockDriverState *bs,uint64_t size); int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,
Signature · 1 line
1.9.3
PATCH v2 4/21 qcow2: Respect error in qcow2_alloc_bytes()
In reply to Max Reitzqcow2_update_cluster_refcount() may fail, and qcow2_alloc_bytes() should mind that case.
--- block/qcow2-refcount.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-)
block/qcow2-refcount.c +21 -11
@@ -761,7 +761,8 @@ int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,int64_t qcow2_alloc_bytes (BlockDriverState *bs,int size) {BDRVQcowState *s = bs->opaque; -int64_t offset, cluster_offset; +int64_t offset, cluster_offset, new_cluster; +int64_t ret;int free_in_cluster;BLKDBG_EVENT (bs->file,BLKDBG_CLUSTER_ALLOC_BYTES );
@@ -783,23 +784,32 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) free_in_cluster -= size;if (free_in_cluster ==0 ) s->free_byte_offset =0 ; -if (offset_into_cluster (s, offset) !=0 ) -qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits,1 , -QCOW2_DISCARD_NEVER ); +if (offset_into_cluster (s, offset) !=0 ) { + ret =qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits, +1 ,QCOW2_DISCARD_NEVER ); +if (ret <0 ) { +return ret; + } + } }else { - offset =qcow2_alloc_clusters (bs, s->cluster_size); -if (offset <0 ) { -return offset; + new_cluster =qcow2_alloc_clusters (bs, s->cluster_size); +if (new_cluster <0 ) { +return new_cluster; } cluster_offset =start_of_cluster (s, s->free_byte_offset); -if ((cluster_offset + s->cluster_size) == offset) { +if ((cluster_offset + s->cluster_size) == new_cluster) {/* we are lucky: contiguous data */ offset = s->free_byte_offset; -qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits,1 , -QCOW2_DISCARD_NEVER ); + ret =qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits, +1 ,QCOW2_DISCARD_NEVER ); +if (ret <0 ) { +qcow2_free_clusters (bs, new_cluster, s->cluster_size, +QCOW2_DISCARD_NEVER ); +return ret; + } s->free_byte_offset += size; }else { - s->free_byte_offset = offset; + s->free_byte_offset = new_cluster; goto redo; } }
Signature · 1 line
1.9.3
PATCH v2 5/21 qcow2: Refcount overflow and qcow2_alloc_bytes()
In reply to Max Reitzqcow2_alloc_bytes() may reuse a cluster multiple times, in which case the refcount is increased accordingly. However, if this would lead to an overflow the function should instead just not reuse this cluster and allocate a new one.
--- block/qcow2-refcount.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
block/qcow2-refcount.c +30 -2
@@ -761,12 +761,13 @@ int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,int64_t qcow2_alloc_bytes (BlockDriverState *bs,int size) {BDRVQcowState *s = bs->opaque; -int64_t offset, cluster_offset, new_cluster; +int64_t offset, cluster_offset, new_cluster, refcount;int64_t ret;int free_in_cluster;BLKDBG_EVENT (bs->file,BLKDBG_CLUSTER_ALLOC_BYTES );assert (size >0 && size <= s->cluster_size); + redo:if (s->free_byte_offset ==0 ) { offset =qcow2_alloc_clusters (bs, s->cluster_size);if (offset <0 ) {
@@ -774,12 +775,25 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) } s->free_byte_offset = offset; } - redo: + free_in_cluster = s->cluster_size -offset_into_cluster (s, s->free_byte_offset);if (size <= free_in_cluster) {/* enough space in current cluster */ offset = s->free_byte_offset; + +if (offset_into_cluster (s, offset) !=0 ) { +/* We will have to increase the refcount of this cluster; if the +* maximum has been reached already, this cluster cannot be used */ + refcount =qcow2_get_refcount (bs, offset >> s->cluster_bits); +if (refcount <0 ) { +return refcount; + }else if (refcount == s->refcount_max) { + s->free_byte_offset =0 ; + goto redo; + } + } + s->free_byte_offset += size; free_in_cluster -= size;if (free_in_cluster ==0 )
@@ -800,6 +814,20 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)if ((cluster_offset + s->cluster_size) == new_cluster) {/* we are lucky: contiguous data */ offset = s->free_byte_offset; + +/* Same as above: In order to reuse the cluster, the refcount has to +* be increased; if that will not work, we are not so lucky after +* all */ + refcount =qcow2_get_refcount (bs, offset >> s->cluster_bits); +if (refcount <0 ) { +qcow2_free_clusters (bs, new_cluster, s->cluster_size, +QCOW2_DISCARD_NEVER ); +return refcount; + }else if (refcount == s->refcount_max) { + s->free_byte_offset = offset; + goto redo; + } + ret =qcow2_update_cluster_refcount (bs, offset >> s->cluster_bits,1 ,QCOW2_DISCARD_NEVER );if (ret <0 ) {
Signature · 1 line
1.9.3
PATCH v2 6/21 qcow2: Helper for refcount array reallocation
In reply to Max ReitzAdd a helper function for reallocating a refcount array, independently of the refcount order. The newly allocated space is zeroed and the function handles failed reallocations gracefully.
--- block/qcow2-refcount.c | 121 +++++++++++++++++++++++++++++-------------------- 1 file changed, 72 insertions(+), 49 deletions(-)
block/qcow2-refcount.c +72 -49
@@ -1108,6 +1108,54 @@ fail:/* refcount checking functions */ +static size_t refcount_array_byte_size (BDRVQcowState *s,uint64_t entries) +{ +if (s->refcount_order <3 ) { +/* sub-byte width */ +int shift =3 - s->refcount_order; +return (entries + (1 << shift) -1 ) >> shift; + }else if (s->refcount_order ==3 ) { +/* byte width */ +return entries; + }else { +/* multiple bytes wide */ + +/* This assertion holds because there is no way we can address more than +* 2^(64 - 9) clusters at once (with cluster size 512 = 2^9, and because +* offsets have to be representable in bytes); due to every cluster +* corresponding to one refcount entry and because refcount_order has to +* be below 7, we are far below that limit */ +assert (!(entries >> (64 - (s->refcount_order -3 )))); + +return entries << (s->refcount_order -3 ); + } +} + +static int realloc_refcount_array (BDRVQcowState *s,uint16_t **array, +int64_t *size,int64_t new_size) +{ +/* Round to clusters so the array can be directly written to disk */ +size_t old_byte_size =ROUND_UP (refcount_array_byte_size (s, *size), + s->cluster_size); +size_t new_byte_size =ROUND_UP (refcount_array_byte_size (s, new_size), + s->cluster_size); +uint16_t *new_ptr; + + new_ptr =g_try_realloc (*array, new_byte_size); +if (new_byte_size && !new_ptr) { +return -ENOMEM ; + } + +if (new_ptr) { +memset ((void *)((uintptr_t )new_ptr + old_byte_size),0 , + new_byte_size - old_byte_size); + } + + *array = new_ptr; + *size = new_size; + +return 0 ; +}/* * Increases the refcount for a range of clusters in a given refcount table.
@@ -1124,6 +1172,7 @@ static int inc_refcounts(BlockDriverState *bs, {BDRVQcowState *s = bs->opaque;uint64_t start, last, cluster_offset, k; +int ret;if (size <=0 ) {return 0 ;
@@ -1135,23 +1184,12 @@ static int inc_refcounts(BlockDriverState *bs, cluster_offset += s->cluster_size) { k = cluster_offset >> s->cluster_bits;if (k >= *refcount_table_size) { -int64_t old_refcount_table_size = *refcount_table_size; -uint16_t *new_refcount_table; - - *refcount_table_size = k +1 ; - new_refcount_table =g_try_realloc (*refcount_table, - *refcount_table_size * -sizeof (**refcount_table)); -if (!new_refcount_table) { - *refcount_table_size = old_refcount_table_size; + ret =realloc_refcount_array (s, refcount_table, + refcount_table_size, k +1 ); +if (ret <0 ) { res->check_errors++; -return -ENOMEM ; +return ret; } - *refcount_table = new_refcount_table; - -memset (*refcount_table + old_refcount_table_size,0 , - (*refcount_table_size - old_refcount_table_size) * -sizeof (**refcount_table)); }if (++(*refcount_table)[k] ==0 ) {
@@ -1518,8 +1556,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", i);if (fix &BDRV_FIX_ERRORS ) { -int64_t old_nb_clusters = *nb_clusters; -uint16_t *new_refcount_table; +int64_t new_nb_clusters;if (offset >INT64_MAX - s->cluster_size) { ret = -EINVAL ;
@@ -1536,22 +1573,15 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res, goto resize_fail; } - *nb_clusters =size_to_clusters (s, size); -assert (*nb_clusters >= old_nb_clusters); + new_nb_clusters =size_to_clusters (s, size); +assert (new_nb_clusters >= *nb_clusters); - new_refcount_table =g_try_realloc (*refcount_table, - *nb_clusters * -sizeof (**refcount_table)); -if (!new_refcount_table) { - *nb_clusters = old_nb_clusters; + ret =realloc_refcount_array (s, refcount_table, + nb_clusters, new_nb_clusters); +if (ret <0 ) { res->check_errors++; -return -ENOMEM ; +return ret; } - *refcount_table = new_refcount_table; - -memset (*refcount_table + old_nb_clusters,0 , - (*nb_clusters - old_nb_clusters) * -sizeof (**refcount_table));if (cluster >= *nb_clusters) { ret = -EINVAL ;
@@ -1611,10 +1641,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,int ret;if (!*refcount_table) { - *refcount_table =g_try_new0 (uint16_t, *nb_clusters); -if (*nb_clusters && *refcount_table ==NULL ) { +int64_t old_size =0 ; + ret =realloc_refcount_array (s, refcount_table, + &old_size, *nb_clusters); +if (ret <0 ) { res->check_errors++; -return -ENOMEM ; +return ret; } }
@@ -1746,6 +1778,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,int64_t cluster = *first_free_cluster, i;bool first_gap = true;int contiguous_free_clusters; +int ret;/* Starting at *first_free_cluster, find a range of at least cluster_count * continuously free clusters */
@@ -1775,28 +1808,18 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,/* If no such range could be found, grow the in-memory refcount table * accordingly to append free clusters at the end of the image */ if (contiguous_free_clusters < cluster_count) { -int64_t old_imrt_nb_clusters = *imrt_nb_clusters; -uint16_t *new_refcount_table; -/* contiguous_free_clusters clusters are already empty at the image end; * we need cluster_count clusters; therefore, we have to allocate * cluster_count - contiguous_free_clusters new clusters at the end of * the image (which is the current value of cluster; note that cluster * may exceed old_imrt_nb_clusters if *first_free_cluster pointed beyond * the image end) */ - *imrt_nb_clusters = cluster + cluster_count - contiguous_free_clusters; - new_refcount_table =g_try_realloc (*refcount_table, - *imrt_nb_clusters * -sizeof (**refcount_table)); -if (!new_refcount_table) { - *imrt_nb_clusters = old_imrt_nb_clusters; -return -ENOMEM ; - } - *refcount_table = new_refcount_table; - -memset (*refcount_table + old_imrt_nb_clusters,0 , - (*imrt_nb_clusters - old_imrt_nb_clusters) * -sizeof (**refcount_table)); + ret =realloc_refcount_array (s, refcount_table, imrt_nb_clusters, + cluster + cluster_count + - contiguous_free_clusters); +if (ret <0 ) { +return ret; + } }/* Go back to the first free cluster */
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v2 06/21] qcow2: Helper for refcount array reallocation
Quoted 1 line · Max Reitz
Add a helper function for reallocating a refcount array, independently
s/independently/independent/
Quoted 2 lines
of the refcount order. The newly allocated space is zeroed and the function handles failed reallocations gracefully.
This patch is doing two things: it is refactoring things into a nice helper function (mentioned), AND it is adding a guarantee that you now always allocate a table on cluster boundaries, even when you aren't using the full table (hinted at elsewhere in the series, but noticeably absent here). I think you want to add more comments to the commit message making that more obvious, since it looks like you rely on that guarantee later.
block/qcow2-refcount.c fail: · line 1135
Quoted 10 lines
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 121 +++++++++++++++++++++++++++++-------------------- 1 file changed, 72 insertions(+), 49 deletions(-)block/qcow2-refcount.c +3
-0+ +static int realloc_refcount_array (BDRVQcowState *s,uint16_t **array, +int64_t *size,int64_t new_size)
I think this function deserves a comment stating that *array is actually allocated to full cluster size with a 0 tail, so that it can be written straight to disk.
block/qcow2-refcount.c fail: · line 1142
Quoted 7 lines
block/qcow2-refcount.c +7
-0+{ +/* Round to clusters so the array can be directly written to disk */ +size_t old_byte_size =ROUND_UP (refcount_array_byte_size (s, *size), + s->cluster_size); +size_t new_byte_size =ROUND_UP (refcount_array_byte_size (s, new_size), + s->cluster_size); +uint16_t *new_ptr;
Can old_byte_size ever equal new_byte_size? Or are we guaranteed that this will only be called when we really need to add another cluster to the reftable?
[reading further]
Yes, it looks like *size and new_size are not necessarily cluster-aligned, so as an example, it is very likely that we might call realloc_refcount_array with the existing size of 20 and a new size of 21, both of which fit within the same byte size when rounded up to cluster boundary. But that means that the realloc is a no-op in that case; might it be worth special-casing rather than wasting time on the g_try_realloc and no-op memset? [at least the code works correctly even without a special case shortcut]
block/qcow2-refcount.c fail: · line 1147
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ + new_ptr =g_try_realloc (*array, new_byte_size); +if (new_byte_size && !new_ptr) { +return -ENOMEM ; + }
Is it worth asserting that new_byte_size is non-zero? Why would anyone ever call this to resize down to 0? (But I can see where you DO call it with old_byte_size of zero, when initializing data structures and using this function for the first allocation.)
Review
Quoted 2 lines
(quoted diff) +2
-0+ + if (new_ptr) {
If we assert that new_byte_size is non-zero, then at this point, new_ptr is non-NULL and this condition is pointless.
block/qcow2-refcount.c fail: · line 1158
Quoted 10 lines
block/qcow2-refcount.c +9
-0+memset ((void *)((uintptr_t )new_ptr + old_byte_size),0 , + new_byte_size - old_byte_size); + } + + *array = new_ptr; + *size = new_size; + +return 0 ; +}
Code looks correct as written, whether or not you also add more comments, asserts, and/or shortcuts for no-op situations. So:
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v2 06/21] qcow2: Helper for refcount array reallocation
Quoted 6 lines · Eric Blake
Quoted 1 line · Max Reitz
Add a helper function for reallocating a refcount array, independently
s/independently/independent/
Quoted 2 lines
of the refcount order. The newly allocated space is zeroed and the function handles failed reallocations gracefully.
Quoted 7 lines
This patch is doing two things: it is refactoring things into a nice helper function (mentioned), AND it is adding a guarantee that you now always allocate a table on cluster boundaries, even when you aren't using the full table (hinted at elsewhere in the series, but noticeably absent here). I think you want to add more comments to the commit message making that more obvious, since it looks like you rely on that guarantee later.
Will do.
Quoted 11 lines
block/qcow2-refcount.c fail: · line 1135
Quoted 8 lines
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 121 +++++++++++++++++++++++++++++-------------------- 1 file changed, 72 insertions(+), 49 deletions(-)block/qcow2-refcount.c +3
-0+ +static int realloc_refcount_array (BDRVQcowState *s,uint16_t **array, +int64_t *size,int64_t new_size)I think this function deserves a comment stating that *array is actually allocated to full cluster size with a 0 tail, so that it can be written straight to disk.
OK, will add a comment.
Quoted 13 lines
block/qcow2-refcount.c fail: · line 1142
Quoted 7 lines
block/qcow2-refcount.c +7
-0+{ +/* Round to clusters so the array can be directly written to disk */ +size_t old_byte_size =ROUND_UP (refcount_array_byte_size (s, *size), + s->cluster_size); +size_t new_byte_size =ROUND_UP (refcount_array_byte_size (s, new_size), + s->cluster_size); +uint16_t *new_ptr;Can
old_byte_sizeever equalnew_byte_size? Or are we guaranteed that this will only be called when we really need to add another cluster to the reftable?[reading further]
Quoted 8 lines
Yes, it looks like *size and
new_sizeare not necessarily cluster-aligned, so as an example, it is very likely that we might callrealloc_refcount_arraywith the existing size of 20 and a new size of 21, both of which fit within the same byte size when rounded up to cluster boundary. But that means that the realloc is a no-op in that case; might it be worth special-casing rather than wasting time on theg_try_reallocand no-op memset? [at least the code works correctly even without a special case shortcut]
Well, it's probably not necessary, but it will look most likely look better to catch that case.
Quoted 9 lines
block/qcow2-refcount.c fail: · line 1147
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ + new_ptr =g_try_realloc (*array, new_byte_size); +if (new_byte_size && !new_ptr) { +return -ENOMEM ; + }Is it worth asserting that
new_byte_sizeis non-zero? Why would anyone ever call this to resize down to 0? (But I can see where you DO call it withold_byte_sizeof zero, when initializing data structures and using this function for the first allocation.)
Hm, considering every image that can be opened using the qcow2 driver needs at least one cluster (the header), we can outrule that this is called with new_size == 0 (which would be the only way new_byte_size could ever be 0 either).
Quoted 18 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ + if (new_ptr) {If we assert that
new_byte_sizeis non-zero, then at this point,new_ptris non-NULL and this condition is pointless.block/qcow2-refcount.c fail: · line 1158
Quoted 10 lines
block/qcow2-refcount.c +9
-0+memset ((void *)((uintptr_t )new_ptr + old_byte_size),0 , + new_byte_size - old_byte_size); + } + + *array = new_ptr; + *size = new_size; + +return 0 ; +}Code looks correct as written, whether or not you also add more comments, asserts, and/or shortcuts for no-op situations. So:
Quoted 1 line
Reviewed-by Eric Blake <[email protected]>
Thanks!
Max
PATCH v2 7/21 qcow2: Helper function for refcount modification
In reply to Max ReitzSince refcounts do not always have to be a uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.
--- block/qcow2-refcount.c | 128 ++++++++++++++++++++++++++++++------------------- block/qcow2.h | 8 ++++ 2 files changed, 87 insertions(+), 49 deletions(-)
block/qcow2-refcount.c +79 -49
@@ -32,6 +32,11 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs, int64_t offset, int64_t length, int addend, enum qcow2_discard_type type); +static uint64_t get_refcount_ro4 (const void *refcount_array,uint64_t index); + +static void set_refcount_ro4 (void *refcount_array,uint64_t index, +uint64_t value); +/*********************************************************/ /* refcount handling */
@@ -42,6 +47,9 @@ int qcow2_refcount_init(BlockDriverState *bs)unsigned int refcount_table_size2, i;int ret; + s->get_refcount = &get_refcount_ro4; + s->set_refcount = &set_refcount_ro4; +assert (s->refcount_table_size <=INT_MAX /sizeof (uint64_t )); refcount_table_size2 = s->refcount_table_size *sizeof (uint64_t ); s->refcount_table =g_try_malloc (refcount_table_size2);
@@ -72,6 +80,19 @@ void qcow2_refcount_close(BlockDriverState *bs) } +static uint64_t get_refcount_ro4 (const void *refcount_array,uint64_t index) +{ +return be16_to_cpu (((const uint16_t *)refcount_array)[index]); +} + +static void set_refcount_ro4 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>16 )); + ((uint16_t *)refcount_array)[index] =cpu_to_be16 (value); +} + +static int load_refcount_block (BlockDriverState *bs,int64_t refcount_block_offset,void **refcount_block)
@@ -97,7 +118,7 @@ int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index)uint64_t refcount_table_index, block_index;int64_t refcount_block_offset;int ret; -uint16_t *refcount_block; +void *refcount_block;int64_t refcount; refcount_table_index = cluster_index >> s->refcount_block_bits;
@@ -116,20 +137,24 @@ int64_t qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index) } ret =qcow2_cache_get (bs, s->refcount_block_cache, refcount_block_offset, - (void **) &refcount_block); + &refcount_block);if (ret <0 ) {return ret; } block_index = cluster_index & (s->refcount_block_size -1 ); - refcount =be16_to_cpu (refcount_block[block_index]); + refcount = s->get_refcount (refcount_block, block_index); - ret =qcow2_cache_put (bs, s->refcount_block_cache, - (void **) &refcount_block); + ret =qcow2_cache_put (bs, s->refcount_block_cache, &refcount_block);if (ret <0 ) {return ret; } +if (refcount <0 ) { +/* overflow */ +return -ERANGE ; + } +return refcount; }
@@ -169,7 +194,7 @@ static int in_same_refcount_block(BDRVQcowState *s, uint64_t offset_a,* Returns 0 on success or -errno in error case */ static int alloc_refcount_block (BlockDriverState *bs, -int64_t cluster_index,uint16_t **refcount_block) +int64_t cluster_index,void **refcount_block) {BDRVQcowState *s = bs->opaque;unsigned int refcount_table_index;
@@ -196,7 +221,7 @@ static int alloc_refcount_block(BlockDriverState *bs, }return load_refcount_block (bs, refcount_block_offset, - (void **) refcount_block); + refcount_block); } }
@@ -256,7 +281,7 @@ static int alloc_refcount_block(BlockDriverState *bs,/* The block describes itself, need to update the cache */ int block_index = (new_block >> s->cluster_bits) & (s->refcount_block_size -1 ); - (*refcount_block)[block_index] =cpu_to_be16 (1 ); + s->set_refcount (*refcount_block, block_index,1 ); }else {/* Described somewhere else. This can recurse at most twice before we * arrive at a block that describes itself. */
@@ -274,7 +299,7 @@ static int alloc_refcount_block(BlockDriverState *bs,/* Initialize the new refcount block only after updating its refcount, * update_refcount uses the refcount cache itself */ ret =qcow2_cache_get_empty (bs, s->refcount_block_cache, new_block, - (void **) refcount_block); + refcount_block);if (ret <0 ) { goto fail_block; }
@@ -308,7 +333,7 @@ static int alloc_refcount_block(BlockDriverState *bs,return -EAGAIN ; } - ret =qcow2_cache_put (bs, s->refcount_block_cache, (void **) refcount_block); + ret =qcow2_cache_put (bs, s->refcount_block_cache, refcount_block);if (ret <0 ) { goto fail_block; }
@@ -362,7 +387,7 @@ static int alloc_refcount_block(BlockDriverState *bs, s->cluster_size;uint64_t table_offset = meta_offset + blocks_clusters * s->cluster_size;uint64_t *new_table =g_try_new0 (uint64_t, table_size); -uint16_t *new_blocks =g_try_malloc0 (blocks_clusters * s->cluster_size); +void *new_blocks =g_try_malloc0_n (blocks_clusters, s->cluster_size);assert (table_size >0 && blocks_clusters >0 );if (new_table ==NULL || new_blocks ==NULL ) {
@@ -384,7 +409,7 @@ static int alloc_refcount_block(BlockDriverState *bs,uint64_t table_clusters =size_to_clusters (s, table_size *sizeof (uint64_t ));int block =0 ;for (i =0 ; i < table_clusters + blocks_clusters; i++) { - new_blocks[block++] =cpu_to_be16 (1 ); + s->set_refcount (new_blocks, block++,1 ); }/* Write refcount blocks to disk */
@@ -437,7 +462,7 @@ static int alloc_refcount_block(BlockDriverState *bs,qcow2_free_clusters (bs, old_table_offset, old_table_size *sizeof (uint64_t ),QCOW2_DISCARD_OTHER ); - ret =load_refcount_block (bs, new_block, (void **) refcount_block); + ret =load_refcount_block (bs, new_block, refcount_block);if (ret <0 ) {return ret; }
@@ -452,7 +477,7 @@ fail_table:g_free (new_table); fail_block:if (*refcount_block !=NULL ) { -qcow2_cache_put (bs, s->refcount_block_cache, (void **) refcount_block); +qcow2_cache_put (bs, s->refcount_block_cache, refcount_block); }return ret; }
@@ -532,7 +557,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs, {BDRVQcowState *s = bs->opaque;int64_t start, last, cluster_offset; -uint16_t *refcount_block =NULL ; +void *refcount_block =NULL ;int64_t old_table_index =-1 ;int ret;
@@ -583,7 +608,12 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,/* we can update the count and save it */ block_index = cluster_index & (s->refcount_block_size -1 ); - refcount =be16_to_cpu (refcount_block[block_index]); + refcount = s->get_refcount (refcount_block, block_index); +if (refcount <0 ) { + ret = -ERANGE ; + goto fail; + } + refcount += addend;if (refcount <0 || refcount > s->refcount_max) { ret = -EINVAL ;
@@ -592,7 +622,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,if (refcount ==0 && cluster_index < s->free_cluster_index) { s->free_cluster_index = cluster_index; } - refcount_block[block_index] =cpu_to_be16 (refcount); + s->set_refcount (refcount_block, block_index, refcount);if (refcount ==0 && s->discard_passthrough[type]) {update_refcount_discard (bs, cluster_offset, s->cluster_size);
@@ -608,8 +638,7 @@ fail:/* Write last changed block to disk */ if (refcount_block) {int wret; - wret =qcow2_cache_put (bs, s->refcount_block_cache, - (void **) &refcount_block); + wret =qcow2_cache_put (bs, s->refcount_block_cache, &refcount_block);if (wret <0 ) {return ret <0 ? ret : wret; }
@@ -1131,7 +1160,7 @@ static size_t refcount_array_byte_size(BDRVQcowState *s, uint64_t entries) } } -static int realloc_refcount_array (BDRVQcowState *s,uint16_t **array, +static int realloc_refcount_array (BDRVQcowState *s,void **array,int64_t *size,int64_t new_size) {/* Round to clusters so the array can be directly written to disk */
@@ -1139,7 +1168,7 @@ static int realloc_refcount_array(BDRVQcowState *s, uint16_t **array, s->cluster_size);size_t new_byte_size =ROUND_UP (refcount_array_byte_size (s, new_size), s->cluster_size); -uint16_t *new_ptr; +void *new_ptr; new_ptr =g_try_realloc (*array, new_byte_size);if (new_byte_size && !new_ptr) {
@@ -1166,12 +1195,13 @@ static int realloc_refcount_array(BDRVQcowState *s, uint16_t **array,*/ static int inc_refcounts (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table, +void **refcount_table,int64_t *refcount_table_size,int64_t offset,int64_t size) {BDRVQcowState *s = bs->opaque;uint64_t start, last, cluster_offset, k; +int64_t refcount;int ret;if (size <=0 ) {
@@ -1192,11 +1222,14 @@ static int inc_refcounts(BlockDriverState *bs, } } -if (++(*refcount_table)[k] ==0 ) { + refcount = s->get_refcount (*refcount_table, k); +if (refcount == s->refcount_max) {fprintf (stderr,"ERROR: overflow cluster offset=0x%" PRIx64"\n" , cluster_offset); res->corruptions++; +continue ; } + s->set_refcount (*refcount_table, k, refcount +1 ); }return 0 ;
@@ -1216,7 +1249,7 @@ enum {* error occurred. */ static int check_refcounts_l2 (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table,int64_t *refcount_table_size,int64_t l2_offset, +void **refcount_table,int64_t *refcount_table_size,int64_t l2_offset,int flags) {BDRVQcowState *s = bs->opaque;
@@ -1334,7 +1367,7 @@ fail:*/ static int check_refcounts_l1 (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table, +void **refcount_table,int64_t *refcount_table_size,int64_t l1_table_offset,int l1_size,int flags)
@@ -1531,7 +1564,7 @@ fail:*/ static int check_refblocks (BlockDriverState *bs,BdrvCheckResult *res,BdrvCheckMode fix,bool *rebuild, -uint16_t **refcount_table,int64_t *nb_clusters) +void **refcount_table,int64_t *nb_clusters) {BDRVQcowState *s = bs->opaque;int64_t i, size;
@@ -1616,9 +1649,10 @@ resize_fail:if (ret <0 ) {return ret; } -if ((*refcount_table)[cluster] !=1 ) { +if (s->get_refcount (*refcount_table, cluster) !=1 ) {fprintf (stderr,"ERROR refcount block %" PRId64 -" refcount=%d\n" , i, (*refcount_table)[cluster]); +" refcount=%" PRIu64"\n" , i, + s->get_refcount (*refcount_table, cluster)); res->corruptions++; *rebuild = true; }
@@ -1633,7 +1667,7 @@ resize_fail:*/ static int calculate_refcounts (BlockDriverState *bs,BdrvCheckResult *res,BdrvCheckMode fix,bool *rebuild, -uint16_t **refcount_table,int64_t *nb_clusters) +void **refcount_table,int64_t *nb_clusters) {BDRVQcowState *s = bs->opaque;int64_t i;
@@ -1697,7 +1731,7 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,static void compare_refcounts (BlockDriverState *bs,BdrvCheckResult *res,BdrvCheckMode fix,bool *rebuild,int64_t *highest_cluster, -uint16_t *refcount_table,int64_t nb_clusters) +void *refcount_table,int64_t nb_clusters) {BDRVQcowState *s = bs->opaque;int64_t i, refcount1, refcount2;
@@ -1712,7 +1746,7 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,continue ; } - refcount2 = refcount_table[i]; + refcount2 = s->get_refcount (refcount_table, i);if (refcount1 >0 || refcount2 >0 ) { *highest_cluster = i;
@@ -1770,7 +1804,7 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,*/ static int64_t alloc_clusters_imrt (BlockDriverState *bs,int cluster_count, -uint16_t **refcount_table, +void **refcount_table,int64_t *imrt_nb_clusters,int64_t *first_free_cluster) {
@@ -1787,7 +1821,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs, contiguous_free_clusters < cluster_count; cluster++) { -if (!(*refcount_table)[cluster]) { +if (!s->get_refcount (*refcount_table, cluster)) { contiguous_free_clusters++;if (first_gap) {/* If this is the first free cluster found, update
@@ -1825,7 +1859,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,/* Go back to the first free cluster */ cluster -= contiguous_free_clusters;for (i =0 ; i < cluster_count; i++) { - (*refcount_table)[cluster + i] =1 ; + s->set_refcount (*refcount_table, cluster + i,1 ); }return cluster << s->cluster_bits;
@@ -1841,7 +1875,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,*/ static int rebuild_refcount_structure (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table, +void **refcount_table,int64_t *nb_clusters) {BDRVQcowState *s = bs->opaque;
@@ -1849,8 +1883,8 @@ static int rebuild_refcount_structure(BlockDriverState *bs,int64_t refblock_offset, refblock_start, refblock_index;uint32_t reftable_size =0 ;uint64_t *on_disk_reftable =NULL ; -uint16_t *on_disk_refblock; -int i, ret =0 ; +void *on_disk_refblock; +int ret =0 ;struct {uint64_t reftable_offset;uint32_t reftable_clusters;
@@ -1860,7 +1894,7 @@ static int rebuild_refcount_structure(BlockDriverState *bs, write_refblocks:for (; cluster < *nb_clusters; cluster++) { -if (!(*refcount_table)[cluster]) { +if (!s->get_refcount (*refcount_table, cluster)) {continue ; }
@@ -1933,17 +1967,13 @@ write_refblocks: goto fail; } - on_disk_refblock =qemu_blockalign0 (bs->file, s->cluster_size); -for (i =0 ; i < s->refcount_block_size && - refblock_start + i < *nb_clusters; i++) - { - on_disk_refblock[i] = -cpu_to_be16 ((*refcount_table)[refblock_start + i]); - } +/* The size of *refcount_table is always cluster-aligned, therefore the +* write operation will not overflow */ + on_disk_refblock = (void *)((uintptr_t )*refcount_table + + (refblock_index << s->refcount_block_bits)); ret =bdrv_write (bs->file, refblock_offset /BDRV_SECTOR_SIZE , - (void *)on_disk_refblock, s->cluster_sectors); -qemu_vfree (on_disk_refblock); + on_disk_refblock, s->cluster_sectors);if (ret <0 ) {fprintf (stderr,"ERROR writing refblock: %s\n" ,strerror (-ret)); goto fail;
@@ -2038,7 +2068,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,BDRVQcowState *s = bs->opaque;BdrvCheckResult pre_compare_res;int64_t size, highest_cluster, nb_clusters; -uint16_t *refcount_table =NULL ; +void *refcount_table =NULL ;bool rebuild = false;int ret;
@@ -2087,7 +2117,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,/* Because the old reftable has been exchanged for a new one the * references have to be recalculated */ rebuild = false; -memset (refcount_table,0 , nb_clusters *sizeof (uint16_t )); +memset (refcount_table,0 , nb_clusters * s->refcount_bits /8 ); ret =calculate_refcounts (bs, res,0 , &rebuild, &refcount_table, &nb_clusters);if (ret <0 ) {
block/qcow2.h +8 -0
@@ -213,6 +213,11 @@ typedef struct Qcow2DiscardRegion {QTAILQ_ENTRY (Qcow2DiscardRegion) next; } Qcow2DiscardRegion; +typedef uint64_t Qcow2GetRefcountFunc (const void *refcount_array, +uint64_t index); +typedef void Qcow2SetRefcountFunc (void *refcount_array, +uint64_t index,uint64_t value); + typedef struct BDRVQcowState { int cluster_bits; int cluster_size;
@@ -261,6 +266,9 @@ typedef struct BDRVQcowState {int refcount_bits;uint64_t refcount_max; +Qcow2GetRefcountFunc *get_refcount; +Qcow2SetRefcountFunc *set_refcount; +bool discard_passthrough[QCOW2_DISCARD_MAX ];int overlap_check;/* bitmask of Qcow2MetadataOverlap values */
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v2 07/21] qcow2: Helper function for refcount modification
Quoted 15 lines · Max Reitz
Since refcounts do not always have to be a
uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 128 ++++++++++++++++++++++++++++++------------------- block/qcow2.h | 8 ++++ 2 files changed, 87 insertions(+), 49 deletions(-)
block/qcow2-refcount.c enum { · line 1254
Quoted 7 lines
block/qcow2-refcount.c +1
-1@@ -1216,7 +1249,7 @@ enum {* error occurred. */ static int check_refcounts_l2 (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table,int64_t *refcount_table_size,int64_t l2_offset, +void **refcount_table,int64_t *refcount_table_size,int64_t l2_offset,int flags) {
Might be worth fixing the indentation here in addition to all the other places you adjusted. But that's minor.
Quoted 5 lines
block/qcow2-refcount.c +0
-1@@ -1933,17 +1967,13 @@ write_refblocks: goto fail; } - on_disk_refblock =qemu_blockalign0 (bs->file, s->cluster_size);
block/qcow2-refcount.c write_refblocks: · line 1973
Quoted 10 lines
block/qcow2-refcount.c +4
-6-for (i =0 ; i < s->refcount_block_size && - refblock_start + i < *nb_clusters; i++) - { - on_disk_refblock[i] = -cpu_to_be16 ((*refcount_table)[refblock_start + i]); - } +/* The size of *refcount_table is always cluster-aligned, therefore the +* write operation will not overflow */ + on_disk_refblock = (void *)((uintptr_t )*refcount_table + + (refblock_index << s->refcount_block_bits));
Here is where you are relying on the guarantee that you added in 6/21, which is why I ask for that one to mention it.
Nice reduction of a bounce buffer, by the way :) Worth mentioning in the commit message as an intentional part of this commit?
block/qcow2-refcount.c int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 2120
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -2087,7 +2117,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,/* Because the old reftable has been exchanged for a new one the * references have to be recalculated */ rebuild = false; -memset (refcount_table,0 , nb_clusters *sizeof (uint16_t )); +memset (refcount_table,0 , nb_clusters * s->refcount_bits /8 );
Phew; we're safe that this won't overflow; and good that you do the * first (if you did the /8 first, it would fail for sub-byte refcounts).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v2 07/21] qcow2: Helper function for refcount modification
Quoted 43 lines · Eric Blake
Quoted 14 lines · Max Reitz
Since refcounts do not always have to be a
uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 128 ++++++++++++++++++++++++++++++------------------- block/qcow2.h | 8 ++++ 2 files changed, 87 insertions(+), 49 deletions(-)block/qcow2-refcount.c enum { · line 1254
Quoted 7 lines
block/qcow2-refcount.c +1
-1@@ -1216,7 +1249,7 @@ enum {* error occurred. */ static int check_refcounts_l2 (BlockDriverState *bs,BdrvCheckResult *res, -uint16_t **refcount_table,int64_t *refcount_table_size,int64_t l2_offset, +void **refcount_table,int64_t *refcount_table_size,int64_t l2_offset,int flags) {Might be worth fixing the indentation here in addition to all the other places you adjusted. But that's minor.
Quoted 5 lines
block/qcow2-refcount.c +0
-1@@ -1933,17 +1967,13 @@ write_refblocks: goto fail; } - on_disk_refblock =qemu_blockalign0 (bs->file, s->cluster_size);block/qcow2-refcount.c write_refblocks: · line 1973
Quoted 10 lines
block/qcow2-refcount.c +4
-6-for (i =0 ; i < s->refcount_block_size && - refblock_start + i < *nb_clusters; i++) - { - on_disk_refblock[i] = -cpu_to_be16 ((*refcount_table)[refblock_start + i]); - } +/* The size of *refcount_table is always cluster-aligned, therefore the +* write operation will not overflow */ + on_disk_refblock = (void *)((uintptr_t )*refcount_table + + (refblock_index << s->refcount_block_bits));Here is where you are relying on the guarantee that you added in 6/21, which is why I ask for that one to mention it.
Quoted 2 lines
Nice reduction of a bounce buffer, by the way :) Worth mentioning in the commit message as an intentional part of this commit?
Why not.
Quoted 8 lines
block/qcow2-refcount.c int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 2120
Quoted 6 lines
block/qcow2-refcount.c +1
-1@@ -2087,7 +2117,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,/* Because the old reftable has been exchanged for a new one the * references have to be recalculated */ rebuild = false; -memset (refcount_table,0 , nb_clusters *sizeof (uint16_t )); +memset (refcount_table,0 , nb_clusters * s->refcount_bits /8 );Phew; we're safe that this won't overflow; and good that you do the * first (if you did the /8 first, it would fail for sub-byte refcounts).
Thanks for catching this, it is wrong (albeit it does the right thing). It should use refcount_array_byte_size(), which was in this version of the series introduced before this patch, so it's an artifact of swapping patch 6 and 7.
Max
Quoted 1 line
Reviewed-by Eric Blake <[email protected]>
PATCH v2 8/21 qcow2: More helpers for refcount modification
In reply to Max ReitzAdd helper functions for getting and setting refcounts in a refcount array for any possible refcount order, and choose the correct one during refcount initialization.
--- block/qcow2-refcount.c | 146 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 144 insertions(+), 2 deletions(-)
block/qcow2-refcount.c +144 -2
@@ -32,10 +32,73 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs, int64_t offset, int64_t length, int addend, enum qcow2_discard_type type); +static uint64_t get_refcount_ro0 (const void *refcount_array,uint64_t index); +static uint64_t get_refcount_ro1 (const void *refcount_array,uint64_t index); +static uint64_t get_refcount_ro2 (const void *refcount_array,uint64_t index); +static uint64_t get_refcount_ro3 (const void *refcount_array,uint64_t index);static uint64_t get_refcount_ro4 (const void *refcount_array,uint64_t index); +static uint64_t get_refcount_ro5 (const void *refcount_array,uint64_t index); +static uint64_t get_refcount_ro6 (const void *refcount_array,uint64_t index); +static void set_refcount_ro0 (void *refcount_array,uint64_t index, +uint64_t value); +static void set_refcount_ro1 (void *refcount_array,uint64_t index, +uint64_t value); +static void set_refcount_ro2 (void *refcount_array,uint64_t index, +uint64_t value); +static void set_refcount_ro3 (void *refcount_array,uint64_t index, +uint64_t value);static void set_refcount_ro4 (void *refcount_array,uint64_t index,uint64_t value); +static void set_refcount_ro5 (void *refcount_array,uint64_t index, +uint64_t value); +static void set_refcount_ro6 (void *refcount_array,uint64_t index, +uint64_t value); + +static void get_refcount_functions (int refcount_order, +Qcow2GetRefcountFunc **get, +Qcow2SetRefcountFunc **set) +{ +switch (refcount_order) { +case 0 : + *get = &get_refcount_ro0; + *set = &set_refcount_ro0; +break ; + +case 1 : + *get = &get_refcount_ro1; + *set = &set_refcount_ro1; +break ; + +case 2 : + *get = &get_refcount_ro2; + *set = &set_refcount_ro2; +break ; + +case 3 : + *get = &get_refcount_ro3; + *set = &set_refcount_ro3; +break ; + +case 4 : + *get = &get_refcount_ro4; + *set = &set_refcount_ro4; +break ; + +case 5 : + *get = &get_refcount_ro5; + *set = &set_refcount_ro5; +break ; + +case 6 : + *get = &get_refcount_ro6; + *set = &set_refcount_ro6; +break ; + +default : +abort (); + } +}/*********************************************************/
@@ -47,8 +110,8 @@ int qcow2_refcount_init(BlockDriverState *bs)unsigned int refcount_table_size2, i;int ret; - s->get_refcount = &get_refcount_ro4; - s->set_refcount = &set_refcount_ro4; +get_refcount_functions (s->refcount_order, + &s->get_refcount, &s->set_refcount);assert (s->refcount_table_size <=INT_MAX /sizeof (uint64_t )); refcount_table_size2 = s->refcount_table_size *sizeof (uint64_t );
@@ -80,6 +143,59 @@ void qcow2_refcount_close(BlockDriverState *bs) } +static uint64_t get_refcount_ro0 (const void *refcount_array,uint64_t index) +{ +return (((const uint8_t *)refcount_array)[index /8 ] >> (index %8 )) &0x1 ; +} + +static void set_refcount_ro0 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>1 )); + ((uint8_t *)refcount_array)[index /8 ] &= ~(0x1 << (index %8 )); + ((uint8_t *)refcount_array)[index /8 ] |= value << (index %8 ); +} + +static uint64_t get_refcount_ro1 (const void *refcount_array,uint64_t index) +{ +return (((const uint8_t *)refcount_array)[index /4 ] >> (2 * (index %4 ))) + &0x3 ; +} + +static void set_refcount_ro1 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>2 )); + ((uint8_t *)refcount_array)[index /4 ] &= ~(0x3 << (2 * (index %4 ))); + ((uint8_t *)refcount_array)[index /4 ] |= value << (2 * (index %4 )); +} + +static uint64_t get_refcount_ro2 (const void *refcount_array,uint64_t index) +{ +return (((const uint8_t *)refcount_array)[index /2 ] >> (4 * (index %2 ))) + &0xf ; +} + +static void set_refcount_ro2 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>4 )); + ((uint8_t *)refcount_array)[index /2 ] &= ~(0xf << (4 * (index %2 ))); + ((uint8_t *)refcount_array)[index /2 ] |= value << (4 * (index %2 )); +} + +static uint64_t get_refcount_ro3 (const void *refcount_array,uint64_t index) +{ +return ((const uint8_t *)refcount_array)[index]; +} + +static void set_refcount_ro3 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>8 )); + ((uint8_t *)refcount_array)[index] = value; +} +static uint64_t get_refcount_ro4 (const void *refcount_array,uint64_t index) {return be16_to_cpu (((const uint16_t *)refcount_array)[index]);
@@ -92,6 +208,32 @@ static void set_refcount_ro4(void *refcount_array, uint64_t index, ((uint16_t *)refcount_array)[index] =cpu_to_be16 (value); } +static uint64_t get_refcount_ro5 (const void *refcount_array,uint64_t index) +{ +return be32_to_cpu (((const uint32_t *)refcount_array)[index]); +} + +static void set_refcount_ro5 (void *refcount_array,uint64_t index, +uint64_t value) +{ +assert (!(value >>32 )); + ((uint32_t *)refcount_array)[index] =cpu_to_be32 (value); +} + +static uint64_t get_refcount_ro6 (const void *refcount_array,uint64_t index) +{ +return be64_to_cpu (((const uint64_t *)refcount_array)[index]); +} + +static void set_refcount_ro6 (void *refcount_array,uint64_t index, +uint64_t value) +{ +/* for 64 bit refcounts, refcount_max is INT64_MAX to prevent signed +* overflows (and to allow for -errno style return values) */ +assert (!(value >>63 )); + ((uint64_t *)refcount_array)[index] =cpu_to_be64 (value); +} + static int load_refcount_block(BlockDriverState *bs, int64_t refcount_block_offset,
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v2 08/21] qcow2: More helpers for refcount modification
Quoted 11 lines · Max Reitz
Add helper functions for getting and setting refcounts in a refcount array for any possible refcount order, and choose the correct one during refcount initialization.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 146 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 144 insertions(+), 2 deletions(-)
block/qcow2-refcount.c static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs, · line 66
Quoted 9 lines
block/qcow2-refcount.c +9
-0+static void get_refcount_functions (int refcount_order, +Qcow2GetRefcountFunc **get, +Qcow2SetRefcountFunc **set) +{ +switch (refcount_order) { +case 0 : + *get = &get_refcount_ro0; + *set = &set_refcount_ro0; +break ;
Bike-shedding: instead of a switch statement and open-coded assignments, is it worth setting up an array of function pointers where you just grab the correct functions by doing array[refcount_order]? But I don't see any strong reason to change style; what you have works.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v2 08/21] qcow2: More helpers for refcount modification
Quoted 10 lines · Eric Blake
Quoted 9 lines · Max Reitz
Add helper functions for getting and setting refcounts in a refcount array for any possible refcount order, and choose the correct one during refcount initialization.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 146 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 144 insertions(+), 2 deletions(-)
Quoted 13 lines
block/qcow2-refcount.c static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs, · line 66
Quoted 9 lines
block/qcow2-refcount.c +9
-0+static void get_refcount_functions (int refcount_order, +Qcow2GetRefcountFunc **get, +Qcow2SetRefcountFunc **set) +{ +switch (refcount_order) { +case 0 : + *get = &get_refcount_ro0; + *set = &set_refcount_ro0; +break ;Bike-shedding: instead of a switch statement and open-coded assignments, is it worth setting up an array of function pointers where you just grab the correct functions by doing array[refcount_order]? But I don't see any strong reason to change style; what you have works.
I thought about it, but it wouldn't get much shorter. But maybe it looks nicer. I ought to think about it again.
Max
Quoted 1 line
Reviewed-by Eric Blake <[email protected]>
PATCH v2 9/21 qcow2: Open images with refcount order != 4
In reply to Max ReitzNo longer refuse to open images with a different refcount entry width than 16 bits; only reject images with a refcount width larger than 64 bits (which is prohibited by the specification).
--- block/qcow2.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
block/qcow2.c +4 -4
@@ -677,10 +677,10 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, }/* Check support for various header values */ -if (header.refcount_order !=4 ) { -report_unsupported (bs, errp,"%d bit reference counts" , -1 << header.refcount_order); - ret = -ENOTSUP ; +if (header.refcount_order >6 ) { +error_setg (errp,"Reference count entry width too large; may not " +"exceed 64 bit" ); + ret = -EINVAL ; goto fail; } s->refcount_order = header.refcount_order;
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v2 09/21] qcow2: Open images with refcount order != 4
Quoted 15 lines · Max Reitz
No longer refuse to open images with a different refcount entry width than 16 bits; only reject images with a refcount width larger than 64 bits (which is prohibited by the specification).
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)block/qcow2.c +0
-0
block/qcow2.c static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, · line 682
Quoted 10 lines
block/qcow2.c +3
-4@@ -677,10 +677,10 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, }/* Check support for various header values */ -if (header.refcount_order !=4 ) { -report_unsupported (bs, errp,"%d bit reference counts" , -1 << header.refcount_order); - ret = -ENOTSUP ; +if (header.refcount_order >6 ) { +error_setg (errp,"Reference count entry width too large; may not " +"exceed 64 bit" );
s/bit/bits/
Maintainer can make that tweak, so:
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v2 10/21 qcow2: refcount_order parameter for qcow2_create2
In reply to Max ReitzAdd a refcount_order parameter to qcow2_create2(), use that value for the image header and for calculating the size required for preallocation.
For now, always pass 4.
--- block/qcow2.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-)
block/qcow2.c +30 -11
@@ -1775,7 +1775,7 @@ static int preallocate(BlockDriverState *bs)static int qcow2_create2 (const char *filename,int64_t total_size,const char *backing_file,const char *backing_format,int flags,size_t cluster_size,PreallocMode prealloc, -QemuOpts *opts,int version, +QemuOpts *opts,int version,int refcount_order,Error **errp) {/* Calculate cluster_bits */
@@ -1811,6 +1811,13 @@ static int qcow2_create2(const char *filename, int64_t total_size,int64_t meta_size =0 ;uint64_t nreftablee, nrefblocke, nl1e, nl2e;int64_t aligned_total_size =align_offset (total_size, cluster_size); +int refblock_bits, refblock_size; +/* refcount entry size in bytes */ +double rces = (1 << refcount_order) /8. ; + +/* see qcow2_open() */ + refblock_bits = cluster_bits - (refcount_order -3 ); + refblock_size =1 << refblock_bits;/* header: 1 cluster */ meta_size += cluster_size;
@@ -1835,20 +1842,20 @@ static int qcow2_create2(const char *filename, int64_t total_size,* c = cluster size * y1 = number of refcount blocks entries * y2 = meta size including everything +* rces = refcount entry size in bytes * then, * y1 = (y2 + a)/c -* y2 = y1 * sizeof(u16) + y1 * sizeof(u16) * sizeof(u64) / c + m +* y2 = y1 * rces + y1 * rces * sizeof(u64) / c + m * we can get y1: -* y1 = (a + m) / (c - sizeof(u16) - sizeof(u16) * sizeof(u64) / c) +* y1 = (a + m) / (c - rces - rces * sizeof(u64) / c) */ - nrefblocke = (aligned_total_size + meta_size + cluster_size) / - (cluster_size -sizeof (uint16_t ) - -1.0 *sizeof (uint16_t ) *sizeof (uint64_t ) / cluster_size); - nrefblocke =align_offset (nrefblocke, cluster_size /sizeof (uint16_t )); - meta_size += nrefblocke *sizeof (uint16_t ); + nrefblocke = (aligned_total_size + meta_size + cluster_size) + / (cluster_size - rces - rces *sizeof (uint64_t ) + / cluster_size); + meta_size +=DIV_ROUND_UP (nrefblocke, refblock_size) * cluster_size;/* total size of refcount tables */ - nreftablee = nrefblocke *sizeof (uint16_t ) / cluster_size; + nreftablee = nrefblocke / refblock_size; nreftablee =align_offset (nreftablee, cluster_size /sizeof (uint64_t )); meta_size += nreftablee *sizeof (uint64_t );
@@ -1883,7 +1890,7 @@ static int qcow2_create2(const char *filename, int64_t total_size, .l1_size = cpu_to_be32(0), .refcount_table_offset = cpu_to_be64(cluster_size), .refcount_table_clusters = cpu_to_be32(1), - .refcount_order = cpu_to_be32(4), + .refcount_order = cpu_to_be32(refcount_order), .header_length = cpu_to_be32(sizeof(*header)), };
@@ -2003,6 +2010,7 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)size_t cluster_size =DEFAULT_CLUSTER_SIZE ;PreallocMode prealloc;int version =3 ; +int refcount_width =16 , refcount_order;Error *local_err =NULL ;int ret;
@@ -2057,8 +2065,19 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) goto finish; } +if (version <3 && refcount_width !=16 ) { +error_setg (errp,"Different refcount widths than 16 bits require " +"compatibility level 1.1 or above (use compat=1.1 or " +"greater)" ); + ret = -EINVAL ; + goto finish; + } + + refcount_order =ffs (refcount_width) -1 ; + ret =qcow2_create2 (filename, size, backing_file, backing_fmt, flags, - cluster_size, prealloc, opts, version, &local_err); + cluster_size, prealloc, opts, version, refcount_order, + &local_err);if (local_err) {error_propagate (errp, local_err); }
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v2 10/21] qcow2: refcount_order parameter for qcow2_create2
Quoted 13 lines · Max Reitz
Add a
refcount_orderparameter toqcow2_create2(), use that value for the image header and for calculating the size required for preallocation.For now, always pass 4.
Signed-off-by Max Reitz <[email protected]>--- block/qcow2.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-)
block/qcow2.c static int qcow2_create2(const char *filename, int64_t total_size, · line 1817
Quoted 7 lines
block/qcow2.c +4
-0@@ -1811,6 +1811,13 @@ static int qcow2_create2(const char *filename, int64_t total_size,int64_t meta_size =0 ;uint64_t nreftablee, nrefblocke, nl1e, nl2e;int64_t aligned_total_size =align_offset (total_size, cluster_size); +int refblock_bits, refblock_size; +/* refcount entry size in bytes */ +double rces = (1 << refcount_order) /8. ; +
Maybe worth a comment that absolute precision is not necessary, and that we are okay that the result gets us to within a fraction of a percent of the right value.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v2 11/21 iotests: Prepare for refcount_width option
In reply to Max ReitzSome tests do not work well with certain refcount widths (i.e. you cannot create internal snapshots with refcount_width=1), so make those widths unsupported.
Furthermore, add another filter to _filter_img_create in common.filter which filters out the refcount_width value.
This is necessary for test 079, which does actually work with any refcount width, but invoking qemu-img directly leads to the refcount_width value being visible in the output; use _make_test_img instead which will filter it out.
--- tests/qemu-iotests/007 | 3 +++ tests/qemu-iotests/015 | 2 ++ tests/qemu-iotests/026 | 7 +++++++ tests/qemu-iotests/029 | 1 + tests/qemu-iotests/051 | 3 +++ tests/qemu-iotests/058 | 2 ++ tests/qemu-iotests/067 | 2 ++ tests/qemu-iotests/079 | 10 ++-------- tests/qemu-iotests/079.out | 38 ++++++++++---------------------------- tests/qemu-iotests/080 | 2 ++ tests/qemu-iotests/089 | 2 ++ tests/qemu-iotests/108 | 2 ++ tests/qemu-iotests/common.filter | 3 ++- 13 files changed, 40 insertions(+), 37 deletions(-)
tests/qemu-iotests/007 +3 -0
@@ -43,6 +43,9 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _supported_os Linux +# refcount_width must be at least 4 bits so we can create ten internal snapshots +# (1 bit supports none, 2 bits support two, 4 bits support 14) +_unsupported_imgopts 'refcount_width=\(1\|2\)[^0-9]' echo echo "creating image"
tests/qemu-iotests/015 +2 -0
@@ -43,6 +43,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _supported_os Linux +# Internal snapshots are (currently) impossible with refcount_width=1 +_unsupported_imgopts 'refcount_width=1[^0-9]' echo echo "creating image"
tests/qemu-iotests/026 +7 -0
@@ -46,6 +46,13 @@ _supported_proto file _supported_os Linux _default_cache_mode "writethrough" _supported_cache_modes "writethrough" "none" +# The refcount table tests expect a certain minimum width for refcount entries +# (so that the refcount table actually needs to grow); that minimum is 16 bits, +# being the default refcount entry width. +# 32 and 64 bits do not work either, however, due to different leaked cluster +# count on error. +# Thus, the only remaining option is refcount_width=16. +_unsupported_imgopts 'refcount_width=\([^1]\|.\([^6]\|$\)\)' echo "Errors while writing 128 kB" echo
tests/qemu-iotests/029 +1 -0
@@ -44,6 +44,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _supported_os Linux +_unsupported_imgopts 'refcount_width=1[^0-9]' offset_size=24 offset_l1_size=36
tests/qemu-iotests/051 +3 -0
@@ -41,6 +41,9 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +# A compat=0.10 image is created in this test which does not support anything +# other than refcount_width=16 +_unsupported_imgopts 'refcount_width=\([^1]\|.\([^6]\|$\)\)' function do_run_qemu() {
tests/qemu-iotests/058 +2 -0
@@ -88,6 +88,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _require_command QEMU_NBD +# Internal snapshots are (currently) impossible with refcount_width=1 +_unsupported_imgopts 'refcount_width=1[^0-9]' echo echo "== preparing image =="
tests/qemu-iotests/067 +2 -0
@@ -35,6 +35,8 @@ status=1 # failure is the default! _supported_fmt qcow2 _supported_proto file _supported_os Linux +# Because anything other than 16 would change the output of query-block +_unsupported_imgopts 'refcount_width=\([^1]\|.\([^6]\|$\)\)' function do_run_qemu() {
tests/qemu-iotests/079 +2 -8
@@ -42,19 +42,13 @@ _supported_fmt qcow2 _supported_proto file nfs _supported_os Linux -function test_qemu_img() -{ - echo qemu-img "$@" | _filter_testdir - $QEMU_IMG "$@" 2>&1 | _filter_testdir - echo -} - echo "=== Check option preallocation and cluster_size ===" echo cluster_sizes="16384 32768 65536 131072 262144 524288 1048576 2097152 4194304" for s in $cluster_sizes; do - test_qemu_img create -f $IMGFMT -o preallocation=metadata,cluster_size=$s "$TEST_IMG" 4G + IMGOPTS=$(_optstr_add "$IMGOPTS" "preallocation=metadata,cluster_size=$s") \ + _make_test_img 4G done # success, all done
tests/qemu-iotests/079.out +10 -28
@@ -1,32 +1,14 @@ QA output created by 079 === Check option preallocation and cluster_size === -qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=16384 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=16384 preallocation='metadata' lazy_refcounts=off - -qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=32768 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=32768 preallocation='metadata' lazy_refcounts=off - -qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=65536 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 preallocation='metadata' lazy_refcounts=off - -qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=131072 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=131072 preallocation='metadata' lazy_refcounts=off - -qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=262144 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=262144 preallocation='metadata' lazy_refcounts=off - -qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=524288 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=524288 preallocation='metadata' lazy_refcounts=off - -qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=1048576 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=1048576 preallocation='metadata' lazy_refcounts=off - -qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=2097152 TEST_DIR/t.qcow2 4G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=2097152 preallocation='metadata' lazy_refcounts=off - -qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=4194304 TEST_DIR/t.qcow2 4G -qemu-img: TEST_DIR/t.qcow2: Cluster size must be a power of two between 512 and 2048k -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=4194304 preallocation='metadata' lazy_refcounts=off - +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata' +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata' +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata' +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata' +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata' +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata' +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata' +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata' +qemu-img: TEST_DIR/t.IMGFMT: Cluster size must be a power of two between 512 and 2048k +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata' *** done
tests/qemu-iotests/080 +2 -0
@@ -42,6 +42,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +# Internal snapshots are (currently) impossible with refcount_width=1 +_unsupported_imgopts 'refcount_width=1[^0-9]' header_size=104
tests/qemu-iotests/089 +2 -0
@@ -41,6 +41,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +# Because anything other than 16 would change the output of qemu_io -c info +_unsupported_imgopts 'refcount_width=\([^1]\|.\([^6]\|$\)\)' # Using an image filename containing quotation marks will render the JSON data # below invalid. In that case, we have little choice but simply not to run this
tests/qemu-iotests/108 +2 -0
@@ -43,6 +43,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file _supported_os Linux +# This test directly modifies a refblock so it relies on refcount_width being 16 +_unsupported_imgopts 'refcount_width=\([^1]\|.\([^6]\|$\)\)' echo echo '=== Repairing an image without any refcount table ==='
tests/qemu-iotests/common.filter +2 -1
@@ -190,7 +190,8 @@ _filter_img_create() -e "s# block_size=[0-9]\\+##g" \ -e "s# block_state_zero=\\(on\\|off\\)##g" \ -e "s# log_size=[0-9]\\+##g" \ - -e "s/archipelago:a/TEST_DIR\//g" + -e "s/archipelago:a/TEST_DIR\//g" \ + -e "s# refcount_width=[0-9]\\+##g" } _filter_img_info()
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v2 11/21] iotests: Prepare for refcount_width option
Quoted 17 lines · Max Reitz
Some tests do not work well with certain refcount widths (i.e. you cannot create internal snapshots with refcount_width=1), so make those widths unsupported.
Furthermore, add another filter to
_filter_img_createin common.filter which filters out therefcount_widthvalue.This is necessary for test 079, which does actually work with any refcount width, but invoking qemu-img directly leads to the
refcount_widthvalue being visible in the output; use_make_test_imginstead which will filter it out.Signed-off-by Max Reitz <[email protected]>---
+++ b/tests/qemu-iotests/029
tests/qemu-iotests/015 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 47
Quoted 4 lines
tests/qemu-iotests/015 +1
-0@@ -44,6 +44,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _supported_os Linux +_unsupported_imgopts 'refcount_width=1[^0-9]'
Missed a comment here.
tests/qemu-iotests/common.filter _filter_img_create() · line 194
Quoted 8 lines
tests/qemu-iotests/common.filter +3
-1+++ b/tests/qemu-iotests/common.filter@@ -190,7 +190,8 @@ _filter_img_create() -e "s# block_size=[0-9]\\+##g" \ -e "s# block_state_zero=\\(on\\|off\\)##g" \ -e "s# log_size=[0-9]\\+##g" \ - -e "s/archipelago:a/TEST_DIR\//g" + -e "s/archipelago:a/TEST_DIR\//g" \ + -e "s# refcount_width=[0-9]\\+##g"
I'm not convinced that \+ is portable sed; but as you are not the first use, it's not worth changing.
Once you add the remaining comment,
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v2 12/21 qcow2: Allow creation with refcount order != 4
In reply to Max ReitzAdd a creation option to qcow2 for setting the refcount order of images to be created, and respect that option's value.
This breaks some test outputs, fix them.
--- block/qcow2.c | 20 ++++++++ include/block/block_int.h | 1 + tests/qemu-iotests/049.out | 112 ++++++++++++++++++++++----------------------- tests/qemu-iotests/082.out | 41 ++++++++++++++--- tests/qemu-iotests/085.out | 38 +++++++-------- 5 files changed, 130 insertions(+), 82 deletions(-)
block/qcow2.c +20 -0
@@ -2065,6 +2065,17 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp) goto finish; } + refcount_width =qemu_opt_get_number_del (opts,BLOCK_OPT_REFCOUNT_WIDTH , + refcount_width); +if (refcount_width <=0 || refcount_width >64 || + !is_power_of_2 (refcount_width)) + { +error_setg (errp,"Refcount width must be a power of two and may not " +"exceed 64 bits" ); + ret = -EINVAL ; + goto finish; + } +if (version <3 && refcount_width !=16 ) {error_setg (errp,"Different refcount widths than 16 bits require " "compatibility level 1.1 or above (use compat=1.1 or "
@@ -2704,6 +2715,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, }else if (!strcmp (desc->name,"lazy_refcounts" )) { lazy_refcounts =qemu_opt_get_bool (opts,"lazy_refcounts" , lazy_refcounts); + }else if (!strcmp (desc->name,"refcount_width" )) { +error_report ("Cannot change refcount entry width" ); +return -ENOTSUP ; }else {/* if this assertion fails, this probably means a new option was * added without having it covered here */
@@ -2873,6 +2887,12 @@ static QemuOptsList qcow2_create_opts = { .help = "Postpone refcount updates", .def_value_str = "off" }, + { + .name = BLOCK_OPT_REFCOUNT_WIDTH, + .type = QEMU_OPT_NUMBER, + .help = "Width of a reference count entry in bits", + .def_value_str = "16" + }, {/* end of list */ } } };
include/block/block_int.h +1 -0
@@ -56,6 +56,7 @@#define BLOCK_OPT_ADAPTER_TYPE "adapter_type"#define BLOCK_OPT_REDUNDANCY "redundancy"#define BLOCK_OPT_NOCOW "nocow" +#define BLOCK_OPT_REFCOUNT_WIDTH "refcount_width" typedef struct BdrvTrackedRequest { BlockDriverState *bs;
tests/qemu-iotests/049.out +56 -56
254 lines not on this page · raw message
tests/qemu-iotests/082.out +34 -7
238 lines not on this page · raw message
tests/qemu-iotests/085.out +19 -19
@@ -11,7 +11,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 === Create a single snapshot on virtio0 === -Formatting 'TEST_DIR/1-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2.orig' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/1-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2.orig' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} === Invalid command - missing device and nodename ===
@@ -25,31 +25,31 @@ Formatting 'TEST_DIR/1-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file === Create several transactional group snapshots === -Formatting 'TEST_DIR/2-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/1-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/2-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/2-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/1-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/2-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/3-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/3-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/3-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/3-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/4-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/4-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/4-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/4-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/5-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/5-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/5-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/5-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/6-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/6-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/6-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/6-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/7-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/7-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/7-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/7-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/8-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/8-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/8-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/8-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/9-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/9-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/9-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/9-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} -Formatting 'TEST_DIR/10-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off -Formatting 'TEST_DIR/10-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off +Formatting 'TEST_DIR/10-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 +Formatting 'TEST_DIR/10-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_width=16 {"return": {}} *** done
Signature · 1 line
1.9.3
PATCH v2 13/21 block: Add opaque value to the amend CB
In reply to Max ReitzAdd an opaque value which is to be passed to the bdrv_amend_options() status callback.
--- block.c | 4 ++-- block/qcow2-cluster.c | 14 ++++++++------ block/qcow2.c | 9 +++++---- block/qcow2.h | 3 ++- include/block/block.h | 4 ++-- include/block/block_int.h | 3 ++- qemu-img.c | 5 +++-- 7 files changed, 24 insertions(+), 18 deletions(-)
block.c +2 -2
@@ -5790,12 +5790,12 @@ void bdrv_add_before_write_notifier(BlockDriverState *bs, }int bdrv_amend_options (BlockDriverState *bs,QemuOpts *opts, -BlockDriverAmendStatusCB *status_cb) +BlockDriverAmendStatusCB *status_cb,void *cb_opaque) {if (!bs->drv->bdrv_amend_options) {return -ENOTSUP ; } -return bs->drv->bdrv_amend_options (bs, opts, status_cb); +return bs->drv->bdrv_amend_options (bs, opts, status_cb, cb_opaque); }/* This function will be called by the bdrv_recurse_is_first_non_filter method
block/qcow2-cluster.c +8 -6
@@ -1620,7 +1620,8 @@ fail:static int expand_zero_clusters_in_l1 (BlockDriverState *bs,uint64_t *l1_table,int l1_size,int64_t *visited_l1_entries,int64_t l1_entries, -BlockDriverAmendStatusCB *status_cb) +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque) {BDRVQcowState *s = bs->opaque;bool is_active_l1 = (l1_table == s->l1_table);
@@ -1646,7 +1647,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,/* unallocated */ (*visited_l1_entries)++;if (status_cb) { -status_cb (bs, *visited_l1_entries, l1_entries); +status_cb (bs, *visited_l1_entries, l1_entries, cb_opaque); }continue ; }
@@ -1768,7 +1769,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, (*visited_l1_entries)++;if (status_cb) { -status_cb (bs, *visited_l1_entries, l1_entries); +status_cb (bs, *visited_l1_entries, l1_entries, cb_opaque); } }
@@ -1797,7 +1798,8 @@ fail:* qcow2 version which doesn't yet support metadata zero clusters. */ int qcow2_expand_zero_clusters (BlockDriverState *bs, -BlockDriverAmendStatusCB *status_cb) +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque) {BDRVQcowState *s = bs->opaque;uint64_t *l1_table =NULL ;
@@ -1814,7 +1816,7 @@ int qcow2_expand_zero_clusters(BlockDriverState *bs, ret =expand_zero_clusters_in_l1 (bs, s->l1_table, s->l1_size, &visited_l1_entries, l1_entries, - status_cb); + status_cb, cb_opaque);if (ret <0 ) { goto fail; }
@@ -1849,7 +1851,7 @@ int qcow2_expand_zero_clusters(BlockDriverState *bs, ret =expand_zero_clusters_in_l1 (bs, l1_table, s->snapshots[i].l1_size, &visited_l1_entries, l1_entries, - status_cb); + status_cb, cb_opaque);if (ret <0 ) { goto fail; }
block/qcow2.c +5 -4
@@ -2592,7 +2592,7 @@ static int qcow2_load_vmstate(BlockDriverState *bs, uint8_t *buf,* have to be removed. */ static int qcow2_downgrade (BlockDriverState *bs,int target_version, -BlockDriverAmendStatusCB *status_cb) +BlockDriverAmendStatusCB *status_cb,void *cb_opaque) {BDRVQcowState *s = bs->opaque;int current_version = s->qcow_version;
@@ -2641,7 +2641,7 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,/* clearing autoclear features is trivial */ s->autoclear_features =0 ; - ret =qcow2_expand_zero_clusters (bs, status_cb); + ret =qcow2_expand_zero_clusters (bs, status_cb, cb_opaque);if (ret <0 ) {return ret; }
@@ -2656,7 +2656,8 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version, }static int qcow2_amend_options (BlockDriverState *bs,QemuOpts *opts, -BlockDriverAmendStatusCB *status_cb) +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque) {BDRVQcowState *s = bs->opaque;int old_version = s->qcow_version, new_version = old_version;
@@ -2737,7 +2738,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,return ret; } }else { - ret =qcow2_downgrade (bs, new_version, status_cb); + ret =qcow2_downgrade (bs, new_version, status_cb, cb_opaque);if (ret <0 ) {return ret; }
block/qcow2.h +2 -1
@@ -551,7 +551,8 @@ int qcow2_discard_clusters(BlockDriverState *bs, uint64_t offset,int qcow2_zero_clusters (BlockDriverState *bs,uint64_t offset,int nb_sectors);int qcow2_expand_zero_clusters (BlockDriverState *bs, -BlockDriverAmendStatusCB *status_cb); +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque);/* qcow2-snapshot.c functions */ int qcow2_snapshot_create (BlockDriverState *bs,QEMUSnapshotInfo *sn_info);
include/block/block.h +2 -2
@@ -272,9 +272,9 @@ int bdrv_check(BlockDriverState *bs, BdrvCheckResult *res, BdrvCheckMode fix);* block driver; total_work_size may change during the course of the amendment * operation */ typedef void BlockDriverAmendStatusCB (BlockDriverState *bs,int64_t offset, -int64_t total_work_size); +int64_t total_work_size,void *opaque);int bdrv_amend_options (BlockDriverState *bs_new,QemuOpts *opts, -BlockDriverAmendStatusCB *status_cb); +BlockDriverAmendStatusCB *status_cb,void *cb_opaque);/* external snapshots */ bool bdrv_recurse_is_first_non_filter(BlockDriverState *bs,
include/block/block_int.h +2 -1
@@ -234,7 +234,8 @@ struct BlockDriver { BdrvCheckMode fix);int (*bdrv_amend_options)(BlockDriverState *bs,QemuOpts *opts, -BlockDriverAmendStatusCB *status_cb); +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque);void (*bdrv_debug_event)(BlockDriverState *bs,BlkDebugEvent event);
qemu-img.c +3 -2
@@ -2869,7 +2869,8 @@ out: }static void amend_status_cb (BlockDriverState *bs, -int64_t offset,int64_t total_work_size) +int64_t offset,int64_t total_work_size, +void *opaque) {qemu_progress_print (100.f * offset / total_work_size,0 ); }
@@ -2982,7 +2983,7 @@ static int img_amend(int argc, char **argv)/* In case the driver does not call amend_status_cb() */ qemu_progress_print (0.f ,0 ); - ret =bdrv_amend_options (bs, opts, &amend_status_cb); + ret =bdrv_amend_options (bs, opts, &amend_status_cb,NULL );qemu_progress_print (100.f ,0 );if (ret <0 ) {error_report ("Error while amending options: %s" ,strerror (-ret));
Signature · 1 line
1.9.3
PATCH v2 14/21 qcow2: Use error_report() in qcow2_amend_options()
In reply to Max Reitz--- block/qcow2.c | 14 ++++++-------- tests/qemu-iotests/061.out | 14 +++++++------- 2 files changed, 13 insertions(+), 15 deletions(-)
block/qcow2.c +6 -8
@@ -2686,11 +2686,11 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, }else if (!strcmp (compat,"1.1" )) { new_version =3 ; }else { -fprintf (stderr,"Unknown compatibility level %s.\n" , compat); +error_report ("Unknown compatibility level %s" , compat);return -EINVAL ; } }else if (!strcmp (desc->name,"preallocation" )) { -fprintf (stderr,"Cannot change preallocation mode.\n" ); +error_report ("Cannot change preallocation mode" );return -ENOTSUP ; }else if (!strcmp (desc->name,"size" )) { new_size =qemu_opt_get_size (opts,"size" ,0 );
@@ -2701,16 +2701,14 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, }else if (!strcmp (desc->name,"encryption" )) { encrypt =qemu_opt_get_bool (opts,"encryption" , s->crypt_method);if (encrypt != !!s->crypt_method) { -fprintf (stderr,"Changing the encryption flag is not " -"supported.\n" ); +error_report ("Changing the encryption flag is not supported" );return -ENOTSUP ; } }else if (!strcmp (desc->name,"cluster_size" )) { cluster_size =qemu_opt_get_size (opts,"cluster_size" , cluster_size);if (cluster_size != s->cluster_size) { -fprintf (stderr,"Changing the cluster size is not " -"supported.\n" ); +error_report ("Changing the cluster size is not supported" );return -ENOTSUP ; } }else if (!strcmp (desc->name,"lazy_refcounts" )) {
@@ -2756,8 +2754,8 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,if (s->use_lazy_refcounts != lazy_refcounts) {if (lazy_refcounts) {if (s->qcow_version <3 ) { -fprintf (stderr,"Lazy refcounts only supported with compatibility " -"level 1.1 and above (use compat=1.1 or greater)\n" ); +error_report ("Lazy refcounts only supported with compatibility " +"level 1.1 and above (use compat=1.1 or greater)" );return -EINVAL ; } s->compatible_features |=QCOW2_COMPAT_LAZY_REFCOUNTS ;
tests/qemu-iotests/061.out +7 -7
@@ -281,19 +281,19 @@ No errors were found on the image. === Testing invalid configurations === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) +qemu-img: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) qemu-img: Error while amending options: Invalid argument -Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) +qemu-img: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) qemu-img: Error while amending options: Invalid argument -Unknown compatibility level 0.42. +qemu-img: Unknown compatibility level 0.42 qemu-img: Error while amending options: Invalid argument qemu-img: Invalid parameter 'foo' qemu-img: Invalid options for file format 'qcow2' -Changing the cluster size is not supported. +qemu-img: Changing the cluster size is not supported qemu-img: Error while amending options: Operation not supported -Changing the encryption flag is not supported. +qemu-img: Changing the encryption flag is not supported qemu-img: Error while amending options: Operation not supported -Cannot change preallocation mode. +qemu-img: Cannot change preallocation mode qemu-img: Error while amending options: Operation not supported === Testing correct handling of unset value ===
@@ -301,7 +301,7 @@ qemu-img: Error while amending options: Operation not supported Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 Should work: Should not work: -Changing the cluster size is not supported. +qemu-img: Changing the cluster size is not supported qemu-img: Error while amending options: Operation not supported === Testing zero expansion on inactive clusters ===
Signature · 1 line
1.9.3
PATCH v2 15/21 qcow2: Use abort() instead of assert(false)
In reply to Max Reitz--- block/qcow2.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block/qcow2.c +2 -2
@@ -2718,9 +2718,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,error_report ("Cannot change refcount entry width" );return -ENOTSUP ; }else { -/* if this assertion fails, this probably means a new option was +/* if this point is reached, this probably means a new option was * added without having it covered here */ -assert (false); +abort (); } desc++;
Signature · 1 line
1.9.3
PATCH v2 16/21 qcow2: Split upgrade/downgrade paths for amend
In reply to Max ReitzIf the image version should be upgraded, that is the first we should do; if it should be downgraded, that is the last we should do. So split the version change block into an upgrade part at the start and a downgrade part at the end.
--- block/qcow2.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-)
block/qcow2.c +16 -15
@@ -2726,20 +2726,13 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, desc++; } -if (new_version != old_version) { -if (new_version > old_version) { -/* Upgrade */ - s->qcow_version = new_version; - ret =qcow2_update_header (bs); -if (ret <0 ) { - s->qcow_version = old_version; -return ret; - } - }else { - ret =qcow2_downgrade (bs, new_version, status_cb, cb_opaque); -if (ret <0 ) { -return ret; - } +/* Upgrade first (some features may require compat=1.1) */ +if (new_version > old_version) { + s->qcow_version = new_version; + ret =qcow2_update_header (bs); +if (ret <0 ) { + s->qcow_version = old_version; +return ret; } }
@@ -2753,7 +2746,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,if (s->use_lazy_refcounts != lazy_refcounts) {if (lazy_refcounts) { -if (s->qcow_version <3 ) { +if (new_version <3 ) {error_report ("Lazy refcounts only supported with compatibility " "level 1.1 and above (use compat=1.1 or greater)" );return -EINVAL ;
@@ -2789,6 +2782,14 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, } } +/* Downgrade last (so unsupported features can be removed before) */ +if (new_version < old_version) { + ret =qcow2_downgrade (bs, new_version, status_cb, cb_opaque); +if (ret <0 ) { +return ret; + } + } +return 0 ; }
Signature · 1 line
1.9.3
PATCH v2 17/21 qcow2: Use intermediate helper CB for amend
In reply to Max ReitzIf there is more than one time-consuming operation to be performed for qcow2_amend_options(), we need an intermediate CB which coordinates the progress of the individual operations and passes the result to the original status callback.
--- block/qcow2.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-)
block/qcow2.c +79 -1
@@ -2655,6 +2655,75 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,return 0 ; } +typedef enum Qcow2AmendOperation { +/* This is the value Qcow2AmendHelperCBInfo::last_operation will be +* statically initialized to so that the helper CB can discern the first +* invocation from an operation change */ +QCOW2_NO_OPERATION =0 , + +QCOW2_DOWNGRADING , +}Qcow2AmendOperation ; + +typedef struct Qcow2AmendHelperCBInfo { +/* The code coordinating the amend operations should only modify +* these four fields; the rest will be managed by the CB */ +BlockDriverAmendStatusCB *original_status_cb; +void *original_cb_opaque; + +Qcow2AmendOperation current_operation; + +/* Total number of operations to perform (only set once) */ +int total_operations; + +/* The following fields are managed by the CB */ + +/* Number of operations completed */ +int operations_completed; + +/* Cumulative offset of all completed operations */ +int64_t offset_completed; + +Qcow2AmendOperation last_operation; +int64_t last_work_size; +}Qcow2AmendHelperCBInfo ; + +static void qcow2_amend_helper_cb (BlockDriverState *bs, +int64_t operation_offset, +int64_t operation_work_size,void *opaque) +{ +Qcow2AmendHelperCBInfo *info = opaque; +int64_t current_work_size; +int64_t projected_work_size; + +if (info->current_operation != info->last_operation) { +if (info->last_operation !=QCOW2_NO_OPERATION ) { + info->offset_completed += info->last_work_size; + info->operations_completed++; + } + + info->last_operation = info->current_operation; + } + +assert (info->total_operations >0 ); +assert (info->operations_completed < info->total_operations); + + info->last_work_size = operation_work_size; + + current_work_size = info->offset_completed + operation_work_size; + +/* current_work_size is the total work size for (operations_completed + 1) +* operations (which includes this one), so multiply it by the number of +* operations not covered and divide it by the number of operations +* covered to get a projection for the operations not covered */ + projected_work_size = current_work_size * (info->total_operations - + info->operations_completed -1 ) + / (info->operations_completed +1 ); + + info->original_status_cb (bs, info->offset_completed + operation_offset, + current_work_size + projected_work_size, + info->original_cb_opaque); +} +static int qcow2_amend_options (BlockDriverState *bs,QemuOpts *opts,BlockDriverAmendStatusCB *status_cb,void *cb_opaque)
@@ -2669,6 +2738,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,bool encrypt;int ret;QemuOptDesc *desc = opts->list->desc; +Qcow2AmendHelperCBInfo helper_cb_info;while (desc && desc->name) {if (!qemu_opt_find (opts, desc->name)) {
@@ -2726,6 +2796,12 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, desc++; } + helper_cb_info = (Qcow2AmendHelperCBInfo ){ + .original_status_cb = status_cb, + .original_cb_opaque = cb_opaque, + .total_operations = (new_version < old_version) + }; +/* Upgrade first (some features may require compat=1.1) */ if (new_version > old_version) { s->qcow_version = new_version;
@@ -2784,7 +2860,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,/* Downgrade last (so unsupported features can be removed before) */ if (new_version < old_version) { - ret =qcow2_downgrade (bs, new_version, status_cb, cb_opaque); + helper_cb_info.current_operation =QCOW2_DOWNGRADING ; + ret =qcow2_downgrade (bs, new_version, &qcow2_amend_helper_cb, + &helper_cb_info);if (ret <0 ) {return ret; }
Signature · 1 line
1.9.3
PATCH v2 18/21 qcow2: Add function for refcount order amendment
In reply to Max ReitzAdd a function qcow2_change_refcount_order() which allows changing the refcount order of a qcow2 image.
--- block/qcow2-refcount.c | 457 +++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 + 2 files changed, 461 insertions(+)
block/qcow2-refcount.c +457 -0
@@ -2484,3 +2484,460 @@ int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset,return 0 ; } + +/* A pointer to a function of this type is given to walk_over_reftable(). That +* function will create refblocks and pass them to a RefblockFinishOp once they +* are completed (@refblock). @refblock_empty is set if the refblock is +* completely empty. +* +* Along with the refblock, a corresponding reftable entry is passed, in the +* reftable @reftable (which may be reallocated) at @reftable_index. +* +* @allocated should be set to true if a new cluster has been allocated. +*/ +typedef int (RefblockFinishOp )(BlockDriverState *bs,uint64_t **reftable, +uint64_t reftable_index,uint64_t *reftable_size, +void *refblock,bool refblock_empty, +bool *allocated,Error **errp); + +/** +* This "operation" for walk_over_reftable() allocates the refblock on disk (if +* it is not empty) and inserts its offset into the new reftable. The size of +* this new reftable is increased as required. +*/ +static int alloc_refblock (BlockDriverState *bs,uint64_t **reftable, +uint64_t reftable_index,uint64_t *reftable_size, +void *refblock,bool refblock_empty,bool *allocated, +Error **errp) +{ +BDRVQcowState *s = bs->opaque; +int64_t offset; + +if (!refblock_empty && reftable_index >= *reftable_size) { +uint64_t *new_reftable; +uint64_t new_reftable_size; + + new_reftable_size =ROUND_UP (reftable_index +1 , + s->cluster_size /sizeof (uint64_t )); +if (new_reftable_size >QCOW_MAX_REFTABLE_SIZE /sizeof (uint64_t )) { +error_setg (errp, +"This operation would make the refcount table grow " +"beyond the maximum size supported by QEMU, aborting" ); +return -ENOTSUP ; + } + + new_reftable =g_try_realloc (*reftable, new_reftable_size * +sizeof (uint64_t )); +if (!new_reftable) { +error_setg (errp,"Failed to increase reftable buffer size" ); +return -ENOMEM ; + } + +memset (new_reftable + *reftable_size,0 , + (new_reftable_size - *reftable_size) *sizeof (uint64_t )); + + *reftable = new_reftable; + *reftable_size = new_reftable_size; + } + +if (!refblock_empty && !(*reftable)[reftable_index]) { + offset =qcow2_alloc_clusters (bs, s->cluster_size); +if (offset <0 ) { +error_setg_errno (errp, -offset,"Failed to allocate refblock" ); +return offset; + } + (*reftable)[reftable_index] = offset; + *allocated = true; + } + +return 0 ; +} + +/** +* This "operation" for walk_over_reftable() writes the refblock to disk at the +* offset specified by the new reftable's entry. It does not modify the new +* reftable or change any refcounts. +*/ +static int flush_refblock (BlockDriverState *bs,uint64_t **reftable, +uint64_t reftable_index,uint64_t *reftable_size, +void *refblock,bool refblock_empty,bool *allocated, +Error **errp) +{ +BDRVQcowState *s = bs->opaque; +int64_t offset; +int ret; + +if (reftable_index < *reftable_size && (*reftable)[reftable_index]) { + offset = (*reftable)[reftable_index]; + + ret =qcow2_pre_write_overlap_check (bs,0 , offset, s->cluster_size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Overlap check failed" ); +return ret; + } + + ret =bdrv_pwrite (bs->file, offset, refblock, s->cluster_size); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to write refblock" ); +return ret; + } + }else { +assert (refblock_empty); + } + +return 0 ; +} + +/** +* This function walks over the existing reftable and every referenced refblock; +* if @new_set_refcount is non-NULL, it is called for every refcount entry to +* create an equal new entry in the passed @new_refblock. Once that +* @new_refblock is completely filled, @operation will be called. +* +* @status_cb and @cb_opaque are used for the amend operation's status callback. +* @index is the index of the walk_over_reftable() calls and @total is the total +* number of walk_over_reftable() calls per amend operation. Both are used for +* calculating the parameters for the status callback. +* +* @allocated is set to true if a new cluster has been allocated. +*/ +static int walk_over_reftable (BlockDriverState *bs,uint64_t **new_reftable, +uint64_t *new_reftable_index, +uint64_t *new_reftable_size, +void *new_refblock,int new_refblock_size, +int new_refcount_bits, +RefblockFinishOp *operation,bool *allocated, +Qcow2SetRefcountFunc *new_set_refcount, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,int index,int total, +Error **errp) +{ +BDRVQcowState *s = bs->opaque; +uint64_t reftable_index; +bool new_refblock_empty = true; +int refblock_index; +int new_refblock_index =0 ; +int ret; + +for (reftable_index =0 ; reftable_index < s->refcount_table_size; + reftable_index++) + { +uint64_t refblock_offset = s->refcount_table[reftable_index] + &REFT_OFFSET_MASK ; + +status_cb (bs, (uint64_t )index * s->refcount_table_size + reftable_index, + (uint64_t )total * s->refcount_table_size, cb_opaque); + +if (refblock_offset) { +void *refblock; + +if (offset_into_cluster (s, refblock_offset)) { +qcow2_signal_corruption (bs, true,-1 ,-1 ,"Refblock offset %#" + PRIx64" unaligned (reftable index: %#" + PRIx64")" , refblock_offset, + reftable_index); +error_setg (errp, +"Image is corrupt (unaligned refblock offset)" ); +return -EIO ; + } + + ret =qcow2_cache_get (bs, s->refcount_block_cache, refblock_offset, + &refblock); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to retrieve refblock" ); +return ret; + } + +for (refblock_index =0 ; refblock_index < s->refcount_block_size; + refblock_index++) + { +uint64_t refcount; + +if (new_refblock_index >= new_refblock_size) { +/* new_refblock is now complete */ + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, + new_refblock_empty, allocated, errp); +if (ret <0 ) { +qcow2_cache_put (bs, s->refcount_block_cache, &refblock); +return ret; + } + + (*new_reftable_index)++; + new_refblock_index =0 ; + new_refblock_empty = true; + } + + refcount = s->get_refcount (refblock, refblock_index); +if (new_refcount_bits <64 && refcount >> new_refcount_bits) { +uint64_t offset; + +qcow2_cache_put (bs, s->refcount_block_cache, &refblock); + + offset = ((reftable_index << s->refcount_block_bits) + + refblock_index) << s->cluster_bits; + +error_setg (errp,"Cannot decrease refcount entry width to " +"%i bits: Cluster at offset %#" PRIx64" has a " +"refcount of %" PRIu64, new_refcount_bits, + offset, refcount); +return -EINVAL ; + } + +if (new_set_refcount) { +new_set_refcount (new_refblock, new_refblock_index++, refcount); + }else { + new_refblock_index++; + } + new_refblock_empty = new_refblock_empty && refcount ==0 ; + } + + ret =qcow2_cache_put (bs, s->refcount_block_cache, &refblock); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to put refblock back into " +"the cache" ); +return ret; + } + }else { +/* No refblock means every refcount is 0 */ +for (refblock_index =0 ; refblock_index < s->refcount_block_size; + refblock_index++) + { +if (new_refblock_index >= new_refblock_size) { +/* new_refblock is now complete */ + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, + new_refblock_empty, allocated, errp); +if (ret <0 ) { +return ret; + } + + (*new_reftable_index)++; + new_refblock_index =0 ; + new_refblock_empty = true; + } + +if (new_set_refcount) { +new_set_refcount (new_refblock, new_refblock_index++,0 ); + }else { + new_refblock_index++; + } + } + } + } + +if (new_refblock_index >0 ) { +/* Complete the potentially existing partially filled final refblock */ +if (new_set_refcount) { +for (; new_refblock_index < new_refblock_size; + new_refblock_index++) + { +new_set_refcount (new_refblock, new_refblock_index,0 ); + } + } + + ret =operation (bs, new_reftable, *new_reftable_index, + new_reftable_size, new_refblock, new_refblock_empty, + allocated, errp); +if (ret <0 ) { +return ret; + } + + (*new_reftable_index)++; + } + +status_cb (bs, (uint64_t )(index +1 ) * s->refcount_table_size, + (uint64_t )total * s->refcount_table_size, cb_opaque); + +return 0 ; +} + +int qcow2_change_refcount_order (BlockDriverState *bs,int refcount_order, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,Error **errp) +{ +BDRVQcowState *s = bs->opaque; +Qcow2GetRefcountFunc *new_get_refcount; +Qcow2SetRefcountFunc *new_set_refcount; +void *new_refblock =qemu_blockalign (bs->file, s->cluster_size); +uint64_t *new_reftable =NULL , new_reftable_size =0 ; +uint64_t *old_reftable, old_reftable_size, old_reftable_offset; +uint64_t new_reftable_index =0 ; +uint64_t i; +int64_t new_reftable_offset =0 , allocated_reftable_size =0 ; +int new_refblock_size, new_refcount_bits =1 << refcount_order; +int old_refcount_order; +int walk_index =0 ; +int ret; +bool new_allocation; + +assert (s->qcow_version >=3 ); +assert (refcount_order >=0 && refcount_order <=6 ); + +/* see qcow2_open() */ + new_refblock_size =1 << (s->cluster_bits - (refcount_order -3 )); + +get_refcount_functions (refcount_order, + &new_get_refcount, &new_set_refcount); + + +do { +int total_walks; + + new_allocation = false; + +/* At least we have to do this walk and the one which writes the +* refblocks; also, at least we have to do this loop here at least +* twice (normally), first to do the allocations, and second to +* determine that everything is correctly allocated, this then makes +* three walks in total */ + total_walks =MIN (walk_index +2 ,3 ); + +/* First, allocate the structures so they are present in the refcount +* structures */ + ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size,NULL , new_refblock_size, + new_refcount_bits, &alloc_refblock, + &new_allocation,NULL , status_cb, cb_opaque, + walk_index++, total_walks, errp); +if (ret <0 ) { + goto done; + } + + new_reftable_index =0 ; + +if (new_allocation) { +if (new_reftable_offset) { +qcow2_free_clusters (bs, new_reftable_offset, + allocated_reftable_size *sizeof (uint64_t ), +QCOW2_DISCARD_NEVER ); + } + + new_reftable_offset =qcow2_alloc_clusters (bs, new_reftable_size * +sizeof (uint64_t )); +if (new_reftable_offset <0 ) { +error_setg_errno (errp, -new_reftable_offset, +"Failed to allocate the new reftable" ); + ret = new_reftable_offset; + goto done; + } + allocated_reftable_size = new_reftable_size; + + new_allocation = true; + } + }while (new_allocation); + +/* Second, write the new refblocks */ + new_allocation = false; + ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size, new_refblock, + new_refblock_size, new_refcount_bits, + &flush_refblock, &new_allocation, new_set_refcount, + status_cb, cb_opaque, walk_index, walk_index +1 , + errp); +if (ret <0 ) { + goto done; + } +assert (!new_allocation); + + +/* Write the new reftable */ + ret =qcow2_pre_write_overlap_check (bs,0 , new_reftable_offset, + new_reftable_size *sizeof (uint64_t )); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Overlap check failed" ); + goto done; + } + +for (i =0 ; i < new_reftable_size; i++) { +cpu_to_be64s (&new_reftable[i]); + } + + ret =bdrv_pwrite (bs->file, new_reftable_offset, new_reftable, + new_reftable_size *sizeof (uint64_t )); + +for (i =0 ; i < new_reftable_size; i++) { +be64_to_cpus (&new_reftable[i]); + } + +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to write the new reftable" ); + goto done; + } + + +/* Empty the refcount cache */ + ret =qcow2_cache_flush (bs, s->refcount_block_cache); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to flush the refblock cache" ); + goto done; + } + +/* Update the image header to point to the new reftable; this only updates +* the fields which are relevant to qcow2_update_header(); other fields +* such as s->refcount_table or s->refcount_bits stay stale for now +* (because we have to restore everything if qcow2_update_header() fails) */ + old_refcount_order = s->refcount_order; + old_reftable_size = s->refcount_table_size; + old_reftable_offset = s->refcount_table_offset; + + s->refcount_order = refcount_order; + s->refcount_table_size = new_reftable_size; + s->refcount_table_offset = new_reftable_offset; + + ret =qcow2_update_header (bs); +if (ret <0 ) { + s->refcount_order = old_refcount_order; + s->refcount_table_size = old_reftable_size; + s->refcount_table_offset = old_reftable_offset; +error_setg_errno (errp, -ret,"Failed to update the qcow2 header" ); + goto done; + } + +/* Now update the rest of the in-memory information */ + old_reftable = s->refcount_table; + s->refcount_table = new_reftable; + + s->refcount_bits =1 << refcount_order; +if (refcount_order <6 ) { + s->refcount_max = (UINT64_C (1 ) << s->refcount_bits) -1 ; + }else { + s->refcount_max =INT64_MAX ; + } + + s->refcount_block_bits = s->cluster_bits - (refcount_order -3 ); + s->refcount_block_size =1 << s->refcount_block_bits; + + s->get_refcount = new_get_refcount; + s->set_refcount = new_set_refcount; + +/* For cleaning up all old refblocks and the old reftable below the "done" +* label */ + new_reftable = old_reftable; + new_reftable_size = old_reftable_size; + new_reftable_offset = old_reftable_offset; + +done: +if (new_reftable) { +/* On success, new_reftable actually points to the old reftable (and +* new_reftable_size is the old reftable's size); but that is just +* fine */ +for (i =0 ; i < new_reftable_size; i++) { +uint64_t offset = new_reftable[i] &REFT_OFFSET_MASK ; +if (offset) { +qcow2_free_clusters (bs, offset, s->cluster_size, +QCOW2_DISCARD_NEVER ); + } + } +g_free (new_reftable); + +if (new_reftable_offset >0 ) { +qcow2_free_clusters (bs, new_reftable_offset, + new_reftable_size *sizeof (uint64_t ), +QCOW2_DISCARD_NEVER ); + } + } + +qemu_vfree (new_refblock); +return ret; +}
block/qcow2.h +4 -0
@@ -526,6 +526,10 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,int qcow2_pre_write_overlap_check (BlockDriverState *bs,int ign,int64_t offset,int64_t size); +int qcow2_change_refcount_order (BlockDriverState *bs,int refcount_order, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,Error **errp); +/* qcow2-cluster.c functions */ int qcow2_grow_l1_table (BlockDriverState *bs,uint64_t min_size,bool exact_size);
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v2 18/21] qcow2: Add function for refcount order amendment
Quoted 10 lines · Max Reitz
Add a function
qcow2_change_refcount_order()which allows changing the refcount order of a qcow2 image.Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 457 +++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 + 2 files changed, 461 insertions(+)
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2629
Quoted 5 lines
block/qcow2-refcount.c +4
-0+static int walk_over_reftable (BlockDriverState *bs,uint64_t **new_reftable, + + status_cb(bs, (uint64_t)index * s->refcount_table_size + reftable_index, + (uint64_t)total * s->refcount_table_size, cb_opaque);
Not sure if the casts are needed (isn't s->refcount_table_size already uint64_t, and 'int * uint64_t' does the right thing); but I guess it doesn't hurt to leave them.
Quoted 6 lines
block/qcow2-refcount.c +5
-0+int qcow2_change_refcount_order (BlockDriverState *bs,int refcount_order, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,Error **errp) +{ + do {
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2794
Quoted 10 lines
block/qcow2-refcount.c +10
-0+int total_walks; + + new_allocation = false; + +/* At least we have to do this walk and the one which writes the +* refblocks; also, at least we have to do this loop here at least +* twice (normally), first to do the allocations, and second to +* determine that everything is correctly allocated, this then makes +* three walks in total */ + total_walks =MIN (walk_index +2 ,3 );
This feels wrong...
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2802
Quoted 8 lines
block/qcow2-refcount.c +8
-0+ +/* First, allocate the structures so they are present in the refcount +* structures */ + ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size,NULL , new_refblock_size, + new_refcount_bits, &alloc_refblock, + &new_allocation,NULL , status_cb, cb_opaque, + walk_index++, total_walks, errp);
...In the common case of just two iterations of the do loop (second iteration confirms no allocations needed), you call with index 0/2, 1/3, and then the later non-allocation walk is index 2/3.
In the rare case of three iterations of the do loop, you call with index 0/2, 1/3, 2/3, and then the later non-allocation walk is 3/4.
I highly doubt that it is possible to trigger four iterations of the do loop, but if it were, you would call with 0/2, 1/3, 2/3, 3/3, and then 4/5.
I think you instead want to have:
total_walks = MAX(walk_index + 2, 3)
then the common case will call with 0/3, 1/3, and the later walk as 2/3
the three-iteration loop will call with 0/3, 1/3, 2/4, and the later walk as 3/4
the unlikely four-iteration loop will call with 0/3, 1/3, 2/4, 3/5, and the later walk as 4/5.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2813
Quoted 8 lines
block/qcow2-refcount.c +8
-0+ + new_reftable_index =0 ; + +if (new_allocation) { +if (new_reftable_offset) { +qcow2_free_clusters (bs, new_reftable_offset, + allocated_reftable_size *sizeof (uint64_t ), +QCOW2_DISCARD_NEVER );
Any reason you picked QCOW2_DISCARD_NEVER instead of some other policy?
Why not punch holes in the file when throwing out a failed too-small
new table, or when cleaning up the old table once the new table is good?
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2826
Quoted 13 lines
block/qcow2-refcount.c +13
-0+ } + + new_reftable_offset =qcow2_alloc_clusters (bs, new_reftable_size * +sizeof (uint64_t )); +if (new_reftable_offset <0 ) { +error_setg_errno (errp, -new_reftable_offset, +"Failed to allocate the new reftable" ); + ret = new_reftable_offset; + goto done; + } + allocated_reftable_size = new_reftable_size; + + new_allocation = true;
This assignment is dead code (it already occurs inside an 'if (new_allocation)' condition).
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2831
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ } + }while (new_allocation); + +/* Second, write the new refblocks */ + new_allocation = false;
This assignment is dead code (it can only be reached if the earlier do loop ended, which is only possible when no allocations are recorded).
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2842
Quoted 11 lines
block/qcow2-refcount.c +11
-0+ ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size, new_refblock, + new_refblock_size, new_refcount_bits, + &flush_refblock, &new_allocation, new_set_refcount, + status_cb, cb_opaque, walk_index, walk_index +1 , + errp); +if (ret <0 ) { + goto done; + } +assert (!new_allocation); +
Correct.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2929
Quoted 10 lines
block/qcow2-refcount.c +10
-0+done: +if (new_reftable) { +/* On success, new_reftable actually points to the old reftable (and +* new_reftable_size is the old reftable's size); but that is just +* fine */ +for (i =0 ; i < new_reftable_size; i++) { +uint64_t offset = new_reftable[i] &REFT_OFFSET_MASK ; +if (offset) { +qcow2_free_clusters (bs, offset, s->cluster_size, +QCOW2_DISCARD_NEVER );
Again, why the QCOW2_DISCARD_NEVER policy?
Fix the MIN vs. MAX bug, and the two dead assignment statements, and you can have:
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v2 18/21] qcow2: Add function for refcount order amendment
Quoted 10 lines · Eric Blake
Quoted 9 lines · Max Reitz
Add a function
qcow2_change_refcount_order()which allows changing the refcount order of a qcow2 image.Signed-off-by Max Reitz <[email protected]>--- block/qcow2-refcount.c | 457 +++++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 4 + 2 files changed, 461 insertions(+)
Quoted 6 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2629
Quoted 4 lines
block/qcow2-refcount.c +4
-0+static int walk_over_reftable (BlockDriverState *bs,uint64_t **new_reftable, + + status_cb(bs, (uint64_t)index * s->refcount_table_size + reftable_index, + (uint64_t)total * s->refcount_table_size, cb_opaque);Not sure if the casts are needed (isn't
s->refcount_table_sizealreadyuint64_t,
Surprise, it isn't. I thought otherwise, too, but then got told by clang_complete (it's uint32_t).
Quoted 19 lines
and 'int *
uint64_t' does the right thing); but I guess it doesn't hurt to leave them.Quoted 5 lines
block/qcow2-refcount.c +5
-0+int qcow2_change_refcount_order (BlockDriverState *bs,int refcount_order, +BlockDriverAmendStatusCB *status_cb, +void *cb_opaque,Error **errp) +{ + do {block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2794
Quoted 10 lines
block/qcow2-refcount.c +10
-0+int total_walks; + + new_allocation = false; + +/* At least we have to do this walk and the one which writes the +* refblocks; also, at least we have to do this loop here at least +* twice (normally), first to do the allocations, and second to +* determine that everything is correctly allocated, this then makes +* three walks in total */ + total_walks =MIN (walk_index +2 ,3 );This feels wrong...
Yes, I noticed already when preparing v3 and it's already fixed in my local v3 branch. *cough*
Quoted 30 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2802
Quoted 8 lines
block/qcow2-refcount.c +8
-0+ +/* First, allocate the structures so they are present in the refcount +* structures */ + ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size,NULL , new_refblock_size, + new_refcount_bits, &alloc_refblock, + &new_allocation,NULL , status_cb, cb_opaque, + walk_index++, total_walks, errp);...In the common case of just two iterations of the do loop (second iteration confirms no allocations needed), you call with index 0/2, 1/3, and then the later non-allocation walk is index 2/3.
In the rare case of three iterations of the do loop, you call with index 0/2, 1/3, 2/3, and then the later non-allocation walk is 3/4.
I highly doubt that it is possible to trigger four iterations of the do loop, but if it were, you would call with 0/2, 1/3, 2/3, 3/3, and then 4/5.
I think you instead want to have:
total_walks=MAX(walk_index + 2, 3)then the common case will call with 0/3, 1/3, and the later walk as 2/3
the three-iteration loop will call with 0/3, 1/3, 2/4, and the later walk as 3/4
the unlikely four-iteration loop will call with 0/3, 1/3, 2/4, 3/5, and the later walk as 4/5.
Quoted 9 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2813
Quoted 8 lines
block/qcow2-refcount.c +8
-0+ + new_reftable_index =0 ; + +if (new_allocation) { +if (new_reftable_offset) { +qcow2_free_clusters (bs, new_reftable_offset, + allocated_reftable_size *sizeof (uint64_t ), +QCOW2_DISCARD_NEVER );Any reason you picked
QCOW2_DISCARD_NEVERinstead of some other policy?
Ah, discarding is always interesting... Last year I used QCOW2_DISCARD_ALWAYS, then asked Kevin and he basically said never to use ALWAYS unless one is really sure about it. I could have used QCOW2_DISCARD_OTHER... But the idea behind using NEVER in cases like this is that the clusters may get picked up by the following allocation, in which case having discarded them is not a good idea (there are some other places in the qcow2 code which use NEVER for the same reason).
So, in this case, I think NEVER is good.
Quoted 18 lines
Why not punch holes in the file when throwing out a failed too-small new table, or when cleaning up the old table once the new table is good?
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2826
Quoted 13 lines
block/qcow2-refcount.c +13
-0+ } + + new_reftable_offset =qcow2_alloc_clusters (bs, new_reftable_size * +sizeof (uint64_t )); +if (new_reftable_offset <0 ) { +error_setg_errno (errp, -new_reftable_offset, +"Failed to allocate the new reftable" ); + ret = new_reftable_offset; + goto done; + } + allocated_reftable_size = new_reftable_size; + + new_allocation = true;This assignment is dead code (it already occurs inside an 'if (
new_allocation)' condition).
Right. Though I somehow like its explicitness... I'll remove it.
Quoted 7 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2831
Quoted 5 lines
block/qcow2-refcount.c +5
-0+ } + }while (new_allocation); + +/* Second, write the new refblocks */ + new_allocation = false;This assignment is dead code (it can only be reached if the earlier do loop ended, which is only possible when no allocations are recorded).
Right again.
Quoted 23 lines
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2842
Quoted 11 lines
block/qcow2-refcount.c +11
-0+ ret =walk_over_reftable (bs, &new_reftable, &new_reftable_index, + &new_reftable_size, new_refblock, + new_refblock_size, new_refcount_bits, + &flush_refblock, &new_allocation, new_set_refcount, + status_cb, cb_opaque, walk_index, walk_index +1 , + errp); +if (ret <0 ) { + goto done; + } +assert (!new_allocation); +Correct.
block/qcow2-refcount.c int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, · line 2929
Quoted 10 lines
block/qcow2-refcount.c +10
-0+done: +if (new_reftable) { +/* On success, new_reftable actually points to the old reftable (and +* new_reftable_size is the old reftable's size); but that is just +* fine */ +for (i =0 ; i < new_reftable_size; i++) { +uint64_t offset = new_reftable[i] &REFT_OFFSET_MASK ; +if (offset) { +qcow2_free_clusters (bs, offset, s->cluster_size, +QCOW2_DISCARD_NEVER );
Quoted 1 line
Again, why the
QCOW2_DISCARD_NEVERpolicy?
Here, I have nothing to justify it. I'll use QCOW2_DISCARD_OTHER in v3.
Quoted 4 lines
Fix the MIN vs. MAX bug, and the two dead assignment statements, and you can have:
Reviewed-by Eric Blake <[email protected]>
I'll also use QCOW2_DISCARD_OTHER for freeing the refblocks and the reftable after the "done" label, if you're fine with that.
Once again, thanks a lot!
Max
Re: [Qemu-devel] [PATCH v2 18/21] qcow2: Add function for refcount order amendment
Quoted 24 lines · Max Reitz
Quoted 7 lines · Eric Blake
Quoted 5 lines · Max Reitz
Add a function
qcow2_change_refcount_order()which allows changing the refcount order of a qcow2 image.Signed-off-by Max Reitz <[email protected]>---
Quoted 7 lines
Quoted 6 lines
block/qcow2-refcount.c +4
-0+if (new_allocation) { +if (new_reftable_offset) { + qcow2_free_clusters(bs, new_reftable_offset, + allocated_reftable_size *
sizeof(uint64_t),
+QCOW2_DISCARD_NEVER);Any reason you picked
QCOW2_DISCARD_NEVERinstead of some other policy?Ah, discarding is always interesting... Last year I used
QCOW2_DISCARD_ALWAYS, then asked Kevin and he basically said never to use ALWAYS unless one is really sure about it. I could have usedQCOW2_DISCARD_OTHER... But the idea behind using NEVER in cases like this is that the clusters may get picked up by the following allocation, in which case having discarded them is not a good idea (there are some other places in the qcow2 code which use NEVER for the same reason).
Quoted 1 line
So, in this case, I think NEVER is good.
Makes sense. Yes, for THIS case, we are probably going to reuse the just-discarded cluster on the very next walk, so it's not worth punching a hole just to reinstate it.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
block/qcow2-refcount.c +3
-0+done: +if (new_reftable) { +/* On success, new_reftable actually points to the old reftable (and
+ *new_reftable_sizeis the old reftable's size); but that
is justblock/qcow2-refcount.c +6
-0+* fine */ +for (i =0 ; i < new_reftable_size; i++) { +uint64_t offset = new_reftable[i] &REFT_OFFSET_MASK ; +if (offset) { +qcow2_free_clusters (bs, offset, s->cluster_size, +QCOW2_DISCARD_NEVER );
Quoted 3 lines
Quoted 1 line
Again, why the
QCOW2_DISCARD_NEVERpolicy?Here, I have nothing to justify it. I'll use
QCOW2_DISCARD_OTHERin v3.
Thanks, and now I know a bit more about discard policy.
Quoted 8 lines
Quoted 5 lines
Fix the MIN vs. MAX bug, and the two dead assignment statements, and you can have:
Reviewed-by Eric Blake <[email protected]>I'll also use
QCOW2_DISCARD_OTHERfor freeing the refblocks and the reftable after the "done" label, if you're fine with that.
Yes, works for me.
Quoted 2 lines
Once again, thanks a lot!
And thank you for a mentally engaging review :) I'm still in the middle of an email on a possible test you can write to provoke a different 3-pass scenario thanks to all-zero refblocks, so you may want to wait for that before posting v3...
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v2 19/21 qcow2: Invoke refcount order amendment function
In reply to Max ReitzMake use of qcow2_change_refcount_order() to support changing the refcount order with qemu-img amend.
--- block/qcow2.c | 44 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 9 deletions(-)
block/qcow2.c +35 -9
@@ -2607,13 +2607,7 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version, }if (s->refcount_order !=4 ) { -/* we would have to convert the image to a refcount_order == 4 image -* here; however, since qemu (at the time of writing this) does not -* support anything different than 4 anyway, there is no point in doing -* so right now; however, we should error out (if qemu supports this in -* the future and this code has not been adapted) */ -error_report ("qcow2_downgrade: Image refcount orders other than 4 are " -"currently not supported." ); +error_report ("compat=0.10 requires refcount_width=16" );return -ENOTSUP ; }
@@ -2661,6 +2655,7 @@ typedef enum Qcow2AmendOperation {* invocation from an operation change */ QCOW2_NO_OPERATION =0 , +QCOW2_CHANGING_REFCOUNT_ORDER ,QCOW2_DOWNGRADING , } Qcow2AmendOperation;
@@ -2736,6 +2731,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,const char *compat =NULL ;uint64_t cluster_size = s->cluster_size;bool encrypt; +int refcount_width = s->refcount_bits;int ret;QemuOptDesc *desc = opts->list->desc;Qcow2AmendHelperCBInfo helper_cb_info;
@@ -2785,8 +2781,16 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, lazy_refcounts =qemu_opt_get_bool (opts,"lazy_refcounts" , lazy_refcounts); }else if (!strcmp (desc->name,"refcount_width" )) { -error_report ("Cannot change refcount entry width" ); -return -ENOTSUP ; + refcount_width =qemu_opt_get_number (opts,"refcount_width" , + refcount_width); + +if (refcount_width <=0 || refcount_width >64 || + !is_power_of_2 (refcount_width)) + { +error_report ("Refcount width must be a power of two and may " +"not exceed 64 bits" ); +return -EINVAL ; + } }else {/* if this point is reached, this probably means a new option was * added without having it covered here */
@@ -2800,6 +2804,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, .original_status_cb = status_cb, .original_cb_opaque = cb_opaque, .total_operations = (new_version < old_version) + + (s->refcount_bits != refcount_width) };/* Upgrade first (some features may require compat=1.1) */
@@ -2812,6 +2817,27 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, } } +if (s->refcount_bits != refcount_width) { +int refcount_order =ffs (refcount_width) -1 ; +Error *local_error =NULL ; + +if (new_version <3 && refcount_width !=16 ) { +error_report ("Different refcount widths than 16 bits require " +"compatibility level 1.1 or above (use compat=1.1 or " +"greater)" ); +return -EINVAL ; + } + + helper_cb_info.current_operation =QCOW2_CHANGING_REFCOUNT_ORDER ; + ret =qcow2_change_refcount_order (bs, refcount_order, + &qcow2_amend_helper_cb, + &helper_cb_info, &local_error); +if (ret <0 ) { +qerror_report_err (local_error); +return ret; + } + } +if (backing_file || backing_format) { ret =qcow2_change_backing_file (bs, backing_file ?: bs->backing_file, backing_format ?: bs->backing_format);
Signature · 1 line
1.9.3
PATCH v2 20/21 qcow2: Point to amend function in check
In reply to Max ReitzIf a reference count is not representable with the current refcount order, the image check should point to qemu-img amend for increasing the refcount order. However, qemu-img amend needs write access to the image which cannot be provided if the image is marked corrupt; and the image check will not mark the image consistent unless everything actually is consistent.
Therefore, if an image is marked corrupt and the image check encounters a reference count overflow, it cannot be fixed by using qemu-img amend to increase the refcount order. Instead, one has to use qemu-img convert to create a completely new copy of the image in this case.
Alternatively, we may want to give the user a way of manually removing the corrupt flag, maybe through qemu-img amend, but this is not part of this patch.
--- block/qcow2-refcount.c | 3 +++ 1 file changed, 3 insertions(+)
block/qcow2-refcount.c +3 -0
@@ -1368,6 +1368,9 @@ static int inc_refcounts(BlockDriverState *bs,if (refcount == s->refcount_max) {fprintf (stderr,"ERROR: overflow cluster offset=0x%" PRIx64"\n" , cluster_offset); +fprintf (stderr,"Use qemu-img amend to increase the refcount entry " +"width or qemu-img convert to create a clean copy if the " +"image cannot be opened for writing\n" ); res->corruptions++;continue ; }
Signature · 1 line
1.9.3
PATCH v2 21/21 iotests: Add test for different refcount widths
In reply to Max ReitzAdd a test for conversion between different refcount widths and errors specific to certain widths (i.e. snapshots with refcount_width=1).
--- tests/qemu-iotests/112 | 252 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/112.out | 131 +++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 384 insertions(+) create mode 100755 tests/qemu-iotests/112 create mode 100644 tests/qemu-iotests/112.out
tests/qemu-iotests/112 new file +252 -0
@@ -0,0 +1,252 @@ +#!/bin/bash +# +# Test cases for different refcount_widths +# +# Copyright (C) 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq="$(basename $0)" +echo "QA output created by $seq" + +here="$PWD" +tmp=/tmp/$$ +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +# This tests qcow2-specific low-level functionality +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux +# This test will set refcount_width on its own which would conflict with the +# manual setting; compat will be overridden as well +_unsupported_imgopts refcount_width 'compat=0.10' + +function print_refcount_width() +{ + $QEMU_IMG info "$TEST_IMG" | sed -n '/refcount width:/ s/^ *//p' +} + +echo +echo '=== refcount_width limits ===' +echo + +# Must be positive (non-zero) +IMGOPTS="$IMGOPTS,refcount_width=0" _make_test_img 64M +# Must be positive (non-negative) +IMGOPTS="$IMGOPTS,refcount_width=-1" _make_test_img 64M +# May not exceed 64 +IMGOPTS="$IMGOPTS,refcount_width=128" _make_test_img 64M +# Must be a power of two +IMGOPTS="$IMGOPTS,refcount_width=42" _make_test_img 64M + +# 1 is the minimum +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# 64 is the maximum +IMGOPTS="$IMGOPTS,refcount_width=64" _make_test_img 64M +print_refcount_width + +# 16 is the default +_make_test_img 64M +print_refcount_width + +echo +echo '=== refcount_width and compat=0.10 ===' +echo + +# Should work +IMGOPTS="$IMGOPTS,compat=0.10,refcount_width=16" _make_test_img 64M +print_refcount_width + +# Should not work +IMGOPTS="$IMGOPTS,compat=0.10,refcount_width=1" _make_test_img 64M +IMGOPTS="$IMGOPTS,compat=0.10,refcount_width=64" _make_test_img 64M + + +echo +echo '=== Snapshot limit on refcount_width=1 ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io + +# Should fail for now; in the future, this might be supported by automatically +# copying all clusters with overflowing refcount +$QEMU_IMG snapshot -c foo "$TEST_IMG" + +# The new L1 table could/should be leaked +_check_test_img + +echo +echo '=== Snapshot limit on refcount_width=2 ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=2" _make_test_img 64M +print_refcount_width + +$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io + +# Should succeed +$QEMU_IMG snapshot -c foo "$TEST_IMG" +$QEMU_IMG snapshot -c bar "$TEST_IMG" +# Should fail (4th reference) +$QEMU_IMG snapshot -c baz "$TEST_IMG" + +# The new L1 table could/should be leaked +_check_test_img + +echo +echo '=== Compressed clusters with refcount_width=1 ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# Both should fit into a single host cluster; instead of failing to increase the +# refcount of that cluster, qemu should just allocate a new cluster and make +# this operation succeed +$QEMU_IO -c 'write -P 0 -c 0 64k' \ + -c 'write -P 1 -c 64k 64k' \ + "$TEST_IMG" | _filter_qemu_io + +_check_test_img + +echo +echo '=== Amend from refcount_width=16 to refcount_width=1 ===' +echo + +_make_test_img 64M +print_refcount_width + +$QEMU_IO -c 'write 16M 32M' "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG amend -o refcount_width=1 "$TEST_IMG" +_check_test_img +print_refcount_width + +echo +echo '=== Amend from refcount_width=1 to refcount_width=64 ===' +echo + +$QEMU_IMG amend -o refcount_width=64 "$TEST_IMG" +_check_test_img +print_refcount_width + +echo +echo '=== Amend to compat=0.10 ===' +echo + +# Should not work because refcount_width needs to be 16 for compat=0.10 +$QEMU_IMG amend -o compat=0.10 "$TEST_IMG" +print_refcount_width +# Should work +$QEMU_IMG amend -o compat=0.10,refcount_width=16 "$TEST_IMG" +_check_test_img +print_refcount_width + +# Get back to compat=1.1 and refcount_width=16 +$QEMU_IMG amend -o compat=1.1 "$TEST_IMG" +print_refcount_width +# Should not work +$QEMU_IMG amend -o refcount_width=32,compat=0.10 "$TEST_IMG" +print_refcount_width + +echo +echo '=== Amend with snapshot ===' +echo + +$QEMU_IMG snapshot -c foo "$TEST_IMG" +# Just to have different refcounts across the image +$QEMU_IO -c 'write 0 16M' "$TEST_IMG" | _filter_qemu_io + +# Should not work (may work in the future by first decreasing all refcounts so +# they fit into the target range by copying them) +$QEMU_IMG amend -o refcount_width=1 "$TEST_IMG" +_check_test_img +print_refcount_width + +# Should work +$QEMU_IMG amend -o refcount_width=2 "$TEST_IMG" +_check_test_img +print_refcount_width + +echo +echo '=== Testing too many references for check ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# This cluster should be created at 0x50000 +$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io +# Now make the second L2 entry (the L2 table should be at 0x40000) point to that +# cluster, so we have two references +poke_file "$TEST_IMG" $((0x40008)) "\x80\x00\x00\x00\x00\x05\x00\x00" + +# This should say "please use amend" +_check_test_img -r all + +# So we do that +$QEMU_IMG amend -o refcount_width=2 "$TEST_IMG" +print_refcount_width + +# And try again +_check_test_img -r all + +echo +echo '=== Multiple walks necessary during amend ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1,cluster_size=512" _make_test_img 64k + +# Cluster 0 is the image header, clusters 1 to 4 are used by the L1 table, a +# single L2 table, the reftable and a single refblock. This creates 58 data +# clusters (actually, the L2 table is created here, too), so in total there are +# then 63 used clusters in the image. With a refcount width of 64, one refblock +# describes 64 clusters (512 bytes / 64 bits/entry = 64 entries), so this will +# make the first target refblock have exactly one free entry. +$QEMU_IO -c "write 0 $((58 * 512))" "$TEST_IMG" | _filter_qemu_io + +# Now change the refcount width; since the first target refblock has exactly one +# free entry, that entry will be used to store its own reference. No other +# refblocks are needed, so then the new reftable will be allocated; since the +# first target refblock is completely filled up, this will require a new +# refblock which is why the refcount width changing function will need to run +# through everything one more time until the allocations are stable. +$QEMU_IMG amend -o refcount_width=64 "$TEST_IMG" +print_refcount_width + +_check_test_img + + +# success, all done +echo '*** done' +rm -f $seq.full +status=0
tests/qemu-iotests/112.out new file +131 -0
131 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -112,3 +112,4 @@ 107 rw auto quick 108 rw auto quick 111 rw auto quick +112 rw auto
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
Quoted 24 lines · Max Reitz
Add a test for conversion between different refcount widths and errors specific to certain widths (i.e. snapshots with refcount_width=1).
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/112 | 252 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/112.out | 131 +++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 384 insertions(+) create mode 100755 tests/qemu-iotests/112 create mode 100644 tests/qemu-iotests/112.outtests/qemu-iotests/112 new file +12
-0+echo +echo '=== Testing too many references for check ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# This cluster should be created at 0x50000 +$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io +# Now make the second L2 entry (the L2 table should be at 0x40000) point to that +# cluster, so we have two references +poke_file "$TEST_IMG" $((0x40008)) "\x80\x00\x00\x00\x00\x05\x00\x00"
tests/qemu-iotests/112 line 221
Quoted 10 lines
tests/qemu-iotests/112 new file +10
-0+ +# This should say "please use amend" +_check_test_img -r all + +# So we do that +$QEMU_IMG amend -o refcount_width=2 "$TEST_IMG" +print_refcount_width + +# And try again +_check_test_img -r all
I think this section also deserves a test that fuzzes an image with width=64 to intentionally set the most significant bit of one of the refcounts, and make sure that we gracefully diagnose it as invalid.
Quoted 13 lines
tests/qemu-iotests/112 new file +13
-0+ +echo +echo '=== Multiple walks necessary during amend ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1,cluster_size=512" _make_test_img 64k + +# Cluster 0 is the image header, clusters 1 to 4 are used by the L1 table, a +# single L2 table, the reftable and a single refblock. This creates 58 data +# clusters (actually, the L2 table is created here, too), so in total there are +# then 63 used clusters in the image. With a refcount width of 64, one refblock +# describes 64 clusters (512 bytes / 64 bits/entry = 64 entries), so this will +# make the first target refblock have exactly one free entry.
tests/qemu-iotests/112 line 244
Quoted 10 lines
tests/qemu-iotests/112 new file +10
-0+$QEMU_IO -c "write 0 $((58 * 512))" "$TEST_IMG" | _filter_qemu_io + +# Now change the refcount width; since the first target refblock has exactly one +# free entry, that entry will be used to store its own reference. No other +# refblocks are needed, so then the new reftable will be allocated; since the +# first target refblock is completely filled up, this will require a new +# refblock which is why the refcount width changing function will need to run +# through everything one more time until the allocations are stable. +$QEMU_IMG amend -o refcount_width=64 "$TEST_IMG" +print_refcount_width
Umm, that sounds backwards from what you document. It's a good test of the _new_ reftable needing a second round of allocations. So keep it with corrected comments. But I think you _intended_ to write a test that starts with a refcount_width=64 image and resize to a refcount_width=1, where the _old_ reftable then suffers a reallocation as part of allocating refblocks for the new table. It may even help if you add a tracepoint for every iteration through the walk function callback, to prove we are indeed executing it 3 times instead of the usual 2, for these test cases.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
Quoted 60 lines · Eric Blake
Quoted 24 lines · Max Reitz
Add a test for conversion between different refcount widths and errors specific to certain widths (i.e. snapshots with refcount_width=1).
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/112 | 252 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/112.out | 131 +++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 384 insertions(+)create mode 100755 tests/qemu-iotests/112
create mode 100644 tests/qemu-iotests/112.outtests/qemu-iotests/112 new file +12
-0+echo +echo '=== Testing too many references for check ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# This cluster should be created at 0x50000 +$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io +# Now make the second L2 entry (the L2 table should be at 0x40000) point to that +# cluster, so we have two references +poke_file "$TEST_IMG" $((0x40008)) "\x80\x00\x00\x00\x00\x05\x00\x00"tests/qemu-iotests/112 line 221
Quoted 10 lines
tests/qemu-iotests/112 new file +10
-0+ +# This should say "please use amend" +_check_test_img -r all + +# So we do that +$QEMU_IMG amend -o refcount_width=2 "$TEST_IMG" +print_refcount_width + +# And try again +_check_test_img -r allI think this section also deserves a test that fuzzes an image with width=64 to intentionally set the most significant bit of one of the refcounts, and make sure that we gracefully diagnose it as invalid.
Quoted 21 lines
tests/qemu-iotests/112 new file +21
-0+ +echo +echo '=== Multiple walks necessary during amend ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1,cluster_size=512" _make_test_img 64k + +# Cluster 0 is the image header, clusters 1 to 4 are used by the L1 table, a +# single L2 table, the reftable and a single refblock. This creates 58 data +# clusters (actually, the L2 table is created here, too), so in total there are +# then 63 used clusters in the image. With a refcount width of 64, one refblock +# describes 64 clusters (512 bytes / 64 bits/entry = 64 entries), so this will +# make the first target refblock have exactly one free entry. +$QEMU_IO -c "write 0 $((58 * 512))" "$TEST_IMG" | _filter_qemu_io + +# Now change the refcount width; since the first target refblock has exactly one +# free entry, that entry will be used to store its own reference. No other +# refblocks are needed, so then the new reftable will be allocated; since the +# first target refblock is completely filled up, this will require a new +# refblock which is why the refcount width changing function will need to run +# through everything one more time until the allocations are stable.
Quoted 8 lines
Quoted 2 lines
+$QEMU_IMG amend -o refcount_width=64 "$TEST_IMG"
+print_refcount_widthUmm, that sounds backwards from what you document. It's a good test of the _new_ reftable needing a second round of allocations. So keep it with corrected comments. But I think you _intended_ to write a test that starts with a refcount_width=64 image and resize to a refcount_width=1, where the _old_ reftable then suffers a reallocation as part of allocating refblocks for the new table.
That's what you intended, but that's harder to test, so I settled for this (and the comments are appropriate (note that "target refblock" refers to the refblocks after amendment); note that this does indeed fail with v1 of this series.
Quoted 4 lines
It may even help if
you add a tracepoint for every iteration through the walk function callback, to prove we are indeed executing it 3 times instead of the usual 2, for these test cases.
That's a good idea! I thought about adding some info, but totally forgot about trace points.
I'll see whether I can add a test for increasing the size of the original reftable during amendment, too.
Max
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
Quoted 83 lines · Max Reitz
Quoted 73 lines · Eric Blake
Quoted 26 lines · Max Reitz
Add a test for conversion between different refcount widths and errors specific to certain widths (i.e. snapshots with refcount_width=1).
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/112 | 252 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/112.out | 131 +++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 384 insertions(+)create mode 100755 tests/qemu-iotests/112
create mode 100644 tests/qemu-iotests/112.outtests/qemu-iotests/112 new file +10
-0+echo +echo '=== Testing too many references for check ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# This cluster should be created at 0x50000 +$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io +# Now make the second L2 entry (the L2 table should be at 0x40000)point to that
tests/qemu-iotests/112 new file +2
-0+# cluster, so we have two references +poke_file "$TEST_IMG" $((0x40008)) "\x80\x00\x00\x00\x00\x05\x00\x00"tests/qemu-iotests/112 line 221
Quoted 10 lines
tests/qemu-iotests/112 new file +10
-0+ +# This should say "please use amend" +_check_test_img -r all + +# So we do that +$QEMU_IMG amend -o refcount_width=2 "$TEST_IMG" +print_refcount_width + +# And try again +_check_test_img -r allI think this section also deserves a test that fuzzes an image with width=64 to intentionally set the most significant bit of one of the refcounts, and make sure that we gracefully diagnose it as invalid.
Quoted 32 lines
tests/qemu-iotests/112 new file +6
-0+ +echo +echo '=== Multiple walks necessary during amend ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1,cluster_size=512" _make_test_img64k
(quoted diff) +2
-0+ +# Cluster 0 is the image header, clusters 1 to 4 are used by the L1table, a
+# single L2 table, the reftable and a single refblock. This creates 58 data
+# clusters (actually, the L2 table is created here, too), so in total there are
+# then 63 used clusters in the image. With a refcount width of 64, one refblock
+# describes 64 clusters (512 bytes / 64 bits/entry = 64 entries), so this willtests/qemu-iotests/112 new file +4
-0+# make the first target refblock have exactly one free entry. +$QEMU_IO -c "write 0 $((58 * 512))" "$TEST_IMG" | _filter_qemu_io + +# Now change the refcount width; since the first target refblockhas exactly one
+# free entry, that entry will be used to store its own reference. No other
+# refblocks are needed, so then the new reftable will be allocated; since the
+# first target refblock is completely filled up, this will require a new
+# refblock which is why the refcount width changing function will need to run
+# through everything one more time until the allocations are stable.Quoted 8 lines
Quoted 2 lines
+$QEMU_IMG amend -o refcount_width=64 "$TEST_IMG"
+print_refcount_widthUmm, that sounds backwards from what you document. It's a good test of the _new_ reftable needing a second round of allocations. So keep it with corrected comments. But I think you _intended_ to write a test that starts with a refcount_width=64 image and resize to a refcount_width=1, where the _old_ reftable then suffers a reallocation as part of allocating refblocks for the new table.
Quoted 8 lines
That's what you intended, but that's harder to test, so I settled for this (and the comments are appropriate (note that "target refblock" refers to the refblocks after amendment); note that this does indeed fail with v1 of this series.
Quoted 4 lines
It may even help if
you add a tracepoint for every iteration through the walk function callback, to prove we are indeed executing it 3 times instead of the usual 2, for these test cases.That's a good idea! I thought about adding some info, but totally forgot about trace points.
...On second thought, trace doesn't work so well with qemu-img. My best bet would be blkdebug, but that seems kind of ugly to me...
Max
Quoted 3 lines
I'll see whether I can add a test for increasing the size of the original reftable during amendment, too.
Max
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
Quoted 86 lines · Max Reitz
Quoted 85 lines · Max Reitz
Quoted 73 lines · Eric Blake
Quoted 26 lines · Max Reitz
Add a test for conversion between different refcount widths and errors specific to certain widths (i.e. snapshots with refcount_width=1).
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/112 | 252 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/112.out | 131 +++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 384 insertions(+)create mode 100755 tests/qemu-iotests/112
create mode 100644 tests/qemu-iotests/112.outtests/qemu-iotests/112 new file +10
-0+echo +echo '=== Testing too many references for check ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# This cluster should be created at 0x50000 +$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io +# Now make the second L2 entry (the L2 table should be at 0x40000)point to that
tests/qemu-iotests/112 new file +2
-0+# cluster, so we have two references +poke_file "$TEST_IMG" $((0x40008)) "\x80\x00\x00\x00\x00\x05\x00\x00"tests/qemu-iotests/112 line 221
Quoted 10 lines
tests/qemu-iotests/112 new file +10
-0+ +# This should say "please use amend" +_check_test_img -r all + +# So we do that +$QEMU_IMG amend -o refcount_width=2 "$TEST_IMG" +print_refcount_width + +# And try again +_check_test_img -r allI think this section also deserves a test that fuzzes an image with width=64 to intentionally set the most significant bit of one of the refcounts, and make sure that we gracefully diagnose it as invalid.
Quoted 32 lines
tests/qemu-iotests/112 new file +6
-0+ +echo +echo '=== Multiple walks necessary during amend ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1,cluster_size=512"
_make_test_img64k(quoted diff) +2
-0+ +# Cluster 0 is the image header, clusters 1 to 4 are used by theL1 table, a
+# single L2 table, the reftable and a single refblock. This creates 58 data
+# clusters (actually, the L2 table is created here, too), so in total there are
+# then 63 used clusters in the image. With a refcount width of 64, one refblock
+# describes 64 clusters (512 bytes / 64 bits/entry = 64 entries), so this willtests/qemu-iotests/112 new file +4
-0+# make the first target refblock have exactly one free entry. +$QEMU_IO -c "write 0 $((58 * 512))" "$TEST_IMG" | _filter_qemu_io + +# Now change the refcount width; since the first target refblockhas exactly one
+# free entry, that entry will be used to store its own reference. No other
+# refblocks are needed, so then the new reftable will be allocated; since the
+# first target refblock is completely filled up, this will require a new
+# refblock which is why the refcount width changing function will need to run
+# through everything one more time until the allocations are stable.Quoted 8 lines
Quoted 2 lines
+$QEMU_IMG amend -o refcount_width=64 "$TEST_IMG"
+print_refcount_widthUmm, that sounds backwards from what you document. It's a good test of the _new_ reftable needing a second round of allocations. So keep it with corrected comments. But I think you _intended_ to write a test that starts with a refcount_width=64 image and resize to a refcount_width=1, where the _old_ reftable then suffers a reallocation as part of allocating refblocks for the new table.
That's what you intended, but that's harder to test, so I settled for this (and the comments are appropriate (note that "target refblock" refers to the refblocks after amendment); note that this does indeed fail with v1 of this series.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
It may even help if
you add a tracepoint for every iteration through the walk function callback, to prove we are indeed executing it 3 times instead of the usual 2, for these test cases.That's a good idea! I thought about adding some info, but totally forgot about trace points.
...On second thought, trace doesn't work so well with qemu-img. My best bet would be blkdebug, but that seems kind of ugly to me...
Problem "solved": If there will be more walks than originally thought (3+1 instead of 2+1), progress will regress at one point. I'll just grep for that point and that should be enough (progress jumping from 66.67 % to 50.00 %).
Max
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
In reply to Eric BlakeQuoted 63 lines · Eric Blake
Quoted 24 lines · Max Reitz
Add a test for conversion between different refcount widths and errors specific to certain widths (i.e. snapshots with refcount_width=1).
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/112 | 252 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/112.out | 131 +++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 384 insertions(+)create mode 100755 tests/qemu-iotests/112
create mode 100644 tests/qemu-iotests/112.outtests/qemu-iotests/112 new file +12
-0+echo +echo '=== Testing too many references for check ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1" _make_test_img 64M +print_refcount_width + +# This cluster should be created at 0x50000 +$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io +# Now make the second L2 entry (the L2 table should be at 0x40000) point to that +# cluster, so we have two references +poke_file "$TEST_IMG" $((0x40008)) "\x80\x00\x00\x00\x00\x05\x00\x00"tests/qemu-iotests/112 line 221
Quoted 10 lines
tests/qemu-iotests/112 new file +10
-0+ +# This should say "please use amend" +_check_test_img -r all + +# So we do that +$QEMU_IMG amend -o refcount_width=2 "$TEST_IMG" +print_refcount_width + +# And try again +_check_test_img -r allI think this section also deserves a test that fuzzes an image with width=64 to intentionally set the most significant bit of one of the refcounts, and make sure that we gracefully diagnose it as invalid.
Quoted 13 lines
tests/qemu-iotests/112 new file +13
-0+ +echo +echo '=== Multiple walks necessary during amend ===' +echo + +IMGOPTS="$IMGOPTS,refcount_width=1,cluster_size=512" _make_test_img 64k + +# Cluster 0 is the image header, clusters 1 to 4 are used by the L1 table, a +# single L2 table, the reftable and a single refblock. This creates 58 data +# clusters (actually, the L2 table is created here, too), so in total there are +# then 63 used clusters in the image. With a refcount width of 64, one refblock +# describes 64 clusters (512 bytes / 64 bits/entry = 64 entries), so this will +# make the first target refblock have exactly one free entry.tests/qemu-iotests/112 line 244
Quoted 10 lines
tests/qemu-iotests/112 new file +10
-0+$QEMU_IO -c "write 0 $((58 * 512))" "$TEST_IMG" | _filter_qemu_io + +# Now change the refcount width; since the first target refblock has exactly one +# free entry, that entry will be used to store its own reference. No other +# refblocks are needed, so then the new reftable will be allocated; since the +# first target refblock is completely filled up, this will require a new +# refblock which is why the refcount width changing function will need to run +# through everything one more time until the allocations are stable. +$QEMU_IMG amend -o refcount_width=64 "$TEST_IMG" +print_refcount_widthUmm, that sounds backwards from what you document. It's a good test of
Quoted 8 lines
the _new_ reftable needing a second round of allocations. So keep it with corrected comments. But I think you _intended_ to write a test that starts with a refcount_width=64 image and resize to a refcount_width=1, where the _old_ reftable then suffers a reallocation as part of allocating refblocks for the new table. It may even help if you add a tracepoint for every iteration through the walk function callback, to prove we are indeed executing it 3 times instead of the usual 2, for these test cases.
I'm currently thinking about a way to test the old reftable reallocation issue, and I can't find any. So, for the old reftable to require a reallocation it must grow. For it to grow we need some allocation beyond what it can currently represent. For this to happen during the refblock allocation walk, this allocation must be the allocation of a new refblock.
If the refblock is allocated beyond the current reftable's limit, this means that either all clusters between free_cluster_index and that point are already taken. If the reftable is then reallocated, it will therefore *always* be allocated behind that refblock, which is beyond its old limit. Therefore, that walk through the old reftable will never miss that new allocation.
So the issue can only occur if the old reftable is resized after the walk through it, that is, when allocating the new reftable. That is indeed an issue but I think it manifests itself basically like the issue I'm testing here: There is now an area in the old refcount structures which was free before but has is used now, and the allocation causing that was the allocation of the new reftable. The only difference is whether the it's the old or the new reftable that resides in the previously free area. Thus, I think I'll leave it at this test – but if you can describe to me how to create an image for a different "rewalk" path, I'm all ears.
Max
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
Quoted 23 lines · Max Reitz
Quoted 9 lines
Umm, that sounds backwards from what you document. It's a good test of the _new_ reftable needing a second round of allocations. So keep it with corrected comments. But I think you _intended_ to write a test that starts with a refcount_width=64 image and resize to a refcount_width=1, where the _old_ reftable then suffers a reallocation as part of allocating refblocks for the new table. It may even help if you add a tracepoint for every iteration through the walk function callback, to prove we are indeed executing it 3 times instead of the usual 2, for these test cases.
I'm currently thinking about a way to test the old reftable reallocation issue, and I can't find any. So, for the old reftable to require a reallocation it must grow. For it to grow we need some allocation beyond what it can currently represent. For this to happen during the refblock allocation walk, this allocation must be the allocation of a new refblock.
If the refblock is allocated beyond the current reftable's limit, this means that either all clusters between
free_cluster_indexand that point are already taken. If the reftable is then reallocated, it will therefore *always* be allocated behind that refblock, which is beyond its old limit. Therefore, that walk through the old reftable will never miss that new allocation.
Quoted 10 lines
So the issue can only occur if the old reftable is resized after the walk through it, that is, when allocating the new reftable. That is indeed an issue but I think it manifests itself basically like the issue I'm testing here: There is now an area in the old refcount structures which was free before but has is used now, and the allocation causing that was the allocation of the new reftable. The only difference is whether the it's the old or the new reftable that resides in the previously free area. Thus, I think I'll leave it at this test – but if you can describe to me how to create an image for a different "rewalk" path, I'm all ears.
=====
The test you wrote does:
original image, pre-walk:
reftable is one cluster; with one refblock and 63 zero entries
that refblock holds 4096 width-1 refcounts; of those, the first 63 are
non-zero, the remaining are zero. Image is 32256 bytes long
During the first walk, we call operation() 64 times - the first time with refblock_empty false, the remaining 63 times with refblock_empty true.
after first walk but before reftable allocation, we have allocated one refblock that holds 64 width-64 refcounts (all zero, because we don't populate them until the final walk); and the old table now has 64 refcounts populated. Image is 32768 bytes long.
Then we allocating a new reftable; so far, we only created one refblock for it to hold, so one cluster is sufficient. The allocation causes the old table to now have 65 refcounts populated. Image is now 33280 bytes long.
On the second pass, we call operation() 64 times; now the first two walks have refblock_empty as false, which means we allocate a new refblock. This allocation causes the old table to now have 66 refcounts populated. Image is now 33792 bytes long.
So we free our first attempt at a new reftable, and allocate another (a single cluster is still sufficient to hold two refblocks); I'm not sure whether this free/realloc will reuse cluster 65 or if it will pick up cluster 67 and leave a hole in 65. [I guess it depends on whether cluster allocation is done by first-fit analysis or whether it blindly favors allocating at the end of the image]. Either way, we have to do a third iteration, because the second iteration allocated a refblock and "reallocated" a reftable.
On the third pass, operation() is still called 64 times, but because the only two calls with refblock_empty as false already have an allocated refblock, no further allocations are needed, and we are done with the do loop; the fourth walk can set refcounts.
=====
The test I thought you were writing would start
original image, pre-walk:
reftable is one cluster; with one refblock and 63 zero entries
that refblock holds 64 width-64 refcounts; of those, the first 63 are
non-zero, the remaining are zero. Image is 32256 bytes long
During the first walk, we call operation() 1 time, with refblock_empty false.
after first walk but before reftable allocation, we have allocated one refblock that holds 4096 width-1 refcounts (all zero, because we don't populate them until the final walk); and the old table now has 64 refcounts populated. Image is 32768 bytes long.
Then we allocating a new reftable; so far, we only created one refblock for it to hold, so one cluster is sufficient. The allocation causes the old table to now have 66 refcounts populated (one for the new refblock, but also one for an additional refblock in the old table because the first refblock was full). Image is now 33792 bytes long.
On the second pass, we call operation() 1 time with refblock_empty as false, so we don't need any allocation.
Which means the test you wrote is correct, while my idea does NOT trigger the third walk, at least not for the initial file size of 32256.
You've been vindicated, you did it correctly :)
=====
Now, in response to your question about some other 3-pass inducing pattern, let's think back to v1, where you questioned what would happen if a hole in the reftable gets turned into data due to a later allocation. Let's see if I can come up with a scenario for that...
Let's stick with a cluster size of 512, and use 32-bit and 64-bit widths as our two sizes. If we downsize from 64 to 32 bits, then every two refblock clusters in the old table results in one call to operation() for the new table; conversely, if we upsize, then every refblock cluster in the old table gives two calls to operation() in the new table. The trick at hand is to come up with some image where we punch a hole so that on the first pass, we call operation() with refblock_empty true for one iteration (necessarily a call later than the first, since the image header guarantees the first refblock is not empty), but where we have data after the hole, where it is the later data that triggers the allocation that will finally start to fill the hole.
How about starting with an image that occupies between 1.5 and 2 refblocks worth of 32-width clusters (so an image anywhere between 193 and 256 clusters, or between 98816 and 131072 bytes). You should be able to figure out how many clusters this consumes for L1, L2, plus 1 for header, reftable, and 2 for refblocks, in order to figure out how many remaining clusters are dedicated to data; ideally, the data clusters are contiguous, and occupy a swath that covers at least clusters 126 through 192. Widening to 64-bit width will require 4 refblocks instead of 2, if all refblocks are needed. But the whole idea of punching a hole is that we don't need a refblock if it will be all-zero entries. So take this original image, and discard the data clusters from physical index 126 through 192, (this is NOT the data visible at guest offset 31744, but whatever actual offset of guest data that maps to physical offset 31744). The old reftable now looks like { refblock_o1 [0-125 occupied, 126 and 127 empty], refblock_o2 [128-191 empty, 192-whatever occupied, tail empty] }. With no allocations required, this would in turn would map to the following new refblocks: { refblock_n1 [0-64 occupied], refblock_n2 [65-125 occupied, 126-127 empty], NULL, refblock_n4 [192-whatever occupied] }. Note that we do not need to allocate refblock_n3 because of the hole in the old refblock; we DO end up allocating three refblocks, but in the sequence of things, refblock_n1 and refblock_n2 are allocated while we are visiting refblock_o1 and still fit in refblock_o1, while refblock_n4 is not allocated until after we have already passed over the first half of refblock_o2.
Thus, the second walk over the image will see that we need to allocate refblock_n3 because it now contains entries (in particular, the entry for refblock_n4, but also the 1-cluster entry for the proposed reftable that is allocated between the walks). So, while your test used the allocation of the reftable as the spillover point, my scenario here uses the allocation of later refblocks as the spillover point that got missed during the first iteration.
which means the reftable now looks like { refblock1, NULL, refblock3, NULL... }; and where refblock1 now has at least two free entries (possibly three, if the just-freed refblock2 happened to live before cluster 62). is we can also free refblock2
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
Quoted 43 lines · Eric Blake
Now, in response to your question about some other 3-pass inducing pattern, let's think back to v1, where you questioned what would happen if a hole in the reftable gets turned into data due to a later allocation. Let's see if I can come up with a scenario for that...
Let's stick with a cluster size of 512, and use 32-bit and 64-bit widths as our two sizes. If we downsize from 64 to 32 bits, then every two refblock clusters in the old table results in one call to
operation()for the new table; conversely, if we upsize, then every refblock cluster in the old table gives two calls tooperation()in the new table. The trick at hand is to come up with some image where we punch a hole so that on the first pass, we calloperation()withrefblock_emptytrue for one iteration (necessarily a call later than the first, since the image header guarantees the first refblock is not empty), but where we have data after the hole, where it is the later data that triggers the allocation that will finally start to fill the hole.How about starting with an image that occupies between 1.5 and 2 refblocks worth of 32-width clusters (so an image anywhere between 193 and 256 clusters, or between 98816 and 131072 bytes). You should be able to figure out how many clusters this consumes for L1, L2, plus 1 for header, reftable, and 2 for refblocks, in order to figure out how many remaining clusters are dedicated to data; ideally, the data clusters are contiguous, and occupy a swath that covers at least clusters 126 through 192. Widening to 64-bit width will require 4 refblocks instead of 2, if all refblocks are needed. But the whole idea of punching a hole is that we don't need a refblock if it will be all-zero entries. So take this original image, and discard the data clusters from physical index 126 through 192, (this is NOT the data visible at guest offset 31744, but whatever actual offset of guest data that maps to physical offset 31744). The old reftable now looks like {
refblock_o1[0-125 occupied, 126 and 127 empty],refblock_o2[128-191 empty, 192-whatever occupied, tail empty] }. With no allocations required, this would in turn would map to the following new refblocks: {refblock_n1[0-64 occupied],refblock_n2[65-125 occupied, 126-127 empty], NULL,refblock_n4[192-whatever occupied] }. Note that we do not need to allocaterefblock_n3because of the hole in the old refblock; we DO end up allocating three refblocks, but in the sequence of things,refblock_n1andrefblock_n2are allocated while we are visitingrefblock_o1and still fit inrefblock_o1, whilerefblock_n4is not allocated until after we have already passed over the first half ofrefblock_o2.
Quoted 8 lines
Thus, the second walk over the image will see that we need to allocate
refblock_n3because it now contains entries (in particular, the entry forrefblock_n4, but also the 1-cluster entry for the proposed reftable that is allocated between the walks). So, while your test used the allocation of the reftable as the spillover point, my scenario here uses the allocation of later refblocks as the spillover point that got missed during the first iteration.
Oops,...
Quoted 6 lines
which means the reftable now looks like { refblock1, NULL, refblock3, NULL... }; and where refblock1 now has at least two free entries (possibly three, if the just-freed refblock2 happened to live before cluster 62). is we can also free refblock2
...forgot to delete these random thoughts that I typed up but no longer needed after reworking the above text.
At any rate, I'm not certain we can come up with a four-pass scenario; if it is even possible, it would be quite complex. Back in v1, I questioned what would happen with a completely full reftable, where the mere allocation of the new reftable causes a spillover not only in the size of the new reftable, but also to the old. Let's try to find a scenario where the reftable does NOT spill on the first pass, but DOES spill on the second. A 32-bit width image spills from 1 to 2 clusters for the reftable at the boundary of 64->65 refblocks (8192->8193 clusters); at that same cluster boundary, the reftable for a 64-bit width table spills from 2 to 3 clusters. Furthermore, a 64-bit width refblock misses an allocation on the first pass if there is a hole of 64 aligned clusters.
First try:
We want an image that has roughly 128 free clusters on the first pass, including a hole of at least 64 aligned clusters, to where a second pass is needed to cover the refblock that was treated as a hole on the first pass. Furthermore, we want the second pass to completely fill the image, so that the reftable allocation of 2 clusters after the walk is the trigger of the spill, then the third pass will allocate another refblock because of the spill, and a fourth pass is required to ensure no allocations before the final pass of assigning refcounts.
Start with a 32-bit width image with clusters 0-125 allocated, 126-191 empty, then 192-8127 allocated (then 8128-8191 are unallocated to pad out to cluster boundary) - describing a file of size 4161536. This image has 130 clusters spare before it spills, with a hole of 66 clusters near the front, and a tail of 64 clusters; since there is no block of 128 aligned unallocated clusters, all 64 refblocks are in use.
Now widen this image to 64-bit width refcounts.
On the first walk, we allocate two refblocks for clusters 0-127 (using free slots 126 and 127), then leave an unallocated refblock for clusters 128-191, then allocate 124 more refblocks for clusters 192-8127 (using free slots 128-191, as well as 8128-8187); this walk also picks up a refblock for clusters 8128-8191 (using free slot 8188) because that area of the file is allocated before the walk reaches that point, even if it was a large enough hole to not need a refblock at the beginning of the walk. So we end the walk with 3 free clusters, and proceed to allocate a 2-cluster reftable for the 127 refblocks that we created (using free slots 8189-8190).
On the second walk, we allocate a refblock for clusters 128-191 (slot 8191). At the end of the walk, we free the 2-cluster reftable, then reallocate; but as we still only need 2 clusters, we reuse slots 8189-8190. Bummer - a third pass doesn't need allocation.
Second try:
Tweak the input by one cluster: 0-125 allocated, 126-191 empty, 192-8128 allocated (8129-8191 unallocated); file size 4162048. 129 spare clusters, with hole of 66 and tail of 63. On the first walk, we allocate 127 refblocks (ending with the use of free slots 8129-8189), then allocate a 2-cluster reftable (free slots 8190-8191). On the second walk, we allocate a refblock for clusters 128-191 (slot 8192 - which triggers a spill of the old reftable), and a refblock for clusters 8192-8255 (since we already spilled). After the walk, we now allocate a 3-cluster reftable, but it fits nicely within 8192-8255. Bummer - a third pass still doesn't need an allocation.
Third try:
Tweak the input by an entire reftable cluster. Let's start with a 32-bit image with a 2-cluster reftable, and try for the spill of a 64-bit image from 3->4 clusters (happens at the boundary from cluster 12287->12288). We want around 192 free clusters, plus the space for the reftable, and still want a hole of at least 64 aligned clusters to trigger the second pass as the one that fills to the spilling point.
So the image has clusters 0-125 allocated, 126-191 empty, then 192-12157 allocated (then 12158-16383 are unallocated to pad out to cluster boundary) - describing a file of size 6224896. This image has 196 clusters spare before it spills the 64-bit reftable, with a hole of 66 clusters near the front, and a tail of 130 clusters.
On the first walk, we allocate two refblocks for clusters 0-127 (using free slots 126 and 127), then leave an unallocated refblock for clusters 128-191, then allocate 187 more refblocks for clusters 192-12157 (using free slots 128-191, as well as 12158-12280); this walk also triggers a refblock allocation for the 32-bit table (slot 12281 for clusters 12160-12287) and 2 refblocks for the 64-bit table (slots 12282-12283 for clusters 12158-12287) because that area of the file is allocated before the walk reaches that point. So we end the walk with 4 free clusters, and proceed to allocate a 3-cluster reftable for the 191 refblocks that we created (using free slots 12284-12286).
On the second walk, we allocate a refblock for clusters 128-191 (slot 12287). At the end of the walk, we free the 3-cluster reftable, then reallocate; but as we still only need 3 clusters, we reuse slots 12284-12286. Bummer - a third pass doesn't need allocation.
I'm sensing a pattern here - trying to go for reftable spills isn't going to help - by the second pass, we already have a reftable reservation (which we free and then reallocate), and the only time we'd need a larger reftable after the second walk is if we encountered extra refblocks during the walk, but the only way to get extra refblocks during the walk is to spill the old table during the second walk, but in that case the second walk picks up the spill.
Fourth try:
Even trying to play with larger files with larger holes won't easily help. Suppose I have a file that has 0-62 allocated, then 63-4159 unallocated, followed by 262144 clusters allocated (an image over 128M in size). On the first walk, we allocate a refblock (slot 63) for clusters 0-63, then pass over 64 refblocks that don't need allocation, then allocate 4096 refblocks (slots 64-4159). Not even considering the slots that are added in the old table, this means the second pass will allocate an additional 64 refblocks. But this allocation will NOT spill the size of the reftable, because the reftable is already sized large enough to cover the refblocks for the 128M tail of the file.
What if we have a larger hole, and also tweak the input file so that there are no missing refblocks of the old table (the holes are only possible in the new table). That is, start with 0-62 allocated, 63-127 free, then a repeating pattern of 64 allocated clusters and 64 free, over the course of 130 repetitions. Follow that with 262144 allocated clusters. This means we have 65 + 130*64 == 8385 free clusters, to dedicate solely to the new reftable; while the image occupies (131*128)+262144 == 278912 clusters (requires 4358 refblock entries, which in turn requires 69 contiguous clusters for the reftable).
On the first walk, we alternate between allocating a refblock and leaving a hole for a repetition of 131 times, then allocate 4096 refblocks. Thus, we have consumed 4227 allocations out of the 8385 free clusters; where the old table is now using the space of 67 of the allocation holes. Furthermore, the reftable allocation is too large to fit in any of the holes, so it gets appended at the end of the image (the image is now 278981 clusters, requiring 4360 refblock entries, but still only 69 clusters for the reftable). On the second walk, we allocate 67 more refblocks for the refcounts in the old table that we first treated as holes, plus 2 more refblocks for the tail of the file holding the new reftable; these allocations still fit in the holes, and are sufficient to fill up another hole. However, the question remains - does the hole in the old table get filled before or after the new table has already passed by the hole? If it is after, a third iteration would allocate yet another refblock or two; but if it is before, then the 2nd walk will already allocate refblocks when it encounters that spot in the old reftable. Remember, on the first iteration, we allocated a refblock for 0-63 (slot 63), for 128-191 (slot 64), and so on - the reason the second pass allocates a refblock for 64-127 is because the first pass didn't populate that area of the file until it was already beyond cluster 127. But on the second pass, the first refblock we allocate for clusters 64-127 will live somewhere around slot 8576, so it will be seen during the second walk. Even the refblock for clusters 8512-8575 will end up being allocated somewhere around slot 8640, which still gets visited during the second walk. You may still be able to tweak things to the point that we trigger an allocating third walk, but I'm not readily seeing what that tweak would be.
At this point, I've spent far too long writing this email. I haven't completely ruled out the possibility of a corner case needing four passes through the do loop, but the image sizes required to get there are starting to be quite large compared to your simpler test of needing three passes through the do loop. I won't be bothered if we call it good, and quit trying to come up with any other "interesting" allocation sequencing.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
Quoted 60 lines · Eric Blake
Quoted 43 lines · Eric Blake
Now, in response to your question about some other 3-pass inducing pattern, let's think back to v1, where you questioned what would happen if a hole in the reftable gets turned into data due to a later allocation. Let's see if I can come up with a scenario for that...
Let's stick with a cluster size of 512, and use 32-bit and 64-bit widths as our two sizes. If we downsize from 64 to 32 bits, then every two refblock clusters in the old table results in one call to
operation()for the new table; conversely, if we upsize, then every refblock cluster in the old table gives two calls tooperation()in the new table. The trick at hand is to come up with some image where we punch a hole so that on the first pass, we calloperation()withrefblock_emptytrue for one iteration (necessarily a call later than the first, since the image header guarantees the first refblock is not empty), but where we have data after the hole, where it is the later data that triggers the allocation that will finally start to fill the hole.How about starting with an image that occupies between 1.5 and 2 refblocks worth of 32-width clusters (so an image anywhere between 193 and 256 clusters, or between 98816 and 131072 bytes). You should be able to figure out how many clusters this consumes for L1, L2, plus 1 for header, reftable, and 2 for refblocks, in order to figure out how many remaining clusters are dedicated to data; ideally, the data clusters are contiguous, and occupy a swath that covers at least clusters 126 through 192. Widening to 64-bit width will require 4 refblocks instead of 2, if all refblocks are needed. But the whole idea of punching a hole is that we don't need a refblock if it will be all-zero entries. So take this original image, and discard the data clusters from physical index 126 through 192, (this is NOT the data visible at guest offset 31744, but whatever actual offset of guest data that maps to physical offset 31744). The old reftable now looks like {
refblock_o1[0-125 occupied, 126 and 127 empty],refblock_o2[128-191 empty, 192-whatever occupied, tail empty] }. With no allocations required, this would in turn would map to the following new refblocks: {refblock_n1[0-64 occupied],refblock_n2[65-125 occupied, 126-127 empty], NULL,refblock_n4[192-whatever occupied] }. Note that we do not need to allocaterefblock_n3because of the hole in the old refblock; we DO end up allocating three refblocks, but in the sequence of things,refblock_n1andrefblock_n2are allocated while we are visitingrefblock_o1and still fit inrefblock_o1, whilerefblock_n4is not allocated until after we have already passed over the first half ofrefblock_o2.Quoted 8 lines
Thus, the second walk over the image will see that we need to allocate
refblock_n3because it now contains entries (in particular, the entry forrefblock_n4, but also the 1-cluster entry for the proposed reftable that is allocated between the walks). So, while your test used the allocation of the reftable as the spillover point, my scenario here uses the allocation of later refblocks as the spillover point that got missed during the first iteration.Oops,...
Quoted 5 lines
which means the reftable now looks like { refblock1, NULL, refblock3, NULL... }; and where refblock1 now has at least two free entries (possibly three, if the just-freed refblock2 happened to live before cluster 62). is we can also free refblock2
Quoted 5 lines
...forgot to delete these random thoughts that I typed up but no longer needed after reworking the above text.
At any rate, I'm not certain we can come up with a four-pass scenario; if it is even possible, it would be quite complex.
[snip] (But rest assured, I read it all ;-))
Quoted 5 lines
At this point, I've spent far too long writing this email. I haven't completely ruled out the possibility of a corner case needing four passes through the do loop, but the image sizes required to get there are starting to be quite large compared to your simpler test of needing three passes through the do loop.
Right, see test 026. Without an SSD, it takes more than ten minutes, not least because it tests resizing the reftable which means writing a lot of data to an image with 512 byte clusters.
Quoted 3 lines
I won't be bothered if we call it
good, and quit trying to come up with any other "interesting" allocation sequencing.
The problem is, in my opinion, that we won't gain a whole lot from proving that there are cases where you need a fourth pass and test these cases. Fundamentally, they are not different from cases with three passes (technically, not even different from two pass cases). You scan through the refcounts, you detect that you need refblocks which you have not yet allocated, you allocate them, then you respin until all allocations are done. The only problem would be whether it'd be possible to run into an infinite loop: Can allocating new refblocks lead to a case where we have to allocate even more refblocks? Well, just judging from how complicated it is to even find a case where the number of new allocations hasn't gone down to zero in the fourth pass, we can safely rule that out.
Some people may ask why the walks are performed in a loop without a fixed limit (because they can't find cases where allocations haven't settled at the third pass). But I doubt that'll be a serious problem. It's much easier to have such a basically unlimited loop with the reasoning "We don't know exactly how many loops it'll take, but it will definitely settle at some point in time" than limiting the loop and then having to explain why we know exactly that it won't take more than X passes. The only problem with not limiting is that we need one walk to verify that all allocations have settled. But we need that for the common case (two passes) anyway, so that's not an issue.
The code from this version does not care whether it takes one, two, three, four or 42 passes. It's all the same. It will never take one and it will probably never take 42 passes; but if it does, well, it will work. Therefore, I think testing one non-standard number of passes (three) is enough. I'd like to test more, but the effort's just not worth it. I think.
Max
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
Quoted 11 lines · Max Reitz
Some people may ask why the walks are performed in a loop without a fixed limit (because they can't find cases where allocations haven't settled at the third pass). But I doubt that'll be a serious problem. It's much easier to have such a basically unlimited loop with the reasoning "We don't know exactly how many loops it'll take, but it will definitely settle at some point in time" than limiting the loop and then having to explain why we know exactly that it won't take more than X passes. The only problem with not limiting is that we need one walk to verify that all allocations have settled. But we need that for the common case (two passes) anyway, so that's not an issue.
Quoted 6 lines
The code from this version does not care whether it takes one, two, three, four or 42 passes. It's all the same. It will never take one and it will probably never take 42 passes; but if it does, well, it will work. Therefore, I think testing one non-standard number of passes (three) is enough. I'd like to test more, but the effort's just not worth it. I think.
Yep, I agree. I've pretty much convinced myself that the REASON we are guaranteed that things converge is that each successive iteration allocates fewer clusters than the one before, and that in later iterations, refblocks are not fully populated by these fewer allocations (that is, on recursion, we are allocating geometrically less).
I think I may have found a case that needs four passes. What if between the first and second pass, we have enough refblocks to require allocating 2752 or more contiguous clusters for the new reftable (again continuing with my 64-bit from 32-bit example, this means at least 1376 contiguous clusters in the old reftable). That's a huge image already (176128 refblocks, 11,272,192 clusters, or 5,771,362,304 bytes). If we time things so that the first pass ends without spilling the old reftable (which by now seems fairly tractable to compute how many spare clusters to start with), then allocating the new reftable will also spill the old reftable, and based on the reftables alone, will result in more than 4096 newly-referenced clusters on the second pass (or more than 64 new refblocks). This in turn is enough to require another full refblock just to describe the reftable, but that spills the size of the new reftable, so between the second and third iteration we now have to allocate 2753 instead of 2752 contiguous clusters. And _that_ reallocation is enough for the third pass to have to allocate yet more clusters. But like you say, testing this is going to be prohibitively slow (it's not worth a 5 gigabyte test).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
In reply to Eric BlakeQuoted 59 lines · Eric Blake
Quoted 22 lines · Max Reitz
Quoted 9 lines
Umm, that sounds backwards from what you document. It's a good test of the _new_ reftable needing a second round of allocations. So keep it with corrected comments. But I think you _intended_ to write a test that starts with a refcount_width=64 image and resize to a refcount_width=1, where the _old_ reftable then suffers a reallocation as part of allocating refblocks for the new table. It may even help if you add a tracepoint for every iteration through the walk function callback, to prove we are indeed executing it 3 times instead of the usual 2, for these test cases.
I'm currently thinking about a way to test the old reftable reallocation issue, and I can't find any. So, for the old reftable to require a reallocation it must grow. For it to grow we need some allocation beyond what it can currently represent. For this to happen during the refblock allocation walk, this allocation must be the allocation of a new refblock.
If the refblock is allocated beyond the current reftable's limit, this means that either all clusters between
free_cluster_indexand that point are already taken. If the reftable is then reallocated, it will therefore *always* be allocated behind that refblock, which is beyond its old limit. Therefore, that walk through the old reftable will never miss that new allocation.Quoted 10 lines
So the issue can only occur if the old reftable is resized after the walk through it, that is, when allocating the new reftable. That is indeed an issue but I think it manifests itself basically like the issue I'm testing here: There is now an area in the old refcount structures which was free before but has is used now, and the allocation causing that was the allocation of the new reftable. The only difference is whether the it's the old or the new reftable that resides in the previously free area. Thus, I think I'll leave it at this test – but if you can describe to me how to create an image for a different "rewalk" path, I'm all ears.
=====
The test you wrote does:original image, pre-walk:
reftable is one cluster; with one refblock and 63 zero entries
that refblock holds 4096 width-1 refcounts; of those, the first 63 are
non-zero, the remaining are zero. Image is 32256 bytes longDuring the first walk, we call
operation()64 times - the first time withrefblock_emptyfalse, the remaining 63 times withrefblock_emptytrue.after first walk but before reftable allocation, we have allocated one refblock that holds 64 width-64 refcounts (all zero, because we don't populate them until the final walk); and the old table now has 64 refcounts populated. Image is 32768 bytes long.
Then we allocating a new reftable; so far, we only created one refblock for it to hold, so one cluster is sufficient. The allocation causes the old table to now have 65 refcounts populated. Image is now 33280 bytes long.
On the second pass, we call
operation()64 times; now the first two walks haverefblock_emptyas false, which means we allocate a new refblock. This allocation causes the old table to now have 66 refcounts populated. Image is now 33792 bytes long.
Quoted 6 lines
So we free our first attempt at a new reftable, and allocate another (a single cluster is still sufficient to hold two refblocks); I'm not sure whether this free/realloc will reuse cluster 65 or if it will pick up cluster 67 and leave a hole in 65. [I guess it depends on whether cluster allocation is done by first-fit analysis or whether it blindly favors allocating at the end of the image].
There is a free_cluster_index to speed up finding the first fit. It's reset when freeing clusters before that index, therefore cluster 65 should be reused.
Quoted 83 lines
Either way, we have to do a
third iteration, because the second iteration allocated a refblock and "reallocated" a reftable.On the third pass,
operation()is still called 64 times, but because the only two calls withrefblock_emptyas false already have an allocated refblock, no further allocations are needed, and we are done with the do loop; the fourth walk can set refcounts.=====
The test I thought you were writing would startoriginal image, pre-walk:
reftable is one cluster; with one refblock and 63 zero entries
that refblock holds 64 width-64 refcounts; of those, the first 63 are
non-zero, the remaining are zero. Image is 32256 bytes longDuring the first walk, we call
operation()1 time, withrefblock_emptyfalse.after first walk but before reftable allocation, we have allocated one refblock that holds 4096 width-1 refcounts (all zero, because we don't populate them until the final walk); and the old table now has 64 refcounts populated. Image is 32768 bytes long.
Then we allocating a new reftable; so far, we only created one refblock for it to hold, so one cluster is sufficient. The allocation causes the old table to now have 66 refcounts populated (one for the new refblock, but also one for an additional refblock in the old table because the first refblock was full). Image is now 33792 bytes long.
On the second pass, we call
operation()1 time withrefblock_emptyas false, so we don't need any allocation.Which means the test you wrote is correct, while my idea does NOT trigger the third walk, at least not for the initial file size of 32256.
You've been vindicated, you did it correctly :)=====
Now, in response to your question about some other 3-pass inducing pattern, let's think back to v1, where you questioned what would happen if a hole in the reftable gets turned into data due to a later allocation. Let's see if I can come up with a scenario for that...Let's stick with a cluster size of 512, and use 32-bit and 64-bit widths as our two sizes. If we downsize from 64 to 32 bits, then every two refblock clusters in the old table results in one call to
operation()for the new table; conversely, if we upsize, then every refblock cluster in the old table gives two calls tooperation()in the new table. The trick at hand is to come up with some image where we punch a hole so that on the first pass, we calloperation()withrefblock_emptytrue for one iteration (necessarily a call later than the first, since the image header guarantees the first refblock is not empty), but where we have data after the hole, where it is the later data that triggers the allocation that will finally start to fill the hole.How about starting with an image that occupies between 1.5 and 2 refblocks worth of 32-width clusters (so an image anywhere between 193 and 256 clusters, or between 98816 and 131072 bytes). You should be able to figure out how many clusters this consumes for L1, L2, plus 1 for header, reftable, and 2 for refblocks, in order to figure out how many remaining clusters are dedicated to data; ideally, the data clusters are contiguous, and occupy a swath that covers at least clusters 126 through 192. Widening to 64-bit width will require 4 refblocks instead of 2, if all refblocks are needed. But the whole idea of punching a hole is that we don't need a refblock if it will be all-zero entries. So take this original image, and discard the data clusters from physical index 126 through 192, (this is NOT the data visible at guest offset 31744, but whatever actual offset of guest data that maps to physical offset 31744). The old reftable now looks like {
refblock_o1[0-125 occupied, 126 and 127 empty],refblock_o2[128-191 empty, 192-whatever occupied, tail empty] }. With no allocations required, this would in turn would map to the following new refblocks: {refblock_n1[0-64 occupied],refblock_n2[65-125 occupied, 126-127 empty], NULL,refblock_n4[192-whatever occupied] }. Note that we do not need to allocaterefblock_n3because of the hole in the old refblock; we DO end up allocating three refblocks, but in the sequence of things,refblock_n1andrefblock_n2are allocated while we are visitingrefblock_o1and still fit inrefblock_o1, whilerefblock_n4is not allocated until after we have already passed over the first half ofrefblock_o2.
Quoted 7 lines
Thus, the second walk over the image will see that we need to allocate
refblock_n3because it now contains entries (in particular, the entry forrefblock_n4, but also the 1-cluster entry for the proposed reftable that is allocated between the walks). So, while your test used the allocation of the reftable as the spillover point, my scenario here uses the allocation of later refblocks as the spillover point that got missed during the first iteration.
Sounds good, the only problem is that I'd have to hand-craft the image myself, because qemu generally uses self-references for refblocks (when allocating new refblocks, they will contain their own refcount).
I think this already would be too much effort (I'll reply to your second email right away ;-)). There is no fundamental difference in how new allocations for the new reftable and for the new refblocks are treated: If there's a new allocation, respin. If that works for the new reftable, that's enough to convince me it will work for new refblocks as well.
Max
Re: [Qemu-devel] [PATCH v2 21/21] iotests: Add test for different refcount widths
Quoted 9 lines · Max Reitz
Sounds good, the only problem is that I'd have to hand-craft the image myself, because qemu generally uses self-references for refblocks (when allocating new refblocks, they will contain their own refcount).
I think this already would be too much effort (I'll reply to your second email right away ;-)). There is no fundamental difference in how new allocations for the new reftable and for the new refblocks are treated:
If there's a new allocation, respin. If that works for the new reftable, that's enough to convince me it will work for new refblocks as well.
Agreed - one test of needing a third pass is sufficient to prove we handle allocations until convergence.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org