Skip to contentSkip to conversation

qcow2: Support refcount orders != 4

PATCH Superseded Max Reitz 3 participants 44 messages Dec 15, 2014 – Feb 4, 2015 Block layer

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

v5 cover qcow2: Support refcount orders != 4

As of version 3, the qcow2 file format supports different widths for refcount entries, ranging from 1 to 64 bit (only powers of two). Currently, qemu only supports 16 bit, which is the only width supported by version 2 (compat=0.10) images.

This series adds support to qemu for all other valid refcount orders. This is mainly done by adding two function pointers into the BDRVQcowState structure for reading and writing refcount values independently of the current refcount entry width; all in-memory refcount arrays (mostly cached refcount blocks) now are void pointers and are accessed through these functions alone.

Thanks to previous work of making the qemu code agnostic of e.g. the number of refcount entries per refcount block, the rest is fairly trivial. The most complex patch in this series is patch 23 which implements changing the refcount order through qemu-img amend.

To test different refcount widths, simply invoke the qemu-iotests check program with -o refcount_width=${your_desired_width}. The final test in this series adds some tests for operations which do not work with certain refcount orders and for refcount order amendment.

v5:
- Patch 5: Use absolute value for addend instead of the two's complement for negative values [Stefan]
- Patch 6:
- Conflicts due to the changes in patch 5
- Dropped the "if (ret >= 0)" -> "if (ret == 0)" hunk; since the update_refcount() return value was never touched in this series there is no point in modifying the callers' checks for their return value [Eric]
- Patch 9: Use DIV_ROUND_UP() [Eric]
- Patch 16: Rebase conflicts due to Fam's patch removing trailing whitespace in all qemu-iotests/*.out files (7486458c33cd3e5714368f181a5fea39c54444ca); I kept the R-bs because these are only trailing whitespace conflicts in *.out files which are really trivial.
There are two not-so-trivial conflicts: Two hunks are missing from v5 which were part of v4 which did nothing but remove trailing whitespace. Obviously those are no longer necessary, so removing these hunks was actually trivial after all.

git-backport-diff against v4:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/26:[----] [--] 'qcow2: Add two new fields to BDRVQcowState' 002/26:[----] [--] 'qcow2: Add refcount_bits to format-specific info' 003/26:[----] [--] 'qcow2: Do not return new value after refcount update' 004/26:[----] [--] 'qcow2: Only return status from qcow2_get_refcount' 005/26:[0040] [FC] 'qcow2: Use unsigned addend for update_refcount()' 006/26:[0028] [FC] 'qcow2: Use 64 bits for refcount values' 007/26:[----] [--] 'qcow2: Respect error in qcow2_alloc_bytes()' 008/26:[----] [--] 'qcow2: Refcount overflow and qcow2_alloc_bytes()' 009/26:[0002] [FC] 'qcow2: Helper for refcount array reallocation' 010/26:[----] [-C] 'qcow2: Helper function for refcount modification' 011/26:[----] [--] 'qcow2: More helpers for refcount modification' 012/26:[----] [--] 'qcow2: Open images with refcount order != 4' 013/26:[----] [--] 'qcow2: refcount_order parameter for qcow2_create2' 014/26:[----] [--] 'qcow2: Use symbolic macros in qcow2_amend_options' 015/26:[----] [--] 'iotests: Prepare for refcount_bits option' 016/26:[0126] [FC] 'qcow2: Allow creation with refcount order != 4' 017/26:[----] [--] 'progress: Allow regressing progress' 018/26:[----] [--] 'block: Add opaque value to the amend CB' 019/26:[----] [-C] 'qcow2: Use error_report() in qcow2_amend_options()' 020/26:[----] [--] 'qcow2: Use abort() instead of assert(false)' 021/26:[----] [--] 'qcow2: Split upgrade/downgrade paths for amend' 022/26:[----] [--] 'qcow2: Use intermediate helper CB for amend' 023/26:[----] [--] 'qcow2: Add function for refcount order amendment' 024/26:[----] [--] 'qcow2: Invoke refcount order amendment function' 025/26:[----] [--] 'qcow2: Point to amend function in check' 026/26:[----] [--] 'iotests: Add test for different refcount widths'

Max Reitz (26):
qcow2: Add two new fields to BDRVQcowState
qcow2: Add refcount_bits to format-specific info
qcow2: Do not return new value after refcount update
qcow2: Only return status from qcow2_get_refcount
qcow2: Use unsigned addend for update_refcount()
qcow2: Use 64 bits for refcount values
qcow2: Respect error in qcow2_alloc_bytes()
qcow2: Refcount overflow and qcow2_alloc_bytes()
qcow2: Helper for refcount array reallocation
qcow2: Helper function for refcount modification
qcow2: More helpers for refcount modification
qcow2: Open images with refcount order != 4
qcow2: refcount_order parameter for qcow2_create2
qcow2: Use symbolic macros in qcow2_amend_options
iotests: Prepare for refcount_bits option
qcow2: Allow creation with refcount order != 4
progress: Allow regressing progress
block: Add opaque value to the amend CB
qcow2: Use error_report() in qcow2_amend_options()
qcow2: Use abort() instead of assert(false)
qcow2: Split upgrade/downgrade paths for amend
qcow2: Use intermediate helper CB for amend
qcow2: Add function for refcount order amendment
qcow2: Invoke refcount order amendment function
qcow2: Point to amend function in check
iotests: Add test for different refcount widths

 block.c                          |    4 +-
 block/qcow2-cluster.c            |   24 +-
 block/qcow2-refcount.c           | 1054 +++++++++++++++++++++++++++++++-------
 block/qcow2.c                    |  286 ++++++++---
 block/qcow2.h                    |   23 +-
 include/block/block.h            |    4 +-
 include/block/block_int.h        |    4 +-
 qapi/block-core.json             |    5 +-
 qemu-img.c                       |    5 +-
 tests/qemu-iotests/007           |    3 +
 tests/qemu-iotests/015           |    2 +
 tests/qemu-iotests/026           |    7 +
 tests/qemu-iotests/029           |    2 +
 tests/qemu-iotests/049.out       |  108 ++--
 tests/qemu-iotests/051           |    3 +
 tests/qemu-iotests/058           |    2 +
 tests/qemu-iotests/060.out       |    1 +
 tests/qemu-iotests/061.out       |   14 +-
 tests/qemu-iotests/065           |   23 +-
 tests/qemu-iotests/067           |    2 +
 tests/qemu-iotests/067.out       |    5 +
 tests/qemu-iotests/079           |   10 +-
 tests/qemu-iotests/079.out       |   38 +-
 tests/qemu-iotests/080           |    2 +
 tests/qemu-iotests/082.out       |   48 +-
 tests/qemu-iotests/085.out       |   38 +-
 tests/qemu-iotests/089           |    2 +
 tests/qemu-iotests/089.out       |    2 +
 tests/qemu-iotests/108           |    2 +
 tests/qemu-iotests/112           |  296 +++++++++++
 tests/qemu-iotests/112.out       |  155 ++++++
 tests/qemu-iotests/common.filter |    3 +-
 tests/qemu-iotests/group         |    1 +
 util/qemu-progress.c             |    3 +-
 34 files changed, 1782 insertions(+), 399 deletions(-)
 create mode 100755 tests/qemu-iotests/112
 create mode 100644 tests/qemu-iotests/112.out
Signature · 1 line
1.9.3
author
#1 lore raw

PATCH v5 1/26 qcow2: Add two new fields to BDRVQcowState

Add two new fields regarding refcount information (the bit width of every entry and the maximum refcount value) to the BDRVQcowState.

---
 block/qcow2-refcount.c | 2 +-
 block/qcow2.c          | 3 +++
 block/qcow2.h          | 2 ++
 3 files changed, 6 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +1 -1
index 9afdb40..6016211 100644
@@ -584,7 +584,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
 
         refcount = be16_to_cpu(refcount_block[block_index]);
         refcount += addend;
-        if (refcount < 0 || refcount > 0xffff) {
+        if (refcount < 0 || refcount > s->refcount_max) {
             ret = -EINVAL;
             goto fail;
         }
block/qcow2.c +3 -0
index e4e690a..848b984 100644
@@ -684,6 +684,9 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
         goto fail;
     }
     s->refcount_order = header.refcount_order;
+    s->refcount_bits = 1 << s->refcount_order;
+    s->refcount_max = UINT64_C(1) << (s->refcount_bits - 1);
+    s->refcount_max += s->refcount_max - 1;
 
     if (header.crypt_method > QCOW_CRYPT_AES) {
         error_setg(errp, "Unsupported encryption method: %" PRIu32,
block/qcow2.h +2 -0
index 6e39a1b..4d8c902 100644
@@ -258,6 +258,8 @@ typedef struct BDRVQcowState {
     int qcow_version;
     bool use_lazy_refcounts;
     int refcount_order;
+    int refcount_bits;
+    uint64_t refcount_max;
 
     bool discard_passthrough[QCOW2_DISCARD_MAX];
 
Signature · 1 line
1.9.3
author
#2 lore raw

PATCH v5 2/26 qcow2: Add refcount_bits to format-specific info

In reply to Max Reitz

Add the bit width of every refcount entry to the format-specific information.

In contrast to lazy_refcounts and the corrupt flag, this should be always emitted, even for compat=0.10 although it does not support any refcount width other than 16 bits. This is because if a boolean is optional, one normally assumes it to be false when omitted; but if an integer is not specified, it is rather difficult to guess its value.

This new field breaks some test outputs, fix them.

---
 block/qcow2.c              |  4 +++-
 qapi/block-core.json       |  5 ++++-
 tests/qemu-iotests/060.out |  1 +
 tests/qemu-iotests/065     | 23 +++++++++++++++--------
 tests/qemu-iotests/067.out |  5 +++++
 tests/qemu-iotests/082.out |  7 +++++++
 tests/qemu-iotests/089.out |  2 ++
 7 files changed, 37 insertions(+), 10 deletions(-)
block/qcow2.c +3 -1
index 848b984..d14e7c4 100644
@@ -2480,7 +2480,8 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs)
     };
     if (s->qcow_version == 2) {
         *spec_info->qcow2 = (ImageInfoSpecificQCow2){
-            .compat = g_strdup("0.10"),
+            .compat             = g_strdup("0.10"),
+            .refcount_bits      = s->refcount_bits,
         };
     } else if (s->qcow_version == 3) {
         *spec_info->qcow2 = (ImageInfoSpecificQCow2){
@@ -2491,6 +2492,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs)
             .corrupt            = s->incompatible_features &
                                   QCOW2_INCOMPAT_CORRUPT,
             .has_corrupt        = true,
+            .refcount_bits      = s->refcount_bits,
         };
     }
 
qapi/block-core.json +4 -1
index 6e8db15..008215e 100644
@@ -41,13 +41,16 @@
 # @corrupt: #optional true if the image has been marked corrupt; only valid for
 #           compat >= 1.1 (since 2.2)
 #
+# @refcount-bits: width of a refcount entry in bits (since 2.3)
+#
 # Since: 1.7
 ##
 { 'type': 'ImageInfoSpecificQCow2',
   'data': {
       'compat': 'str',
       '*lazy-refcounts': 'bool',
-      '*corrupt': 'bool'
+      '*corrupt': 'bool',
+      'refcount-bits': 'int'
   } }
 
 ##
tests/qemu-iotests/060.out +1 -0
index 7d493bb..e8d002e 100644
@@ -18,6 +18,7 @@ cluster_size: 65536
 Format specific information:
     compat: 1.1
     lazy refcounts: false
+    refcount bits: 16
     corrupt: true
 qemu-io: can't open device TEST_DIR/t.IMGFMT: IMGFMT: Image is corrupt; cannot be opened read/write
 read 512/512 bytes at offset 0
tests/qemu-iotests/065 +15 -8
index 8d3a9c9..72aa970 100755
@@ -88,34 +88,41 @@ class TestQMP(TestImageInfoSpecific):
 class TestQCow2(TestQemuImgInfo):
     '''Testing a qcow2 version 2 image'''
     img_options = 'compat=0.10'
-    json_compare = { 'compat': '0.10' }
-    human_compare = [ 'compat: 0.10' ]
+    json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
+    human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
 
 class TestQCow3NotLazy(TestQemuImgInfo):
     '''Testing a qcow2 version 3 image with lazy refcounts disabled'''
     img_options = 'compat=1.1,lazy_refcounts=off'
-    json_compare = { 'compat': '1.1', 'lazy-refcounts': False, 'corrupt': False }
-    human_compare = [ 'compat: 1.1', 'lazy refcounts: false', 'corrupt: false' ]
+    json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
+                     'refcount-bits': 16, 'corrupt': False }
+    human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
+                      'refcount bits: 16', 'corrupt: false' ]
 
 class TestQCow3Lazy(TestQemuImgInfo):
     '''Testing a qcow2 version 3 image with lazy refcounts enabled'''
     img_options = 'compat=1.1,lazy_refcounts=on'
-    json_compare = { 'compat': '1.1', 'lazy-refcounts': True, 'corrupt': False }
-    human_compare = [ 'compat: 1.1', 'lazy refcounts: true', 'corrupt: false' ]
+    json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
+                     'refcount-bits': 16, 'corrupt': False }
+    human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
+                      'refcount bits: 16', 'corrupt: false' ]
 
 class TestQCow3NotLazyQMP(TestQMP):
     '''Testing a qcow2 version 3 image with lazy refcounts disabled, opening
        with lazy refcounts enabled'''
     img_options = 'compat=1.1,lazy_refcounts=off'
     qemu_options = 'lazy-refcounts=on'
-    compare = { 'compat': '1.1', 'lazy-refcounts': False, 'corrupt': False }
+    compare = { 'compat': '1.1', 'lazy-refcounts': False,
+                'refcount-bits': 16, 'corrupt': False }
+
 
 class TestQCow3LazyQMP(TestQMP):
     '''Testing a qcow2 version 3 image with lazy refcounts enabled, opening
        with lazy refcounts disabled'''
     img_options = 'compat=1.1,lazy_refcounts=on'
     qemu_options = 'lazy-refcounts=off'
-    compare = { 'compat': '1.1', 'lazy-refcounts': True, 'corrupt': False }
+    compare = { 'compat': '1.1', 'lazy-refcounts': True,
+                'refcount-bits': 16, 'corrupt': False }
 
 TestImageInfoSpecific = None
 TestQemuImgInfo = None
tests/qemu-iotests/067.out +5 -0
index 13ff3cd..196b59b 100644
@@ -32,6 +32,7 @@ Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,if=none,id=disk -device virti
                         "data": {
                             "compat": "1.1",
                             "lazy-refcounts": false,
+                            "refcount-bits": 16,
                             "corrupt": false
                         }
                     },
@@ -207,6 +208,7 @@ Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,if=none,id=disk
                         "data": {
                             "compat": "1.1",
                             "lazy-refcounts": false,
+                            "refcount-bits": 16,
                             "corrupt": false
                         }
                     },
@@ -412,6 +414,7 @@ Testing:
                         "data": {
                             "compat": "1.1",
                             "lazy-refcounts": false,
+                            "refcount-bits": 16,
                             "corrupt": false
                         }
                     },
@@ -596,6 +599,7 @@ Testing:
                         "data": {
                             "compat": "1.1",
                             "lazy-refcounts": false,
+                            "refcount-bits": 16,
                             "corrupt": false
                         }
                     },
@@ -706,6 +710,7 @@ Testing:
                         "data": {
                             "compat": "1.1",
                             "lazy-refcounts": false,
+                            "refcount-bits": 16,
                             "corrupt": false
                         }
                     },
tests/qemu-iotests/082.out +7 -0
index d0234e6..38729dc 100644
@@ -18,6 +18,7 @@ cluster_size: 4096
 Format specific information:
     compat: 1.1
     lazy refcounts: true
+    refcount bits: 16
     corrupt: false
 
 Testing: create -f qcow2 -o cluster_size=4k -o lazy_refcounts=on -o cluster_size=8k TEST_DIR/t.qcow2 128M
@@ -29,6 +30,7 @@ cluster_size: 8192
 Format specific information:
     compat: 1.1
     lazy refcounts: true
+    refcount bits: 16
     corrupt: false
 
 Testing: create -f qcow2 -o cluster_size=4k,cluster_size=8k TEST_DIR/t.qcow2 128M
@@ -190,6 +192,7 @@ cluster_size: 4096
 Format specific information:
     compat: 1.1
     lazy refcounts: true
+    refcount bits: 16
     corrupt: false
 
 Testing: convert -O qcow2 -o cluster_size=4k -o lazy_refcounts=on -o cluster_size=8k TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
@@ -200,6 +203,7 @@ cluster_size: 8192
 Format specific information:
     compat: 1.1
     lazy refcounts: true
+    refcount bits: 16
     corrupt: false
 
 Testing: convert -O qcow2 -o cluster_size=4k,cluster_size=8k TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
@@ -346,6 +350,7 @@ cluster_size: 65536
 Format specific information:
     compat: 1.1
     lazy refcounts: true
+    refcount bits: 16
     corrupt: false
 
 Testing: amend -f qcow2 -o size=130M -o lazy_refcounts=off TEST_DIR/t.qcow2
@@ -356,6 +361,7 @@ cluster_size: 65536
 Format specific information:
     compat: 1.1
     lazy refcounts: false
+    refcount bits: 16
     corrupt: false
 
 Testing: amend -f qcow2 -o size=8M -o lazy_refcounts=on -o size=132M TEST_DIR/t.qcow2
@@ -366,6 +372,7 @@ cluster_size: 65536
 Format specific information:
     compat: 1.1
     lazy refcounts: true
+    refcount bits: 16
     corrupt: false
 
 Testing: amend -f qcow2 -o size=4M,size=148M TEST_DIR/t.qcow2
tests/qemu-iotests/089.out +2 -0
index a6d3711..5b541a3 100644
@@ -43,6 +43,7 @@ vm state offset: 512 MiB
 Format specific information:
     compat: 1.1
     lazy refcounts: false
+    refcount bits: 16
     corrupt: false
 format name: IMGFMT
 cluster size: 64 KiB
@@ -50,5 +51,6 @@ vm state offset: 512 MiB
 Format specific information:
     compat: 1.1
     lazy refcounts: false
+    refcount bits: 16
     corrupt: false
 *** done
Signature · 1 line
1.9.3
author
#3 lore raw

PATCH v5 3/26 qcow2: Do not return new value after refcount update

In reply to Max Reitz

qcow2_update_cluster_refcount() does not have any quick access to the new refcount value, it has to call qcow2_get_refcount(). Some callers do not need that new value at all, others call qcow2_get_refcount() themselves anyway (albeit in a different code path, which can however be easily changed), therefore there is no advantage in making qcow2_update_cluster_refcount() return the new value. Drop it.

---
 block/qcow2-refcount.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)
block/qcow2-refcount.c +15 -10
index 6016211..7556384 100644
@@ -631,8 +631,7 @@ fail:
 /*
  * Increases or decreases the refcount of a given cluster.
  *
- * If the return value is non-negative, it is the new refcount of the cluster.
- * If it is negative, it is -errno and indicates an error.
+ * On success 0 is returned; on failure -errno is returned.
  */
 int qcow2_update_cluster_refcount(BlockDriverState *bs,
                                   int64_t cluster_index,
@@ -648,7 +647,7 @@ int qcow2_update_cluster_refcount(BlockDriverState *bs,
         return ret;
     }
 
-    return qcow2_get_refcount(bs, cluster_index);
+    return 0;
 }
 
 
@@ -976,13 +975,15 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
                             break;
                         }
                         if (addend != 0) {
-                            refcount = qcow2_update_cluster_refcount(bs,
+                            ret = qcow2_update_cluster_refcount(bs,
                                     cluster_index, addend,
                                     QCOW2_DISCARD_SNAPSHOT);
-                        } else {
-                            refcount = qcow2_get_refcount(bs, cluster_index);
+                            if (ret < 0) {
+                                goto fail;
+                            }
                         }
 
+                        refcount = qcow2_get_refcount(bs, cluster_index);
                         if (refcount < 0) {
                             ret = refcount;
                             goto fail;
@@ -1017,11 +1018,15 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
 
 
             if (addend != 0) {
-                refcount = qcow2_update_cluster_refcount(bs, l2_offset >>
-                        s->cluster_bits, addend, QCOW2_DISCARD_SNAPSHOT);
-            } else {
-                refcount = qcow2_get_refcount(bs, l2_offset >> s->cluster_bits);
+                ret = qcow2_update_cluster_refcount(bs, l2_offset >>
+                                                        s->cluster_bits,
+                                                    addend,
+                                                    QCOW2_DISCARD_SNAPSHOT);
+                if (ret < 0) {
+                    goto fail;
+                }
             }
+            refcount = qcow2_get_refcount(bs, l2_offset >> s->cluster_bits);
             if (refcount < 0) {
                 ret = refcount;
                 goto fail;
Signature · 1 line
1.9.3
author
#4 lore raw

PATCH v5 4/26 qcow2: Only return status from qcow2_get_refcount

In reply to Max Reitz

Refcounts can theoretically be of type uint64_t; in order to be able to represent the full range, qcow2_get_refcount() cannot use a single variable to represent both all refcount values and also keep some values reserved for errors.

One solution would be to add an Error pointer parameter to qcow2_get_refcount(); however, no caller could (currently) pass that error message, so it would have to be emitted immediately and be passed to the next caller by returning -EIO or something similar. Therefore, an Error parameter does not offer any advantages here.

The solution applied by this patch is simpler to use. Because no caller would be able to pass the error message, they would have to print it and free it, whereas with this patch the caller only needs to pass the returned integer (which is often a no-op from the code perspective, because that integer will be stored in a variable "ret" which will be returned by the fail path of many callers).

---
 block/qcow2-cluster.c  |  8 ++---
 block/qcow2-refcount.c | 79 +++++++++++++++++++++++++++-----------------------
 block/qcow2.h          |  3 +-
 3 files changed, 49 insertions(+), 41 deletions(-)
block/qcow2-cluster.c +4 -4
index 1fea514..133aad3 100644
@@ -1640,7 +1640,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
     for (i = 0; i < l1_size; i++) {
         uint64_t l2_offset = l1_table[i] & L1E_OFFSET_MASK;
         bool l2_dirty = false;
-        int l2_refcount;
+        uint16_t l2_refcount;
 
         if (!l2_offset) {
             /* unallocated */
@@ -1664,9 +1664,9 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
             goto fail;
         }
 
-        l2_refcount = qcow2_get_refcount(bs, l2_offset >> s->cluster_bits);
-        if (l2_refcount < 0) {
-            ret = l2_refcount;
+        ret = qcow2_get_refcount(bs, l2_offset >> s->cluster_bits,
+                                 &l2_refcount);
+        if (ret < 0) {
             goto fail;
         }
 
block/qcow2-refcount.c +43 -36

185 lines not on this page · raw message

block/qcow2.h +2 -1
index 4d8c902..1e59277 100644
@@ -489,7 +489,8 @@ void qcow2_signal_corruption(BlockDriverState *bs, bool fatal, int64_t offset,
 int qcow2_refcount_init(BlockDriverState *bs);
 void qcow2_refcount_close(BlockDriverState *bs);
 
-int qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index);
+int qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index,
+                       uint16_t *refcount);
 
 int qcow2_update_cluster_refcount(BlockDriverState *bs, int64_t cluster_index,
                                   int addend, enum qcow2_discard_type type);
Signature · 1 line
1.9.3
author
#5 lore raw

PATCH v5 5/26 qcow2: Use unsigned addend for update_refcount()

In reply to Max Reitz

update_refcount() and qcow2_update_cluster_refcount() currently take a signed addend. At least one caller passes a value directly derived from an absolute refcount that should be reached ("l2_refcount - 1" in expand_zero_clusters_in_l1()). Therefore, the addend should be unsigned because unsigned overflow is well-defined in contrast to signed overflow. This will be especially important for 64 bit refcounts.

Because update_refcount() then no longer knows whether the refcount should be increased or decreased (which is important for setting the refblock-L2-table cache dependency and for overflow/underflow checks), it now requires an additional flag which specified exactly that. The same applies to qcow2_update_cluster_refcount().

---
 block/qcow2-cluster.c  |  2 +-
 block/qcow2-refcount.c | 65 ++++++++++++++++++++++++++++++++------------------
 block/qcow2.h          |  3 ++-
 3 files changed, 45 insertions(+), 25 deletions(-)
block/qcow2-cluster.c +1 -1
index 133aad3..b664d2f 100644
@@ -1699,7 +1699,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
                     /* For shared L2 tables, set the refcount accordingly (it is
                      * already 1 and needs to be l2_refcount) */
                     ret = qcow2_update_cluster_refcount(bs,
-                            offset >> s->cluster_bits, l2_refcount - 1,
+                            offset >> s->cluster_bits, l2_refcount - 1, false,
                             QCOW2_DISCARD_OTHER);
                     if (ret < 0) {
                         qcow2_free_clusters(bs, offset, s->cluster_size,
block/qcow2-refcount.c +42 -23

187 lines not on this page · raw message

block/qcow2.h +2 -1
index 1e59277..0240ee8 100644
@@ -493,7 +493,8 @@ int qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index,
                        uint16_t *refcount);
 
 int qcow2_update_cluster_refcount(BlockDriverState *bs, int64_t cluster_index,
-                                  int addend, enum qcow2_discard_type type);
+                                  uint16_t addend, bool decrease,
+                                  enum qcow2_discard_type type);
 
 int64_t qcow2_alloc_clusters(BlockDriverState *bs, uint64_t size);
 int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,
Signature · 1 line
1.9.3
Reviewed-by on v5 5/26
#6 lore raw

Re: [Qemu-devel] [PATCH v5 05/26] qcow2: Use unsigned addend for update_refcount()

Quoted 13 lines · Max Reitz

update_refcount() and qcow2_update_cluster_refcount() currently take a signed addend. At least one caller passes a value directly derived from an absolute refcount that should be reached ("l2_refcount - 1" in expand_zero_clusters_in_l1()). Therefore, the addend should be unsigned because unsigned overflow is well-defined in contrast to signed overflow. This will be especially important for 64 bit refcounts.

Because update_refcount() then no longer knows whether the refcount should be increased or decreased (which is important for setting the refblock-L2-table cache dependency and for overflow/underflow checks), it now requires an additional flag which specified exactly that. The same applies to qcow2_update_cluster_refcount().

Quoted 7 lines
---
 block/qcow2-cluster.c  |  2 +-
 block/qcow2-refcount.c | 65 ++++++++++++++++++++++++++++++++------------------
 block/qcow2.h          |  3 ++-
 3 files changed, 45 insertions(+), 25 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#7 lore raw

PATCH v5 6/26 qcow2: Use 64 bits for refcount values

In reply to Max Reitz

Refcounts may have a width of up to 64 bits, so qemu should use the same width to represent refcount values internally.

---
 block/qcow2-cluster.c  |  2 +-
 block/qcow2-refcount.c | 46 ++++++++++++++++++++++------------------------
 block/qcow2.h          |  4 ++--
 3 files changed, 25 insertions(+), 27 deletions(-)
block/qcow2-cluster.c +1 -1
index b664d2f..a86bb5e 100644
@@ -1640,7 +1640,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
     for (i = 0; i < l1_size; i++) {
         uint64_t l2_offset = l1_table[i] & L1E_OFFSET_MASK;
         bool l2_dirty = false;
-        uint16_t l2_refcount;
+        uint64_t l2_refcount;
 
         if (!l2_offset) {
             /* unallocated */
block/qcow2-refcount.c +22 -24

164 lines not on this page · raw message

block/qcow2.h +2 -2
index 0240ee8..adf515c 100644
@@ -490,10 +490,10 @@ int qcow2_refcount_init(BlockDriverState *bs);
 void qcow2_refcount_close(BlockDriverState *bs);
 
 int qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index,
-                       uint16_t *refcount);
+                       uint64_t *refcount);
 
 int qcow2_update_cluster_refcount(BlockDriverState *bs, int64_t cluster_index,
-                                  uint16_t addend, bool decrease,
+                                  uint64_t addend, bool decrease,
                                   enum qcow2_discard_type type);
 
 int64_t qcow2_alloc_clusters(BlockDriverState *bs, uint64_t size);
Signature · 1 line
1.9.3
Reviewed-by on v5 6/26
#8 lore raw

Re: [Qemu-devel] [PATCH v5 06/26] qcow2: Use 64 bits for refcount values

Quoted 10 lines · Max Reitz

Refcounts may have a width of up to 64 bits, so qemu should use the same width to represent refcount values internally.

---
 block/qcow2-cluster.c  |  2 +-
 block/qcow2-refcount.c | 46 ++++++++++++++++++++++------------------------
 block/qcow2.h          |  4 ++--
 3 files changed, 25 insertions(+), 27 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#9 lore raw

Re: [Qemu-devel] [PATCH v5 06/26] qcow2: Use 64 bits for refcount values

In reply to Max Reitz

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

block/qcow2-refcount.c int qcow2_update_snapshot_refcount(BlockDriverState *bs, · line 896

Quoted 13 lines

Refcounts may have a width of up to 64 bits, so qemu should use the same width to represent refcount values internally.

---
 block/qcow2-cluster.c  |  2 +-
 block/qcow2-refcount.c | 46 ++++++++++++++++++++++------------------------
 block/qcow2.h          |  4 ++--
 3 files changed, 25 insertions(+), 27 deletions(-)
block/qcow2-refcount.c +0 -0
@@ -897,11 +895,10 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
     int64_t l1_table_offset, int l1_size, int addend)
 {

Your leaving addend an int here...

Quoted 29 lines
block/qcow2-refcount.c +3 -4
     BDRVQcowState *s = bs->opaque;
-    uint64_t *l1_table, *l2_table, l2_offset, offset, l1_size2;
+    uint64_t *l1_table, *l2_table, l2_offset, offset, l1_size2, refcount;
     bool l1_allocated = false;
     int64_t old_offset, old_l2_offset;
     int i, j, l1_modified = 0, nb_csectors;
-    uint16_t refcount;
     int ret;
 
     l2_table = NULL;
@@ -968,7 +965,7 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
                         if (addend != 0) {
                             ret = update_refcount(bs,
                                 (offset & s->cluster_offset_mask) & ~511,
-                                nb_csectors * 512, abs(addend), addend < 0,
+                                nb_csectors * 512, imaxabs(addend), addend < 0,
                                 QCOW2_DISCARD_SNAPSHOT);
                             if (ret < 0) {
                                 goto fail;
@@ -999,7 +996,7 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
                         }
                         if (addend != 0) {
                             ret = qcow2_update_cluster_refcount(bs,
-                                    cluster_index, abs(addend), addend < 0,
+                                    cluster_index, imaxabs(addend), addend < 0,
                                     QCOW2_DISCARD_SNAPSHOT);
                             if (ret < 0) {
                                 goto fail;

block/qcow2-refcount.c int qcow2_update_snapshot_refcount(BlockDriverState *bs, · line 1045

Quoted 8 lines
block/qcow2-refcount.c +1 -1
@@ -1042,7 +1039,7 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
             if (addend != 0) {
                 ret = qcow2_update_cluster_refcount(bs, l2_offset >>
                                                         s->cluster_bits,
-                                                    abs(addend), addend < 0,
+                                                    imaxabs(addend), addend < 0,
                                                     QCOW2_DISCARD_SNAPSHOT);
                 if (ret < 0) {
                     goto fail;

...but still replace abs() by imaxabs(). Did you intend to convert addend or why this change?

Quoted 20 lines
block/qcow2-refcount.c +3 -2
@@ -1658,7 +1655,7 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
 {
     BDRVQcowState *s = bs->opaque;
     int64_t i;
-    uint16_t refcount1, refcount2;
+    uint64_t refcount1, refcount2;
     int ret;
 
     for (i = 0, *highest_cluster = 0; i < nb_clusters; i++) {
@@ -1687,7 +1684,8 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
                 num_fixed = &res->corruptions_fixed;
             }
 
-            fprintf(stderr, "%s cluster %" PRId64 " refcount=%d reference=%d\n",
+            fprintf(stderr, "%s cluster %" PRId64 " refcount=%" PRIu64
+                    " reference=%" PRIu64 "\n",
                    num_fixed != NULL     ? "Repairing" :
                    refcount1 < refcount2 ? "ERROR" :
                                            "Leaked",

block/qcow2-refcount.c static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 1697

Quoted 6 lines
block/qcow2-refcount.c +1 -1
@@ -1695,7 +1693,7 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
 
             if (num_fixed) {
                 ret = update_refcount(bs, i << s->cluster_bits, 1,
-                                      abs(refcount2 - refcount1),
+                                      imaxabs(refcount2 - refcount1),
                                       refcount1 > refcount2,

Hope I got that right. Here's my analysis:

Before: refcount{1,2} were both uint16_t. Promoted to int for the subtraction. Therefore a negative result could occur. abs() takes the absolute value and the sign is passed separately.

After: refcount{1,2} are both uint64_t. No integer promotion happens, we perform an unsigned subtraction. The separate passed sign is okay. For the absolute value, there are two cases:

1. refcount2 >= refcount1: No overflow occurs, everything fine.

2. refcount2 < refcount1: (refcount2 - refcount1) wraps around, but is still an uint64_t. imaxabs() takes an intmax_t, which is signed. The conversion is implementation defined, but let's assume the obvious one. imaxabs() has two cases again:

diff := refcount2 - refcount1 + UINT64_MAX

a. diff > INTMAX_MAX:
We get diff converted back to signed, which undoes the wraparound. The absolute value of the signed difference is:

-(refcount2 - refcount1) = refcount1 - refcount2

This is what we wanted. Good.

b. diff <= INTMAX_MAX:
diff is again converted back to signed, however its value is unchanged because diff can be represented by intmax_t. This is a positive value, so taking the absolute value changes nothing.

This is _not_ refcount1 - refcount2!

I suggest using a function that calculates the absolute value of the difference of two unsigned values the naive way with an if statement. Gets us rid of the implementation defined conversion, too.

block/qcow2-refcount.c static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 1699

Quoted 2 lines
block/qcow2-refcount.c +0 -0
                                       QCOW2_DISCARD_ALWAYS);
                 if (ret >= 0) {

Kevin

author
#10 lore raw

Re: [Qemu-devel] [PATCH v5 06/26] qcow2: Use 64 bits for refcount values

Quoted 46 lines · Kevin Wolf

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

block/qcow2-refcount.c int qcow2_update_snapshot_refcount(BlockDriverState *bs, · line 896

Quoted 12 lines

Refcounts may have a width of up to 64 bits, so qemu should use the same width to represent refcount values internally.

---
  block/qcow2-cluster.c  |  2 +-
  block/qcow2-refcount.c | 46 ++++++++++++++++++++++------------------------
  block/qcow2.h          |  4 ++--
  3 files changed, 25 insertions(+), 27 deletions(-)
block/qcow2-refcount.c +0 -0
@@ -897,11 +895,10 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
      int64_t l1_table_offset, int l1_size, int addend)
  {

Your leaving addend an int here...

Quoted 31 lines
block/qcow2-refcount.c +3 -4
      BDRVQcowState *s = bs->opaque;
-    uint64_t *l1_table, *l2_table, l2_offset, offset, l1_size2;
+    uint64_t *l1_table, *l2_table, l2_offset, offset, l1_size2, refcount;
      bool l1_allocated = false;
      int64_t old_offset, old_l2_offset;
      int i, j, l1_modified = 0, nb_csectors;
-    uint16_t refcount;
      int ret;
 
      l2_table = NULL;
@@ -968,7 +965,7 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
                          if (addend != 0) {
                              ret = update_refcount(bs,
                                  (offset & s->cluster_offset_mask) & ~511,
-                                nb_csectors * 512, abs(addend), addend < 0,
+                                nb_csectors * 512, imaxabs(addend), addend < 0,
                                  QCOW2_DISCARD_SNAPSHOT);
                              if (ret < 0) {
                                  goto fail;
@@ -999,7 +996,7 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
                          }
                          if (addend != 0) {
                              ret = qcow2_update_cluster_refcount(bs,
-                                    cluster_index, abs(addend), addend < 0,
+                                    cluster_index, imaxabs(addend), addend < 0,
                                      QCOW2_DISCARD_SNAPSHOT);
                              if (ret < 0) {
                                  goto fail;
@@ -1042,7 +1039,7 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
              if (addend != 0) {
                  ret = qcow2_update_cluster_refcount(bs, l2_offset >>
Quoted 8 lines

block/qcow2-refcount.c int qcow2_update_snapshot_refcount(BlockDriverState *bs, · line 1045

Quoted 6 lines
block/qcow2-refcount.c +1 -1
                                                          s->cluster_bits,
-                                                    abs(addend), addend < 0,
+                                                    imaxabs(addend), addend < 0,
                                                      QCOW2_DISCARD_SNAPSHOT);
                  if (ret < 0) {
                      goto fail;

...but still replace abs() by imaxabs(). Did you intend to convert addend or why this change?

Mechanical replacement of every abs(addend) most likely.

Considering that qcow2_update_snapshot_refcount() is only called with @addend \in { -1, 0, 1 }, it doesn't seem to make any technical sense to convert @addend to something else than an int; and thus it doesn't make any sense to use imaxabs() instead of abs() (although it doesn't hurt, it just looks bad). Also, if I were to convert qcow2_update_snapshot_refcount() to the "full 64 bit difference interface", I'd need an additional argument for the sign of the addend.

Therefore, I'll drop the imaxabs() hunks for qcow2_update_snapshot_refcount(), if you're fine with that.

Quoted 36 lines
Quoted 20 lines
block/qcow2-refcount.c +3 -2
@@ -1658,7 +1655,7 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
  {
      BDRVQcowState *s = bs->opaque;
      int64_t i;
-    uint16_t refcount1, refcount2;
+    uint64_t refcount1, refcount2;
      int ret;
 
      for (i = 0, *highest_cluster = 0; i < nb_clusters; i++) {
@@ -1687,7 +1684,8 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
                  num_fixed = &res->corruptions_fixed;
              }
 
-            fprintf(stderr, "%s cluster %" PRId64 " refcount=%d reference=%d\n",
+            fprintf(stderr, "%s cluster %" PRId64 " refcount=%" PRIu64
+                    " reference=%" PRIu64 "\n",
                     num_fixed != NULL     ? "Repairing" :
                     refcount1 < refcount2 ? "ERROR" :
                                             "Leaked",

block/qcow2-refcount.c static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 1697

Quoted 6 lines
block/qcow2-refcount.c +1 -1
@@ -1695,7 +1693,7 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
 
              if (num_fixed) {
                  ret = update_refcount(bs, i << s->cluster_bits, 1,
-                                      abs(refcount2 - refcount1),
+                                      imaxabs(refcount2 - refcount1),
                                        refcount1 > refcount2,

Hope I got that right. Here's my analysis:

Before: refcount{1,2} were both uint16_t. Promoted to int for the subtraction. Therefore a negative result could occur. abs() takes the absolute value and the sign is passed separately.

After: refcount{1,2} are both uint64_t. No integer promotion happens, we perform an unsigned subtraction. The separate passed sign is okay. For the absolute value, there are two cases:

Quoted 8 lines

1. refcount2 >= refcount1: No overflow occurs, everything fine.

2. refcount2 < refcount1: (refcount2 - refcount1) wraps around, but is still an uint64_t. imaxabs() takes an intmax_t, which is signed. The conversion is implementation defined, but let's assume the obvious one. imaxabs() has two cases again:

diff := refcount2 - refcount1 + UINT64_MAX

Actually it's + UINT64_MAX + 1, but it doesn't matter for the point you're making.

Quoted 6 lines

a. diff > INTMAX_MAX:
We get diff converted back to signed, which undoes the wraparound. The absolute value of the signed difference is:

-(refcount2 - refcount1) = refcount1 - refcount2

Quoted 8 lines

This is what we wanted. Good.

b. diff <= INTMAX_MAX:
diff is again converted back to signed, however its value is unchanged because diff can be represented by intmax_t. This is a positive value, so taking the absolute value changes nothing.

This is _not_ refcount1 - refcount2!

You're completely right. Actually, I won absolutely nothing by separating the sign if using imaxabs() because the latter will only return values in 0 .. 2^63 - 1, which makes it (using imaxabs()) a very bad idea in the first place.

Quoted 3 lines

I suggest using a function that calculates the absolute value of the difference of two unsigned values the naive way with an if statement. Gets us rid of the implementation defined conversion, too.

Indeed, will do. Thanks!

Max

Quoted 3 lines

block/qcow2-refcount.c static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 1699

Quoted 2 lines
block/qcow2-refcount.c +0 -0
                                        QCOW2_DISCARD_ALWAYS);
                  if (ret >= 0) {

Kevin

author
#11 lore raw

PATCH v5 7/26 qcow2: Respect error in qcow2_alloc_bytes()

In reply to Max Reitz

qcow2_update_cluster_refcount() may fail, and qcow2_alloc_bytes() should mind that case.

---
 block/qcow2-refcount.c | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)
block/qcow2-refcount.c +21 -12
index 0308a7e..db81647 100644
@@ -778,8 +778,8 @@ int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,
 int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
 {
     BDRVQcowState *s = bs->opaque;
-    int64_t offset, cluster_offset;
-    int free_in_cluster;
+    int64_t offset, cluster_offset, new_cluster;
+    int free_in_cluster, ret;
 
     BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_BYTES);
     assert(size > 0 && size <= s->cluster_size);
@@ -800,23 +800,32 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
         free_in_cluster -= size;
         if (free_in_cluster == 0)
             s->free_byte_offset = 0;
-        if (offset_into_cluster(s, offset) != 0)
-            qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits, 1,
-                                          false, QCOW2_DISCARD_NEVER);
+        if (offset_into_cluster(s, offset) != 0) {
+            ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
+                                                1, false, QCOW2_DISCARD_NEVER);
+            if (ret < 0) {
+                return ret;
+            }
+        }
     } else {
-        offset = qcow2_alloc_clusters(bs, s->cluster_size);
-        if (offset < 0) {
-            return offset;
+        new_cluster = qcow2_alloc_clusters(bs, s->cluster_size);
+        if (new_cluster < 0) {
+            return new_cluster;
         }
         cluster_offset = start_of_cluster(s, s->free_byte_offset);
-        if ((cluster_offset + s->cluster_size) == offset) {
+        if ((cluster_offset + s->cluster_size) == new_cluster) {
             /* we are lucky: contiguous data */
             offset = s->free_byte_offset;
-            qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits, 1,
-                                          false, QCOW2_DISCARD_NEVER);
+            ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
+                                                1, false, QCOW2_DISCARD_NEVER);
+            if (ret < 0) {
+                qcow2_free_clusters(bs, new_cluster, s->cluster_size,
+                                    QCOW2_DISCARD_NEVER);
+                return ret;
+            }
             s->free_byte_offset += size;
         } else {
-            s->free_byte_offset = offset;
+            s->free_byte_offset = new_cluster;
             goto redo;
         }
     }
Signature · 1 line
1.9.3
#12 lore raw

Re: [Qemu-devel] [PATCH v5 07/26] qcow2: Respect error in qcow2_alloc_bytes()

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 27 lines

qcow2_update_cluster_refcount() may fail, and qcow2_alloc_bytes() should mind that case.

---
 block/qcow2-refcount.c | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)
block/qcow2-refcount.c +2 -2
index 0308a7e..db81647 100644
@@ -778,8 +778,8 @@ int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,
 int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
 {
     BDRVQcowState *s = bs->opaque;
-    int64_t offset, cluster_offset;
-    int free_in_cluster;
+    int64_t offset, cluster_offset, new_cluster;
+    int free_in_cluster, ret;
 
     BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_BYTES);
     assert(size > 0 && size <= s->cluster_size);
@@ -800,23 +800,32 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
         free_in_cluster -= size;

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 807

Quoted 10 lines
block/qcow2-refcount.c +5 -3
         if (free_in_cluster == 0)
             s->free_byte_offset = 0;
-        if (offset_into_cluster(s, offset) != 0)
-            qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits, 1,
-                                          false, QCOW2_DISCARD_NEVER);
+        if (offset_into_cluster(s, offset) != 0) {
+            ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
+                                                1, false, QCOW2_DISCARD_NEVER);
+            if (ret < 0) {
+                return ret;

Not sure how relevant it is, but s->free_byte_offset has already been increased, so we're leaving sub-cluster space unused. (It's not really leaking as freeing all references still frees the cluster.)

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 814

Quoted 10 lines
block/qcow2-refcount.c +5 -3
+            }
+        }
     } else {
-        offset = qcow2_alloc_clusters(bs, s->cluster_size);
-        if (offset < 0) {
-            return offset;
+        new_cluster = qcow2_alloc_clusters(bs, s->cluster_size);
+        if (new_cluster < 0) {
+            return new_cluster;
         }

offset is the return value of this function, and now there are cases where it isn't set to new_cluster any more (I wonder why gcc doesn't warn).

Why can't we keep offset where it was used and only assign new_cluster additionally so we can do the cleanup?

Quoted 11 lines
block/qcow2-refcount.c +5 -3
         cluster_offset = start_of_cluster(s, s->free_byte_offset);
-        if ((cluster_offset + s->cluster_size) == offset) {
+        if ((cluster_offset + s->cluster_size) == new_cluster) {
             /* we are lucky: contiguous data */
             offset = s->free_byte_offset;
-            qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits, 1,
-                                          false, QCOW2_DISCARD_NEVER);
+            ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
+                                                1, false, QCOW2_DISCARD_NEVER);
+            if (ret < 0) {
+                qcow2_free_clusters(bs, new_cluster, s->cluster_size,

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 831

Quoted 10 lines
block/qcow2-refcount.c +4 -1
+                                    QCOW2_DISCARD_NEVER);
+                return ret;
+            }
             s->free_byte_offset += size;
         } else {
-            s->free_byte_offset = offset;
+            s->free_byte_offset = new_cluster;
             goto redo;
         }
     }

Kevin

author
#13 lore raw

Re: [Qemu-devel] [PATCH v5 07/26] qcow2: Respect error in qcow2_alloc_bytes()

Quoted 33 lines · Kevin Wolf

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 32 lines

qcow2_update_cluster_refcount() may fail, and qcow2_alloc_bytes() should mind that case.

---
  block/qcow2-refcount.c | 33 +++++++++++++++++++++------------
  1 file changed, 21 insertions(+), 12 deletions(-)
block/qcow2-refcount.c +2 -5
index 0308a7e..db81647 100644
@@ -778,8 +778,8 @@ int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,
  int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
  {
      BDRVQcowState *s = bs->opaque;
-    int64_t offset, cluster_offset;
-    int free_in_cluster;
+    int64_t offset, cluster_offset, new_cluster;
+    int free_in_cluster, ret;
 
      BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_BYTES);
      assert(size > 0 && size <= s->cluster_size);
@@ -800,23 +800,32 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
          free_in_cluster -= size;
          if (free_in_cluster == 0)
              s->free_byte_offset = 0;
-        if (offset_into_cluster(s, offset) != 0)
-            qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits, 1,
-                                          false, QCOW2_DISCARD_NEVER);
Quoted 8 lines

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 807

Quoted 5 lines
block/qcow2-refcount.c +5 -0
+        if (offset_into_cluster(s, offset) != 0) {
+            ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
+                                                1, false, QCOW2_DISCARD_NEVER);
+            if (ret < 0) {
+                return ret;

Not sure how relevant it is, but s->free_byte_offset has already been increased, so we're leaving sub-cluster space unused. (It's not really leaking as freeing all references still frees the cluster.)

Right, will fix.

Quoted 10 lines

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 814

Quoted 10 lines
block/qcow2-refcount.c +5 -3
+            }
+        }
      } else {
-        offset = qcow2_alloc_clusters(bs, s->cluster_size);
-        if (offset < 0) {
-            return offset;
+        new_cluster = qcow2_alloc_clusters(bs, s->cluster_size);
+        if (new_cluster < 0) {
+            return new_cluster;
          }
Quoted 3 lines

offset is the return value of this function, and now there are cases where it isn't set to new_cluster any more (I wonder why gcc doesn't warn).

Because @offset is always set. In case the next condition is true, it is set to s->free_byte_offset, just like it was before. In case it isn't, s->free_byte_offset will be set to @new_cluster and the loop will be started again (probably always resulting in size <= free_in_cluster being true and thus @offset being set to s->free_byte_offset).

Quoted 2 lines

Why can't we keep offset where it was used and only assign new_cluster additionally so we can do the cleanup?

Why should we? If I were a reader of this code, I think it would confuse me to have two variables holding the same value but for some reason only using @offset, but once it's been overwritten, suddenly using @new_cluster (whereas one could just use @new_cluster everywhere, and then writing @offset would be superfluous). Also, the idea was having @offset hold the offset of where the compressed data is to be stored, whereas @new_cluster is just the offset of a new cluster, but not necessarily the offset where compressed data will be stored (it won't be if (cluster_offset + s->cluster_size) == new_cluster).

Max

Quoted 22 lines
Quoted 11 lines
block/qcow2-refcount.c +5 -3
          cluster_offset = start_of_cluster(s, s->free_byte_offset);
-        if ((cluster_offset + s->cluster_size) == offset) {
+        if ((cluster_offset + s->cluster_size) == new_cluster) {
              /* we are lucky: contiguous data */
              offset = s->free_byte_offset;
-            qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits, 1,
-                                          false, QCOW2_DISCARD_NEVER);
+            ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
+                                                1, false, QCOW2_DISCARD_NEVER);
+            if (ret < 0) {
+                qcow2_free_clusters(bs, new_cluster, s->cluster_size,

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 831

Quoted 10 lines
block/qcow2-refcount.c +4 -1
+                                    QCOW2_DISCARD_NEVER);
+                return ret;
+            }
              s->free_byte_offset += size;
          } else {
-            s->free_byte_offset = offset;
+            s->free_byte_offset = new_cluster;
              goto redo;
          }
      }

Kevin

#14 lore raw

Re: [Qemu-devel] [PATCH v5 07/26] qcow2: Respect error in qcow2_alloc_bytes()

Am 04.02.2015 um 16:04 hat Max Reitz geschrieben:

Quoted 58 lines · Kevin Wolf
Quoted 32 lines

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 31 lines

qcow2_update_cluster_refcount() may fail, and qcow2_alloc_bytes() should mind that case.

---
 block/qcow2-refcount.c | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)
block/qcow2-refcount.c +2 -5
index 0308a7e..db81647 100644
@@ -778,8 +778,8 @@ int qcow2_alloc_clusters_at(BlockDriverState *bs, uint64_t offset,
 int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
 {
     BDRVQcowState *s = bs->opaque;
-    int64_t offset, cluster_offset;
-    int free_in_cluster;
+    int64_t offset, cluster_offset, new_cluster;
+    int free_in_cluster, ret;
     BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_BYTES);
     assert(size > 0 && size <= s->cluster_size);
@@ -800,23 +800,32 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
         free_in_cluster -= size;
         if (free_in_cluster == 0)
             s->free_byte_offset = 0;
-        if (offset_into_cluster(s, offset) != 0)
-            qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits, 1,
-                                          false, QCOW2_DISCARD_NEVER);
Quoted 8 lines

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 807

Quoted 5 lines
block/qcow2-refcount.c +5 -0
+        if (offset_into_cluster(s, offset) != 0) {
+            ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
+                                                1, false, QCOW2_DISCARD_NEVER);
+            if (ret < 0) {
+                return ret;

Not sure how relevant it is, but s->free_byte_offset has already been increased, so we're leaving sub-cluster space unused. (It's not really leaking as freeing all references still frees the cluster.)

Right, will fix.

Quoted 10 lines

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 814

Quoted 10 lines
block/qcow2-refcount.c +5 -3
+            }
+        }
     } else {
-        offset = qcow2_alloc_clusters(bs, s->cluster_size);
-        if (offset < 0) {
-            return offset;
+        new_cluster = qcow2_alloc_clusters(bs, s->cluster_size);
+        if (new_cluster < 0) {
+            return new_cluster;
         }
Quoted 3 lines

offset is the return value of this function, and now there are cases where it isn't set to new_cluster any more (I wonder why gcc doesn't warn).

Quoted 6 lines

Because @offset is always set. In case the next condition is true, it is set to s->free_byte_offset, just like it was before. In case it isn't, s->free_byte_offset will be set to @new_cluster and the loop will be started again (probably always resulting in size <= free_in_cluster being true and thus @offset being set to s->free_byte_offset).

Yes, I misread. Hooray for goto. One more reason for a cleanup of the whole function.

Kevin

author
#15 lore raw

PATCH v5 8/26 qcow2: Refcount overflow and qcow2_alloc_bytes()

In reply to Max Reitz

qcow2_alloc_bytes() may reuse a cluster multiple times, in which case the refcount is increased accordingly. However, if this would lead to an overflow the function should instead just not reuse this cluster and allocate a new one.

---
 block/qcow2-refcount.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +30 -1
index db81647..fd28a13 100644
@@ -780,9 +780,11 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
     BDRVQcowState *s = bs->opaque;
     int64_t offset, cluster_offset, new_cluster;
     int free_in_cluster, ret;
+    uint64_t refcount;
 
     BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_BYTES);
     assert(size > 0 && size <= s->cluster_size);
+ redo:
     if (s->free_byte_offset == 0) {
         offset = qcow2_alloc_clusters(bs, s->cluster_size);
         if (offset < 0) {
@@ -790,12 +792,25 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
         }
         s->free_byte_offset = offset;
     }
- redo:
+
     free_in_cluster = s->cluster_size -
         offset_into_cluster(s, s->free_byte_offset);
     if (size <= free_in_cluster) {
         /* enough space in current cluster */
         offset = s->free_byte_offset;
+
+        if (offset_into_cluster(s, offset) != 0) {
+            /* We will have to increase the refcount of this cluster; if the
+             * maximum has been reached already, this cluster cannot be used */
+            ret = qcow2_get_refcount(bs, offset >> s->cluster_bits, &refcount);
+            if (ret < 0) {
+                return ret;
+            } else if (refcount == s->refcount_max) {
+                s->free_byte_offset = 0;
+                goto redo;
+            }
+        }
+
         s->free_byte_offset += size;
         free_in_cluster -= size;
         if (free_in_cluster == 0)
@@ -816,6 +831,20 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
         if ((cluster_offset + s->cluster_size) == new_cluster) {
             /* we are lucky: contiguous data */
             offset = s->free_byte_offset;
+
+            /* Same as above: In order to reuse the cluster, the refcount has to
+             * be increased; if that will not work, we are not so lucky after
+             * all */
+            ret = qcow2_get_refcount(bs, offset >> s->cluster_bits, &refcount);
+            if (ret < 0) {
+                qcow2_free_clusters(bs, new_cluster, s->cluster_size,
+                                    QCOW2_DISCARD_NEVER);
+                return ret;
+            } else if (refcount == s->refcount_max) {
+                s->free_byte_offset = offset;
+                goto redo;
+            }
+
             ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
                                                 1, false, QCOW2_DISCARD_NEVER);
             if (ret < 0) {
Signature · 1 line
1.9.3
#16 lore raw

Re: [Qemu-devel] [PATCH v5 08/26] qcow2: Refcount overflow and qcow2_alloc_bytes()

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 60 lines

qcow2_alloc_bytes() may reuse a cluster multiple times, in which case the refcount is increased accordingly. However, if this would lead to an overflow the function should instead just not reuse this cluster and allocate a new one.

---
 block/qcow2-refcount.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +17 -1
index db81647..fd28a13 100644
@@ -780,9 +780,11 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
     BDRVQcowState *s = bs->opaque;
     int64_t offset, cluster_offset, new_cluster;
     int free_in_cluster, ret;
+    uint64_t refcount;
 
     BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_BYTES);
     assert(size > 0 && size <= s->cluster_size);
+ redo:
     if (s->free_byte_offset == 0) {
         offset = qcow2_alloc_clusters(bs, s->cluster_size);
         if (offset < 0) {
@@ -790,12 +792,25 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
         }
         s->free_byte_offset = offset;
     }
- redo:
+
     free_in_cluster = s->cluster_size -
         offset_into_cluster(s, s->free_byte_offset);
     if (size <= free_in_cluster) {
         /* enough space in current cluster */
         offset = s->free_byte_offset;
+
+        if (offset_into_cluster(s, offset) != 0) {
+            /* We will have to increase the refcount of this cluster; if the
+             * maximum has been reached already, this cluster cannot be used */
+            ret = qcow2_get_refcount(bs, offset >> s->cluster_bits, &refcount);
+            if (ret < 0) {
+                return ret;
+            } else if (refcount == s->refcount_max) {
+                s->free_byte_offset = 0;
+                goto redo;
+            }
+        }
+
         s->free_byte_offset += size;
         free_in_cluster -= size;
         if (free_in_cluster == 0)
@@ -816,6 +831,20 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
         if ((cluster_offset + s->cluster_size) == new_cluster) {
             /* we are lucky: contiguous data */
             offset = s->free_byte_offset;
+

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 844

Quoted 10 lines
block/qcow2-refcount.c +10 -0
+            /* Same as above: In order to reuse the cluster, the refcount has to
+             * be increased; if that will not work, we are not so lucky after
+             * all */
+            ret = qcow2_get_refcount(bs, offset >> s->cluster_bits, &refcount);
+            if (ret < 0) {
+                qcow2_free_clusters(bs, new_cluster, s->cluster_size,
+                                    QCOW2_DISCARD_NEVER);
+                return ret;
+            } else if (refcount == s->refcount_max) {
+                s->free_byte_offset = offset;

I think you mean 0. offset is already the old value.

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 850

Quoted 6 lines
block/qcow2-refcount.c +3 -0
+                goto redo;
+            }
+
             ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
                                                 1, false, QCOW2_DISCARD_NEVER);
             if (ret < 0) {

I wonder if the code duplication is necessary. I was already thinking that there was some duplication when I reviewed the previous patch, but now it seems to become even more obvious that the three parts of this function are:

1. Allocate a new cluster
2. Allocate space in the already allocated cluster
3. Allocate a new cluster and space inside it, which is just 1. + 2.

Kevin

author
#17 lore raw

Re: [Qemu-devel] [PATCH v5 08/26] qcow2: Refcount overflow and qcow2_alloc_bytes()

Quoted 64 lines · Kevin Wolf

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 63 lines

qcow2_alloc_bytes() may reuse a cluster multiple times, in which case the refcount is increased accordingly. However, if this would lead to an overflow the function should instead just not reuse this cluster and allocate a new one.

---
  block/qcow2-refcount.c | 31 ++++++++++++++++++++++++++++++-
  1 file changed, 30 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +20 -1
index db81647..fd28a13 100644
@@ -780,9 +780,11 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
      BDRVQcowState *s = bs->opaque;
      int64_t offset, cluster_offset, new_cluster;
      int free_in_cluster, ret;
+    uint64_t refcount;
 
      BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_BYTES);
      assert(size > 0 && size <= s->cluster_size);
+ redo:
      if (s->free_byte_offset == 0) {
          offset = qcow2_alloc_clusters(bs, s->cluster_size);
          if (offset < 0) {
@@ -790,12 +792,25 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
          }
          s->free_byte_offset = offset;
      }
- redo:
+
      free_in_cluster = s->cluster_size -
          offset_into_cluster(s, s->free_byte_offset);
      if (size <= free_in_cluster) {
          /* enough space in current cluster */
          offset = s->free_byte_offset;
+
+        if (offset_into_cluster(s, offset) != 0) {
+            /* We will have to increase the refcount of this cluster; if the
+             * maximum has been reached already, this cluster cannot be used */
+            ret = qcow2_get_refcount(bs, offset >> s->cluster_bits, &refcount);
+            if (ret < 0) {
+                return ret;
+            } else if (refcount == s->refcount_max) {
+                s->free_byte_offset = 0;
+                goto redo;
+            }
+        }
+
          s->free_byte_offset += size;
          free_in_cluster -= size;
          if (free_in_cluster == 0)
@@ -816,6 +831,20 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
          if ((cluster_offset + s->cluster_size) == new_cluster) {
              /* we are lucky: contiguous data */
              offset = s->free_byte_offset;
+
+            /* Same as above: In order to reuse the cluster, the refcount has to
+             * be increased; if that will not work, we are not so lucky after
+             * all */
Quoted 8 lines

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 844

Quoted 7 lines
block/qcow2-refcount.c +7 -0
+            ret = qcow2_get_refcount(bs, offset >> s->cluster_bits, &refcount);
+            if (ret < 0) {
+                qcow2_free_clusters(bs, new_cluster, s->cluster_size,
+                                    QCOW2_DISCARD_NEVER);
+                return ret;
+            } else if (refcount == s->refcount_max) {
+                s->free_byte_offset = offset;

I think you mean 0. offset is already the old value.

Oh, right. Thanks for catching!

Quoted 6 lines

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 850

Quoted 6 lines
block/qcow2-refcount.c +3 -0
+                goto redo;
+            }
+
              ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
                                                  1, false, QCOW2_DISCARD_NEVER);
              if (ret < 0) {
Quoted 8 lines

I wonder if the code duplication is necessary. I was already thinking that there was some duplication when I reviewed the previous patch, but now it seems to become even more obvious that the three parts of this function are:

1. Allocate a new cluster
2. Allocate space in the already allocated cluster
3. Allocate a new cluster and space inside it, which is just 1. + 2.

Well, I can try rewriting this function, but I guess the diffcount will be even higher.

I think I'll try to write an independent patch which rewrites this function and then drop these two patches from this series.

Max

#18 lore raw

Re: [Qemu-devel] [PATCH v5 08/26] qcow2: Refcount overflow and qcow2_alloc_bytes()

Am 04.02.2015 um 16:33 hat Max Reitz geschrieben:

Quoted 90 lines · Kevin Wolf
Quoted 63 lines

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 62 lines

qcow2_alloc_bytes() may reuse a cluster multiple times, in which case the refcount is increased accordingly. However, if this would lead to an overflow the function should instead just not reuse this cluster and allocate a new one.

---
 block/qcow2-refcount.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +20 -1
index db81647..fd28a13 100644
@@ -780,9 +780,11 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
     BDRVQcowState *s = bs->opaque;
     int64_t offset, cluster_offset, new_cluster;
     int free_in_cluster, ret;
+    uint64_t refcount;
     BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_BYTES);
     assert(size > 0 && size <= s->cluster_size);
+ redo:
     if (s->free_byte_offset == 0) {
         offset = qcow2_alloc_clusters(bs, s->cluster_size);
         if (offset < 0) {
@@ -790,12 +792,25 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
         }
         s->free_byte_offset = offset;
     }
- redo:
+
     free_in_cluster = s->cluster_size -
         offset_into_cluster(s, s->free_byte_offset);
     if (size <= free_in_cluster) {
         /* enough space in current cluster */
         offset = s->free_byte_offset;
+
+        if (offset_into_cluster(s, offset) != 0) {
+            /* We will have to increase the refcount of this cluster; if the
+             * maximum has been reached already, this cluster cannot be used */
+            ret = qcow2_get_refcount(bs, offset >> s->cluster_bits, &refcount);
+            if (ret < 0) {
+                return ret;
+            } else if (refcount == s->refcount_max) {
+                s->free_byte_offset = 0;
+                goto redo;
+            }
+        }
+
         s->free_byte_offset += size;
         free_in_cluster -= size;
         if (free_in_cluster == 0)
@@ -816,6 +831,20 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size)
         if ((cluster_offset + s->cluster_size) == new_cluster) {
             /* we are lucky: contiguous data */
             offset = s->free_byte_offset;
+
+            /* Same as above: In order to reuse the cluster, the refcount has to
+             * be increased; if that will not work, we are not so lucky after
+             * all */
Quoted 8 lines

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 844

Quoted 7 lines
block/qcow2-refcount.c +7 -0
+            ret = qcow2_get_refcount(bs, offset >> s->cluster_bits, &refcount);
+            if (ret < 0) {
+                qcow2_free_clusters(bs, new_cluster, s->cluster_size,
+                                    QCOW2_DISCARD_NEVER);
+                return ret;
+            } else if (refcount == s->refcount_max) {
+                s->free_byte_offset = offset;

I think you mean 0. offset is already the old value.

Oh, right. Thanks for catching!

Quoted 6 lines

block/qcow2-refcount.c int64_t qcow2_alloc_bytes(BlockDriverState *bs, int size) · line 850

Quoted 6 lines
block/qcow2-refcount.c +3 -0
+                goto redo;
+            }
+
             ret = qcow2_update_cluster_refcount(bs, offset >> s->cluster_bits,
                                                 1, false, QCOW2_DISCARD_NEVER);
             if (ret < 0) {
Quoted 8 lines

I wonder if the code duplication is necessary. I was already thinking that there was some duplication when I reviewed the previous patch, but now it seems to become even more obvious that the three parts of this function are:

1. Allocate a new cluster
2. Allocate space in the already allocated cluster
3. Allocate a new cluster and space inside it, which is just 1. + 2.

Quoted 5 lines

Well, I can try rewriting this function, but I guess the diffcount will be even higher.

I think I'll try to write an independent patch which rewrites this function and then drop these two patches from this series.

Anything that makes the series shorter is welcome. ;-)

In fact, I've been considering merging the first few patches even if they aren't valuable in themselves, just to make v6 shorter.

Kevin

author
#19 lore raw

PATCH v5 9/26 qcow2: Helper for refcount array reallocation

In reply to Max Reitz

Add a helper function for reallocating a refcount array, independent of the refcount order. The newly allocated space is zeroed and the function handles failed reallocations gracefully.

The helper function will always align the buffer size to a cluster boundary; if storing the refcounts in such an array in big endian byte order, this makes it possible to write parts of the array directly as refcount blocks into the image file.

---
 block/qcow2-refcount.c | 137 +++++++++++++++++++++++++++++++------------------
 1 file changed, 88 insertions(+), 49 deletions(-)
block/qcow2-refcount.c +88 -49
index fd28a13..4ede971 100644
@@ -1130,6 +1130,70 @@ fail:
 /* refcount checking functions */
 
 
+static size_t refcount_array_byte_size(BDRVQcowState *s, uint64_t entries)
+{
+    if (s->refcount_order < 3) {
+        /* sub-byte width */
+        int shift = 3 - s->refcount_order;
+        return DIV_ROUND_UP(entries, 1 << shift);
+    } else if (s->refcount_order == 3) {
+        /* byte width */
+        return entries;
+    } else {
+        /* multiple bytes wide */
+
+        /* This assertion holds because there is no way we can address more than
+         * 2^(64 - 9) clusters at once (with cluster size 512 = 2^9, and because
+         * offsets have to be representable in bytes); due to every cluster
+         * corresponding to one refcount entry and because refcount_order has to
+         * be below 7, we are far below that limit */
+        assert(!(entries >> (64 - (s->refcount_order - 3))));
+
+        return entries << (s->refcount_order - 3);
+    }
+}
+
+/**
+ * Reallocates *array so that it can hold new_size entries. *size must contain
+ * the current number of entries in *array. If the reallocation fails, *array
+ * and *size will not be modified and -errno will be returned. If the
+ * reallocation is successful, *array will be set to the new buffer and *size
+ * will be set to new_size. The size of the reallocated refcount array buffer
+ * will be aligned to a cluster boundary, and the newly allocated area will be
+ * zeroed.
+ */
+static int realloc_refcount_array(BDRVQcowState *s, uint16_t **array,
+                                  int64_t *size, int64_t new_size)
+{
+    /* Round to clusters so the array can be directly written to disk */
+    size_t old_byte_size = ROUND_UP(refcount_array_byte_size(s, *size),
+                                    s->cluster_size);
+    size_t new_byte_size = ROUND_UP(refcount_array_byte_size(s, new_size),
+                                    s->cluster_size);
+    uint16_t *new_ptr;
+
+    if (new_byte_size == old_byte_size) {
+        *size = new_size;
+        return 0;
+    }
+
+    assert(new_byte_size > 0);
+
+    new_ptr = g_try_realloc(*array, new_byte_size);
+    if (!new_ptr) {
+        return -ENOMEM;
+    }
+
+    if (new_byte_size > old_byte_size) {
+        memset((void *)((uintptr_t)new_ptr + old_byte_size), 0,
+               new_byte_size - old_byte_size);
+    }
+
+    *array = new_ptr;
+    *size  = new_size;
+
+    return 0;
+}
 
 /*
  * Increases the refcount for a range of clusters in a given refcount table.
@@ -1146,6 +1210,7 @@ static int inc_refcounts(BlockDriverState *bs,
 {
     BDRVQcowState *s = bs->opaque;
     uint64_t start, last, cluster_offset, k;
+    int ret;
 
     if (size <= 0) {
         return 0;
@@ -1157,23 +1222,12 @@ static int inc_refcounts(BlockDriverState *bs,
         cluster_offset += s->cluster_size) {
         k = cluster_offset >> s->cluster_bits;
         if (k >= *refcount_table_size) {
-            int64_t old_refcount_table_size = *refcount_table_size;
-            uint16_t *new_refcount_table;
-
-            *refcount_table_size = k + 1;
-            new_refcount_table = g_try_realloc(*refcount_table,
-                                               *refcount_table_size *
-                                               sizeof(**refcount_table));
-            if (!new_refcount_table) {
-                *refcount_table_size = old_refcount_table_size;
+            ret = realloc_refcount_array(s, refcount_table,
+                                         refcount_table_size, k + 1);
+            if (ret < 0) {
                 res->check_errors++;
-                return -ENOMEM;
+                return ret;
             }
-            *refcount_table = new_refcount_table;
-
-            memset(*refcount_table + old_refcount_table_size, 0,
-                   (*refcount_table_size - old_refcount_table_size) *
-                   sizeof(**refcount_table));
         }
 
         if (++(*refcount_table)[k] == 0) {
@@ -1542,8 +1596,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,
                     fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", i);
 
             if (fix & BDRV_FIX_ERRORS) {
-                int64_t old_nb_clusters = *nb_clusters;
-                uint16_t *new_refcount_table;
+                int64_t new_nb_clusters;
 
                 if (offset > INT64_MAX - s->cluster_size) {
                     ret = -EINVAL;
@@ -1560,22 +1613,15 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,
                     goto resize_fail;
                 }
 
-                *nb_clusters = size_to_clusters(s, size);
-                assert(*nb_clusters >= old_nb_clusters);
+                new_nb_clusters = size_to_clusters(s, size);
+                assert(new_nb_clusters >= *nb_clusters);
 
-                new_refcount_table = g_try_realloc(*refcount_table,
-                                                   *nb_clusters *
-                                                   sizeof(**refcount_table));
-                if (!new_refcount_table) {
-                    *nb_clusters = old_nb_clusters;
+                ret = realloc_refcount_array(s, refcount_table,
+                                             nb_clusters, new_nb_clusters);
+                if (ret < 0) {
                     res->check_errors++;
-                    return -ENOMEM;
+                    return ret;
                 }
-                *refcount_table = new_refcount_table;
-
-                memset(*refcount_table + old_nb_clusters, 0,
-                       (*nb_clusters - old_nb_clusters) *
-                       sizeof(**refcount_table));
 
                 if (cluster >= *nb_clusters) {
                     ret = -EINVAL;
@@ -1635,10 +1681,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
     int ret;
 
     if (!*refcount_table) {
-        *refcount_table = g_try_new0(uint16_t, *nb_clusters);
-        if (*nb_clusters && *refcount_table == NULL) {
+        int64_t old_size = 0;
+        ret = realloc_refcount_array(s, refcount_table,
+                                     &old_size, *nb_clusters);
+        if (ret < 0) {
             res->check_errors++;
-            return -ENOMEM;
+            return ret;
         }
     }
 
@@ -1772,6 +1820,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,
     int64_t cluster = *first_free_cluster, i;
     bool first_gap = true;
     int contiguous_free_clusters;
+    int ret;
 
     /* Starting at *first_free_cluster, find a range of at least cluster_count
      * continuously free clusters */
@@ -1801,28 +1850,18 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,
     /* If no such range could be found, grow the in-memory refcount table
      * accordingly to append free clusters at the end of the image */
     if (contiguous_free_clusters < cluster_count) {
-        int64_t old_imrt_nb_clusters = *imrt_nb_clusters;
-        uint16_t *new_refcount_table;
-
         /* contiguous_free_clusters clusters are already empty at the image end;
          * we need cluster_count clusters; therefore, we have to allocate
          * cluster_count - contiguous_free_clusters new clusters at the end of
          * the image (which is the current value of cluster; note that cluster
          * may exceed old_imrt_nb_clusters if *first_free_cluster pointed beyond
          * the image end) */
-        *imrt_nb_clusters = cluster + cluster_count - contiguous_free_clusters;
-        new_refcount_table = g_try_realloc(*refcount_table,
-                                           *imrt_nb_clusters *
-                                           sizeof(**refcount_table));
-        if (!new_refcount_table) {
-            *imrt_nb_clusters = old_imrt_nb_clusters;
-            return -ENOMEM;
-        }
-        *refcount_table = new_refcount_table;
-
-        memset(*refcount_table + old_imrt_nb_clusters, 0,
-               (*imrt_nb_clusters - old_imrt_nb_clusters) *
-               sizeof(**refcount_table));
+        ret = realloc_refcount_array(s, refcount_table, imrt_nb_clusters,
+                                     cluster + cluster_count
+                                     - contiguous_free_clusters);
+        if (ret < 0) {
+            return ret;
+        }
     }
 
     /* Go back to the first free cluster */
Signature · 1 line
1.9.3
#20 lore raw

Re: [Qemu-devel] [PATCH v5 09/26] qcow2: Helper for refcount array reallocation

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 28 lines

Add a helper function for reallocating a refcount array, independent of the refcount order. The newly allocated space is zeroed and the function handles failed reallocations gracefully.

The helper function will always align the buffer size to a cluster boundary; if storing the refcounts in such an array in big endian byte order, this makes it possible to write parts of the array directly as refcount blocks into the image file.

---
 block/qcow2-refcount.c | 137 +++++++++++++++++++++++++++++++------------------
 1 file changed, 88 insertions(+), 49 deletions(-)
block/qcow2-refcount.c +5 -0
index fd28a13..4ede971 100644
@@ -1130,6 +1130,70 @@ fail:
 /* refcount checking functions */
 
 
+static size_t refcount_array_byte_size(BDRVQcowState *s, uint64_t entries)
+{
+    if (s->refcount_order < 3) {
+        /* sub-byte width */
+        int shift = 3 - s->refcount_order;

block/qcow2-refcount.c fail: · line 1147

Quoted 10 lines
block/qcow2-refcount.c +10 -0
+        return DIV_ROUND_UP(entries, 1 << shift);
+    } else if (s->refcount_order == 3) {
+        /* byte width */
+        return entries;
+    } else {
+        /* multiple bytes wide */
+
+        /* This assertion holds because there is no way we can address more than
+         * 2^(64 - 9) clusters at once (with cluster size 512 = 2^9, and because
+         * offsets have to be representable in bytes); due to every cluster

Considering this, which implies that a multiplication by 64 can't overflow, why can't this function be as simple as the following?

    assert(entries <= (1 << (64 - 9)));
    return DIV_ROUND_UP(entries * s->refcount_bits, 8)
Quoted 4 lines
block/qcow2-refcount.c +4 -0
+         * corresponding to one refcount entry and because refcount_order has to
+         * be below 7, we are far below that limit */
+        assert(!(entries >> (64 - (s->refcount_order - 3))));
+

block/qcow2-refcount.c fail:

Quoted 10 lines
block/qcow2-refcount.c +10 -0
+        return entries << (s->refcount_order - 3);
+    }
+}
+
+/**
+ * Reallocates *array so that it can hold new_size entries. *size must contain
+ * the current number of entries in *array. If the reallocation fails, *array
+ * and *size will not be modified and -errno will be returned. If the
+ * reallocation is successful, *array will be set to the new buffer and *size
+ * will be set to new_size.

And 0 is returned.

block/qcow2-refcount.c fail: · line 1172

Quoted 12 lines

The size of the reallocated refcount array buffer
+ * will be aligned to a cluster boundary, and the newly allocated area will be
+ * zeroed.
+ */

block/qcow2-refcount.c +8 -0
+static int realloc_refcount_array(BDRVQcowState *s, uint16_t **array,
+                                  int64_t *size, int64_t new_size)
+{
+    /* Round to clusters so the array can be directly written to disk */
+    size_t old_byte_size = ROUND_UP(refcount_array_byte_size(s, *size),
+                                    s->cluster_size);
+    size_t new_byte_size = ROUND_UP(refcount_array_byte_size(s, new_size),
+                                    s->cluster_size);

size_to_clusters()? (Unfortunately still not short enough to keep each initialisation on a single line...)

block/qcow2-refcount.c fail: · line 1178

Quoted 6 lines
block/qcow2-refcount.c +6 -0
+    uint16_t *new_ptr;
+
+    if (new_byte_size == old_byte_size) {
+        *size = new_size;
+        return 0;
+    }

This is only correct if the array was previously allocated by the same function, or at least rounded up to a cluster boundary. What do we win by keeping a smaller byte count in *size than is actually allocated? If we had the real allocation size in it, we wouldn't have to make assumptions about the real array size.

Quoted 96 lines
block/qcow2-refcount.c +29 -30
+    assert(new_byte_size > 0);
+
+    new_ptr = g_try_realloc(*array, new_byte_size);
+    if (!new_ptr) {
+        return -ENOMEM;
+    }
+
+    if (new_byte_size > old_byte_size) {
+        memset((void *)((uintptr_t)new_ptr + old_byte_size), 0,
+               new_byte_size - old_byte_size);
+    }
+
+    *array = new_ptr;
+    *size  = new_size;
+
+    return 0;
+}
 
 /*
  * Increases the refcount for a range of clusters in a given refcount table.
@@ -1146,6 +1210,7 @@ static int inc_refcounts(BlockDriverState *bs,
 {
     BDRVQcowState *s = bs->opaque;
     uint64_t start, last, cluster_offset, k;
+    int ret;
 
     if (size <= 0) {
         return 0;
@@ -1157,23 +1222,12 @@ static int inc_refcounts(BlockDriverState *bs,
         cluster_offset += s->cluster_size) {
         k = cluster_offset >> s->cluster_bits;
         if (k >= *refcount_table_size) {
-            int64_t old_refcount_table_size = *refcount_table_size;
-            uint16_t *new_refcount_table;
-
-            *refcount_table_size = k + 1;
-            new_refcount_table = g_try_realloc(*refcount_table,
-                                               *refcount_table_size *
-                                               sizeof(**refcount_table));
-            if (!new_refcount_table) {
-                *refcount_table_size = old_refcount_table_size;
+            ret = realloc_refcount_array(s, refcount_table,
+                                         refcount_table_size, k + 1);
+            if (ret < 0) {
                 res->check_errors++;
-                return -ENOMEM;
+                return ret;
             }
-            *refcount_table = new_refcount_table;
-
-            memset(*refcount_table + old_refcount_table_size, 0,
-                   (*refcount_table_size - old_refcount_table_size) *
-                   sizeof(**refcount_table));
         }
 
         if (++(*refcount_table)[k] == 0) {
@@ -1542,8 +1596,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,
                     fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", i);
 
             if (fix & BDRV_FIX_ERRORS) {
-                int64_t old_nb_clusters = *nb_clusters;
-                uint16_t *new_refcount_table;
+                int64_t new_nb_clusters;
 
                 if (offset > INT64_MAX - s->cluster_size) {
                     ret = -EINVAL;
@@ -1560,22 +1613,15 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,
                     goto resize_fail;
                 }
 
-                *nb_clusters = size_to_clusters(s, size);
-                assert(*nb_clusters >= old_nb_clusters);
+                new_nb_clusters = size_to_clusters(s, size);
+                assert(new_nb_clusters >= *nb_clusters);
 
-                new_refcount_table = g_try_realloc(*refcount_table,
-                                                   *nb_clusters *
-                                                   sizeof(**refcount_table));
-                if (!new_refcount_table) {
-                    *nb_clusters = old_nb_clusters;
+                ret = realloc_refcount_array(s, refcount_table,
+                                             nb_clusters, new_nb_clusters);
+                if (ret < 0) {
                     res->check_errors++;
-                    return -ENOMEM;
+                    return ret;
                 }
-                *refcount_table = new_refcount_table;
-
-                memset(*refcount_table + old_nb_clusters, 0,
-                       (*nb_clusters - old_nb_clusters) *
-                       sizeof(**refcount_table));
 
                 if (cluster >= *nb_clusters) {
                     ret = -EINVAL;

block/qcow2-refcount.c static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 1686

Quoted 8 lines
block/qcow2-refcount.c +3 -2
@@ -1635,10 +1681,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
     int ret;
 
     if (!*refcount_table) {
-        *refcount_table = g_try_new0(uint16_t, *nb_clusters);
-        if (*nb_clusters && *refcount_table == NULL) {
+        int64_t old_size = 0;
+        ret = realloc_refcount_array(s, refcount_table,
+                                     &old_size, *nb_clusters);

Here the returned new size is thrown away.

With the implementation of realloc_refcount_array() as above this is not incorrect because it is *nb_clusters anyway when the function succeeds, but it's a bit sloppy.

If you decide to allow realloc_refcount_array() returning a size bigger than was requested (i.e. the rounded value is returned) as I suggested above, then you need to change this call.

block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1231

Quoted 6 lines
block/qcow2-refcount.c +2 -1
+        if (ret < 0) {
             res->check_errors++;
-            return -ENOMEM;
+            return ret;
         }
     }

Kevin

author
#21 lore raw

Re: [Qemu-devel] [PATCH v5 09/26] qcow2: Helper for refcount array reallocation

Quoted 42 lines · Kevin Wolf

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 28 lines

Add a helper function for reallocating a refcount array, independent of the refcount order. The newly allocated space is zeroed and the function handles failed reallocations gracefully.

The helper function will always align the buffer size to a cluster boundary; if storing the refcounts in such an array in big endian byte order, this makes it possible to write parts of the array directly as refcount blocks into the image file.

---
  block/qcow2-refcount.c | 137 +++++++++++++++++++++++++++++++------------------
  1 file changed, 88 insertions(+), 49 deletions(-)
block/qcow2-refcount.c +5 -0
index fd28a13..4ede971 100644
@@ -1130,6 +1130,70 @@ fail:
  /* refcount checking functions */
 
 
+static size_t refcount_array_byte_size(BDRVQcowState *s, uint64_t entries)
+{
+    if (s->refcount_order < 3) {
+        /* sub-byte width */
+        int shift = 3 - s->refcount_order;

block/qcow2-refcount.c fail: · line 1147

Quoted 10 lines
block/qcow2-refcount.c +10 -0
+        return DIV_ROUND_UP(entries, 1 << shift);
+    } else if (s->refcount_order == 3) {
+        /* byte width */
+        return entries;
+    } else {
+        /* multiple bytes wide */
+
+        /* This assertion holds because there is no way we can address more than
+         * 2^(64 - 9) clusters at once (with cluster size 512 = 2^9, and because
+         * offsets have to be representable in bytes); due to every cluster

Considering this, which implies that a multiplication by 64 can't overflow, why can't this function be as simple as the following?

Quoted 2 lines
     assert(entries <= (1 << (64 - 9)));
     return DIV_ROUND_UP(entries * s->refcount_bits, 8)

Because that was too simple for me to think of.

I'm not sure whether Eric said something about this before, but I think he said he liked separating the cases...

Anyway, I'll take your suggestion, thanks!

Max

Quoted 16 lines
Quoted 4 lines
block/qcow2-refcount.c +4 -0
+         * corresponding to one refcount entry and because refcount_order has to
+         * be below 7, we are far below that limit */
+        assert(!(entries >> (64 - (s->refcount_order - 3))));
+

block/qcow2-refcount.c fail:

Quoted 10 lines
block/qcow2-refcount.c +10 -0
+        return entries << (s->refcount_order - 3);
+    }
+}
+
+/**
+ * Reallocates *array so that it can hold new_size entries. *size must contain
+ * the current number of entries in *array. If the reallocation fails, *array
+ * and *size will not be modified and -errno will be returned. If the
+ * reallocation is successful, *array will be set to the new buffer and *size
+ * will be set to new_size.

And 0 is returned.

Quoted 14 lines

block/qcow2-refcount.c fail: · line 1172

Quoted 12 lines

The size of the reallocated refcount array buffer
+ * will be aligned to a cluster boundary, and the newly allocated area will be
+ * zeroed.
+ */

block/qcow2-refcount.c +8 -0
+static int realloc_refcount_array(BDRVQcowState *s, uint16_t **array,
+                                  int64_t *size, int64_t new_size)
+{
+    /* Round to clusters so the array can be directly written to disk */
+    size_t old_byte_size = ROUND_UP(refcount_array_byte_size(s, *size),
+                                    s->cluster_size);
+    size_t new_byte_size = ROUND_UP(refcount_array_byte_size(s, new_size),
+                                    s->cluster_size);

size_to_clusters()? (Unfortunately still not short enough to keep each initialisation on a single line...)

Right, I'll use it.

Quoted 11 lines

block/qcow2-refcount.c fail: · line 1178

Quoted 6 lines
block/qcow2-refcount.c +6 -0
+    uint16_t *new_ptr;
+
+    if (new_byte_size == old_byte_size) {
+        *size = new_size;
+        return 0;
+    }

This is only correct if the array was previously allocated by the same function, or at least rounded up to a cluster boundary. What do we win by keeping a smaller byte count in *size than is actually allocated? If we had the real allocation size in it, we wouldn't have to make assumptions about the real array size.

Historical reasons. *cough*

Other than that, I don't see how assuming the array was previously allocated by this function is a bad thing. I'll gladly continue under that assumption. And by having the number of entries in *size instead of the byte size of the refcount array, this patch is kept shorter (I have no proof of that, though) because the size of that array was (before this patch and before this series) not kept in bytes, but in entries. This is why keeping that unit here makes sense; also I don't like having different units across function parameters (in this case, bytes and entries).

So the first answer to your question "What do we win by keeping a smaller byte count in *size than is actually allocated?" is "it isn't a byte count but an entry count". The answer to the (naturally?) ensuing question "So why do we keep a smaller entry count in *size than is actually allocated?" is "because there is no refcount_array_entry_count_to_byte_size() and I don't see what's so bad about this block that we have to introduce that function".

Thus, to me it looks trying to have the maximum number of entries stored in *size (or the byte size of the table) will result in more code, and that is what we win by not doing that.

As far as I can see from your response, what we're losing is that we have to assume that the refcount array is always allocated by this function; as I've said above, I'll gladly do so.

Max

Quoted 121 lines
Quoted 96 lines
block/qcow2-refcount.c +29 -30
+    assert(new_byte_size > 0);
+
+    new_ptr = g_try_realloc(*array, new_byte_size);
+    if (!new_ptr) {
+        return -ENOMEM;
+    }
+
+    if (new_byte_size > old_byte_size) {
+        memset((void *)((uintptr_t)new_ptr + old_byte_size), 0,
+               new_byte_size - old_byte_size);
+    }
+
+    *array = new_ptr;
+    *size  = new_size;
+
+    return 0;
+}
 
  /*
   * Increases the refcount for a range of clusters in a given refcount table.
@@ -1146,6 +1210,7 @@ static int inc_refcounts(BlockDriverState *bs,
  {
      BDRVQcowState *s = bs->opaque;
      uint64_t start, last, cluster_offset, k;
+    int ret;
 
      if (size <= 0) {
          return 0;
@@ -1157,23 +1222,12 @@ static int inc_refcounts(BlockDriverState *bs,
          cluster_offset += s->cluster_size) {
          k = cluster_offset >> s->cluster_bits;
          if (k >= *refcount_table_size) {
-            int64_t old_refcount_table_size = *refcount_table_size;
-            uint16_t *new_refcount_table;
-
-            *refcount_table_size = k + 1;
-            new_refcount_table = g_try_realloc(*refcount_table,
-                                               *refcount_table_size *
-                                               sizeof(**refcount_table));
-            if (!new_refcount_table) {
-                *refcount_table_size = old_refcount_table_size;
+            ret = realloc_refcount_array(s, refcount_table,
+                                         refcount_table_size, k + 1);
+            if (ret < 0) {
                  res->check_errors++;
-                return -ENOMEM;
+                return ret;
              }
-            *refcount_table = new_refcount_table;
-
-            memset(*refcount_table + old_refcount_table_size, 0,
-                   (*refcount_table_size - old_refcount_table_size) *
-                   sizeof(**refcount_table));
          }
 
          if (++(*refcount_table)[k] == 0) {
@@ -1542,8 +1596,7 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,
                      fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", i);
 
              if (fix & BDRV_FIX_ERRORS) {
-                int64_t old_nb_clusters = *nb_clusters;
-                uint16_t *new_refcount_table;
+                int64_t new_nb_clusters;
 
                  if (offset > INT64_MAX - s->cluster_size) {
                      ret = -EINVAL;
@@ -1560,22 +1613,15 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,
                      goto resize_fail;
                  }
 
-                *nb_clusters = size_to_clusters(s, size);
-                assert(*nb_clusters >= old_nb_clusters);
+                new_nb_clusters = size_to_clusters(s, size);
+                assert(new_nb_clusters >= *nb_clusters);
 
-                new_refcount_table = g_try_realloc(*refcount_table,
-                                                   *nb_clusters *
-                                                   sizeof(**refcount_table));
-                if (!new_refcount_table) {
-                    *nb_clusters = old_nb_clusters;
+                ret = realloc_refcount_array(s, refcount_table,
+                                             nb_clusters, new_nb_clusters);
+                if (ret < 0) {
                      res->check_errors++;
-                    return -ENOMEM;
+                    return ret;
                  }
-                *refcount_table = new_refcount_table;
-
-                memset(*refcount_table + old_nb_clusters, 0,
-                       (*nb_clusters - old_nb_clusters) *
-                       sizeof(**refcount_table));
 
                  if (cluster >= *nb_clusters) {
                      ret = -EINVAL;

block/qcow2-refcount.c static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, · line 1686

Quoted 8 lines
block/qcow2-refcount.c +3 -2
@@ -1635,10 +1681,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
      int ret;
 
      if (!*refcount_table) {
-        *refcount_table = g_try_new0(uint16_t, *nb_clusters);
-        if (*nb_clusters && *refcount_table == NULL) {
+        int64_t old_size = 0;
+        ret = realloc_refcount_array(s, refcount_table,
+                                     &old_size, *nb_clusters);

Here the returned new size is thrown away.

With the implementation of realloc_refcount_array() as above this is not incorrect because it is *nb_clusters anyway when the function succeeds, but it's a bit sloppy.

If you decide to allow realloc_refcount_array() returning a size bigger than was requested (i.e. the rounded value is returned) as I suggested above, then you need to change this call.

block/qcow2-refcount.c static int inc_refcounts(BlockDriverState *bs, · line 1231

Quoted 6 lines
block/qcow2-refcount.c +2 -1
+        if (ret < 0) {
              res->check_errors++;
-            return -ENOMEM;
+            return ret;
          }
      }

Kevin

author
#22 lore raw

PATCH v5 10/26 qcow2: Helper function for refcount modification

In reply to Max Reitz

Since refcounts do not always have to be a uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.

With the check and repair functions using this function, the refcount array they are creating will be in big endian byte order; additionally, using realloc_refcount_array() makes the size of this refcount array always cluster-aligned. Both combined allow rebuild_refcount_structure() to drop the bounce buffer which was used to convert parts of the refcount array to big endian byte order and store them on disk. Instead, those parts can now be written directly.

---
 block/qcow2-refcount.c | 122 ++++++++++++++++++++++++++++---------------------
 block/qcow2.h          |   8 ++++
 2 files changed, 79 insertions(+), 51 deletions(-)
block/qcow2-refcount.c +71 -51
index 4ede971..dd6840e 100644
@@ -32,6 +32,11 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
                             int64_t offset, int64_t length, uint64_t addend,
                             bool decrease, enum qcow2_discard_type type);
 
+static uint64_t get_refcount_ro4(const void *refcount_array, uint64_t index);
+
+static void set_refcount_ro4(void *refcount_array, uint64_t index,
+                             uint64_t value);
+
 
 /*********************************************************/
 /* refcount handling */
@@ -42,6 +47,9 @@ int qcow2_refcount_init(BlockDriverState *bs)
     unsigned int refcount_table_size2, i;
     int ret;
 
+    s->get_refcount = &get_refcount_ro4;
+    s->set_refcount = &set_refcount_ro4;
+
     assert(s->refcount_table_size <= INT_MAX / sizeof(uint64_t));
     refcount_table_size2 = s->refcount_table_size * sizeof(uint64_t);
     s->refcount_table = g_try_malloc(refcount_table_size2);
@@ -72,6 +80,19 @@ void qcow2_refcount_close(BlockDriverState *bs)
 }
 
 
+static uint64_t get_refcount_ro4(const void *refcount_array, uint64_t index)
+{
+    return be16_to_cpu(((const uint16_t *)refcount_array)[index]);
+}
+
+static void set_refcount_ro4(void *refcount_array, uint64_t index,
+                             uint64_t value)
+{
+    assert(!(value >> 16));
+    ((uint16_t *)refcount_array)[index] = cpu_to_be16(value);
+}
+
+
 static int load_refcount_block(BlockDriverState *bs,
                                int64_t refcount_block_offset,
                                void **refcount_block)
@@ -97,7 +118,7 @@ int qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index,
     uint64_t refcount_table_index, block_index;
     int64_t refcount_block_offset;
     int ret;
-    uint16_t *refcount_block;
+    void *refcount_block;
 
     refcount_table_index = cluster_index >> s->refcount_block_bits;
     if (refcount_table_index >= s->refcount_table_size) {
@@ -119,16 +140,15 @@ int qcow2_get_refcount(BlockDriverState *bs, int64_t cluster_index,
     }
 
     ret = qcow2_cache_get(bs, s->refcount_block_cache, refcount_block_offset,
-        (void**) &refcount_block);
+                          &refcount_block);
     if (ret < 0) {
         return ret;
     }
 
     block_index = cluster_index & (s->refcount_block_size - 1);
-    *refcount = be16_to_cpu(refcount_block[block_index]);
+    *refcount = s->get_refcount(refcount_block, block_index);
 
-    ret = qcow2_cache_put(bs, s->refcount_block_cache,
-        (void**) &refcount_block);
+    ret = qcow2_cache_put(bs, s->refcount_block_cache, &refcount_block);
     if (ret < 0) {
         return ret;
     }
@@ -172,7 +192,7 @@ static int in_same_refcount_block(BDRVQcowState *s, uint64_t offset_a,
  * Returns 0 on success or -errno in error case
  */
 static int alloc_refcount_block(BlockDriverState *bs,
-    int64_t cluster_index, uint16_t **refcount_block)
+                                int64_t cluster_index, void **refcount_block)
 {
     BDRVQcowState *s = bs->opaque;
     unsigned int refcount_table_index;
@@ -199,7 +219,7 @@ static int alloc_refcount_block(BlockDriverState *bs,
             }
 
              return load_refcount_block(bs, refcount_block_offset,
-                 (void**) refcount_block);
+                                        refcount_block);
         }
     }
 
@@ -259,7 +279,7 @@ static int alloc_refcount_block(BlockDriverState *bs,
         /* The block describes itself, need to update the cache */
         int block_index = (new_block >> s->cluster_bits) &
             (s->refcount_block_size - 1);
-        (*refcount_block)[block_index] = cpu_to_be16(1);
+        s->set_refcount(*refcount_block, block_index, 1);
     } else {
         /* Described somewhere else. This can recurse at most twice before we
          * arrive at a block that describes itself. */
@@ -277,7 +297,7 @@ static int alloc_refcount_block(BlockDriverState *bs,
         /* Initialize the new refcount block only after updating its refcount,
          * update_refcount uses the refcount cache itself */
         ret = qcow2_cache_get_empty(bs, s->refcount_block_cache, new_block,
-            (void**) refcount_block);
+                                    refcount_block);
         if (ret < 0) {
             goto fail_block;
         }
@@ -311,7 +331,7 @@ static int alloc_refcount_block(BlockDriverState *bs,
         return -EAGAIN;
     }
 
-    ret = qcow2_cache_put(bs, s->refcount_block_cache, (void**) refcount_block);
+    ret = qcow2_cache_put(bs, s->refcount_block_cache, refcount_block);
     if (ret < 0) {
         goto fail_block;
     }
@@ -365,7 +385,7 @@ static int alloc_refcount_block(BlockDriverState *bs,
         s->cluster_size;
     uint64_t table_offset = meta_offset + blocks_clusters * s->cluster_size;
     uint64_t *new_table = g_try_new0(uint64_t, table_size);
-    uint16_t *new_blocks = g_try_malloc0(blocks_clusters * s->cluster_size);
+    void *new_blocks = g_try_malloc0_n(blocks_clusters, s->cluster_size);
 
     assert(table_size > 0 && blocks_clusters > 0);
     if (new_table == NULL || new_blocks == NULL) {
@@ -387,7 +407,7 @@ static int alloc_refcount_block(BlockDriverState *bs,
     uint64_t table_clusters = size_to_clusters(s, table_size * sizeof(uint64_t));
     int block = 0;
     for (i = 0; i < table_clusters + blocks_clusters; i++) {
-        new_blocks[block++] = cpu_to_be16(1);
+        s->set_refcount(new_blocks, block++, 1);
     }
 
     /* Write refcount blocks to disk */
@@ -440,7 +460,7 @@ static int alloc_refcount_block(BlockDriverState *bs,
     qcow2_free_clusters(bs, old_table_offset, old_table_size * sizeof(uint64_t),
                         QCOW2_DISCARD_OTHER);
 
-    ret = load_refcount_block(bs, new_block, (void**) refcount_block);
+    ret = load_refcount_block(bs, new_block, refcount_block);
     if (ret < 0) {
         return ret;
     }
@@ -455,7 +475,7 @@ fail_table:
     g_free(new_table);
 fail_block:
     if (*refcount_block != NULL) {
-        qcow2_cache_put(bs, s->refcount_block_cache, (void**) refcount_block);
+        qcow2_cache_put(bs, s->refcount_block_cache, refcount_block);
     }
     return ret;
 }
@@ -541,7 +561,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
 {
     BDRVQcowState *s = bs->opaque;
     int64_t start, last, cluster_offset;
-    uint16_t *refcount_block = NULL;
+    void *refcount_block = NULL;
     int64_t old_table_index = -1;
     int ret;
 
@@ -593,7 +613,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
         /* we can update the count and save it */
         block_index = cluster_index & (s->refcount_block_size - 1);
 
-        refcount = be16_to_cpu(refcount_block[block_index]);
+        refcount = s->get_refcount(refcount_block, block_index);
         if (decrease ? (refcount - addend > refcount)
                      : (refcount + addend < refcount ||
                         refcount + addend > s->refcount_max))
@@ -609,7 +629,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
         if (refcount == 0 && cluster_index < s->free_cluster_index) {
             s->free_cluster_index = cluster_index;
         }
-        refcount_block[block_index] = cpu_to_be16(refcount);
+        s->set_refcount(refcount_block, block_index, refcount);
 
         if (refcount == 0 && s->discard_passthrough[type]) {
             update_refcount_discard(bs, cluster_offset, s->cluster_size);
@@ -625,8 +645,7 @@ fail:
     /* Write last changed block to disk */
     if (refcount_block) {
         int wret;
-        wret = qcow2_cache_put(bs, s->refcount_block_cache,
-            (void**) &refcount_block);
+        wret = qcow2_cache_put(bs, s->refcount_block_cache, &refcount_block);
         if (wret < 0) {
             return ret < 0 ? ret : wret;
         }
@@ -1162,7 +1181,7 @@ static size_t refcount_array_byte_size(BDRVQcowState *s, uint64_t entries)
  * will be aligned to a cluster boundary, and the newly allocated area will be
  * zeroed.
  */
-static int realloc_refcount_array(BDRVQcowState *s, uint16_t **array,
+static int realloc_refcount_array(BDRVQcowState *s, void **array,
                                   int64_t *size, int64_t new_size)
 {
     /* Round to clusters so the array can be directly written to disk */
@@ -1170,7 +1189,7 @@ static int realloc_refcount_array(BDRVQcowState *s, uint16_t **array,
                                     s->cluster_size);
     size_t new_byte_size = ROUND_UP(refcount_array_byte_size(s, new_size),
                                     s->cluster_size);
-    uint16_t *new_ptr;
+    void *new_ptr;
 
     if (new_byte_size == old_byte_size) {
         *size = new_size;
@@ -1204,12 +1223,12 @@ static int realloc_refcount_array(BDRVQcowState *s, uint16_t **array,
  */
 static int inc_refcounts(BlockDriverState *bs,
                          BdrvCheckResult *res,
-                         uint16_t **refcount_table,
+                         void **refcount_table,
                          int64_t *refcount_table_size,
                          int64_t offset, int64_t size)
 {
     BDRVQcowState *s = bs->opaque;
-    uint64_t start, last, cluster_offset, k;
+    uint64_t start, last, cluster_offset, k, refcount;
     int ret;
 
     if (size <= 0) {
@@ -1230,11 +1249,14 @@ static int inc_refcounts(BlockDriverState *bs,
             }
         }
 
-        if (++(*refcount_table)[k] == 0) {
+        refcount = s->get_refcount(*refcount_table, k);
+        if (refcount == s->refcount_max) {
             fprintf(stderr, "ERROR: overflow cluster offset=0x%" PRIx64
                     "\n", cluster_offset);
             res->corruptions++;
+            continue;
         }
+        s->set_refcount(*refcount_table, k, refcount + 1);
     }
 
     return 0;
@@ -1254,8 +1276,9 @@ enum {
  * error occurred.
  */
 static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
-    uint16_t **refcount_table, int64_t *refcount_table_size, int64_t l2_offset,
-    int flags)
+                              void **refcount_table,
+                              int64_t *refcount_table_size, int64_t l2_offset,
+                              int flags)
 {
     BDRVQcowState *s = bs->opaque;
     uint64_t *l2_table, l2_entry;
@@ -1372,7 +1395,7 @@ fail:
  */
 static int check_refcounts_l1(BlockDriverState *bs,
                               BdrvCheckResult *res,
-                              uint16_t **refcount_table,
+                              void **refcount_table,
                               int64_t *refcount_table_size,
                               int64_t l1_table_offset, int l1_size,
                               int flags)
@@ -1571,7 +1594,7 @@ fail:
  */
 static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,
                            BdrvCheckMode fix, bool *rebuild,
-                           uint16_t **refcount_table, int64_t *nb_clusters)
+                           void **refcount_table, int64_t *nb_clusters)
 {
     BDRVQcowState *s = bs->opaque;
     int64_t i, size;
@@ -1656,9 +1679,10 @@ resize_fail:
             if (ret < 0) {
                 return ret;
             }
-            if ((*refcount_table)[cluster] != 1) {
+            if (s->get_refcount(*refcount_table, cluster) != 1) {
                 fprintf(stderr, "ERROR refcount block %" PRId64
-                        " refcount=%d\n", i, (*refcount_table)[cluster]);
+                        " refcount=%" PRIu64 "\n", i,
+                        s->get_refcount(*refcount_table, cluster));
                 res->corruptions++;
                 *rebuild = true;
             }
@@ -1673,7 +1697,7 @@ resize_fail:
  */
 static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
                                BdrvCheckMode fix, bool *rebuild,
-                               uint16_t **refcount_table, int64_t *nb_clusters)
+                               void **refcount_table, int64_t *nb_clusters)
 {
     BDRVQcowState *s = bs->opaque;
     int64_t i;
@@ -1737,7 +1761,7 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
 static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
                               BdrvCheckMode fix, bool *rebuild,
                               int64_t *highest_cluster,
-                              uint16_t *refcount_table, int64_t nb_clusters)
+                              void *refcount_table, int64_t nb_clusters)
 {
     BDRVQcowState *s = bs->opaque;
     int64_t i;
@@ -1753,7 +1777,7 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
             continue;
         }
 
-        refcount2 = refcount_table[i];
+        refcount2 = s->get_refcount(refcount_table, i);
 
         if (refcount1 > 0 || refcount2 > 0) {
             *highest_cluster = i;
@@ -1812,7 +1836,7 @@ static void compare_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
  */
 static int64_t alloc_clusters_imrt(BlockDriverState *bs,
                                    int cluster_count,
-                                   uint16_t **refcount_table,
+                                   void **refcount_table,
                                    int64_t *imrt_nb_clusters,
                                    int64_t *first_free_cluster)
 {
@@ -1829,7 +1853,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,
          contiguous_free_clusters < cluster_count;
          cluster++)
     {
-        if (!(*refcount_table)[cluster]) {
+        if (!s->get_refcount(*refcount_table, cluster)) {
             contiguous_free_clusters++;
             if (first_gap) {
                 /* If this is the first free cluster found, update
@@ -1867,7 +1891,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,
     /* Go back to the first free cluster */
     cluster -= contiguous_free_clusters;
     for (i = 0; i < cluster_count; i++) {
-        (*refcount_table)[cluster + i] = 1;
+        s->set_refcount(*refcount_table, cluster + i, 1);
     }
 
     return cluster << s->cluster_bits;
@@ -1883,7 +1907,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,
  */
 static int rebuild_refcount_structure(BlockDriverState *bs,
                                       BdrvCheckResult *res,
-                                      uint16_t **refcount_table,
+                                      void **refcount_table,
                                       int64_t *nb_clusters)
 {
     BDRVQcowState *s = bs->opaque;
@@ -1891,8 +1915,8 @@ static int rebuild_refcount_structure(BlockDriverState *bs,
     int64_t refblock_offset, refblock_start, refblock_index;
     uint32_t reftable_size = 0;
     uint64_t *on_disk_reftable = NULL;
-    uint16_t *on_disk_refblock;
-    int i, ret = 0;
+    void *on_disk_refblock;
+    int ret = 0;
     struct {
         uint64_t reftable_offset;
         uint32_t reftable_clusters;
@@ -1902,7 +1926,7 @@ static int rebuild_refcount_structure(BlockDriverState *bs,
 
 write_refblocks:
     for (; cluster < *nb_clusters; cluster++) {
-        if (!(*refcount_table)[cluster]) {
+        if (!s->get_refcount(*refcount_table, cluster)) {
             continue;
         }
 
@@ -1975,17 +1999,13 @@ write_refblocks:
             goto fail;
         }
 
-        on_disk_refblock = qemu_blockalign0(bs->file, s->cluster_size);
-        for (i = 0; i < s->refcount_block_size &&
-                    refblock_start + i < *nb_clusters; i++)
-        {
-            on_disk_refblock[i] =
-                cpu_to_be16((*refcount_table)[refblock_start + i]);
-        }
+        /* The size of *refcount_table is always cluster-aligned, therefore the
+         * write operation will not overflow */
+        on_disk_refblock = (void *)((uintptr_t)*refcount_table +
+                                    (refblock_index << s->refcount_block_bits));
 
         ret = bdrv_write(bs->file, refblock_offset / BDRV_SECTOR_SIZE,
-                         (void *)on_disk_refblock, s->cluster_sectors);
-        qemu_vfree(on_disk_refblock);
+                         on_disk_refblock, s->cluster_sectors);
         if (ret < 0) {
             fprintf(stderr, "ERROR writing refblock: %s\n", strerror(-ret));
             goto fail;
@@ -2080,7 +2100,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
     BDRVQcowState *s = bs->opaque;
     BdrvCheckResult pre_compare_res;
     int64_t size, highest_cluster, nb_clusters;
-    uint16_t *refcount_table = NULL;
+    void *refcount_table = NULL;
     bool rebuild = false;
     int ret;
 
@@ -2129,7 +2149,7 @@ int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
         /* Because the old reftable has been exchanged for a new one the
          * references have to be recalculated */
         rebuild = false;
-        memset(refcount_table, 0, nb_clusters * sizeof(uint16_t));
+        memset(refcount_table, 0, refcount_array_byte_size(s, nb_clusters));
         ret = calculate_refcounts(bs, res, 0, &rebuild, &refcount_table,
                                   &nb_clusters);
         if (ret < 0) {
block/qcow2.h +8 -0
index adf515c..6070e74 100644
@@ -213,6 +213,11 @@ typedef struct Qcow2DiscardRegion {
     QTAILQ_ENTRY(Qcow2DiscardRegion) next;
 } Qcow2DiscardRegion;
 
+typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array,
+                                      uint64_t index);
+typedef void Qcow2SetRefcountFunc(void *refcount_array,
+                                  uint64_t index, uint64_t value);
+
 typedef struct BDRVQcowState {
     int cluster_bits;
     int cluster_size;
@@ -261,6 +266,9 @@ typedef struct BDRVQcowState {
     int refcount_bits;
     uint64_t refcount_max;
 
+    Qcow2GetRefcountFunc *get_refcount;
+    Qcow2SetRefcountFunc *set_refcount;
+
     bool discard_passthrough[QCOW2_DISCARD_MAX];
 
     int overlap_check; /* bitmask of Qcow2MetadataOverlap values */
Signature · 1 line
1.9.3
#23 lore raw

Re: [Qemu-devel] [PATCH v5 10/26] qcow2: Helper function for refcount modification

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 51 lines

Since refcounts do not always have to be a uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.

With the check and repair functions using this function, the refcount array they are creating will be in big endian byte order; additionally, using realloc_refcount_array() makes the size of this refcount array always cluster-aligned. Both combined allow rebuild_refcount_structure() to drop the bounce buffer which was used to convert parts of the refcount array to big endian byte order and store them on disk. Instead, those parts can now be written directly.

---
 block/qcow2-refcount.c | 122 ++++++++++++++++++++++++++++---------------------
 block/qcow2.h          |   8 ++++
 2 files changed, 79 insertions(+), 51 deletions(-)
block/qcow2-refcount.c +3 -3
@@ -541,7 +561,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
 {
     BDRVQcowState *s = bs->opaque;
     int64_t start, last, cluster_offset;
-    uint16_t *refcount_block = NULL;
+    void *refcount_block = NULL;
     int64_t old_table_index = -1;
     int ret;
 
@@ -593,7 +613,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
         /* we can update the count and save it */
         block_index = cluster_index & (s->refcount_block_size - 1);
 
-        refcount = be16_to_cpu(refcount_block[block_index]);
+        refcount = s->get_refcount(refcount_block, block_index);
         if (decrease ? (refcount - addend > refcount)
                      : (refcount + addend < refcount ||
                         refcount + addend > s->refcount_max))
@@ -609,7 +629,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
         if (refcount == 0 && cluster_index < s->free_cluster_index) {
             s->free_cluster_index = cluster_index;
         }
-        refcount_block[block_index] = cpu_to_be16(refcount);
+        s->set_refcount(refcount_block, block_index, refcount);
 
         if (refcount == 0 && s->discard_passthrough[type]) {
             update_refcount_discard(bs, cluster_offset, s->cluster_size);

block/qcow2-refcount.c fail: · line 651

Quoted 9 lines
block/qcow2-refcount.c +1 -2
@@ -625,8 +645,7 @@ fail:
     /* Write last changed block to disk */
     if (refcount_block) {
         int wret;
-        wret = qcow2_cache_put(bs, s->refcount_block_cache,
-            (void**) &refcount_block);
+        wret = qcow2_cache_put(bs, s->refcount_block_cache, &refcount_block);
         if (wret < 0) {
             return ret < 0 ? ret : wret;
         }

There is a (void**) cast left in the other qcow2_cache_put() call in update_refcount().

Quoted 34 lines
block/qcow2-refcount.c +4 -5
@@ -1883,7 +1907,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,
  */
 static int rebuild_refcount_structure(BlockDriverState *bs,
                                       BdrvCheckResult *res,
-                                      uint16_t **refcount_table,
+                                      void **refcount_table,
                                       int64_t *nb_clusters)
 {
     BDRVQcowState *s = bs->opaque;
@@ -1891,8 +1915,8 @@ static int rebuild_refcount_structure(BlockDriverState *bs,
     int64_t refblock_offset, refblock_start, refblock_index;
     uint32_t reftable_size = 0;
     uint64_t *on_disk_reftable = NULL;
-    uint16_t *on_disk_refblock;
-    int i, ret = 0;
+    void *on_disk_refblock;
+    int ret = 0;
     struct {
         uint64_t reftable_offset;
         uint32_t reftable_clusters;
@@ -1902,7 +1926,7 @@ static int rebuild_refcount_structure(BlockDriverState *bs,
 
 write_refblocks:
     for (; cluster < *nb_clusters; cluster++) {
-        if (!(*refcount_table)[cluster]) {
+        if (!s->get_refcount(*refcount_table, cluster)) {
             continue;
         }
 
@@ -1975,17 +1999,13 @@ write_refblocks:
             goto fail;
         }
 
-        on_disk_refblock = qemu_blockalign0(bs->file, s->cluster_size);

block/qcow2-refcount.c write_refblocks: · line 2005

Quoted 10 lines
block/qcow2-refcount.c +4 -6
-        for (i = 0; i < s->refcount_block_size &&
-                    refblock_start + i < *nb_clusters; i++)
-        {
-            on_disk_refblock[i] =
-                cpu_to_be16((*refcount_table)[refblock_start + i]);
-        }
+        /* The size of *refcount_table is always cluster-aligned, therefore the
+         * write operation will not overflow */
+        on_disk_refblock = (void *)((uintptr_t)*refcount_table +
+                                    (refblock_index << s->refcount_block_bits));

Are you sure about s->refcount_block_bits? You're now calculating in bytes and not in entries any more like before with uint16_t*. So I think this should be s->cluster_bits now (or refblock_index * s->cluster_size if you don't want to confuse people more than necessary :-)).

block/qcow2-refcount.c write_refblocks: · line 2011

Quoted 7 lines
block/qcow2-refcount.c +1 -2
         ret = bdrv_write(bs->file, refblock_offset / BDRV_SECTOR_SIZE,
-                         (void *)on_disk_refblock, s->cluster_sectors);
-        qemu_vfree(on_disk_refblock);
+                         on_disk_refblock, s->cluster_sectors);
         if (ret < 0) {
             fprintf(stderr, "ERROR writing refblock: %s\n", strerror(-ret));
             goto fail;

Kevin

author
#24 lore raw

Re: [Qemu-devel] [PATCH v5 10/26] qcow2: Helper function for refcount modification

Quoted 48 lines · Kevin Wolf

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 47 lines

Since refcounts do not always have to be a uint16_t, all refcount blocks and arrays in memory should not have a specific type (thus they become pointers to void) and for accessing them, two helper functions are used (a getter and a setter). Those functions are called indirectly through function pointers in the BDRVQcowState so they may later be exchanged for different refcount orders.

With the check and repair functions using this function, the refcount array they are creating will be in big endian byte order; additionally, using realloc_refcount_array() makes the size of this refcount array always cluster-aligned. Both combined allow rebuild_refcount_structure() to drop the bounce buffer which was used to convert parts of the refcount array to big endian byte order and store them on disk. Instead, those parts can now be written directly.

---
  block/qcow2-refcount.c | 122 ++++++++++++++++++++++++++++---------------------
  block/qcow2.h          |   8 ++++
  2 files changed, 79 insertions(+), 51 deletions(-)
block/qcow2-refcount.c +3 -3
@@ -541,7 +561,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
  {
      BDRVQcowState *s = bs->opaque;
      int64_t start, last, cluster_offset;
-    uint16_t *refcount_block = NULL;
+    void *refcount_block = NULL;
      int64_t old_table_index = -1;
      int ret;
 
@@ -593,7 +613,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
          /* we can update the count and save it */
          block_index = cluster_index & (s->refcount_block_size - 1);
 
-        refcount = be16_to_cpu(refcount_block[block_index]);
+        refcount = s->get_refcount(refcount_block, block_index);
          if (decrease ? (refcount - addend > refcount)
                       : (refcount + addend < refcount ||
                          refcount + addend > s->refcount_max))
@@ -609,7 +629,7 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
          if (refcount == 0 && cluster_index < s->free_cluster_index) {
              s->free_cluster_index = cluster_index;
          }
-        refcount_block[block_index] = cpu_to_be16(refcount);
+        s->set_refcount(refcount_block, block_index, refcount);
 
Quoted 14 lines

block/qcow2-refcount.c fail: · line 651

Quoted 12 lines
block/qcow2-refcount.c +1 -2
          if (refcount == 0 && s->discard_passthrough[type]) {
              update_refcount_discard(bs, cluster_offset, s->cluster_size);
@@ -625,8 +645,7 @@ fail:
      /* Write last changed block to disk */
      if (refcount_block) {
          int wret;
-        wret = qcow2_cache_put(bs, s->refcount_block_cache,
-            (void**) &refcount_block);
+        wret = qcow2_cache_put(bs, s->refcount_block_cache, &refcount_block);
          if (wret < 0) {
              return ret < 0 ? ret : wret;
          }

There is a (void**) cast left in the other qcow2_cache_put() call in update_refcount().

Will remove.

Quoted 33 lines
Quoted 33 lines
block/qcow2-refcount.c +4 -4
@@ -1883,7 +1907,7 @@ static int64_t alloc_clusters_imrt(BlockDriverState *bs,
   */
  static int rebuild_refcount_structure(BlockDriverState *bs,
                                        BdrvCheckResult *res,
-                                      uint16_t **refcount_table,
+                                      void **refcount_table,
                                        int64_t *nb_clusters)
  {
      BDRVQcowState *s = bs->opaque;
@@ -1891,8 +1915,8 @@ static int rebuild_refcount_structure(BlockDriverState *bs,
      int64_t refblock_offset, refblock_start, refblock_index;
      uint32_t reftable_size = 0;
      uint64_t *on_disk_reftable = NULL;
-    uint16_t *on_disk_refblock;
-    int i, ret = 0;
+    void *on_disk_refblock;
+    int ret = 0;
      struct {
          uint64_t reftable_offset;
          uint32_t reftable_clusters;
@@ -1902,7 +1926,7 @@ static int rebuild_refcount_structure(BlockDriverState *bs,
 
  write_refblocks:
      for (; cluster < *nb_clusters; cluster++) {
-        if (!(*refcount_table)[cluster]) {
+        if (!s->get_refcount(*refcount_table, cluster)) {
              continue;
          }
 
@@ -1975,17 +1999,13 @@ write_refblocks:
              goto fail;
          }
 
Quoted 15 lines

block/qcow2-refcount.c write_refblocks: · line 2005

Quoted 11 lines
block/qcow2-refcount.c +4 -7
-        on_disk_refblock = qemu_blockalign0(bs->file, s->cluster_size);
-        for (i = 0; i < s->refcount_block_size &&
-                    refblock_start + i < *nb_clusters; i++)
-        {
-            on_disk_refblock[i] =
-                cpu_to_be16((*refcount_table)[refblock_start + i]);
-        }
+        /* The size of *refcount_table is always cluster-aligned, therefore the
+         * write operation will not overflow */
+        on_disk_refblock = (void *)((uintptr_t)*refcount_table +
+                                    (refblock_index << s->refcount_block_bits));

Are you sure about s->refcount_block_bits? You're now calculating in bytes and not in entries any more like before with uint16_t*.So I think this should be s->cluster_bits now (or refblock_index * s->cluster_size if you don't want to confuse people more than necessary :-)).

You're right. Now I'm wondering why it worked, though. Maybe the tests only covered the refblock_index == 0 case...

Good catch, thanks!

Max

Quoted 8 lines

block/qcow2-refcount.c write_refblocks: · line 2011

Quoted 7 lines
block/qcow2-refcount.c +1 -2
          ret = bdrv_write(bs->file, refblock_offset / BDRV_SECTOR_SIZE,
-                         (void *)on_disk_refblock, s->cluster_sectors);
-        qemu_vfree(on_disk_refblock);
+                         on_disk_refblock, s->cluster_sectors);
          if (ret < 0) {
              fprintf(stderr, "ERROR writing refblock: %s\n", strerror(-ret));
              goto fail;

Kevin

author
#25 lore raw

PATCH v5 11/26 qcow2: More helpers for refcount modification

In reply to Max Reitz

Add helper functions for getting and setting refcounts in a refcount array for any possible refcount order, and choose the correct one during refcount initialization.

---
 block/qcow2-refcount.c | 121 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 119 insertions(+), 2 deletions(-)
block/qcow2-refcount.c +119 -2
index dd6840e..e37fcdf 100644
@@ -32,10 +32,49 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(BlockDriverState *bs,
                             int64_t offset, int64_t length, uint64_t addend,
                             bool decrease, enum qcow2_discard_type type);
 
+static uint64_t get_refcount_ro0(const void *refcount_array, uint64_t index);
+static uint64_t get_refcount_ro1(const void *refcount_array, uint64_t index);
+static uint64_t get_refcount_ro2(const void *refcount_array, uint64_t index);
+static uint64_t get_refcount_ro3(const void *refcount_array, uint64_t index);
 static uint64_t get_refcount_ro4(const void *refcount_array, uint64_t index);
+static uint64_t get_refcount_ro5(const void *refcount_array, uint64_t index);
+static uint64_t get_refcount_ro6(const void *refcount_array, uint64_t index);
 
+static void set_refcount_ro0(void *refcount_array, uint64_t index,
+                             uint64_t value);
+static void set_refcount_ro1(void *refcount_array, uint64_t index,
+                             uint64_t value);
+static void set_refcount_ro2(void *refcount_array, uint64_t index,
+                             uint64_t value);
+static void set_refcount_ro3(void *refcount_array, uint64_t index,
+                             uint64_t value);
 static void set_refcount_ro4(void *refcount_array, uint64_t index,
                              uint64_t value);
+static void set_refcount_ro5(void *refcount_array, uint64_t index,
+                             uint64_t value);
+static void set_refcount_ro6(void *refcount_array, uint64_t index,
+                             uint64_t value);
+
+
+static Qcow2GetRefcountFunc *const get_refcount_funcs[] = {
+    &get_refcount_ro0,
+    &get_refcount_ro1,
+    &get_refcount_ro2,
+    &get_refcount_ro3,
+    &get_refcount_ro4,
+    &get_refcount_ro5,
+    &get_refcount_ro6
+};
+
+static Qcow2SetRefcountFunc *const set_refcount_funcs[] = {
+    &set_refcount_ro0,
+    &set_refcount_ro1,
+    &set_refcount_ro2,
+    &set_refcount_ro3,
+    &set_refcount_ro4,
+    &set_refcount_ro5,
+    &set_refcount_ro6
+};
 
 
 /*********************************************************/
@@ -47,8 +86,10 @@ int qcow2_refcount_init(BlockDriverState *bs)
     unsigned int refcount_table_size2, i;
     int ret;
 
-    s->get_refcount = &get_refcount_ro4;
-    s->set_refcount = &set_refcount_ro4;
+    assert(s->refcount_order >= 0 && s->refcount_order <= 6);
+
+    s->get_refcount = get_refcount_funcs[s->refcount_order];
+    s->set_refcount = set_refcount_funcs[s->refcount_order];
 
     assert(s->refcount_table_size <= INT_MAX / sizeof(uint64_t));
     refcount_table_size2 = s->refcount_table_size * sizeof(uint64_t);
@@ -80,6 +121,59 @@ void qcow2_refcount_close(BlockDriverState *bs)
 }
 
 
+static uint64_t get_refcount_ro0(const void *refcount_array, uint64_t index)
+{
+    return (((const uint8_t *)refcount_array)[index / 8] >> (index % 8)) & 0x1;
+}
+
+static void set_refcount_ro0(void *refcount_array, uint64_t index,
+                             uint64_t value)
+{
+    assert(!(value >> 1));
+    ((uint8_t *)refcount_array)[index / 8] &= ~(0x1 << (index % 8));
+    ((uint8_t *)refcount_array)[index / 8] |= value << (index % 8);
+}
+
+static uint64_t get_refcount_ro1(const void *refcount_array, uint64_t index)
+{
+    return (((const uint8_t *)refcount_array)[index / 4] >> (2 * (index % 4)))
+           & 0x3;
+}
+
+static void set_refcount_ro1(void *refcount_array, uint64_t index,
+                             uint64_t value)
+{
+    assert(!(value >> 2));
+    ((uint8_t *)refcount_array)[index / 4] &= ~(0x3 << (2 * (index % 4)));
+    ((uint8_t *)refcount_array)[index / 4] |= value << (2 * (index % 4));
+}
+
+static uint64_t get_refcount_ro2(const void *refcount_array, uint64_t index)
+{
+    return (((const uint8_t *)refcount_array)[index / 2] >> (4 * (index % 2)))
+           & 0xf;
+}
+
+static void set_refcount_ro2(void *refcount_array, uint64_t index,
+                             uint64_t value)
+{
+    assert(!(value >> 4));
+    ((uint8_t *)refcount_array)[index / 2] &= ~(0xf << (4 * (index % 2)));
+    ((uint8_t *)refcount_array)[index / 2] |= value << (4 * (index % 2));
+}
+
+static uint64_t get_refcount_ro3(const void *refcount_array, uint64_t index)
+{
+    return ((const uint8_t *)refcount_array)[index];
+}
+
+static void set_refcount_ro3(void *refcount_array, uint64_t index,
+                             uint64_t value)
+{
+    assert(!(value >> 8));
+    ((uint8_t *)refcount_array)[index] = value;
+}
+
 static uint64_t get_refcount_ro4(const void *refcount_array, uint64_t index)
 {
     return be16_to_cpu(((const uint16_t *)refcount_array)[index]);
@@ -92,6 +186,29 @@ static void set_refcount_ro4(void *refcount_array, uint64_t index,
     ((uint16_t *)refcount_array)[index] = cpu_to_be16(value);
 }
 
+static uint64_t get_refcount_ro5(const void *refcount_array, uint64_t index)
+{
+    return be32_to_cpu(((const uint32_t *)refcount_array)[index]);
+}
+
+static void set_refcount_ro5(void *refcount_array, uint64_t index,
+                             uint64_t value)
+{
+    assert(!(value >> 32));
+    ((uint32_t *)refcount_array)[index] = cpu_to_be32(value);
+}
+
+static uint64_t get_refcount_ro6(const void *refcount_array, uint64_t index)
+{
+    return be64_to_cpu(((const uint64_t *)refcount_array)[index]);
+}
+
+static void set_refcount_ro6(void *refcount_array, uint64_t index,
+                             uint64_t value)
+{
+    ((uint64_t *)refcount_array)[index] = cpu_to_be64(value);
+}
+
 
 static int load_refcount_block(BlockDriverState *bs,
                                int64_t refcount_block_offset,
Signature · 1 line
1.9.3
#26 lore raw

Re: [Qemu-devel] [PATCH v5 11/26] qcow2: More helpers for refcount modification

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 7 lines

Add helper functions for getting and setting refcounts in a refcount array for any possible refcount order, and choose the correct one during refcount initialization.

Hm... Wouldn't it be better to have a single generic implementation that is based on uint64_t* and calculates the right shifts and bitmasks on the fly?

Kevin

author
#27 lore raw

Re: [Qemu-devel] [PATCH v5 11/26] qcow2: More helpers for refcount modification

Quoted 11 lines · Kevin Wolf

Am 15.12.2014 um 13:50 hat Max Reitz geschrieben:

Quoted 7 lines

Add helper functions for getting and setting refcounts in a refcount array for any possible refcount order, and choose the correct one during refcount initialization.

Hm... Wouldn't it be better to have a single generic implementation that is based on uint64_t* and calculates the right shifts and bitmasks on the fly?

Please do try. I tried it, and what I came up with were horrible unreadable abominations. I think you can do better, but I don't know whether it'll look better than this apparent code duplications.

Max

author
#28 lore raw

PATCH v5 12/26 qcow2: Open images with refcount order != 4

In reply to Max Reitz

No longer refuse to open images with a different refcount entry width than 16 bits; only reject images with a refcount width larger than 64 bits (which is prohibited by the specification).

---
 block/qcow2.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
block/qcow2.c +4 -4
index d14e7c4..e5105f1 100644
@@ -677,10 +677,10 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
     }
 
     /* Check support for various header values */
-    if (header.refcount_order != 4) {
-        report_unsupported(bs, errp, "%d bit reference counts",
-                           1 << header.refcount_order);
-        ret = -ENOTSUP;
+    if (header.refcount_order > 6) {
+        error_setg(errp, "Reference count entry width too large; may not "
+                   "exceed 64 bits");
+        ret = -EINVAL;
         goto fail;
     }
     s->refcount_order = header.refcount_order;
Signature · 1 line
1.9.3
author
#29 lore raw

PATCH v5 13/26 qcow2: refcount_order parameter for qcow2_create2

In reply to Max Reitz

Add a refcount_order parameter to qcow2_create2(), use that value for the image header and for calculating the size required for preallocation.

For now, always pass 4.

This addition requires changes to the calculation of the file size for the "full" and "falloc" preallocation modes. That in turn is a nice opportunity to add a comment about that calculation not necessarily being exact (and that being intentional).

---
 block/qcow2.c | 46 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 11 deletions(-)
block/qcow2.c +35 -11
index e5105f1..1e333bf 100644
@@ -1782,7 +1782,7 @@ static int preallocate(BlockDriverState *bs)
 static int qcow2_create2(const char *filename, int64_t total_size,
                          const char *backing_file, const char *backing_format,
                          int flags, size_t cluster_size, PreallocMode prealloc,
-                         QemuOpts *opts, int version,
+                         QemuOpts *opts, int version, int refcount_order,
                          Error **errp)
 {
     /* Calculate cluster_bits */
@@ -1815,9 +1815,21 @@ static int qcow2_create2(const char *filename, int64_t total_size,
     int ret;
 
     if (prealloc == PREALLOC_MODE_FULL || prealloc == PREALLOC_MODE_FALLOC) {
+        /* Note: The following calculation does not need to be exact; if it is a
+         * bit off, either some bytes will be "leaked" (which is fine) or we
+         * will need to increase the file size by some bytes (which is fine,
+         * too, as long as the bulk is allocated here). Therefore, using
+         * floating point arithmetic is fine. */
         int64_t meta_size = 0;
         uint64_t nreftablee, nrefblocke, nl1e, nl2e;
         int64_t aligned_total_size = align_offset(total_size, cluster_size);
+        int refblock_bits, refblock_size;
+        /* refcount entry size in bytes */
+        double rces = (1 << refcount_order) / 8.;
+
+        /* see qcow2_open() */
+        refblock_bits = cluster_bits - (refcount_order - 3);
+        refblock_size = 1 << refblock_bits;
 
         /* header: 1 cluster */
         meta_size += cluster_size;
@@ -1842,20 +1854,20 @@ static int qcow2_create2(const char *filename, int64_t total_size,
          *   c = cluster size
          *   y1 = number of refcount blocks entries
          *   y2 = meta size including everything
+         *   rces = refcount entry size in bytes
          * then,
          *   y1 = (y2 + a)/c
-         *   y2 = y1 * sizeof(u16) + y1 * sizeof(u16) * sizeof(u64) / c + m
+         *   y2 = y1 * rces + y1 * rces * sizeof(u64) / c + m
          * we can get y1:
-         *   y1 = (a + m) / (c - sizeof(u16) - sizeof(u16) * sizeof(u64) / c)
+         *   y1 = (a + m) / (c - rces - rces * sizeof(u64) / c)
          */
-        nrefblocke = (aligned_total_size + meta_size + cluster_size) /
-            (cluster_size - sizeof(uint16_t) -
-             1.0 * sizeof(uint16_t) * sizeof(uint64_t) / cluster_size);
-        nrefblocke = align_offset(nrefblocke, cluster_size / sizeof(uint16_t));
-        meta_size += nrefblocke * sizeof(uint16_t);
+        nrefblocke = (aligned_total_size + meta_size + cluster_size)
+                   / (cluster_size - rces - rces * sizeof(uint64_t)
+                                                 / cluster_size);
+        meta_size += DIV_ROUND_UP(nrefblocke, refblock_size) * cluster_size;
 
         /* total size of refcount tables */
-        nreftablee = nrefblocke * sizeof(uint16_t) / cluster_size;
+        nreftablee = nrefblocke / refblock_size;
         nreftablee = align_offset(nreftablee, cluster_size / sizeof(uint64_t));
         meta_size += nreftablee * sizeof(uint64_t);
 
@@ -1890,7 +1902,7 @@ static int qcow2_create2(const char *filename, int64_t total_size,
         .l1_size                    = cpu_to_be32(0),
         .refcount_table_offset      = cpu_to_be64(cluster_size),
         .refcount_table_clusters    = cpu_to_be32(1),
-        .refcount_order             = cpu_to_be32(4),
+        .refcount_order             = cpu_to_be32(refcount_order),
         .header_length              = cpu_to_be32(sizeof(*header)),
     };
 
@@ -2009,6 +2021,7 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
     size_t cluster_size = DEFAULT_CLUSTER_SIZE;
     PreallocMode prealloc;
     int version = 3;
+    int refcount_bits = 16, refcount_order;
     Error *local_err = NULL;
     int ret;
 
@@ -2063,8 +2076,19 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
         goto finish;
     }
 
+    if (version < 3 && refcount_bits != 16) {
+        error_setg(errp, "Different refcount widths than 16 bits require "
+                   "compatibility level 1.1 or above (use compat=1.1 or "
+                   "greater)");
+        ret = -EINVAL;
+        goto finish;
+    }
+
+    refcount_order = ffs(refcount_bits) - 1;
+
     ret = qcow2_create2(filename, size, backing_file, backing_fmt, flags,
-                        cluster_size, prealloc, opts, version, &local_err);
+                        cluster_size, prealloc, opts, version, refcount_order,
+                        &local_err);
     if (local_err) {
         error_propagate(errp, local_err);
     }
Signature · 1 line
1.9.3
author
#30 lore raw

PATCH v5 14/26 qcow2: Use symbolic macros in qcow2_amend_options

In reply to Max Reitz

qcow2_amend_options() should not compare options against some inline strings but rather use the symbolic macros available for each of the creation options.

---
 block/qcow2.c | 31 ++++++++++++++++---------------
 1 file changed, 16 insertions(+), 15 deletions(-)
block/qcow2.c +16 -15
index 1e333bf..c0c0333 100644
@@ -2675,8 +2675,8 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
             continue;
         }
 
-        if (!strcmp(desc->name, "compat")) {
-            compat = qemu_opt_get(opts, "compat");
+        if (!strcmp(desc->name, BLOCK_OPT_COMPAT_LEVEL)) {
+            compat = qemu_opt_get(opts, BLOCK_OPT_COMPAT_LEVEL);
             if (!compat) {
                 /* preserve default */
             } else if (!strcmp(compat, "0.10")) {
@@ -2687,32 +2687,33 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
                 fprintf(stderr, "Unknown compatibility level %s.\n", compat);
                 return -EINVAL;
             }
-        } else if (!strcmp(desc->name, "preallocation")) {
+        } else if (!strcmp(desc->name, BLOCK_OPT_PREALLOC)) {
             fprintf(stderr, "Cannot change preallocation mode.\n");
             return -ENOTSUP;
-        } else if (!strcmp(desc->name, "size")) {
-            new_size = qemu_opt_get_size(opts, "size", 0);
-        } else if (!strcmp(desc->name, "backing_file")) {
-            backing_file = qemu_opt_get(opts, "backing_file");
-        } else if (!strcmp(desc->name, "backing_fmt")) {
-            backing_format = qemu_opt_get(opts, "backing_fmt");
-        } else if (!strcmp(desc->name, "encryption")) {
-            encrypt = qemu_opt_get_bool(opts, "encryption", s->crypt_method);
+        } else if (!strcmp(desc->name, BLOCK_OPT_SIZE)) {
+            new_size = qemu_opt_get_size(opts, BLOCK_OPT_SIZE, 0);
+        } else if (!strcmp(desc->name, BLOCK_OPT_BACKING_FILE)) {
+            backing_file = qemu_opt_get(opts, BLOCK_OPT_BACKING_FILE);
+        } else if (!strcmp(desc->name, BLOCK_OPT_BACKING_FMT)) {
+            backing_format = qemu_opt_get(opts, BLOCK_OPT_BACKING_FMT);
+        } else if (!strcmp(desc->name, BLOCK_OPT_ENCRYPT)) {
+            encrypt = qemu_opt_get_bool(opts, BLOCK_OPT_ENCRYPT,
+                                        s->crypt_method);
             if (encrypt != !!s->crypt_method) {
                 fprintf(stderr, "Changing the encryption flag is not "
                         "supported.\n");
                 return -ENOTSUP;
             }
-        } else if (!strcmp(desc->name, "cluster_size")) {
-            cluster_size = qemu_opt_get_size(opts, "cluster_size",
+        } else if (!strcmp(desc->name, BLOCK_OPT_CLUSTER_SIZE)) {
+            cluster_size = qemu_opt_get_size(opts, BLOCK_OPT_CLUSTER_SIZE,
                                              cluster_size);
             if (cluster_size != s->cluster_size) {
                 fprintf(stderr, "Changing the cluster size is not "
                         "supported.\n");
                 return -ENOTSUP;
             }
-        } else if (!strcmp(desc->name, "lazy_refcounts")) {
-            lazy_refcounts = qemu_opt_get_bool(opts, "lazy_refcounts",
+        } else if (!strcmp(desc->name, BLOCK_OPT_LAZY_REFCOUNTS)) {
+            lazy_refcounts = qemu_opt_get_bool(opts, BLOCK_OPT_LAZY_REFCOUNTS,
                                                lazy_refcounts);
         } else {
             /* if this assertion fails, this probably means a new option was
Signature · 1 line
1.9.3
author
#31 lore raw

PATCH v5 15/26 iotests: Prepare for refcount_bits option

In reply to Max Reitz

Some tests do not work well with certain refcount widths (i.e. you cannot create internal snapshots with refcount_bits=1), so make those widths unsupported.

Furthermore, add another filter to _filter_img_create in common.filter which filters out the refcount_bits value.

This is necessary for test 079, which does actually work with any refcount width, but invoking qemu-img directly leads to the refcount_bits value being visible in the output; use _make_test_img instead which will filter it out.

---
 tests/qemu-iotests/007           |  3 +++
 tests/qemu-iotests/015           |  2 ++
 tests/qemu-iotests/026           |  7 +++++++
 tests/qemu-iotests/029           |  2 ++
 tests/qemu-iotests/051           |  3 +++
 tests/qemu-iotests/058           |  2 ++
 tests/qemu-iotests/067           |  2 ++
 tests/qemu-iotests/079           | 10 ++--------
 tests/qemu-iotests/079.out       | 38 ++++++++++----------------------------
 tests/qemu-iotests/080           |  2 ++
 tests/qemu-iotests/089           |  2 ++
 tests/qemu-iotests/108           |  2 ++
 tests/qemu-iotests/common.filter |  3 ++-
 13 files changed, 41 insertions(+), 37 deletions(-)
tests/qemu-iotests/007 +3 -0
index fe1a743..7b5aff5 100755
@@ -43,6 +43,9 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto generic
 _supported_os Linux
+# refcount_bits must be at least 4 so we can create ten internal snapshots
+# (1 bit supports none, 2 bits support two, 4 bits support 14)
+_unsupported_imgopts 'refcount_bits=\(1\|2\)[^0-9]'
 
 echo
 echo "creating image"
tests/qemu-iotests/015 +2 -0
index 099d757..6f26095 100755
@@ -43,6 +43,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto generic
 _supported_os Linux
+# Internal snapshots are (currently) impossible with refcount_bits=1
+_unsupported_imgopts 'refcount_bits=1[^0-9]'
 
 echo
 echo "creating image"
tests/qemu-iotests/026 +7 -0
index df2884b..0fc3244 100755
@@ -46,6 +46,13 @@ _supported_proto file
 _supported_os Linux
 _default_cache_mode "writethrough"
 _supported_cache_modes "writethrough" "none"
+# The refcount table tests expect a certain minimum width for refcount entries
+# (so that the refcount table actually needs to grow); that minimum is 16 bits,
+# being the default refcount entry width.
+# 32 and 64 bits do not work either, however, due to different leaked cluster
+# count on error.
+# Thus, the only remaining option is refcount_bits=16.
+_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)'
 
 echo "Errors while writing 128 kB"
 echo
tests/qemu-iotests/029 +2 -0
index fa46ace..b9cd826 100755
@@ -44,6 +44,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto generic
 _supported_os Linux
+# Internal snapshots are (currently) impossible with refcount_bits=1
+_unsupported_imgopts 'refcount_bits=1[^0-9]'
 
 offset_size=24
 offset_l1_size=36
tests/qemu-iotests/051 +3 -0
index 11c858f..1677592 100755
@@ -41,6 +41,9 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
+# A compat=0.10 image is created in this test which does not support anything
+# other than refcount_bits=16
+_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)'
 
 function do_run_qemu()
 {
tests/qemu-iotests/058 +2 -0
index 2d5ca85..5f884f1 100755
@@ -88,6 +88,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _require_command QEMU_NBD
+# Internal snapshots are (currently) impossible with refcount_bits=1
+_unsupported_imgopts 'refcount_bits=1[^0-9]'
 
 # Use -f raw instead of -f $IMGFMT for the NBD connection
 QEMU_IO_NBD="$QEMU_IO -f raw --cache=$CACHEMODE"
tests/qemu-iotests/067 +2 -0
index 29cd6b5..88e8352 100755
@@ -35,6 +35,8 @@ status=1	# failure is the default!
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
+# Because anything other than 16 would change the output of query-block
+_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)'
 
 function do_run_qemu()
 {
tests/qemu-iotests/079 +2 -8
index 6613cfb..ade6efa 100755
@@ -42,19 +42,13 @@ _supported_fmt qcow2
 _supported_proto file nfs
 _supported_os Linux
 
-function test_qemu_img()
-{
-    echo qemu-img "$@" | _filter_testdir
-    $QEMU_IMG "$@" 2>&1 | _filter_testdir
-    echo
-}
-
 echo "=== Check option preallocation and cluster_size ==="
 echo
 cluster_sizes="16384 32768 65536 131072 262144 524288 1048576 2097152 4194304"
 
 for s in $cluster_sizes; do
-    test_qemu_img create -f $IMGFMT -o preallocation=metadata,cluster_size=$s "$TEST_IMG" 4G
+    IMGOPTS=$(_optstr_add "$IMGOPTS" "preallocation=metadata,cluster_size=$s") \
+        _make_test_img 4G
 done
 
 # success, all done
tests/qemu-iotests/079.out +10 -28
index ef4b8c9..6dc5d57 100644
@@ -1,32 +1,14 @@
 QA output created by 079
 === Check option preallocation and cluster_size ===
 
-qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=16384 TEST_DIR/t.qcow2 4G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=16384 preallocation='metadata' lazy_refcounts=off
-
-qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=32768 TEST_DIR/t.qcow2 4G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=32768 preallocation='metadata' lazy_refcounts=off
-
-qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=65536 TEST_DIR/t.qcow2 4G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 preallocation='metadata' lazy_refcounts=off
-
-qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=131072 TEST_DIR/t.qcow2 4G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=131072 preallocation='metadata' lazy_refcounts=off
-
-qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=262144 TEST_DIR/t.qcow2 4G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=262144 preallocation='metadata' lazy_refcounts=off
-
-qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=524288 TEST_DIR/t.qcow2 4G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=524288 preallocation='metadata' lazy_refcounts=off
-
-qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=1048576 TEST_DIR/t.qcow2 4G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=1048576 preallocation='metadata' lazy_refcounts=off
-
-qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=2097152 TEST_DIR/t.qcow2 4G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=2097152 preallocation='metadata' lazy_refcounts=off
-
-qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=4194304 TEST_DIR/t.qcow2 4G
-qemu-img: TEST_DIR/t.qcow2: Cluster size must be a power of two between 512 and 2048k
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=4194304 preallocation='metadata' lazy_refcounts=off
-
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata'
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata'
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata'
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata'
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata'
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata'
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata'
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata'
+qemu-img: TEST_DIR/t.IMGFMT: Cluster size must be a power of two between 512 and 2048k
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation='metadata'
 *** done
tests/qemu-iotests/080 +2 -0
index 73795f1..a2c58ae 100755
@@ -42,6 +42,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
+# Internal snapshots are (currently) impossible with refcount_bits=1
+_unsupported_imgopts 'refcount_bits=1[^0-9]'
 
 header_size=104
 
tests/qemu-iotests/089 +2 -0
index 6f74cec..3e0038d 100755
@@ -41,6 +41,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
+# Because anything other than 16 would change the output of qemu_io -c info
+_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)'
 
 # Using an image filename containing quotation marks will render the JSON data
 # below invalid. In that case, we have little choice but simply not to run this
tests/qemu-iotests/108 +2 -0
index 12fc92a..ce44749 100755
@@ -43,6 +43,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
+# This test directly modifies a refblock so it relies on refcount_bits being 16
+_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)'
 
 echo
 echo '=== Repairing an image without any refcount table ==='
tests/qemu-iotests/common.filter +2 -1
index 6c14590..a78a3b4 100644
@@ -191,7 +191,8 @@ _filter_img_create()
         -e "s# block_size=[0-9]\\+##g" \
         -e "s# block_state_zero=\\(on\\|off\\)##g" \
         -e "s# log_size=[0-9]\\+##g" \
-        -e "s/archipelago:a/TEST_DIR\//g"
+        -e "s/archipelago:a/TEST_DIR\//g" \
+        -e "s# refcount_bits=[0-9]\\+##g"
 }
 
 _filter_img_info()
Signature · 1 line
1.9.3
author
#32 lore raw

PATCH v5 16/26 qcow2: Allow creation with refcount order != 4

In reply to Max Reitz

Add a creation option to qcow2 for setting the refcount order of images to be created, and respect that option's value.

This breaks some test outputs, fix them.

---
 block/qcow2.c              |  20 +++++++++
 include/block/block_int.h  |   1 +
 tests/qemu-iotests/049.out | 108 ++++++++++++++++++++++-----------------------
 tests/qemu-iotests/082.out |  41 ++++++++++++++---
 tests/qemu-iotests/085.out |  38 ++++++++--------
 5 files changed, 128 insertions(+), 80 deletions(-)
block/qcow2.c +20 -0
index c0c0333..e6c2805 100644
@@ -2076,6 +2076,17 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
         goto finish;
     }
 
+    refcount_bits = qemu_opt_get_number_del(opts, BLOCK_OPT_REFCOUNT_BITS,
+                                            refcount_bits);
+    if (refcount_bits <= 0 || refcount_bits > 64 ||
+        !is_power_of_2(refcount_bits))
+    {
+        error_setg(errp, "Refcount width must be a power of two and may not "
+                   "exceed 64 bits");
+        ret = -EINVAL;
+        goto finish;
+    }
+
     if (version < 3 && refcount_bits != 16) {
         error_setg(errp, "Different refcount widths than 16 bits require "
                    "compatibility level 1.1 or above (use compat=1.1 or "
@@ -2715,6 +2726,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
         } else if (!strcmp(desc->name, BLOCK_OPT_LAZY_REFCOUNTS)) {
             lazy_refcounts = qemu_opt_get_bool(opts, BLOCK_OPT_LAZY_REFCOUNTS,
                                                lazy_refcounts);
+        } else if (!strcmp(desc->name, BLOCK_OPT_REFCOUNT_BITS)) {
+            error_report("Cannot change refcount entry width");
+            return -ENOTSUP;
         } else {
             /* if this assertion fails, this probably means a new option was
              * added without having it covered here */
@@ -2884,6 +2898,12 @@ static QemuOptsList qcow2_create_opts = {
             .help = "Postpone refcount updates",
             .def_value_str = "off"
         },
+        {
+            .name = BLOCK_OPT_REFCOUNT_BITS,
+            .type = QEMU_OPT_NUMBER,
+            .help = "Width of a reference count entry in bits",
+            .def_value_str = "16"
+        },
         { /* end of list */ }
     }
 };
include/block/block_int.h +1 -0
index 06a21dd..cfa0b8b 100644
@@ -56,6 +56,7 @@
 #define BLOCK_OPT_ADAPTER_TYPE      "adapter_type"
 #define BLOCK_OPT_REDUNDANCY        "redundancy"
 #define BLOCK_OPT_NOCOW             "nocow"
+#define BLOCK_OPT_REFCOUNT_BITS     "refcount_bits"
 
 #define BLOCK_PROBE_BUF_SIZE        512
 
tests/qemu-iotests/049.out +54 -54

249 lines not on this page · raw message

tests/qemu-iotests/082.out +34 -7

238 lines not on this page · raw message

tests/qemu-iotests/085.out +19 -19
index 0f2b17f..5eb8b94 100644
@@ -11,7 +11,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
 
 === Create a single snapshot on virtio0 ===
 
-Formatting 'TEST_DIR/1-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2.orig' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
+Formatting 'TEST_DIR/1-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2.orig' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
 {"return": {}}
 
 === Invalid command - missing device and nodename ===
@@ -25,31 +25,31 @@ Formatting 'TEST_DIR/1-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file
 
 === Create several transactional group snapshots ===
 
-Formatting 'TEST_DIR/2-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/1-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
-Formatting 'TEST_DIR/2-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
+Formatting 'TEST_DIR/2-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/1-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/2-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/t.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
 {"return": {}}
-Formatting 'TEST_DIR/3-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
-Formatting 'TEST_DIR/3-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
+Formatting 'TEST_DIR/3-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/3-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/2-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
 {"return": {}}
-Formatting 'TEST_DIR/4-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
-Formatting 'TEST_DIR/4-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
+Formatting 'TEST_DIR/4-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/4-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/3-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
 {"return": {}}
-Formatting 'TEST_DIR/5-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
-Formatting 'TEST_DIR/5-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
+Formatting 'TEST_DIR/5-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/5-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/4-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
 {"return": {}}
-Formatting 'TEST_DIR/6-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
-Formatting 'TEST_DIR/6-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
+Formatting 'TEST_DIR/6-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/6-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/5-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
 {"return": {}}
-Formatting 'TEST_DIR/7-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
-Formatting 'TEST_DIR/7-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
+Formatting 'TEST_DIR/7-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/7-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/6-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
 {"return": {}}
-Formatting 'TEST_DIR/8-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
-Formatting 'TEST_DIR/8-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
+Formatting 'TEST_DIR/8-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/8-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/7-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
 {"return": {}}
-Formatting 'TEST_DIR/9-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
-Formatting 'TEST_DIR/9-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
+Formatting 'TEST_DIR/9-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/9-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/8-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
 {"return": {}}
-Formatting 'TEST_DIR/10-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
-Formatting 'TEST_DIR/10-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off
+Formatting 'TEST_DIR/10-snapshot-v0.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v0.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/10-snapshot-v1.qcow2', fmt=qcow2 size=134217728 backing_file='TEST_DIR/9-snapshot-v1.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
 {"return": {}}
 *** done
Signature · 1 line
1.9.3
author
#33 lore raw

PATCH v5 17/26 progress: Allow regressing progress

In reply to Max Reitz

Progress may regress; this should be displayed correctly by qemu_progress_print().

While touching that area of code, drop the redundant parentheses in the same condition.

---
 util/qemu-progress.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
util/qemu-progress.c +2 -1
index 4ee5cd0..532333e 100644
@@ -152,7 +152,8 @@ void qemu_progress_print(float delta, int max)
     state.current = current;
 
     if (current > (state.last_print + state.min_skip) ||
-        (current == 100) || (current == 0)) {
+        current < (state.last_print - state.min_skip) ||
+        current == 100 || current == 0) {
         state.last_print = state.current;
         state.print();
     }
Signature · 1 line
1.9.3
author
#34 lore raw

PATCH v5 18/26 block: Add opaque value to the amend CB

In reply to Max Reitz

Add an opaque value which is to be passed to the bdrv_amend_options() status callback.

---
 block.c                   |  4 ++--
 block/qcow2-cluster.c     | 14 ++++++++------
 block/qcow2.c             |  9 +++++----
 block/qcow2.h             |  3 ++-
 include/block/block.h     |  4 ++--
 include/block/block_int.h |  3 ++-
 qemu-img.c                |  5 +++--
 7 files changed, 24 insertions(+), 18 deletions(-)
block.c +2 -2
index 4165d42..c2fcf39 100644
@@ -5817,12 +5817,12 @@ void bdrv_add_before_write_notifier(BlockDriverState *bs,
 }
 
 int bdrv_amend_options(BlockDriverState *bs, QemuOpts *opts,
-                       BlockDriverAmendStatusCB *status_cb)
+                       BlockDriverAmendStatusCB *status_cb, void *cb_opaque)
 {
     if (!bs->drv->bdrv_amend_options) {
         return -ENOTSUP;
     }
-    return bs->drv->bdrv_amend_options(bs, opts, status_cb);
+    return bs->drv->bdrv_amend_options(bs, opts, status_cb, cb_opaque);
 }
 
 /* This function will be called by the bdrv_recurse_is_first_non_filter method
block/qcow2-cluster.c +8 -6
index a86bb5e..6d06058 100644
@@ -1620,7 +1620,8 @@ fail:
 static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
                                       int l1_size, int64_t *visited_l1_entries,
                                       int64_t l1_entries,
-                                      BlockDriverAmendStatusCB *status_cb)
+                                      BlockDriverAmendStatusCB *status_cb,
+                                      void *cb_opaque)
 {
     BDRVQcowState *s = bs->opaque;
     bool is_active_l1 = (l1_table == s->l1_table);
@@ -1646,7 +1647,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
             /* unallocated */
             (*visited_l1_entries)++;
             if (status_cb) {
-                status_cb(bs, *visited_l1_entries, l1_entries);
+                status_cb(bs, *visited_l1_entries, l1_entries, cb_opaque);
             }
             continue;
         }
@@ -1765,7 +1766,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
 
         (*visited_l1_entries)++;
         if (status_cb) {
-            status_cb(bs, *visited_l1_entries, l1_entries);
+            status_cb(bs, *visited_l1_entries, l1_entries, cb_opaque);
         }
     }
 
@@ -1794,7 +1795,8 @@ fail:
  * qcow2 version which doesn't yet support metadata zero clusters.
  */
 int qcow2_expand_zero_clusters(BlockDriverState *bs,
-                               BlockDriverAmendStatusCB *status_cb)
+                               BlockDriverAmendStatusCB *status_cb,
+                               void *cb_opaque)
 {
     BDRVQcowState *s = bs->opaque;
     uint64_t *l1_table = NULL;
@@ -1811,7 +1813,7 @@ int qcow2_expand_zero_clusters(BlockDriverState *bs,
 
     ret = expand_zero_clusters_in_l1(bs, s->l1_table, s->l1_size,
                                      &visited_l1_entries, l1_entries,
-                                     status_cb);
+                                     status_cb, cb_opaque);
     if (ret < 0) {
         goto fail;
     }
@@ -1846,7 +1848,7 @@ int qcow2_expand_zero_clusters(BlockDriverState *bs,
 
         ret = expand_zero_clusters_in_l1(bs, l1_table, s->snapshots[i].l1_size,
                                          &visited_l1_entries, l1_entries,
-                                         status_cb);
+                                         status_cb, cb_opaque);
         if (ret < 0) {
             goto fail;
         }
block/qcow2.c +5 -4
index e6c2805..dfdab18 100644
@@ -2602,7 +2602,7 @@ static int qcow2_load_vmstate(BlockDriverState *bs, uint8_t *buf,
  * have to be removed.
  */
 static int qcow2_downgrade(BlockDriverState *bs, int target_version,
-                           BlockDriverAmendStatusCB *status_cb)
+                           BlockDriverAmendStatusCB *status_cb, void *cb_opaque)
 {
     BDRVQcowState *s = bs->opaque;
     int current_version = s->qcow_version;
@@ -2651,7 +2651,7 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,
     /* clearing autoclear features is trivial */
     s->autoclear_features = 0;
 
-    ret = qcow2_expand_zero_clusters(bs, status_cb);
+    ret = qcow2_expand_zero_clusters(bs, status_cb, cb_opaque);
     if (ret < 0) {
         return ret;
     }
@@ -2666,7 +2666,8 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,
 }
 
 static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
-                               BlockDriverAmendStatusCB *status_cb)
+                               BlockDriverAmendStatusCB *status_cb,
+                               void *cb_opaque)
 {
     BDRVQcowState *s = bs->opaque;
     int old_version = s->qcow_version, new_version = old_version;
@@ -2748,7 +2749,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
                 return ret;
             }
         } else {
-            ret = qcow2_downgrade(bs, new_version, status_cb);
+            ret = qcow2_downgrade(bs, new_version, status_cb, cb_opaque);
             if (ret < 0) {
                 return ret;
             }
block/qcow2.h +2 -1
index 6070e74..c6b48c0 100644
@@ -552,7 +552,8 @@ int qcow2_discard_clusters(BlockDriverState *bs, uint64_t offset,
 int qcow2_zero_clusters(BlockDriverState *bs, uint64_t offset, int nb_sectors);
 
 int qcow2_expand_zero_clusters(BlockDriverState *bs,
-                               BlockDriverAmendStatusCB *status_cb);
+                               BlockDriverAmendStatusCB *status_cb,
+                               void *cb_opaque);
 
 /* qcow2-snapshot.c functions */
 int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info);
include/block/block.h +2 -2
index 6e7275d..bd1fdde 100644
@@ -274,9 +274,9 @@ int bdrv_check(BlockDriverState *bs, BdrvCheckResult *res, BdrvCheckMode fix);
  * block driver; total_work_size may change during the course of the amendment
  * operation */
 typedef void BlockDriverAmendStatusCB(BlockDriverState *bs, int64_t offset,
-                                      int64_t total_work_size);
+                                      int64_t total_work_size, void *opaque);
 int bdrv_amend_options(BlockDriverState *bs_new, QemuOpts *opts,
-                       BlockDriverAmendStatusCB *status_cb);
+                       BlockDriverAmendStatusCB *status_cb, void *cb_opaque);
 
 /* external snapshots */
 bool bdrv_recurse_is_first_non_filter(BlockDriverState *bs,
include/block/block_int.h +2 -1
index cfa0b8b..7cc5fe1 100644
@@ -236,7 +236,8 @@ struct BlockDriver {
         BdrvCheckMode fix);
 
     int (*bdrv_amend_options)(BlockDriverState *bs, QemuOpts *opts,
-                              BlockDriverAmendStatusCB *status_cb);
+                              BlockDriverAmendStatusCB *status_cb,
+                              void *cb_opaque);
 
     void (*bdrv_debug_event)(BlockDriverState *bs, BlkDebugEvent event);
 
qemu-img.c +3 -2
index 7876258..c6d3400 100644
@@ -2883,7 +2883,8 @@ out:
 }
 
 static void amend_status_cb(BlockDriverState *bs,
-                            int64_t offset, int64_t total_work_size)
+                            int64_t offset, int64_t total_work_size,
+                            void *opaque)
 {
     qemu_progress_print(100.f * offset / total_work_size, 0);
 }
@@ -3003,7 +3004,7 @@ static int img_amend(int argc, char **argv)
 
     /* In case the driver does not call amend_status_cb() */
     qemu_progress_print(0.f, 0);
-    ret = bdrv_amend_options(bs, opts, &amend_status_cb);
+    ret = bdrv_amend_options(bs, opts, &amend_status_cb, NULL);
     qemu_progress_print(100.f, 0);
     if (ret < 0) {
         error_report("Error while amending options: %s", strerror(-ret));
Signature · 1 line
1.9.3
author
#35 lore raw

PATCH v5 19/26 qcow2: Use error_report() in qcow2_amend_options()

In reply to Max Reitz
---
 block/qcow2.c              | 14 ++++++--------
 tests/qemu-iotests/061.out | 14 +++++++-------
 2 files changed, 13 insertions(+), 15 deletions(-)
block/qcow2.c +6 -8
index dfdab18..c174a23 100644
@@ -2696,11 +2696,11 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
             } else if (!strcmp(compat, "1.1")) {
                 new_version = 3;
             } else {
-                fprintf(stderr, "Unknown compatibility level %s.\n", compat);
+                error_report("Unknown compatibility level %s", compat);
                 return -EINVAL;
             }
         } else if (!strcmp(desc->name, BLOCK_OPT_PREALLOC)) {
-            fprintf(stderr, "Cannot change preallocation mode.\n");
+            error_report("Cannot change preallocation mode");
             return -ENOTSUP;
         } else if (!strcmp(desc->name, BLOCK_OPT_SIZE)) {
             new_size = qemu_opt_get_size(opts, BLOCK_OPT_SIZE, 0);
@@ -2712,16 +2712,14 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
             encrypt = qemu_opt_get_bool(opts, BLOCK_OPT_ENCRYPT,
                                         s->crypt_method);
             if (encrypt != !!s->crypt_method) {
-                fprintf(stderr, "Changing the encryption flag is not "
-                        "supported.\n");
+                error_report("Changing the encryption flag is not supported");
                 return -ENOTSUP;
             }
         } else if (!strcmp(desc->name, BLOCK_OPT_CLUSTER_SIZE)) {
             cluster_size = qemu_opt_get_size(opts, BLOCK_OPT_CLUSTER_SIZE,
                                              cluster_size);
             if (cluster_size != s->cluster_size) {
-                fprintf(stderr, "Changing the cluster size is not "
-                        "supported.\n");
+                error_report("Changing the cluster size is not supported");
                 return -ENOTSUP;
             }
         } else if (!strcmp(desc->name, BLOCK_OPT_LAZY_REFCOUNTS)) {
@@ -2767,8 +2765,8 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
     if (s->use_lazy_refcounts != lazy_refcounts) {
         if (lazy_refcounts) {
             if (s->qcow_version < 3) {
-                fprintf(stderr, "Lazy refcounts only supported with compatibility "
-                        "level 1.1 and above (use compat=1.1 or greater)\n");
+                error_report("Lazy refcounts only supported with compatibility "
+                             "level 1.1 and above (use compat=1.1 or greater)");
                 return -EINVAL;
             }
             s->compatible_features |= QCOW2_COMPAT_LAZY_REFCOUNTS;
tests/qemu-iotests/061.out +7 -7
index e70f983..2cd9eee 100644
@@ -281,19 +281,19 @@ No errors were found on the image.
 === Testing invalid configurations ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater)
+qemu-img: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater)
 qemu-img: Error while amending options: Invalid argument
-Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater)
+qemu-img: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater)
 qemu-img: Error while amending options: Invalid argument
-Unknown compatibility level 0.42.
+qemu-img: Unknown compatibility level 0.42
 qemu-img: Error while amending options: Invalid argument
 qemu-img: Invalid parameter 'foo'
 qemu-img: Invalid options for file format 'qcow2'
-Changing the cluster size is not supported.
+qemu-img: Changing the cluster size is not supported
 qemu-img: Error while amending options: Operation not supported
-Changing the encryption flag is not supported.
+qemu-img: Changing the encryption flag is not supported
 qemu-img: Error while amending options: Operation not supported
-Cannot change preallocation mode.
+qemu-img: Cannot change preallocation mode
 qemu-img: Error while amending options: Operation not supported
 
 === Testing correct handling of unset value ===
@@ -301,7 +301,7 @@ qemu-img: Error while amending options: Operation not supported
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 Should work:
 Should not work:
-Changing the cluster size is not supported.
+qemu-img: Changing the cluster size is not supported
 qemu-img: Error while amending options: Operation not supported
 
 === Testing zero expansion on inactive clusters ===
Signature · 1 line
1.9.3
author
#36 lore raw

PATCH v5 20/26 qcow2: Use abort() instead of assert(false)

In reply to Max Reitz
---
 block/qcow2.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
block/qcow2.c +2 -2
index c174a23..258e0df 100644
@@ -2729,9 +2729,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
             error_report("Cannot change refcount entry width");
             return -ENOTSUP;
         } else {
-            /* if this assertion fails, this probably means a new option was
+            /* if this point is reached, this probably means a new option was
              * added without having it covered here */
-            assert(false);
+            abort();
         }
 
         desc++;
Signature · 1 line
1.9.3
author
#37 lore raw

PATCH v5 21/26 qcow2: Split upgrade/downgrade paths for amend

In reply to Max Reitz

If the image version should be upgraded, that is the first we should do; if it should be downgraded, that is the last we should do. So split the version change block into an upgrade part at the start and a downgrade part at the end.

---
 block/qcow2.c | 31 ++++++++++++++++---------------
 1 file changed, 16 insertions(+), 15 deletions(-)
block/qcow2.c +16 -15
index 258e0df..6f9e4c8 100644
@@ -2737,20 +2737,13 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
         desc++;
     }
 
-    if (new_version != old_version) {
-        if (new_version > old_version) {
-            /* Upgrade */
-            s->qcow_version = new_version;
-            ret = qcow2_update_header(bs);
-            if (ret < 0) {
-                s->qcow_version = old_version;
-                return ret;
-            }
-        } else {
-            ret = qcow2_downgrade(bs, new_version, status_cb, cb_opaque);
-            if (ret < 0) {
-                return ret;
-            }
+    /* Upgrade first (some features may require compat=1.1) */
+    if (new_version > old_version) {
+        s->qcow_version = new_version;
+        ret = qcow2_update_header(bs);
+        if (ret < 0) {
+            s->qcow_version = old_version;
+            return ret;
         }
     }
 
@@ -2764,7 +2757,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
 
     if (s->use_lazy_refcounts != lazy_refcounts) {
         if (lazy_refcounts) {
-            if (s->qcow_version < 3) {
+            if (new_version < 3) {
                 error_report("Lazy refcounts only supported with compatibility "
                              "level 1.1 and above (use compat=1.1 or greater)");
                 return -EINVAL;
@@ -2800,6 +2793,14 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
         }
     }
 
+    /* Downgrade last (so unsupported features can be removed before) */
+    if (new_version < old_version) {
+        ret = qcow2_downgrade(bs, new_version, status_cb, cb_opaque);
+        if (ret < 0) {
+            return ret;
+        }
+    }
+
     return 0;
 }
 
Signature · 1 line
1.9.3
author
#38 lore raw

PATCH v5 22/26 qcow2: Use intermediate helper CB for amend

In reply to Max Reitz

If there is more than one time-consuming operation to be performed for qcow2_amend_options(), we need an intermediate CB which coordinates the progress of the individual operations and passes the result to the original status callback.

---
 block/qcow2.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 79 insertions(+), 1 deletion(-)
block/qcow2.c +79 -1
index 6f9e4c8..387a02e 100644
@@ -2665,6 +2665,75 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,
     return 0;
 }
 
+typedef enum Qcow2AmendOperation {
+    /* This is the value Qcow2AmendHelperCBInfo::last_operation will be
+     * statically initialized to so that the helper CB can discern the first
+     * invocation from an operation change */
+    QCOW2_NO_OPERATION = 0,
+
+    QCOW2_DOWNGRADING,
+} Qcow2AmendOperation;
+
+typedef struct Qcow2AmendHelperCBInfo {
+    /* The code coordinating the amend operations should only modify
+     * these four fields; the rest will be managed by the CB */
+    BlockDriverAmendStatusCB *original_status_cb;
+    void *original_cb_opaque;
+
+    Qcow2AmendOperation current_operation;
+
+    /* Total number of operations to perform (only set once) */
+    int total_operations;
+
+    /* The following fields are managed by the CB */
+
+    /* Number of operations completed */
+    int operations_completed;
+
+    /* Cumulative offset of all completed operations */
+    int64_t offset_completed;
+
+    Qcow2AmendOperation last_operation;
+    int64_t last_work_size;
+} Qcow2AmendHelperCBInfo;
+
+static void qcow2_amend_helper_cb(BlockDriverState *bs,
+                                  int64_t operation_offset,
+                                  int64_t operation_work_size, void *opaque)
+{
+    Qcow2AmendHelperCBInfo *info = opaque;
+    int64_t current_work_size;
+    int64_t projected_work_size;
+
+    if (info->current_operation != info->last_operation) {
+        if (info->last_operation != QCOW2_NO_OPERATION) {
+            info->offset_completed += info->last_work_size;
+            info->operations_completed++;
+        }
+
+        info->last_operation = info->current_operation;
+    }
+
+    assert(info->total_operations > 0);
+    assert(info->operations_completed < info->total_operations);
+
+    info->last_work_size = operation_work_size;
+
+    current_work_size = info->offset_completed + operation_work_size;
+
+    /* current_work_size is the total work size for (operations_completed + 1)
+     * operations (which includes this one), so multiply it by the number of
+     * operations not covered and divide it by the number of operations
+     * covered to get a projection for the operations not covered */
+    projected_work_size = current_work_size * (info->total_operations -
+                                               info->operations_completed - 1)
+                                            / (info->operations_completed + 1);
+
+    info->original_status_cb(bs, info->offset_completed + operation_offset,
+                             current_work_size + projected_work_size,
+                             info->original_cb_opaque);
+}
+
 static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
                                BlockDriverAmendStatusCB *status_cb,
                                void *cb_opaque)
@@ -2679,6 +2748,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
     bool encrypt;
     int ret;
     QemuOptDesc *desc = opts->list->desc;
+    Qcow2AmendHelperCBInfo helper_cb_info;
 
     while (desc && desc->name) {
         if (!qemu_opt_find(opts, desc->name)) {
@@ -2737,6 +2807,12 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
         desc++;
     }
 
+    helper_cb_info = (Qcow2AmendHelperCBInfo){
+        .original_status_cb = status_cb,
+        .original_cb_opaque = cb_opaque,
+        .total_operations = (new_version < old_version)
+    };
+
     /* Upgrade first (some features may require compat=1.1) */
     if (new_version > old_version) {
         s->qcow_version = new_version;
@@ -2795,7 +2871,9 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
 
     /* Downgrade last (so unsupported features can be removed before) */
     if (new_version < old_version) {
-        ret = qcow2_downgrade(bs, new_version, status_cb, cb_opaque);
+        helper_cb_info.current_operation = QCOW2_DOWNGRADING;
+        ret = qcow2_downgrade(bs, new_version, &qcow2_amend_helper_cb,
+                              &helper_cb_info);
         if (ret < 0) {
             return ret;
         }
Signature · 1 line
1.9.3
author
#39 lore raw

PATCH v5 23/26 qcow2: Add function for refcount order amendment

In reply to Max Reitz

Add a function qcow2_change_refcount_order() which allows changing the refcount order of a qcow2 image.

---
 block/qcow2-refcount.c | 452 +++++++++++++++++++++++++++++++++++++++++++++++++
 block/qcow2.h          |   4 +
 2 files changed, 456 insertions(+)
block/qcow2-refcount.c +452 -0
index e37fcdf..7c3f8e0 100644
@@ -2491,3 +2491,455 @@ int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset,
 
     return 0;
 }
+
+/* A pointer to a function of this type is given to walk_over_reftable(). That
+ * function will create refblocks and pass them to a RefblockFinishOp once they
+ * are completed (@refblock). @refblock_empty is set if the refblock is
+ * completely empty.
+ *
+ * Along with the refblock, a corresponding reftable entry is passed, in the
+ * reftable @reftable (which may be reallocated) at @reftable_index.
+ *
+ * @allocated should be set to true if a new cluster has been allocated.
+ */
+typedef int (RefblockFinishOp)(BlockDriverState *bs, uint64_t **reftable,
+                               uint64_t reftable_index, uint64_t *reftable_size,
+                               void *refblock, bool refblock_empty,
+                               bool *allocated, Error **errp);
+
+/**
+ * This "operation" for walk_over_reftable() allocates the refblock on disk (if
+ * it is not empty) and inserts its offset into the new reftable. The size of
+ * this new reftable is increased as required.
+ */
+static int alloc_refblock(BlockDriverState *bs, uint64_t **reftable,
+                          uint64_t reftable_index, uint64_t *reftable_size,
+                          void *refblock, bool refblock_empty, bool *allocated,
+                          Error **errp)
+{
+    BDRVQcowState *s = bs->opaque;
+    int64_t offset;
+
+    if (!refblock_empty && reftable_index >= *reftable_size) {
+        uint64_t *new_reftable;
+        uint64_t new_reftable_size;
+
+        new_reftable_size = ROUND_UP(reftable_index + 1,
+                                     s->cluster_size / sizeof(uint64_t));
+        if (new_reftable_size > QCOW_MAX_REFTABLE_SIZE / sizeof(uint64_t)) {
+            error_setg(errp,
+                       "This operation would make the refcount table grow "
+                       "beyond the maximum size supported by QEMU, aborting");
+            return -ENOTSUP;
+        }
+
+        new_reftable = g_try_realloc(*reftable, new_reftable_size *
+                                                sizeof(uint64_t));
+        if (!new_reftable) {
+            error_setg(errp, "Failed to increase reftable buffer size");
+            return -ENOMEM;
+        }
+
+        memset(new_reftable + *reftable_size, 0,
+               (new_reftable_size - *reftable_size) * sizeof(uint64_t));
+
+        *reftable      = new_reftable;
+        *reftable_size = new_reftable_size;
+    }
+
+    if (!refblock_empty && !(*reftable)[reftable_index]) {
+        offset = qcow2_alloc_clusters(bs, s->cluster_size);
+        if (offset < 0) {
+            error_setg_errno(errp, -offset, "Failed to allocate refblock");
+            return offset;
+        }
+        (*reftable)[reftable_index] = offset;
+        *allocated = true;
+    }
+
+    return 0;
+}
+
+/**
+ * This "operation" for walk_over_reftable() writes the refblock to disk at the
+ * offset specified by the new reftable's entry. It does not modify the new
+ * reftable or change any refcounts.
+ */
+static int flush_refblock(BlockDriverState *bs, uint64_t **reftable,
+                          uint64_t reftable_index, uint64_t *reftable_size,
+                          void *refblock, bool refblock_empty, bool *allocated,
+                          Error **errp)
+{
+    BDRVQcowState *s = bs->opaque;
+    int64_t offset;
+    int ret;
+
+    if (reftable_index < *reftable_size && (*reftable)[reftable_index]) {
+        offset = (*reftable)[reftable_index];
+
+        ret = qcow2_pre_write_overlap_check(bs, 0, offset, s->cluster_size);
+        if (ret < 0) {
+            error_setg_errno(errp, -ret, "Overlap check failed");
+            return ret;
+        }
+
+        ret = bdrv_pwrite(bs->file, offset, refblock, s->cluster_size);
+        if (ret < 0) {
+            error_setg_errno(errp, -ret, "Failed to write refblock");
+            return ret;
+        }
+    } else {
+        assert(refblock_empty);
+    }
+
+    return 0;
+}
+
+/**
+ * This function walks over the existing reftable and every referenced refblock;
+ * if @new_set_refcount is non-NULL, it is called for every refcount entry to
+ * create an equal new entry in the passed @new_refblock. Once that
+ * @new_refblock is completely filled, @operation will be called.
+ *
+ * @status_cb and @cb_opaque are used for the amend operation's status callback.
+ * @index is the index of the walk_over_reftable() calls and @total is the total
+ * number of walk_over_reftable() calls per amend operation. Both are used for
+ * calculating the parameters for the status callback.
+ *
+ * @allocated is set to true if a new cluster has been allocated.
+ */
+static int walk_over_reftable(BlockDriverState *bs, uint64_t **new_reftable,
+                              uint64_t *new_reftable_index,
+                              uint64_t *new_reftable_size,
+                              void *new_refblock, int new_refblock_size,
+                              int new_refcount_bits,
+                              RefblockFinishOp *operation, bool *allocated,
+                              Qcow2SetRefcountFunc *new_set_refcount,
+                              BlockDriverAmendStatusCB *status_cb,
+                              void *cb_opaque, int index, int total,
+                              Error **errp)
+{
+    BDRVQcowState *s = bs->opaque;
+    uint64_t reftable_index;
+    bool new_refblock_empty = true;
+    int refblock_index;
+    int new_refblock_index = 0;
+    int ret;
+
+    for (reftable_index = 0; reftable_index < s->refcount_table_size;
+         reftable_index++)
+    {
+        uint64_t refblock_offset = s->refcount_table[reftable_index]
+                                 & REFT_OFFSET_MASK;
+
+        status_cb(bs, (uint64_t)index * s->refcount_table_size + reftable_index,
+                  (uint64_t)total * s->refcount_table_size, cb_opaque);
+
+        if (refblock_offset) {
+            void *refblock;
+
+            if (offset_into_cluster(s, refblock_offset)) {
+                qcow2_signal_corruption(bs, true, -1, -1, "Refblock offset %#"
+                                        PRIx64 " unaligned (reftable index: %#"
+                                        PRIx64 ")", refblock_offset,
+                                        reftable_index);
+                error_setg(errp,
+                           "Image is corrupt (unaligned refblock offset)");
+                return -EIO;
+            }
+
+            ret = qcow2_cache_get(bs, s->refcount_block_cache, refblock_offset,
+                                  &refblock);
+            if (ret < 0) {
+                error_setg_errno(errp, -ret, "Failed to retrieve refblock");
+                return ret;
+            }
+
+            for (refblock_index = 0; refblock_index < s->refcount_block_size;
+                 refblock_index++)
+            {
+                uint64_t refcount;
+
+                if (new_refblock_index >= new_refblock_size) {
+                    /* new_refblock is now complete */
+                    ret = operation(bs, new_reftable, *new_reftable_index,
+                                    new_reftable_size, new_refblock,
+                                    new_refblock_empty, allocated, errp);
+                    if (ret < 0) {
+                        qcow2_cache_put(bs, s->refcount_block_cache, &refblock);
+                        return ret;
+                    }
+
+                    (*new_reftable_index)++;
+                    new_refblock_index = 0;
+                    new_refblock_empty = true;
+                }
+
+                refcount = s->get_refcount(refblock, refblock_index);
+                if (new_refcount_bits < 64 && refcount >> new_refcount_bits) {
+                    uint64_t offset;
+
+                    qcow2_cache_put(bs, s->refcount_block_cache, &refblock);
+
+                    offset = ((reftable_index << s->refcount_block_bits)
+                              + refblock_index) << s->cluster_bits;
+
+                    error_setg(errp, "Cannot decrease refcount entry width to "
+                               "%i bits: Cluster at offset %#" PRIx64 " has a "
+                               "refcount of %" PRIu64, new_refcount_bits,
+                               offset, refcount);
+                    return -EINVAL;
+                }
+
+                if (new_set_refcount) {
+                    new_set_refcount(new_refblock, new_refblock_index++,
+                                     refcount);
+                } else {
+                    new_refblock_index++;
+                }
+                new_refblock_empty = new_refblock_empty && refcount == 0;
+            }
+
+            ret = qcow2_cache_put(bs, s->refcount_block_cache, &refblock);
+            if (ret < 0) {
+                error_setg_errno(errp, -ret, "Failed to put refblock back into "
+                                 "the cache");
+                return ret;
+            }
+        } else {
+            /* No refblock means every refcount is 0 */
+            for (refblock_index = 0; refblock_index < s->refcount_block_size;
+                 refblock_index++)
+            {
+                if (new_refblock_index >= new_refblock_size) {
+                    /* new_refblock is now complete */
+                    ret = operation(bs, new_reftable, *new_reftable_index,
+                                    new_reftable_size, new_refblock,
+                                    new_refblock_empty, allocated, errp);
+                    if (ret < 0) {
+                        return ret;
+                    }
+
+                    (*new_reftable_index)++;
+                    new_refblock_index = 0;
+                    new_refblock_empty = true;
+                }
+
+                if (new_set_refcount) {
+                    new_set_refcount(new_refblock, new_refblock_index++, 0);
+                } else {
+                    new_refblock_index++;
+                }
+            }
+        }
+    }
+
+    if (new_refblock_index > 0) {
+        /* Complete the potentially existing partially filled final refblock */
+        if (new_set_refcount) {
+            for (; new_refblock_index < new_refblock_size;
+                 new_refblock_index++)
+            {
+                new_set_refcount(new_refblock, new_refblock_index, 0);
+            }
+        }
+
+        ret = operation(bs, new_reftable, *new_reftable_index,
+                        new_reftable_size, new_refblock, new_refblock_empty,
+                        allocated, errp);
+        if (ret < 0) {
+            return ret;
+        }
+
+        (*new_reftable_index)++;
+    }
+
+    status_cb(bs, (uint64_t)(index + 1) * s->refcount_table_size,
+              (uint64_t)total * s->refcount_table_size, cb_opaque);
+
+    return 0;
+}
+
+int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order,
+                                BlockDriverAmendStatusCB *status_cb,
+                                void *cb_opaque, Error **errp)
+{
+    BDRVQcowState *s = bs->opaque;
+    Qcow2GetRefcountFunc *new_get_refcount;
+    Qcow2SetRefcountFunc *new_set_refcount;
+    void *new_refblock = qemu_blockalign(bs->file, s->cluster_size);
+    uint64_t *new_reftable = NULL, new_reftable_size = 0;
+    uint64_t *old_reftable, old_reftable_size, old_reftable_offset;
+    uint64_t new_reftable_index = 0;
+    uint64_t i;
+    int64_t new_reftable_offset = 0, allocated_reftable_size = 0;
+    int new_refblock_size, new_refcount_bits = 1 << refcount_order;
+    int old_refcount_order;
+    int walk_index = 0;
+    int ret;
+    bool new_allocation;
+
+    assert(s->qcow_version >= 3);
+    assert(refcount_order >= 0 && refcount_order <= 6);
+
+    /* see qcow2_open() */
+    new_refblock_size = 1 << (s->cluster_bits - (refcount_order - 3));
+
+    new_get_refcount = get_refcount_funcs[refcount_order];
+    new_set_refcount = set_refcount_funcs[refcount_order];
+
+
+    do {
+        int total_walks;
+
+        new_allocation = false;
+
+        /* At least we have to do this walk and the one which writes the
+         * refblocks; also, at least we have to do this loop here at least
+         * twice (normally), first to do the allocations, and second to
+         * determine that everything is correctly allocated, this then makes
+         * three walks in total */
+        total_walks = MAX(walk_index + 2, 3);
+
+        /* First, allocate the structures so they are present in the refcount
+         * structures */
+        ret = walk_over_reftable(bs, &new_reftable, &new_reftable_index,
+                                 &new_reftable_size, NULL, new_refblock_size,
+                                 new_refcount_bits, &alloc_refblock,
+                                 &new_allocation, NULL, status_cb, cb_opaque,
+                                 walk_index++, total_walks, errp);
+        if (ret < 0) {
+            goto done;
+        }
+
+        new_reftable_index = 0;
+
+        if (new_allocation) {
+            if (new_reftable_offset) {
+                qcow2_free_clusters(bs, new_reftable_offset,
+                                    allocated_reftable_size * sizeof(uint64_t),
+                                    QCOW2_DISCARD_NEVER);
+            }
+
+            new_reftable_offset = qcow2_alloc_clusters(bs, new_reftable_size *
+                                                           sizeof(uint64_t));
+            if (new_reftable_offset < 0) {
+                error_setg_errno(errp, -new_reftable_offset,
+                                 "Failed to allocate the new reftable");
+                ret = new_reftable_offset;
+                goto done;
+            }
+            allocated_reftable_size = new_reftable_size;
+        }
+    } while (new_allocation);
+
+    /* Second, write the new refblocks */
+    ret = walk_over_reftable(bs, &new_reftable, &new_reftable_index,
+                             &new_reftable_size, new_refblock,
+                             new_refblock_size, new_refcount_bits,
+                             &flush_refblock, &new_allocation, new_set_refcount,
+                             status_cb, cb_opaque, walk_index, walk_index + 1,
+                             errp);
+    if (ret < 0) {
+        goto done;
+    }
+    assert(!new_allocation);
+
+
+    /* Write the new reftable */
+    ret = qcow2_pre_write_overlap_check(bs, 0, new_reftable_offset,
+                                        new_reftable_size * sizeof(uint64_t));
+    if (ret < 0) {
+        error_setg_errno(errp, -ret, "Overlap check failed");
+        goto done;
+    }
+
+    for (i = 0; i < new_reftable_size; i++) {
+        cpu_to_be64s(&new_reftable[i]);
+    }
+
+    ret = bdrv_pwrite(bs->file, new_reftable_offset, new_reftable,
+                      new_reftable_size * sizeof(uint64_t));
+
+    for (i = 0; i < new_reftable_size; i++) {
+        be64_to_cpus(&new_reftable[i]);
+    }
+
+    if (ret < 0) {
+        error_setg_errno(errp, -ret, "Failed to write the new reftable");
+        goto done;
+    }
+
+
+    /* Empty the refcount cache */
+    ret = qcow2_cache_flush(bs, s->refcount_block_cache);
+    if (ret < 0) {
+        error_setg_errno(errp, -ret, "Failed to flush the refblock cache");
+        goto done;
+    }
+
+    /* Update the image header to point to the new reftable; this only updates
+     * the fields which are relevant to qcow2_update_header(); other fields
+     * such as s->refcount_table or s->refcount_bits stay stale for now
+     * (because we have to restore everything if qcow2_update_header() fails) */
+    old_refcount_order  = s->refcount_order;
+    old_reftable_size   = s->refcount_table_size;
+    old_reftable_offset = s->refcount_table_offset;
+
+    s->refcount_order        = refcount_order;
+    s->refcount_table_size   = new_reftable_size;
+    s->refcount_table_offset = new_reftable_offset;
+
+    ret = qcow2_update_header(bs);
+    if (ret < 0) {
+        s->refcount_order        = old_refcount_order;
+        s->refcount_table_size   = old_reftable_size;
+        s->refcount_table_offset = old_reftable_offset;
+        error_setg_errno(errp, -ret, "Failed to update the qcow2 header");
+        goto done;
+    }
+
+    /* Now update the rest of the in-memory information */
+    old_reftable = s->refcount_table;
+    s->refcount_table = new_reftable;
+
+    s->refcount_bits = 1 << refcount_order;
+    s->refcount_max = UINT64_C(1) << (s->refcount_bits - 1);
+    s->refcount_max += s->refcount_max - 1;
+
+    s->refcount_block_bits = s->cluster_bits - (refcount_order - 3);
+    s->refcount_block_size = 1 << s->refcount_block_bits;
+
+    s->get_refcount = new_get_refcount;
+    s->set_refcount = new_set_refcount;
+
+    /* For cleaning up all old refblocks and the old reftable below the "done"
+     * label */
+    new_reftable        = old_reftable;
+    new_reftable_size   = old_reftable_size;
+    new_reftable_offset = old_reftable_offset;
+
+done:
+    if (new_reftable) {
+        /* On success, new_reftable actually points to the old reftable (and
+         * new_reftable_size is the old reftable's size); but that is just
+         * fine */
+        for (i = 0; i < new_reftable_size; i++) {
+            uint64_t offset = new_reftable[i] & REFT_OFFSET_MASK;
+            if (offset) {
+                qcow2_free_clusters(bs, offset, s->cluster_size,
+                                    QCOW2_DISCARD_OTHER);
+            }
+        }
+        g_free(new_reftable);
+
+        if (new_reftable_offset > 0) {
+            qcow2_free_clusters(bs, new_reftable_offset,
+                                new_reftable_size * sizeof(uint64_t),
+                                QCOW2_DISCARD_OTHER);
+        }
+    }
+
+    qemu_vfree(new_refblock);
+    return ret;
+}
block/qcow2.h +4 -0
index c6b48c0..c64677e 100644
@@ -527,6 +527,10 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
 int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset,
                                   int64_t size);
 
+int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order,
+                                BlockDriverAmendStatusCB *status_cb,
+                                void *cb_opaque, Error **errp);
+
 /* qcow2-cluster.c functions */
 int qcow2_grow_l1_table(BlockDriverState *bs, uint64_t min_size,
                         bool exact_size);
Signature · 1 line
1.9.3
author
#40 lore raw

PATCH v5 24/26 qcow2: Invoke refcount order amendment function

In reply to Max Reitz

Make use of qcow2_change_refcount_order() to support changing the refcount order with qemu-img amend.

---
 block/qcow2.c | 44 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 9 deletions(-)
block/qcow2.c +35 -9
index 387a02e..8b22564 100644
@@ -2617,13 +2617,7 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,
     }
 
     if (s->refcount_order != 4) {
-        /* we would have to convert the image to a refcount_order == 4 image
-         * here; however, since qemu (at the time of writing this) does not
-         * support anything different than 4 anyway, there is no point in doing
-         * so right now; however, we should error out (if qemu supports this in
-         * the future and this code has not been adapted) */
-        error_report("qcow2_downgrade: Image refcount orders other than 4 are "
-                     "currently not supported.");
+        error_report("compat=0.10 requires refcount_bits=16");
         return -ENOTSUP;
     }
 
@@ -2671,6 +2665,7 @@ typedef enum Qcow2AmendOperation {
      * invocation from an operation change */
     QCOW2_NO_OPERATION = 0,
 
+    QCOW2_CHANGING_REFCOUNT_ORDER,
     QCOW2_DOWNGRADING,
 } Qcow2AmendOperation;
 
@@ -2746,6 +2741,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
     const char *compat = NULL;
     uint64_t cluster_size = s->cluster_size;
     bool encrypt;
+    int refcount_bits = s->refcount_bits;
     int ret;
     QemuOptDesc *desc = opts->list->desc;
     Qcow2AmendHelperCBInfo helper_cb_info;
@@ -2796,8 +2792,16 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
             lazy_refcounts = qemu_opt_get_bool(opts, BLOCK_OPT_LAZY_REFCOUNTS,
                                                lazy_refcounts);
         } else if (!strcmp(desc->name, BLOCK_OPT_REFCOUNT_BITS)) {
-            error_report("Cannot change refcount entry width");
-            return -ENOTSUP;
+            refcount_bits = qemu_opt_get_number(opts, BLOCK_OPT_REFCOUNT_BITS,
+                                                refcount_bits);
+
+            if (refcount_bits <= 0 || refcount_bits > 64 ||
+                !is_power_of_2(refcount_bits))
+            {
+                error_report("Refcount width must be a power of two and may "
+                             "not exceed 64 bits");
+                return -EINVAL;
+            }
         } else {
             /* if this point is reached, this probably means a new option was
              * added without having it covered here */
@@ -2811,6 +2815,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
         .original_status_cb = status_cb,
         .original_cb_opaque = cb_opaque,
         .total_operations = (new_version < old_version)
+                          + (s->refcount_bits != refcount_bits)
     };
 
     /* Upgrade first (some features may require compat=1.1) */
@@ -2823,6 +2828,27 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
         }
     }
 
+    if (s->refcount_bits != refcount_bits) {
+        int refcount_order = ffs(refcount_bits) - 1;
+        Error *local_error = NULL;
+
+        if (new_version < 3 && refcount_bits != 16) {
+            error_report("Different refcount widths than 16 bits require "
+                         "compatibility level 1.1 or above (use compat=1.1 or "
+                         "greater)");
+            return -EINVAL;
+        }
+
+        helper_cb_info.current_operation = QCOW2_CHANGING_REFCOUNT_ORDER;
+        ret = qcow2_change_refcount_order(bs, refcount_order,
+                                          &qcow2_amend_helper_cb,
+                                          &helper_cb_info, &local_error);
+        if (ret < 0) {
+            qerror_report_err(local_error);
+            return ret;
+        }
+    }
+
     if (backing_file || backing_format) {
         ret = qcow2_change_backing_file(bs, backing_file ?: bs->backing_file,
                                         backing_format ?: bs->backing_format);
Signature · 1 line
1.9.3
author
#41 lore raw

PATCH v5 25/26 qcow2: Point to amend function in check

In reply to Max Reitz

If a reference count is not representable with the current refcount order, the image check should point to qemu-img amend for increasing the refcount order. However, qemu-img amend needs write access to the image which cannot be provided if the image is marked corrupt; and the image check will not mark the image consistent unless everything actually is consistent.

Therefore, if an image is marked corrupt and the image check encounters a reference count overflow, it cannot be fixed by using qemu-img amend to increase the refcount order. Instead, one has to use qemu-img convert to create a completely new copy of the image in this case.

Alternatively, we may want to give the user a way of manually removing the corrupt flag, maybe through qemu-img amend, but this is not part of this patch.

---
 block/qcow2-refcount.c | 3 +++
 1 file changed, 3 insertions(+)
block/qcow2-refcount.c +3 -0
index 7c3f8e0..7c9b334 100644
@@ -1370,6 +1370,9 @@ static int inc_refcounts(BlockDriverState *bs,
         if (refcount == s->refcount_max) {
             fprintf(stderr, "ERROR: overflow cluster offset=0x%" PRIx64
                     "\n", cluster_offset);
+            fprintf(stderr, "Use qemu-img amend to increase the refcount entry "
+                    "width or qemu-img convert to create a clean copy if the "
+                    "image cannot be opened for writing\n");
             res->corruptions++;
             continue;
         }
Signature · 1 line
1.9.3
author
#42 lore raw

PATCH v5 26/26 iotests: Add test for different refcount widths

In reply to Max Reitz

Add a test for conversion between different refcount widths and errors specific to certain widths (i.e. snapshots with refcount_bits=1).

---
 tests/qemu-iotests/112     | 296 +++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/112.out | 155 ++++++++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 452 insertions(+)
 create mode 100755 tests/qemu-iotests/112
 create mode 100644 tests/qemu-iotests/112.out
tests/qemu-iotests/112 new file +296 -0
new file mode 100755
index 0000000..74c75ab
@@ -0,0 +1,296 @@
+#!/bin/bash
+#
+# Test cases for different refcount_bits values
+#
+# Copyright (C) 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
[email protected]
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+here="$PWD"
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+_cleanup()
+{
+	_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# This tests qcow2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+# This test will set refcount_bits on its own which would conflict with the
+# manual setting; compat will be overridden as well
+_unsupported_imgopts refcount_bits 'compat=0.10'
+
+function print_refcount_bits()
+{
+    $QEMU_IMG info "$TEST_IMG" | sed -n '/refcount bits:/ s/^ *//p'
+}
+
+echo
+echo '=== refcount_bits limits ==='
+echo
+
+# Must be positive (non-zero)
+IMGOPTS="$IMGOPTS,refcount_bits=0" _make_test_img 64M
+# Must be positive (non-negative)
+IMGOPTS="$IMGOPTS,refcount_bits=-1" _make_test_img 64M
+# May not exceed 64
+IMGOPTS="$IMGOPTS,refcount_bits=128" _make_test_img 64M
+# Must be a power of two
+IMGOPTS="$IMGOPTS,refcount_bits=42" _make_test_img 64M
+
+# 1 is the minimum
+IMGOPTS="$IMGOPTS,refcount_bits=1" _make_test_img 64M
+print_refcount_bits
+
+# 64 is the maximum
+IMGOPTS="$IMGOPTS,refcount_bits=64" _make_test_img 64M
+print_refcount_bits
+
+# 16 is the default
+_make_test_img 64M
+print_refcount_bits
+
+echo
+echo '=== refcount_bits and compat=0.10 ==='
+echo
+
+# Should work
+IMGOPTS="$IMGOPTS,compat=0.10,refcount_bits=16" _make_test_img 64M
+print_refcount_bits
+
+# Should not work
+IMGOPTS="$IMGOPTS,compat=0.10,refcount_bits=1" _make_test_img 64M
+IMGOPTS="$IMGOPTS,compat=0.10,refcount_bits=64" _make_test_img 64M
+
+
+echo
+echo '=== Snapshot limit on refcount_bits=1 ==='
+echo
+
+IMGOPTS="$IMGOPTS,refcount_bits=1" _make_test_img 64M
+print_refcount_bits
+
+$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io
+
+# Should fail for now; in the future, this might be supported by automatically
+# copying all clusters with overflowing refcount
+$QEMU_IMG snapshot -c foo "$TEST_IMG"
+
+# The new L1 table could/should be leaked
+_check_test_img
+
+echo
+echo '=== Snapshot limit on refcount_bits=2 ==='
+echo
+
+IMGOPTS="$IMGOPTS,refcount_bits=2" _make_test_img 64M
+print_refcount_bits
+
+$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io
+
+# Should succeed
+$QEMU_IMG snapshot -c foo "$TEST_IMG"
+$QEMU_IMG snapshot -c bar "$TEST_IMG"
+# Should fail (4th reference)
+$QEMU_IMG snapshot -c baz "$TEST_IMG"
+
+# The new L1 table could/should be leaked
+_check_test_img
+
+echo
+echo '=== Compressed clusters with refcount_bits=1 ==='
+echo
+
+IMGOPTS="$IMGOPTS,refcount_bits=1" _make_test_img 64M
+print_refcount_bits
+
+# Both should fit into a single host cluster; instead of failing to increase the
+# refcount of that cluster, qemu should just allocate a new cluster and make
+# this operation succeed
+$QEMU_IO -c 'write -P 0 -c  0  64k' \
+         -c 'write -P 1 -c 64k 64k' \
+         "$TEST_IMG" | _filter_qemu_io
+
+_check_test_img
+
+echo
+echo '=== MSb set in 64 bit refcount ==='
+echo
+
+IMGOPTS="$IMGOPTS,refcount_bits=64" _make_test_img 64M
+print_refcount_bits
+
+$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io
+
+# Set the MSb in the refblock entry of the data cluster
+poke_file "$TEST_IMG" $((0x20028)) "\x80\x00\x00\x00\x00\x00\x00\x00"
+
+# Clear OFLAG_COPIED in the L2 entry of the data cluster
+poke_file "$TEST_IMG" $((0x40000)) "\x00\x00\x00\x00\x00\x05\x00\x00"
+
+# Try to write to that cluster (should work, even though the MSb is set)
+$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io
+
+echo
+echo '=== Snapshot on maximum 64 bit refcount value ==='
+echo
+
+IMGOPTS="$IMGOPTS,refcount_bits=64" _make_test_img 64M
+print_refcount_bits
+
+$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io
+
+# Set the refblock entry to the maximum value possible
+poke_file "$TEST_IMG" $((0x20028)) "\xff\xff\xff\xff\xff\xff\xff\xff"
+
+# Clear OFLAG_COPIED in the L2 entry of the data cluster
+poke_file "$TEST_IMG" $((0x40000)) "\x00\x00\x00\x00\x00\x05\x00\x00"
+
+# Try a snapshot (should correctly identify the overflow; may work in the future
+# by falling back to COW)
+$QEMU_IMG snapshot -c foo "$TEST_IMG"
+
+# The new L1 table could/should be leaked; and obviously the data cluster is
+# leaked (refcount=UINT64_MAX reference=1)
+_check_test_img
+
+echo
+echo '=== Amend from refcount_bits=16 to refcount_bits=1 ==='
+echo
+
+_make_test_img 64M
+print_refcount_bits
+
+$QEMU_IO -c 'write 16M 32M' "$TEST_IMG" | _filter_qemu_io
+$QEMU_IMG amend -o refcount_bits=1 "$TEST_IMG"
+_check_test_img
+print_refcount_bits
+
+echo
+echo '=== Amend from refcount_bits=1 to refcount_bits=64 ==='
+echo
+
+$QEMU_IMG amend -o refcount_bits=64 "$TEST_IMG"
+_check_test_img
+print_refcount_bits
+
+echo
+echo '=== Amend to compat=0.10 ==='
+echo
+
+# Should not work because refcount_bits needs to be 16 for compat=0.10
+$QEMU_IMG amend -o compat=0.10 "$TEST_IMG"
+print_refcount_bits
+# Should work
+$QEMU_IMG amend -o compat=0.10,refcount_bits=16 "$TEST_IMG"
+_check_test_img
+print_refcount_bits
+
+# Get back to compat=1.1 and refcount_bits=16
+$QEMU_IMG amend -o compat=1.1 "$TEST_IMG"
+print_refcount_bits
+# Should not work
+$QEMU_IMG amend -o refcount_bits=32,compat=0.10 "$TEST_IMG"
+print_refcount_bits
+
+echo
+echo '=== Amend with snapshot ==='
+echo
+
+$QEMU_IMG snapshot -c foo "$TEST_IMG"
+# Just to have different refcounts across the image
+$QEMU_IO -c 'write 0 16M' "$TEST_IMG" | _filter_qemu_io
+
+# Should not work (may work in the future by first decreasing all refcounts so
+# they fit into the target range by copying them)
+$QEMU_IMG amend -o refcount_bits=1 "$TEST_IMG"
+_check_test_img
+print_refcount_bits
+
+# Should work
+$QEMU_IMG amend -o refcount_bits=2 "$TEST_IMG"
+_check_test_img
+print_refcount_bits
+
+echo
+echo '=== Testing too many references for check ==='
+echo
+
+IMGOPTS="$IMGOPTS,refcount_bits=1" _make_test_img 64M
+print_refcount_bits
+
+# This cluster should be created at 0x50000
+$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io
+# Now make the second L2 entry (the L2 table should be at 0x40000) point to that
+# cluster, so we have two references
+poke_file "$TEST_IMG" $((0x40008)) "\x80\x00\x00\x00\x00\x05\x00\x00"
+
+# This should say "please use amend"
+_check_test_img -r all
+
+# So we do that
+$QEMU_IMG amend -o refcount_bits=2 "$TEST_IMG"
+print_refcount_bits
+
+# And try again
+_check_test_img -r all
+
+echo
+echo '=== Multiple walks necessary during amend ==='
+echo
+
+IMGOPTS="$IMGOPTS,refcount_bits=1,cluster_size=512" _make_test_img 64k
+
+# Cluster 0 is the image header, clusters 1 to 4 are used by the L1 table, a
+# single L2 table, the reftable and a single refblock. This creates 58 data
+# clusters (actually, the L2 table is created here, too), so in total there are
+# then 63 used clusters in the image. With a refcount width of 64, one refblock
+# describes 64 clusters (512 bytes / 64 bits/entry = 64 entries), so this will
+# make the first refblock in the amended image have exactly one free entry.
+$QEMU_IO -c "write 0 $((58 * 512))" "$TEST_IMG" | _filter_qemu_io
+
+# Now change the refcount width; since the first new refblock will have exactly
+# one free entry, that entry will be used to store its own reference. No other
+# refblocks are needed, so then the new reftable will be allocated; since the
+# first new refblock is completely filled up, this will require a new refblock
+# which is why the refcount width changing function will need to run through
+# everything one more time until the allocations are stable.
+# Having more walks than usual should be visible as regressing progress (from
+# 66.67 % (2/3 walks) to 50.00 % (2/4 walks)).
+$QEMU_IMG amend -o refcount_bits=64 -p "$TEST_IMG" | tr '\r' '\n' \
+                                                    | grep -A 1 '66.67'
+print_refcount_bits
+
+_check_test_img
+
+
+# success, all done
+echo '*** done'
+rm -f $seq.full
+status=0
tests/qemu-iotests/112.out new file +155 -0

155 lines not on this page · raw message

tests/qemu-iotests/group +1 -0
index 08099b9..2098219 100644
@@ -113,5 +113,6 @@
 108 rw auto quick
 109 rw auto
 111 rw auto quick
+112 rw auto
 113 rw auto quick
 114 rw auto quick
Signature · 1 line
1.9.3
author
#43 lore raw
In reply to Max Reitz
Quoted 143 lines · Max Reitz

As of version 3, the qcow2 file format supports different widths for refcount entries, ranging from 1 to 64 bit (only powers of two). Currently, qemu only supports 16 bit, which is the only width supported by version 2 (compat=0.10) images.

This series adds support to qemu for all other valid refcount orders. This is mainly done by adding two function pointers into the BDRVQcowState structure for reading and writing refcount values independently of the current refcount entry width; all in-memory refcount arrays (mostly cached refcount blocks) now are void pointers and are accessed through these functions alone.

Thanks to previous work of making the qemu code agnostic of e.g. the number of refcount entries per refcount block, the rest is fairly trivial. The most complex patch in this series is patch 23 which implements changing the refcount order through qemu-img amend.

To test different refcount widths, simply invoke the qemu-iotests check program with -o refcount_width=${your_desired_width}. The final test in this series adds some tests for operations which do not work with certain refcount orders and for refcount order amendment.

v5:
- Patch 5: Use absolute value for addend instead of the two's complement for negative values [Stefan]
- Patch 6:
- Conflicts due to the changes in patch 5
- Dropped the "if (ret >= 0)" -> "if (ret == 0)" hunk; since the update_refcount() return value was never touched in this series there is no point in modifying the callers' checks for their return value [Eric]
- Patch 9: Use DIV_ROUND_UP() [Eric]
- Patch 16: Rebase conflicts due to Fam's patch removing trailing whitespace in all qemu-iotests/*.out files (7486458c33cd3e5714368f181a5fea39c54444ca); I kept the R-bs because these are only trailing whitespace conflicts in *.out files which are really trivial.
There are two not-so-trivial conflicts: Two hunks are missing from v5 which were part of v4 which did nothing but remove trailing whitespace. Obviously those are no longer necessary, so removing these hunks was actually trivial after all.

git-backport-diff against v4:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/26:[----] [--] 'qcow2: Add two new fields to BDRVQcowState' 002/26:[----] [--] 'qcow2: Add refcount_bits to format-specific info' 003/26:[----] [--] 'qcow2: Do not return new value after refcount update' 004/26:[----] [--] 'qcow2: Only return status from qcow2_get_refcount' 005/26:[0040] [FC] 'qcow2: Use unsigned addend for update_refcount()' 006/26:[0028] [FC] 'qcow2: Use 64 bits for refcount values' 007/26:[----] [--] 'qcow2: Respect error in qcow2_alloc_bytes()' 008/26:[----] [--] 'qcow2: Refcount overflow and qcow2_alloc_bytes()' 009/26:[0002] [FC] 'qcow2: Helper for refcount array reallocation' 010/26:[----] [-C] 'qcow2: Helper function for refcount modification' 011/26:[----] [--] 'qcow2: More helpers for refcount modification' 012/26:[----] [--] 'qcow2: Open images with refcount order != 4' 013/26:[----] [--] 'qcow2: refcount_order parameter for qcow2_create2' 014/26:[----] [--] 'qcow2: Use symbolic macros in qcow2_amend_options' 015/26:[----] [--] 'iotests: Prepare for refcount_bits option' 016/26:[0126] [FC] 'qcow2: Allow creation with refcount order != 4' 017/26:[----] [--] 'progress: Allow regressing progress' 018/26:[----] [--] 'block: Add opaque value to the amend CB' 019/26:[----] [-C] 'qcow2: Use error_report() in qcow2_amend_options()' 020/26:[----] [--] 'qcow2: Use abort() instead of assert(false)' 021/26:[----] [--] 'qcow2: Split upgrade/downgrade paths for amend' 022/26:[----] [--] 'qcow2: Use intermediate helper CB for amend' 023/26:[----] [--] 'qcow2: Add function for refcount order amendment' 024/26:[----] [--] 'qcow2: Invoke refcount order amendment function' 025/26:[----] [--] 'qcow2: Point to amend function in check' 026/26:[----] [--] 'iotests: Add test for different refcount widths'

Max Reitz (26):
qcow2: Add two new fields to BDRVQcowState
qcow2: Add refcount_bits to format-specific info
qcow2: Do not return new value after refcount update
qcow2: Only return status from qcow2_get_refcount
qcow2: Use unsigned addend for update_refcount()
qcow2: Use 64 bits for refcount values
qcow2: Respect error in qcow2_alloc_bytes()
qcow2: Refcount overflow and qcow2_alloc_bytes()
qcow2: Helper for refcount array reallocation
qcow2: Helper function for refcount modification
qcow2: More helpers for refcount modification
qcow2: Open images with refcount order != 4
qcow2: refcount_order parameter for qcow2_create2
qcow2: Use symbolic macros in qcow2_amend_options
iotests: Prepare for refcount_bits option
qcow2: Allow creation with refcount order != 4
progress: Allow regressing progress
block: Add opaque value to the amend CB
qcow2: Use error_report() in qcow2_amend_options()
qcow2: Use abort() instead of assert(false)
qcow2: Split upgrade/downgrade paths for amend
qcow2: Use intermediate helper CB for amend
qcow2: Add function for refcount order amendment
qcow2: Invoke refcount order amendment function
qcow2: Point to amend function in check
iotests: Add test for different refcount widths

  block.c                          |    4 +-
  block/qcow2-cluster.c            |   24 +-
  block/qcow2-refcount.c           | 1054 +++++++++++++++++++++++++++++++-------
  block/qcow2.c                    |  286 ++++++++---
  block/qcow2.h                    |   23 +-
  include/block/block.h            |    4 +-
  include/block/block_int.h        |    4 +-
  qapi/block-core.json             |    5 +-
  qemu-img.c                       |    5 +-
  tests/qemu-iotests/007           |    3 +
  tests/qemu-iotests/015           |    2 +
  tests/qemu-iotests/026           |    7 +
  tests/qemu-iotests/029           |    2 +
  tests/qemu-iotests/049.out       |  108 ++--
  tests/qemu-iotests/051           |    3 +
  tests/qemu-iotests/058           |    2 +
  tests/qemu-iotests/060.out       |    1 +
  tests/qemu-iotests/061.out       |   14 +-
  tests/qemu-iotests/065           |   23 +-
  tests/qemu-iotests/067           |    2 +
  tests/qemu-iotests/067.out       |    5 +
  tests/qemu-iotests/079           |   10 +-
  tests/qemu-iotests/079.out       |   38 +-
  tests/qemu-iotests/080           |    2 +
  tests/qemu-iotests/082.out       |   48 +-
  tests/qemu-iotests/085.out       |   38 +-
  tests/qemu-iotests/089           |    2 +
  tests/qemu-iotests/089.out       |    2 +
  tests/qemu-iotests/108           |    2 +
  tests/qemu-iotests/112           |  296 +++++++++++
  tests/qemu-iotests/112.out       |  155 ++++++
  tests/qemu-iotests/common.filter |    3 +-
  tests/qemu-iotests/group         |    1 +
  util/qemu-progress.c             |    3 +-
  34 files changed, 1782 insertions(+), 399 deletions(-)
Quoted 2 lines

create mode 100755 tests/qemu-iotests/112
create mode 100644 tests/qemu-iotests/112.out

Ping