Skip to contentSkip to conversation

qcow2: Define and use QCOW2_COMPRESSED_SECTOR_SIZE

PATCH Applied Alberto Garcia 3 participants 6 messages May 10, 2019 – May 13, 2019 Block layer

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 qcow2: Define and use QCOW2_COMPRESSED_SECTOR_SIZE

When an L2 table entry points to a compressed cluster the space used by the data is specified in 512-byte sectors. This size is independent from BDRV_SECTOR_SIZE and is specific to the qcow2 file format.

The QCOW2_COMPRESSED_SECTOR_SIZE constant defined in this patch makes this explicit.

---
 block/qcow2-cluster.c  |  5 +++--
 block/qcow2-refcount.c | 25 ++++++++++++++-----------
 block/qcow2.c          |  3 ++-
 block/qcow2.h          |  4 ++++
 4 files changed, 23 insertions(+), 14 deletions(-)
block/qcow2-cluster.c +3 -2
index 974a4e8656..b36f4aa84a 100644
@@ -796,8 +796,9 @@ int qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
         return cluster_offset;
     }
 
-    nb_csectors = ((cluster_offset + compressed_size - 1) >> 9) -
-                  (cluster_offset >> 9);
+    nb_csectors =
+        (cluster_offset + compressed_size - 1) / QCOW2_COMPRESSED_SECTOR_SIZE -
+        (cluster_offset / QCOW2_COMPRESSED_SECTOR_SIZE);
 
     cluster_offset |= QCOW_OFLAG_COMPRESSED |
                       ((uint64_t)nb_csectors << s->csize_shift);
block/qcow2-refcount.c +14 -11
index fa7ac1f7cb..780bd49a00 100644
@@ -1172,12 +1172,11 @@ void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry,
     switch (ctype) {
     case QCOW2_CLUSTER_COMPRESSED:
         {
-            int nb_csectors;
-            nb_csectors = ((l2_entry >> s->csize_shift) &
-                           s->csize_mask) + 1;
-            qcow2_free_clusters(bs,
-                (l2_entry & s->cluster_offset_mask) & ~511,
-                nb_csectors * 512, type);
+            int64_t offset = (l2_entry & s->cluster_offset_mask)
+                & QCOW2_COMPRESSED_SECTOR_MASK;
+            int size = QCOW2_COMPRESSED_SECTOR_SIZE *
+                (((l2_entry >> s->csize_shift) & s->csize_mask) + 1);
+            qcow2_free_clusters(bs, offset, size, type);
         }
         break;
     case QCOW2_CLUSTER_NORMAL:
@@ -1317,9 +1316,12 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
                         nb_csectors = ((entry >> s->csize_shift) &
                                        s->csize_mask) + 1;
                         if (addend != 0) {
+                            uint64_t coffset = (entry & s->cluster_offset_mask)
+                                & QCOW2_COMPRESSED_SECTOR_MASK;
                             ret = update_refcount(
-                                bs, (entry & s->cluster_offset_mask) & ~511,
-                                nb_csectors * 512, abs(addend), addend < 0,
+                                bs, coffset,
+                                nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE,
+                                abs(addend), addend < 0,
                                 QCOW2_DISCARD_SNAPSHOT);
                             if (ret < 0) {
                                 goto fail;
@@ -1635,9 +1637,10 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
             nb_csectors = ((l2_entry >> s->csize_shift) &
                            s->csize_mask) + 1;
             l2_entry &= s->cluster_offset_mask;
-            ret = qcow2_inc_refcounts_imrt(bs, res,
-                                           refcount_table, refcount_table_size,
-                                           l2_entry & ~511, nb_csectors * 512);
+            ret = qcow2_inc_refcounts_imrt(
+                bs, res, refcount_table, refcount_table_size,
+                l2_entry & QCOW2_COMPRESSED_SECTOR_MASK,
+                nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE);
             if (ret < 0) {
                 goto fail;
             }
block/qcow2.c +2 -1
index a520d116ef..80679a84d2 100644
@@ -4188,7 +4188,8 @@ qcow2_co_preadv_compressed(BlockDriverState *bs,
 
     coffset = file_cluster_offset & s->cluster_offset_mask;
     nb_csectors = ((file_cluster_offset >> s->csize_shift) & s->csize_mask) + 1;
-    csize = nb_csectors * 512 - (coffset & 511);
+    csize = nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE -
+        (coffset & ~QCOW2_COMPRESSED_SECTOR_MASK);
 
     buf = g_try_malloc(csize);
     if (!buf) {
block/qcow2.h +4 -0
index fdee297f33..7e796877d6 100644
@@ -74,6 +74,10 @@
 #define MIN_CLUSTER_BITS 9
 #define MAX_CLUSTER_BITS 21
 
+/* Defined in the qcow2 spec (compressed cluster descriptor) */
+#define QCOW2_COMPRESSED_SECTOR_SIZE 512U
+#define QCOW2_COMPRESSED_SECTOR_MASK (~(QCOW2_COMPRESSED_SECTOR_SIZE - 1))
+
 /* Must be at least 2 to cover COW */
 #define MIN_L2_CACHE_SIZE 2 /* cache entries */
 
Signature · 1 line
2.11.0
#1 lore raw

The patch LGTM, just a comment below.

Quoted 39 lines · Alberto Garcia

When an L2 table entry points to a compressed cluster the space used by the data is specified in 512-byte sectors. This size is independent from BDRV_SECTOR_SIZE and is specific to the qcow2 file format.

The QCOW2_COMPRESSED_SECTOR_SIZE constant defined in this patch makes this explicit.

---
 block/qcow2-cluster.c  |  5 +++--
 block/qcow2-refcount.c | 25 ++++++++++++++-----------
 block/qcow2.c          |  3 ++-
 block/qcow2.h          |  4 ++++
 4 files changed, 23 insertions(+), 14 deletions(-)
block/qcow2-cluster.c +3 -2
index 974a4e8656..b36f4aa84a 100644
@@ -796,8 +796,9 @@ int qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
         return cluster_offset;
     }
 
-    nb_csectors = ((cluster_offset + compressed_size - 1) >> 9) -
-                  (cluster_offset >> 9);
+    nb_csectors =
+        (cluster_offset + compressed_size - 1) / QCOW2_COMPRESSED_SECTOR_SIZE -
+        (cluster_offset / QCOW2_COMPRESSED_SECTOR_SIZE);
 
     cluster_offset |= QCOW_OFLAG_COMPRESSED |
                       ((uint64_t)nb_csectors << s->csize_shift);
block/qcow2-refcount.c +0 -0
index fa7ac1f7cb..780bd49a00 100644
@@ -1172,12 +1172,11 @@ void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry,
     switch (ctype) {
     case QCOW2_CLUSTER_COMPRESSED:
         {

block/qcow2-refcount.c void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry, · line 1178

Quoted 10 lines
block/qcow2-refcount.c +4 -6
-            int nb_csectors;
-            nb_csectors = ((l2_entry >> s->csize_shift) &
-                           s->csize_mask) + 1;
-            qcow2_free_clusters(bs,
-                (l2_entry & s->cluster_offset_mask) & ~511,
-                nb_csectors * 512, type);
+            int64_t offset = (l2_entry & s->cluster_offset_mask)
+                & QCOW2_COMPRESSED_SECTOR_MASK;
+            int size = QCOW2_COMPRESSED_SECTOR_SIZE *
+                (((l2_entry >> s->csize_shift) & s->csize_mask) + 1);

What about using int64_t type for the 'size' variable? (because the qcow2_free_clusters() 'size' parameter is int64_t)

Quoted 62 lines
block/qcow2-refcount.c +10 -5
+            qcow2_free_clusters(bs, offset, size, type);
         }
         break;
     case QCOW2_CLUSTER_NORMAL:
@@ -1317,9 +1316,12 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
                         nb_csectors = ((entry >> s->csize_shift) &
                                        s->csize_mask) + 1;
                         if (addend != 0) {
+                            uint64_t coffset = (entry & s->cluster_offset_mask)
+                                & QCOW2_COMPRESSED_SECTOR_MASK;
                             ret = update_refcount(
-                                bs, (entry & s->cluster_offset_mask) & ~511,
-                                nb_csectors * 512, abs(addend), addend < 0,
+                                bs, coffset,
+                                nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE,
+                                abs(addend), addend < 0,
                                 QCOW2_DISCARD_SNAPSHOT);
                             if (ret < 0) {
                                 goto fail;
@@ -1635,9 +1637,10 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
             nb_csectors = ((l2_entry >> s->csize_shift) &
                            s->csize_mask) + 1;
             l2_entry &= s->cluster_offset_mask;
-            ret = qcow2_inc_refcounts_imrt(bs, res,
-                                           refcount_table, refcount_table_size,
-                                           l2_entry & ~511, nb_csectors * 512);
+            ret = qcow2_inc_refcounts_imrt(
+                bs, res, refcount_table, refcount_table_size,
+                l2_entry & QCOW2_COMPRESSED_SECTOR_MASK,
+                nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE);
             if (ret < 0) {
                 goto fail;
             }
block/qcow2.c +2 -1
index a520d116ef..80679a84d2 100644
@@ -4188,7 +4188,8 @@ qcow2_co_preadv_compressed(BlockDriverState *bs,
 
     coffset = file_cluster_offset & s->cluster_offset_mask;
     nb_csectors = ((file_cluster_offset >> s->csize_shift) & s->csize_mask) + 1;
-    csize = nb_csectors * 512 - (coffset & 511);
+    csize = nb_csectors * QCOW2_COMPRESSED_SECTOR_SIZE -
+        (coffset & ~QCOW2_COMPRESSED_SECTOR_MASK);
 
     buf = g_try_malloc(csize);
     if (!buf) {
block/qcow2.h +4 -0
index fdee297f33..7e796877d6 100644
@@ -74,6 +74,10 @@
 #define MIN_CLUSTER_BITS 9
 #define MAX_CLUSTER_BITS 21
 
+/* Defined in the qcow2 spec (compressed cluster descriptor) */
+#define QCOW2_COMPRESSED_SECTOR_SIZE 512U
+#define QCOW2_COMPRESSED_SECTOR_MASK (~(QCOW2_COMPRESSED_SECTOR_SIZE - 1))
+
 /* Must be at least 2 to cover COW */
 #define MIN_L2_CACHE_SIZE 2 /* cache entries */
 
#2 lore raw
Quoted 5 lines · Stefano Garzarella

block/qcow2-refcount.c void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry, · line 1178

Quoted 2 lines
block/qcow2-refcount.c +2 -0
+            int size = QCOW2_COMPRESSED_SECTOR_SIZE *
+                (((l2_entry >> s->csize_shift) & s->csize_mask) + 1);

What about using int64_t type for the 'size' variable? (because the qcow2_free_clusters() 'size' parameter is int64_t)

The maximum size that can be read from a compressed cluster descriptor using the formula above is twice the cluster size (more information on commit abd3622cc03cf41ed542126a540385f30a4c0175 and on the Compressed Clusters Descriptor spec in docs/interop/qcow2.txt).

Since the maximum allowed cluster size is 2MB, the value of the 'size' variable can never be larger than 4MB, which fits comfortably on a 32-bit integer. We would need to support 512MB clusters in order to have problems with this.

Berto

#3 lore raw
Quoted 12 lines · Alberto Garcia
Quoted 5 lines · Stefano Garzarella

block/qcow2-refcount.c void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry, · line 1178

Quoted 2 lines
block/qcow2-refcount.c +2 -0
+            int size = QCOW2_COMPRESSED_SECTOR_SIZE *
+                (((l2_entry >> s->csize_shift) & s->csize_mask) + 1);

What about using int64_t type for the 'size' variable? (because the qcow2_free_clusters() 'size' parameter is int64_t)

The maximum size that can be read from a compressed cluster descriptor using the formula above is twice the cluster size (more information on commit abd3622cc03cf41ed542126a540385f30a4c0175 and on the Compressed Clusters Descriptor spec in docs/interop/qcow2.txt).

Quoted 4 lines

Since the maximum allowed cluster size is 2MB, the value of the 'size' variable can never be larger than 4MB, which fits comfortably on a 32-bit integer. We would need to support 512MB clusters in order to have problems with this.

Thanks for the explaination and sorry for that!

Cheers,
Stefano

#4 lore raw
Quoted 6 lines · Stefano Garzarella
Quoted 4 lines

Since the maximum allowed cluster size is 2MB, the value of the 'size' variable can never be larger than 4MB, which fits comfortably on a 32-bit integer. We would need to support 512MB clusters in order to have problems with this.

Thanks for the explaination and sorry for that!

No need to say sorry, the question was perfectly valid and it could have been an actual bug, so I appreciate that you wrote the e-mail.

Thanks,

Berto

Applied
#5 lore raw
In reply to Alberto Garcia

Am 10.05.2019 um 18:22 hat Alberto Garcia geschrieben:

Quoted 8 lines

When an L2 table entry points to a compressed cluster the space used by the data is specified in 512-byte sectors. This size is independent from BDRV_SECTOR_SIZE and is specific to the qcow2 file format.

The QCOW2_COMPRESSED_SECTOR_SIZE constant defined in this patch makes this explicit.

Thanks, applied to the block branch.

Kevin