Skip to contentSkip to conversation

block/vpc optimizations

PATCH Applied Peter Lieven 2 participants 11 messages Mar 3, 2015 – Mar 17, 2015 Block layer

lore
Share & tools
lore
#0 lore raw

v2 cover block/vpc optimizations

This series covers VPC format changes discussed during the last weeks.

Peter

v1->v2:
Patch 1: added comment why we can exit after the first allocated sector [Max] Patch 2: Fixed gemotry in commit msg [Max]
Patch 3: Fixed comparision from > to >= [Max]
Fixed exit in case of -EFBIG [Max]
Removed DIV_ROUND_UP [Max]
Patch 5: added

Kevin Wolf (1):
vpc: Ignore geometry for large images

Peter Lieven (4):
block/vpc: optimize vpc_co_get_block_status block/vpc: make calculate_geometry spec conform block/vpc: rename footer->size -> footer->current_size block/vpc: remove disabled code from get_sector_offset

 block/vpc.c | 106 +++++++++++++++++++++---------------------------------------
 1 file changed, 37 insertions(+), 69 deletions(-)
Signature · 1 line
1.9.1
#1 lore raw

PATCH v2 1/5 block/vpc: optimize vpc_co_get_block_status

*pnum can't be greater than s->block_size / BDRV_SECTOR_SIZE for allocated sectors since there is always a bitmap in between.

---
 block/vpc.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
block/vpc.c +8 -10
index 1533b6a..c8e17cb 100644
@@ -602,7 +602,7 @@ static int64_t coroutine_fn vpc_co_get_block_status(BlockDriverState *bs,
 {
     BDRVVPCState *s = bs->opaque;
     VHDFooter *footer = (VHDFooter*) s->footer_buf;
-    int64_t start, offset, next;
+    int64_t start, offset;
     bool allocated;
     int n;
 
@@ -626,20 +626,18 @@ static int64_t coroutine_fn vpc_co_get_block_status(BlockDriverState *bs,
         *pnum += n;
         sector_num += n;
         nb_sectors -= n;
-        next = start + (*pnum * BDRV_SECTOR_SIZE);
-
+        /* *pnum can't be greater than one block for allocated
+         * sectors since there is always a bitmap in between. */
+        if (allocated) {
+            return BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID | start;
+        }
         if (nb_sectors == 0) {
             break;
         }
-
         offset = get_sector_offset(bs, sector_num, 0);
-    } while ((allocated && offset == next) || (!allocated && offset == -1));
+    } while (offset == -1);
 
-    if (allocated) {
-        return BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID | start;
-    } else {
-        return 0;
-    }
+    return 0;
 }
 
 /*
Signature · 1 line
1.9.1
#2 lore raw

PATCH v2 2/5 vpc: Ignore geometry for large images

In reply to Peter Lieven

From: Kevin Wolf <[email protected]>

The CHS calculation as done per the VHD spec imposes a maximum image size of ~127 GB. Real VHD images exist that are larger than that.

Apparently there are two separate non-standard ways to achieve this:
You could use more heads than the spec does - this is the option that qemu-img create chooses.

However, other images exist where the geometry is set to the maximum (65535/16/255), but the actual image size is larger. Until now, such images are truncated at 127 GB when opening them with qemu.

This patch changes the vpc driver to ignore geometry in this case and only trust the size field in the header.

[PL: Fixed maximum geometry in the commit msg]
---
 block/vpc.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
block/vpc.c +4 -6
index c8e17cb..1c9592c 100644
@@ -215,12 +215,10 @@ static int vpc_open(BlockDriverState *bs, QDict *options, int flags,
     bs->total_sectors = (int64_t)
         be16_to_cpu(footer->cyls) * footer->heads * footer->secs_per_cyl;
 
-    /* images created with disk2vhd report a far higher virtual size
-     * than expected with the cyls * heads * sectors_per_cyl formula.
-     * use the footer->size instead if the image was created with
-     * disk2vhd.
-     */
-    if (!strncmp(footer->creator_app, "d2v", 4)) {
+    /* Images that have exactly the maximum geometry are probably bigger and
+     * would be truncated if we adhered to the geometry for them. Rely on
+     * footer->size for them. */
+    if (bs->total_sectors == 65535ULL * 16 * 255) {
         bs->total_sectors = be64_to_cpu(footer->size) / BDRV_SECTOR_SIZE;
     }
 
Signature · 1 line
1.9.1
Reviewed-by on v2 2/5
#3 lore raw

Re: [Qemu-devel] [PATCH v2 2/5] vpc: Ignore geometry for large images

Quoted 16 lines · Peter Lieven

From: Kevin Wolf <[email protected]>

The CHS calculation as done per the VHD spec imposes a maximum image size of ~127 GB. Real VHD images exist that are larger than that.

Apparently there are two separate non-standard ways to achieve this:
You could use more heads than the spec does - this is the option that qemu-img create chooses.

However, other images exist where the geometry is set to the maximum (65535/16/255), but the actual image size is larger. Until now, such images are truncated at 127 GB when opening them with qemu.

This patch changes the vpc driver to ignore geometry in this case and only trust the size field in the header.

Quoted 6 lines
[PL: Fixed maximum geometry in the commit msg]
---
  block/vpc.c | 10 ++++------
  1 file changed, 4 insertions(+), 6 deletions(-)
#4 lore raw

PATCH v2 3/5 block/vpc: make calculate_geometry spec conform

In reply to Peter Lieven

The VHD spec [1] allows for total_sectors of 65535 x 16 x 255 (~127GB) represented by a CHS geometry. If total_sectors is greater than 65535 x 16 x 255 this geometry is set as a maximum.

Qemu, Hyper-V and disk2vhd use this special geometry as an indicator to use the image current size from the footer as disk size.

This patch changes vpc_create to effectively calculate a CxHxS geometry for the given image size if possible while rounding up if necessary. If the image size is to big to be represented in CHS we set the maximum and write the exact requested image size into the footer.

This partly reverts commit 258d2edb, but leaves support for >127G disks intact.

[1] http://download.microsoft.com/download/f/f/e/ffef50a5-07dd-4cf8-aaa3-442c0673a029/Virtual%20Hard%20Disk%20Format%20Spec_10_18_06.doc

---
 block/vpc.c | 41 ++++++++++++++++++++++-------------------
 1 file changed, 22 insertions(+), 19 deletions(-)
block/vpc.c +22 -19
index 1c9592c..7e99a69 100644
@@ -46,6 +46,7 @@ enum vhd_type {
 #define VHD_TIMESTAMP_BASE 946684800
 
 #define VHD_MAX_SECTORS       (65535LL * 255 * 255)
+#define VHD_MAX_GEOMETRY      (65535LL *  16 * 255)
 
 // always big-endian
 typedef struct vhd_footer {
@@ -218,7 +219,7 @@ static int vpc_open(BlockDriverState *bs, QDict *options, int flags,
     /* Images that have exactly the maximum geometry are probably bigger and
      * would be truncated if we adhered to the geometry for them. Rely on
      * footer->size for them. */
-    if (bs->total_sectors == 65535ULL * 16 * 255) {
+    if (bs->total_sectors == VHD_MAX_GEOMETRY) {
         bs->total_sectors = be64_to_cpu(footer->size) / BDRV_SECTOR_SIZE;
     }
 
@@ -655,26 +656,20 @@ static int calculate_geometry(int64_t total_sectors, uint16_t* cyls,
 {
     uint32_t cyls_times_heads;
 
-    /* Allow a maximum disk size of approximately 2 TB */
-    if (total_sectors > 65535LL * 255 * 255) {
-        return -EFBIG;
-    }
+    total_sectors = MIN(total_sectors, VHD_MAX_GEOMETRY);
 
-    if (total_sectors > 65535 * 16 * 63) {
+    if (total_sectors >= 65535LL * 16 * 63) {
         *secs_per_cyl = 255;
-        if (total_sectors > 65535 * 16 * 255) {
-            *heads = 255;
-        } else {
-            *heads = 16;
-        }
+        *heads = 16;
         cyls_times_heads = total_sectors / *secs_per_cyl;
     } else {
         *secs_per_cyl = 17;
         cyls_times_heads = total_sectors / *secs_per_cyl;
         *heads = (cyls_times_heads + 1023) / 1024;
 
-        if (*heads < 4)
+        if (*heads < 4) {
             *heads = 4;
+        }
 
         if (cyls_times_heads >= (*heads * 1024) || *heads > 16) {
             *secs_per_cyl = 31;
@@ -830,20 +825,28 @@ static int vpc_create(const char *filename, QemuOpts *opts, Error **errp)
      * Calculate matching total_size and geometry. Increase the number of
      * sectors requested until we get enough (or fail). This ensures that
      * qemu-img convert doesn't truncate images, but rather rounds up.
+     *
+     * If the image size can't be represented by a spec conform CHS geometry,
+     * we set the geometry to 65535 x 16 x 255 (CxHxS) sectors and use
+     * the image size from the VHD footer to calculate total_sectors.
      */
-    total_sectors = total_size / BDRV_SECTOR_SIZE;
+    total_sectors = MIN(VHD_MAX_GEOMETRY, total_size / BDRV_SECTOR_SIZE);
     for (i = 0; total_sectors > (int64_t)cyls * heads * secs_per_cyl; i++) {
-        if (calculate_geometry(total_sectors + i, &cyls, &heads,
-                               &secs_per_cyl))
-        {
+        calculate_geometry(total_sectors + i, &cyls, &heads, &secs_per_cyl);
+    }
+
+    if ((int64_t)cyls * heads * secs_per_cyl == VHD_MAX_GEOMETRY) {
+        total_sectors = total_size / BDRV_SECTOR_SIZE;
+        /* Allow a maximum disk size of approximately 2 TB */
+        if (total_sectors > VHD_MAX_SECTORS) {
             ret = -EFBIG;
             goto out;
         }
+    } else {
+        total_sectors = (int64_t)cyls * heads * secs_per_cyl;
+        total_size = total_sectors * BDRV_SECTOR_SIZE;
     }
 
-    total_sectors = (int64_t) cyls * heads * secs_per_cyl;
-    total_size = total_sectors * BDRV_SECTOR_SIZE;
-
     /* Prepare the Hard Disk Footer */
     memset(buf, 0, 1024);
 
Signature · 1 line
1.9.1
Reviewed-by on v2 3/5
#5 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] block/vpc: make calculate_geometry spec conform

Quoted 10 lines · Peter Lieven

The VHD spec [1] allows for total_sectors of 65535 x 16 x 255 (~127GB) represented by a CHS geometry. If total_sectors is greater than 65535 x 16 x 255 this geometry is set as a maximum.

Qemu, Hyper-V and disk2vhd use this special geometry as an indicator to use the image current size from the footer as disk size.

This patch changes vpc_create to effectively calculate a CxHxS geometry for the given image size if possible while rounding up if necessary. If the image size is to big to be represented in CHS we set the maximum

"too big" (as it was in v1) was correct, no need to make it wrong. :-)

Quoted 11 lines

and write the exact requested image size into the footer.

This partly reverts commit 258d2edb, but leaves support for >127G disks intact.

[1] http://download.microsoft.com/download/f/f/e/ffef50a5-07dd-4cf8-aaa3-442c0673a029/Virtual%20Hard%20Disk%20Format%20Spec_10_18_06.doc

---
  block/vpc.c | 41 ++++++++++++++++++++++-------------------
  1 file changed, 22 insertions(+), 19 deletions(-)

With s/to big/too big/:

#6 lore raw

PATCH v2 4/5 block/vpc: rename footer->size -> footer->current_size

In reply to Peter Lieven

the field is named current size in the spec. Name it accordingly.

---
 block/vpc.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
block/vpc.c +5 -4
index 7e99a69..226be02 100644
@@ -66,7 +66,7 @@ typedef struct vhd_footer {
     char        creator_os[4]; // "Wi2k"
 
     uint64_t    orig_size;
-    uint64_t    size;
+    uint64_t    current_size;
 
     uint16_t    cyls;
     uint8_t     heads;
@@ -218,9 +218,10 @@ static int vpc_open(BlockDriverState *bs, QDict *options, int flags,
 
     /* Images that have exactly the maximum geometry are probably bigger and
      * would be truncated if we adhered to the geometry for them. Rely on
-     * footer->size for them. */
+     * footer->current_size for them. */
     if (bs->total_sectors == VHD_MAX_GEOMETRY) {
-        bs->total_sectors = be64_to_cpu(footer->size) / BDRV_SECTOR_SIZE;
+        bs->total_sectors = be64_to_cpu(footer->current_size) /
+                            BDRV_SECTOR_SIZE;
     }
 
     /* Allow a maximum disk size of approximately 2 TB */
@@ -868,7 +869,7 @@ static int vpc_create(const char *filename, QemuOpts *opts, Error **errp)
     footer->major = cpu_to_be16(0x0005);
     footer->minor = cpu_to_be16(0x0003);
     footer->orig_size = cpu_to_be64(total_size);
-    footer->size = cpu_to_be64(total_size);
+    footer->current_size = cpu_to_be64(total_size);
     footer->cyls = cpu_to_be16(cyls);
     footer->heads = heads;
     footer->secs_per_cyl = secs_per_cyl;
Signature · 1 line
1.9.1
#7 lore raw

PATCH v2 5/5 block/vpc: remove disabled code from get_sector_offset

In reply to Peter Lieven

The code to check the bitmap for the allocation status of each sector has been "disabled by reason" ever since the vpc driver existed.

The reason might be that we might end up reading sector by sector in vpc_read if we really used it. This would be a performance desaster.

The current code would furthermore not work if the disabled parts get reactivated since vpc_read and vpc_write only use get_sector_offset to check the allocation status of the first sector of a read/write operation. This might lead to sectors incorrectly treated as zero in vpc_read and to sectors getting allocated twice in vpc_write.

---
 block/vpc.c | 32 --------------------------------
 1 file changed, 32 deletions(-)
block/vpc.c +0 -32
index 226be02..43e768e 100644
@@ -376,38 +376,6 @@ static inline int64_t get_sector_offset(BlockDriverState *bs,
         bdrv_pwrite_sync(bs->file, bitmap_offset, bitmap, s->bitmap_size);
     }
 
-//    printf("sector: %" PRIx64 ", index: %x, offset: %x, bioff: %" PRIx64 ", bloff: %" PRIx64 "\n",
-//	sector_num, pagetable_index, pageentry_index,
-//	bitmap_offset, block_offset);
-
-// disabled by reason
-#if 0
-#ifdef CACHE
-    if (bitmap_offset != s->last_bitmap)
-    {
-	lseek(s->fd, bitmap_offset, SEEK_SET);
-
-	s->last_bitmap = bitmap_offset;
-
-	// Scary! Bitmap is stored as big endian 32bit entries,
-	// while we used to look it up byte by byte
-	read(s->fd, s->pageentry_u8, 512);
-	for (i = 0; i < 128; i++)
-	    be32_to_cpus(&s->pageentry_u32[i]);
-    }
-
-    if ((s->pageentry_u8[pageentry_index / 8] >> (pageentry_index % 8)) & 1)
-	return -1;
-#else
-    lseek(s->fd, bitmap_offset + (pageentry_index / 8), SEEK_SET);
-
-    read(s->fd, &bitmap_entry, 1);
-
-    if ((bitmap_entry >> (pageentry_index % 8)) & 1)
-	return -1; // not allocated
-#endif
-#endif
-
     return block_offset;
 }
 
Signature · 1 line
1.9.1
Reviewed-by on v2 5/5
#8 lore raw

Re: [Qemu-devel] [PATCH v2 5/5] block/vpc: remove disabled code from get_sector_offset

Quoted 12 lines · Peter Lieven

The code to check the bitmap for the allocation status of each sector has been "disabled by reason" ever since the vpc driver existed.

The reason might be that we might end up reading sector by sector in vpc_read if we really used it. This would be a performance desaster.

The current code would furthermore not work if the disabled parts get reactivated since vpc_read and vpc_write only use get_sector_offset to check the allocation status of the first sector of a read/write operation. This might lead to sectors incorrectly treated as zero in vpc_read and to sectors getting allocated twice in vpc_write.

Quoted 4 lines
---
  block/vpc.c | 32 --------------------------------
  1 file changed, 32 deletions(-)

Awww, that code nearly had its ten-year anniversary. Too bad it won't live to see it (I hope...).

Applied
#9 lore raw
In reply to Peter Lieven
Quoted 15 lines · Peter Lieven

This series covers VPC format changes discussed during the last weeks.

Peter

v1->v2:
Patch 1: added comment why we can exit after the first allocated sector [Max] Patch 2: Fixed gemotry in commit msg [Max]
Patch 3: Fixed comparision from > to >= [Max]
Fixed exit in case of -EFBIG [Max]
Removed DIV_ROUND_UP [Max]
Patch 5: added

Kevin Wolf (1):
vpc: Ignore geometry for large images

Quoted 8 lines

Peter Lieven (4):
block/vpc: optimize vpc_co_get_block_status block/vpc: make calculate_geometry spec conform block/vpc: rename footer->size -> footer->current_size block/vpc: remove disabled code from get_sector_offset

  block/vpc.c | 106 +++++++++++++++++++++---------------------------------------
  1 file changed, 37 insertions(+), 69 deletions(-)

Thanks, applied to my block-next tree:

https://github.com/XanClic/qemu/commits/block-next

#10 lore raw
Quoted 23 lines · Max Reitz
Quoted 22 lines · Peter Lieven

This series covers VPC format changes discussed during the last weeks.

Peter

v1->v2:
Patch 1: added comment why we can exit after the first allocated
sector [Max]
Patch 2: Fixed gemotry in commit msg [Max]
Patch 3: Fixed comparision from > to >= [Max]
Fixed exit in case of -EFBIG [Max]
Removed DIV_ROUND_UP [Max]
Patch 5: added

Kevin Wolf (1):
vpc: Ignore geometry for large images

Peter Lieven (4):
block/vpc: optimize vpc_co_get_block_status block/vpc: make calculate_geometry spec conform block/vpc: rename footer->size -> footer->current_size block/vpc: remove disabled code from get_sector_offset

Quoted 7 lines
Quoted 3 lines
  block/vpc.c | 106 
+++++++++++++++++++++---------------------------------------
  1 file changed, 37 insertions(+), 69 deletions(-)

Thanks, applied to my block-next tree:

https://github.com/XanClic/qemu/commits/block-next

For the sake of completeness: Moved to the block tree, now merged to master.

Max